#2018-03-1215:17ghadiour depstar tool just got an awesome logo courtesy @quephird#2018-03-1215:18ghadihttps://raw.githubusercontent.com/healthfinch/depstar/master/depstar_logo.png#2018-03-1215:18Alex Miller (Clojure team)I’ll pretend I didn’t see that :)#2018-03-1215:21Alex Miller (Clojure team)does quephird work with you?#2018-03-1215:21ghadiyes @alexmiller#2018-03-1215:21Alex Miller (Clojure team)aw, cool. she’s great.#2018-03-1215:22Alex Miller (Clojure team);)#2018-03-1215:22Alex Miller (Clojure team)I meant to put all that in pm to Ghadi :)#2018-03-1215:41devn🙂#2018-03-1217:39Alex Miller (Clojure team)Added a wiki page with tools that use t.d.a or clj https://github.com/clojure/tools.deps.alpha/wiki/Tools#2018-03-1217:39Alex Miller (Clojure team)linked from the readme too if you’re trying to find it later#2018-03-1217:54seancorfieldI think that last line should be cljs-test-runner, not clj-test-runner (link is correct, text is not).#2018-03-1217:55Alex Miller (Clojure team)thx#2018-03-1218:33ghadidepstar is one word @alexmiller - thx for the mention#2018-03-1218:34Alex Miller (Clojure team)it’s a wiki, fix as you like :)#2018-03-1218:34Alex Miller (Clojure team)and if it’s misconfigured for edits, let me know#2018-03-1218:57ghadiI'm getting a 403 when I clone then push @alexmiller#2018-03-1218:58thheller@alexmiller https://github.com/thheller/shadow-cljs also supports using deps.edn via clojure. can't edit wiki.#2018-03-1218:58ghadi(I can't remember if there is supposed to be an edit button in the UI -- but there isn't one visible)#2018-03-1219:00Alex Miller (Clojure team)I’m sure it’s set wrong - seems like the default is always wrong :)#2018-03-1219:00Alex Miller (Clojure team)@ghadi clone then push what?#2018-03-1219:01ghadithe wiki repo#2018-03-1219:01ghadiI think I was supposed to edit in the UI#2018-03-1219:05Alex Miller (Clojure team)well that’s how normal people do it :) I’m getting Stu to change the setting#2018-03-1221:21dominicm@alexmiller going on the current line of thinking, it would be useful if https://github.com/clojure/tools.deps.alpha/blob/88f8e4f0113ba555c3c667bd5fbf60f3887f8140/src/main/clojure/clojure/tools/deps/alpha/reader.clj#L18-L28 let me specify other options particularly -Srepro#2018-03-1221:27Alex Miller (Clojure team)other options like?#2018-03-1221:28Alex Miller (Clojure team)oh, I see what you mean#2018-03-1221:28Alex Miller (Clojure team)good idea, feel free to jira / patch#2018-03-1221:28Alex Miller (Clojure team)that affects memoization of course so would need to probably expose the scrape function with a better name#2018-03-1221:34dominicmtbh, if I was being very specific, it might be just that I need the version that Stu originally created where the different config files get labelled.#2018-03-1221:36Alex Miller (Clojure team)well that idea is still in play for several reasons#2018-03-1221:37Alex Miller (Clojure team)prob the same reasons you’re interested in it#2018-03-1221:37Alex Miller (Clojure team)that’s going to be a mildly painful change though across the script and tda#2018-03-1221:44dominicmalpha, baby 🙂#2018-03-1221:51dominicm(I can picture Austin powers saying that)#2018-03-1222:07mhuebertcan a github dep in deps.edn specify the root path (into the repo) for the desired library?#2018-03-1222:08mhuebertI have been publishing more than one library from a single repo#2018-03-1222:08Alex Miller (Clojure team)Yeah with :deps/root#2018-03-1222:08mhuebertwould that be a relative or absolute path?#2018-03-1222:08mhuebert/my-lib#2018-03-1222:09Alex Miller (Clojure team)Relative in the repo#2018-03-1222:09Alex Miller (Clojure team)To the base dir#2018-03-1222:12dominicm@mhuebert fyi, when a transitive dependency does :local/root, it is relative to where you're running clj from. So you can't put things in sub-dirs, it has to be flat.#2018-03-1222:12dominicmWe're doing this, you're not alone it's good fun 🙂#2018-03-1222:14mhueberthmm. ok. i am not using :local/root anywhere at the moment#2018-03-1222:14mhuebertbut I am seeing:
Error building classpath. nil
java.lang.NullPointerException
at clojure.string$starts_with_QMARK_.invokeStatic(string.clj:364)
at clojure.string$starts_with_QMARK_.invoke(string.clj:360)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:48)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:43)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_fetch.invokeStatic(impl.clj:64)
at clojure.tools.gitlibs.impl$git_fetch.invoke(impl.clj:61)
at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:105)
at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
#2018-03-1222:14mhuebertfor the dep
lark/tools {:git/url ""
:sha "cc1305534b1371df3962fd46503dd0478d81a08a"
:deps/root "tools"}
#2018-03-1222:15mhuebertI can run deps from that directory locally (“tools”), it has a deps.edn file#2018-03-1222:15dominicmfwiw, that should work, I used it at some point, 100%#2018-03-1222:15mhueberthmm.#2018-03-1222:18mhuebert#2018-03-1222:18seancorfield@mhuebert FWIW, I just ran (! 581)-> clj -Sdeps '{:deps {lark/tools {:git/url ""
:sha "cc1305534b1371df3962fd46503dd0478d81a08a"
:deps/root "tools"}}}
'
and it worked fine and gave me a REPL.#2018-03-1222:18mhueberthmm#2018-03-1222:19mhuebertok thanks for trying that. maybe i need to update my clj#2018-03-1222:21mhuebertwell that’s weird. that exact command doesn’t work on my machine, with latest clj from brew#2018-03-1222:24seancorfieldI'm on the latest clojure 1.9.0.358 already installed#2018-03-1222:24seancorfield(output from brew upgrade clojure)#2018-03-1222:24mhuebertyeah, me too#2018-03-1222:25seancorfieldI created a new folder mhuebert and dropped into there to run that command with no deps.edn file present, just to confirm it works on its own.#2018-03-1222:25seancorfieldIf you have deps.edn, it will read that and maybe the NPE is coming from something else there?#2018-03-1222:26mhuebertyeah I tried the same thing, ran it from a fresh directory in a new terminal window#2018-03-1222:26mhuebertmysteries of the universe#2018-03-1222:26seancorfieldMaybe something in your ~/.clojure/deps.edn then?#2018-03-1222:27mhuebertappears to be all commented out#2018-03-1222:29mfikesTry -Srepro to be sure#2018-03-1222:30mhuebertthat produces a repl#2018-03-1222:30mfikesYou must have something in your deps.edn chain 🙂#2018-03-1222:30mhuebertsomehow the url passed to call-with-auth is nil, when it’s trying to fetch from github#2018-03-1222:31mfikesOther hunch: Maybe a cache gone bad? and it needs to be forced#2018-03-1222:31mhuebert(defn- call-with-auth
([^GitCommand command]
(call-with-auth
(.. command getRepository getConfig (getString "remote" "origin" "url"))
command))
([^String url ^GitCommand command]
(if (and (instance? TransportCommand command)
(not (str/starts-with? url "http")))
(.. ^TransportCommand command (setTransportConfigCallback @ssh-callback) call)
(.call command))))
#2018-03-1222:31mhuebertthird from bottom, the str/starts-with?#2018-03-1222:31mhuebertis there some caching somewhere that I can clear?#2018-03-1222:32mfikesI''d try without -Srepro and add -Sforce (but I'm speculating)#2018-03-1222:34mhuebertoh, I maybe misread/communicated, I tried -Srepro by itself earlier#2018-03-1222:34mhuebertclj -Srepro -Sforce -Sdeps '{:deps {lark/tools {:git/url ""
:sha "cc1305534b1371df3962fd46503dd0478d81a08a"
:deps/root "tools"}}}'
#2018-03-1222:34mhuebertstill null pointer. hmm.#2018-03-1222:39mhueberti wonder if it could have anything to do with git auth#2018-03-1222:41mhuebertwell this will have to wait for another day#2018-03-1307:31mhueberttrying clojure.tools.gitlibs, I get a failure loading slf4j:
user=> (require '[clojure.tools.gitlibs :as gl])
nil
user=> (gl/resolve "" "cc1305534b1371df3962fd46503dd0478d81a08a")
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
NullPointerException clojure.string/starts-with? (string.clj:364)
#2018-03-1307:33mhuebertadding slf4j-simple to class path resolves that warning but doesn’t affect the error#2018-03-1307:41mhuebertok, I removed the braintripping directories from ~/.gitlibs/_repos (inside <mailto:/cdn-cgi/l/email-protection|/cdn-cgi/l/email-protection> and https://github.com) and now it works#2018-03-1307:41mhuebertmaybe there was some corrupt thing there#2018-03-1307:42mhuebertunfortunately I can’t find a way to reproduce this now#2018-03-1307:43mhuebertoh, now i’ve got it. found the old dir in the trash.#2018-03-1307:44mhuebertwell, that is simple.#2018-03-1307:46mhuebertReproduction steps:
1. create an empty directory at ~/.gitlibs/_repos/github.com/<user>/<repo>
2. try loading any version of *main-repl-fn*#2018-03-1320:45bhaumanthen we could set that with an -e "(set! clojure.main/*main-repl-fn* rebel-readline.main/repl)"#2018-03-1320:46bhaumanand then bobs your uncle#2018-03-1320:48Alex Miller (Clojure team)clojure.main is in Clojure, so this would require a change in Clojure 1.10. I think ultimately we’ll want a change that does something like that. But I’d like to look at a shorter range version.#2018-03-1320:50dominicmI feel like I'm missing some context?#2018-03-1320:54bhauman@dominicm I was just trying to think of a lightweight way for clj to inject rebel-readline, adding an inline dep and extra e arg would be easy#2018-03-1320:55seancorfieldCurrently, I have :rebel {:extra-deps {com.bhauman/rebel-readline {:mvn/version "RELEASE"}}
:main-opts ["-m" "rebel-readline.main"]}
in my ~/.clojure/deps.edn file and just say clj -A:rebel when I want a rebel REPL...#2018-03-1320:55dominicmyeah, I have that, but I namespaced it as :user/rebel.#2018-03-1320:55bhaumanbut that doesn't help clj#2018-03-1320:56dominicmno?#2018-03-1320:56dominicmOH. You mean for including rebel in clj?#2018-03-1320:56bhaumanyes 🙂#2018-03-1320:56dominicm:thinking_face: does rebel have any dependencies#2018-03-1320:57bhaumanextremely flat, tools reader is the only one I still want to get rid of#2018-03-1320:57bhaumanthat requires me writing my own indenting code#2018-03-1320:58bhaumanafter that there would be no deps besides Jline of course#2018-03-1320:58dominicmI mean, you could use Mr. Anderson to package it?#2018-03-1321:00bhaumanoh interesting, didn't know about that#2018-03-1321:08dominicmShading is a pretty cool thing 🙂#2018-03-1322:15dominicm@seancorfield do you have your handy snippet for clj + socket server?#2018-03-1322:16seancorfieldThis one? :socket {:jvm-opts ["-Dclojure.server.repl={:port,5555,:accept,clojure.core.server/repl}"]}
#2018-03-1322:19dominicmYep#2018-03-1322:20dominicmclj -J-Dclojure.server.repl={:port,5555,:accept,clojure.core.server/repl} -A:dev:build is giving me
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol
at clojure.lang.RT.seqFrom(RT.java:547)
at clojure.lang.RT.seq(RT.java:527)
at clojure.lang.APersistentMap.cons(APersistentMap.java:40)
at clojure.lang.RT.conj(RT.java:667)
at clojure.core$conj__4932.invokeStatic(core.clj:85)
at clojure.core$merge$fn__5498.invoke(core.clj:3029)
at clojure.core$reduce1.invokeStatic(core.clj:936)
at clojure.core$reduce1.invokeStatic(core.clj:926)
at clojure.core$merge.invokeStatic(core.clj:3028)
at clojure.core.server$parse_props$fn__8321.invoke(server.clj:149)
at clojure.core.protocols$naive_seq_reduce.invokeStatic(protocols.clj:62)
at clojure.core.protocols$interface_or_naive_reduce.invokeStatic(protocols.clj:72)
at clojure.core.protocols$fn__7665.invokeStatic(protocols.clj:168)
at clojure.core.protocols$fn__7665.invoke(protocols.clj:124)
at clojure.core.protocols$fn__7620$G__7615__7629.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
at clojure.core.protocols$fn__7642.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__7642.invoke(protocols.clj:75)
at clojure.core.protocols$fn__7594$G__7589__7607.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6704)
at clojure.core.server$parse_props.invokeStatic(server.clj:143)
at clojure.core.server$start_servers.invokeStatic(server.clj:154)
at clojure.core.server$start_servers.invoke(server.clj:154)
at clojure.lang.Var.invoke(Var.java:381)
at clojure.lang.RT.doInit(RT.java:486)
at clojure.lang.RT.<clinit>(RT.java:333)
... 1 more
java.lang.NoClassDefFoundError: Could not initialize class clojure.lang.RT
at clojure.lang.PersistentQueue.pop(PersistentQueue.java:113)
at clojure.lang.PersistentQueue.pop(PersistentQueue.java:25)
at clojure.lang.Agent$Action.doRun(Agent.java:149)
at clojure.lang.Agent$Action.run(Agent.java:163)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
which is interesting#2018-03-1322:21dominicmWorks in ~/.clojure/deps.edn though.#2018-03-1322:21seancorfieldYou need "..." around it.#2018-03-1322:22seancorfieldclj -J"-Dclojure..."#2018-03-1322:22seancorfieldOr this clj -J-Dclojure.server.repl="{:port,5555,:accept,clojure.core.server/repl}"
#2018-03-1322:24dominicmI did the latter originally. But I got the above error.#2018-03-1322:32seancorfield(! 614)-> clj -J-Dclojure.server.repl="{:port,5555,:accept,clojure.core.server/repl}"
Clojure 1.9.0
user=> ^D
Tue Mar 13 15:32:30
(sean)-(jobs:0)-(~/clojure/lsnape)
(! 615)-> clj -J-Dclojure.server.repl={:port,5555,:accept,clojure.core.server/repl}
Exception in thread "main" java.lang.ExceptionInInitializerError
at clojure.main.<clinit>(main.java:20)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol
#2018-03-1322:33seancorfieldI get the exception without the quotes -- but it works with quotes as seen above.#2018-03-1322:39dominicmI'll explore it later. ¯\(ツ)/¯#2018-03-1403:51jebberjebI’ve just been doing $> JAVA_OPTS="-Dclojure..." clj. Same for lein#2018-03-1403:53jebberjebMy thinking was that it’s nice because it doesn’t depend on how a particular thing handles command line args?#2018-03-1407:50dominicmI didn't know that worked for clj, neat.#2018-03-1411:52niclasnilssonI want to get these types of things working, and I think I’ve tried quoting and escaping the things after clj in all ways possible, but I bet there is a way I haven’t tried, since I can’t get it to work:
#!/usr/bin/env clj -Sdeps '{:deps {clj-time {:mvn/version "0.14.2"}}}'
(require '[clj-time.core :as t])
(println (str "Time is now " (t/now)))
I either get a "Error building classpath. Don't know how to create ISeq from: clojure.lang.Symbol" or a "Error while parsing option "--config-data {:deps": java.lang.RuntimeException: EOF while reading" for a combinations that I’ve tried. I bet someone knows how to do this properly?#2018-03-1412:15dominicm@niclasnilsson what OS are you on? That won't work for Linux.#2018-03-1412:47niclasnilsson@dominicm MacOS#2018-03-1412:48niclasnilssonDo you mean like “there is no way to make it work”?#2018-03-1412:51dominicmThere's a limitation on how shebangs are read. It doesn't apply to mac though.
You may find it works if you don't use env #2018-03-1413:12niclasnilssonAh, ok. Will try again then. Thanks!#2018-03-1416:19seancorfieldAlso: try using , instead of space in the deps string. #2018-03-1416:30dominicm@seancorfield Yes, I think you might have the right of it there. ' doesn't have have the same effect in shebangs as on the shell.#2018-03-1417:09seancorfieldConfirmed that the following works @niclasnilsson (! 622)-> ./time.sh
Time is now 2018-03-14T17:09:15.652Z
Wed Mar 14 10:09:15
(sean)-(jobs:0)-(~/clojure/bin)
(! 623)-> cat time.sh
#!/usr/bin/env clj -Sdeps {:deps,{clj-time,{:mvn/version,"0.14.2"}}}
(require '[clj-time.core :as t])
(println (str "Time is now " (t/now)))
On macOS#2018-03-1417:22hiredmanfootnote to "On macOS" https://unix.stackexchange.com/questions/63979/shebang-line-with-usr-bin-env-command-argument-fails-on-linux#2018-03-1419:56dominicmI didn't know BSD did it right too.#2018-03-1417:11niclasnilssonAh… I wasn’t even close to try that combination. Neat, thanks, @seancorfield!#2018-03-1417:12niclasnilssonWill try myself later tonight, when I get back to this.#2018-03-1506:20niclasnilssonWorked fine! Thanks, @seancorfield#2018-03-1512:16delaguardoIs it possible to redefine local maven repo location with tools-deps?#2018-03-1512:21Alex Miller (Clojure team)Yes, I think the key is :mvn/local-repo#2018-03-1512:21Alex Miller (Clojure team)At the root of deps.edn#2018-03-1512:23Alex Miller (Clojure team)Yeah, that’s it#2018-03-1512:24delaguardoCool! thanks. will check this out#2018-03-1512:37mpenetany eta for TDEPS-9, or if another approach is considered for that issue?#2018-03-1512:44mpenethaving a blank: what's the way to get a dependency from a subdir in a git repo with multiple dependencies ?#2018-03-1512:44mpenetI think I got that answered a while back, but I can't remember the answer (and well... slack history is not helping)#2018-03-1512:51niclasnilssonSometimes I wonder how Slack managed to take over the world so much. The history problem sucks for large open things like Clojurians.#2018-03-1514:24seancorfieldOff-road topic for this channel -- take it to #community-development if you want to discuss Slack/history/etc. #2018-03-1512:47mpenetit might be :deps/root reading the source#2018-03-1512:51Alex Miller (Clojure team)yes#2018-03-1512:55Alex Miller (Clojure team)on TDEPS-9, no eta. I’m consulting right now so only getting a few hrs a week on tdeps. The windows installer is my top priority at the moment, hoping to get that done this week.#2018-03-1514:25delaguardoI found strange bug -
15:24 $ clj -Srepro -Sdeps "{:mvn/local-repo \".m2/repository\"}" -Spom
Downloading: org/clojure/clojure/1.9.0/clojure-1.9.0.pom from
Downloading: org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.pom from
Downloading: org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from
Downloading: org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.pom from
Downloading: org/clojure/clojure/1.9.0/clojure-1.9.0.jar from
Downloading: org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar from
Downloading: org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar from
Unknown option: "--config-data"
#2018-03-1514:26delaguardoIt happens only when I’m trying to regenerate pom.xml file with local-repo#2018-03-1514:28Alex Miller (Clojure team)the local-repo is an absolute path btw, although I don’t think that’s the issue you’re seeing#2018-03-1514:29delaguardobut pom.xml is not generated in that case#2018-03-1514:29Alex Miller (Clojure team)if you can file a jira in https://dev.clojure.org/jira/browse/TDEPS that would be helpful. I can repro locally and I suspect I know where the issue is#2018-03-1514:29delaguardoWill do#2018-03-1514:32Alex Miller (Clojure team)there’s some duplicated code in the make classpath and pom gen stuff and I don’t think the latter was updated the last time the former changed#2018-03-1515:23seancorfieldFWIW, that command ultimately becomes this + /usr/bin/java -Xmx256m -classpath /usr/local/Cellar/clojure/1.9.0.358/libexec/clojure-tools-1.9.0.358.jar clojure.main -m clojure.tools.deps.alpha.script.generate-manifest --config-files /usr/local/Cellar/clojure/1.9.0.358/deps.edn,deps.edn --gen=pom --config-data '{:mvn/local-repo ".m2/repository"}'
and that's what complains about --config-data#2018-03-1515:25Alex Miller (Clojure team)yeah, I’m aware - the generate-manifest hasn’t tracked changes in make-classpath#2018-03-1517:42mhueberthas anyone seen something like this when running -Spom?
clj -Spom
Error generating pom manifest: called children on a leaf node
java.lang.Exception: called children on a leaf node
at clojure.zip$children.invokeStatic(zip.clj:80)
at clojure.zip$append_child.invokeStatic(zip.clj:223)
at clojure.zip$append_child.invoke(zip.clj:223)
at clojure.tools.deps.alpha.gen.pom$xml_update$fn__2247.invoke(pom.clj:94)
at clojure.tools.deps.alpha.gen.pom$xml_update.invokeStatic(pom.clj:87)
at clojure.tools.deps.alpha.gen.pom$xml_update.invoke(pom.clj:83)
at clojure.tools.deps.alpha.gen.pom$replace_deps.invokeStatic(pom.clj:98)
at clojure.tools.deps.alpha.gen.pom$replace_deps.invoke(pom.clj:96)
at clojure.tools.deps.alpha.gen.pom$sync_pom$fn__2265.invoke(pom.clj:120)
at clojure.tools.deps.alpha.gen.pom$sync_pom.invokeStatic(pom.clj:117)
at clojure.tools.deps.alpha.gen.pom$sync_pom.invoke(pom.clj:112)
at clojure.tools.deps.alpha.script.generate_manifest$_main.invokeStatic(generate_manifest.clj:49)
at clojure.tools.deps.alpha.script.generate_manifest$_main.doInvoke(generate_manifest.clj:27)
#2018-03-1517:47Alex Miller (Clojure team)what’s your existing pom.xml?#2018-03-1517:48Alex Miller (Clojure team)does it have a <dependencies>?#2018-03-1517:54mhuebertlet me see#2018-03-1517:55mhuebertyes it does#2018-03-1517:56mhuebertI can run -Spom if I remove the existing pom.xml#2018-03-1518:20Alex Miller (Clojure team)well if you can capture the case when it fails, would be happy to take a ticket for that#2018-03-1518:21Alex Miller (Clojure team)the pom.xml when it fails that is#2018-03-1520:20mhueberthere is a minimal pom.xml which produces the error:
<?xml version="1.0" encoding="UTF-8"?>
<!-- generated by shadow-cljs pom, do not edit -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>editor</groupId>
<artifactId>editor</artifactId>
<version>0.0.1</version>
<name>editor</name>
<dependencies>
<dependency>
<groupId>thheller</groupId>
<artifactId>shadow-cljs</artifactId>
<version>2.2.9</version>
</dependency>
</dependencies>
</project>
#2018-03-1520:20mhuebertI can create a ticket for that if it would be helpful#2018-03-1520:21Alex Miller (Clojure team)yes, also would help to know which jdk you’re using#2018-03-1520:22Alex Miller (Clojure team)java -version#2018-03-1520:23mhuebertjava version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
#2018-03-1520:27mhueberthttps://dev.clojure.org/jira/browse/TDEPS-55#2018-03-1520:27Alex Miller (Clojure team)I also assume you’re using latest clj (`clj -Sverbose`) ?#2018-03-1520:28Alex Miller (Clojure team)latest is 1.9.0.358#2018-03-1520:54mhuebertyes#2018-03-1520:55Alex Miller (Clojure team)k, if I get a chance I’ll look at it tomorrow#2018-03-1520:55Alex Miller (Clojure team)this doesn’t seem much different than everything else I’ve done that works fine so not sure what the deal is#2018-03-1521:07seancorfieldLooks like it's the comment that causes problems @mhuebert /cc @alexmiller#2018-03-1521:07Alex Miller (Clojure team)I was just about to say that :)#2018-03-1521:08Alex Miller (Clojure team)there’s an assumption in the code about the doc structure that’s being foiled there#2018-03-1521:08seancorfieldIf you move the comment inside <project> it's fine. Or delete it.#2018-03-1521:09seancorfield(sorry, it made me too curious to resist trying to figure out what was causing it)#2018-03-1521:09Alex Miller (Clojure team)I will fix it, but that’s a workaround in the meantime#2018-03-1613:06dnolen@alexmiller the behavior of -R seems a big strange to me, shouldn’t it compose any aliases + your local deps.edn deps?#2018-03-1613:37Alex Miller (Clojure team)-R is solely about selecting aliases to be used when resolving deps. The aliases it can select from are those produced by the combined deps.edn files in play, which typically includes your install, user, local deps.edn (and -Sdeps passed on command line)#2018-03-1613:37Alex Miller (Clojure team)aliases are not pulled from transitive deps#2018-03-1613:39Alex Miller (Clojure team)I’m not sure that would even be possible based on the order in which things are resolved#2018-03-1705:59seancorfieldJust FYI, I put my ~/.clojure/deps.edn file on GitHub https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#2018-03-1708:06dominicmI've been using the :user/ namespace for aliases on my home deps file, with the idea that it will prevent collisions.#2018-03-1814:40hypirionSo I have a (pruned) dependency tree that looks like this
.
├── a 1
│ └── b 1
│ └── x 2
│ └── y 1
└─── c 1
└── x 1
└── z 1
and my result when doing clj -Stree is the following:
org.clojure/clojure 1.9.0
org.clojure/core.specs.alpha 0.1.24
org.clojure/spec.alpha 0.1.143
c path-to-c 1
a path-to-a 1
b path-to-b 1
x path-to-x 2
y path-to-y 1
z path-to-z 1
I'm guessing this is a bug? I would love to send a reproducible example, but it's a bit hard without polluting my GitHub profile. Maybe adding an option to include local git repositories for :git/url would be an idea?#2018-03-1815:10gfrederickswould a gist pollute your github profile?#2018-03-1815:13Alex Miller (Clojure team)some things that would be helpful:
- version info from clj -Sverbose (I’m assuming you’re on latest, but if not, then there were significant changes to resolution in 1.9.0.309)
- output from adding -Sdeps '{:aliases {:v {:verbose true}}}' -R:v#2018-03-1815:14Alex Miller (Clojure team)with that I probably don’t need the reproducible deps.edn#2018-03-1815:16Alex Miller (Clojure team)this is the trickiest kind of version resolution issue - when you have different versions of a dep and the lib sets (not just the versions) are changing across them.#2018-03-1815:17Alex Miller (Clojure team)I’m assuming that what you expect here is to have a1, b1, x2, y1, c1 as the output (and no z1)#2018-03-1815:31hypirionversion = 1.9.0.358
install_dir = /usr/local/lib/clojure
config_dir = /home/jeannikl/.clojure
config_paths = /usr/local/lib/clojure/deps.edn /home/jeannikl/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/674722798.cp
Nothing in /usr/local/lib/clojure/deps.edn or /home/jeannikl/.clojure/deps.edn
@gfredericks: That's a good idea, forgot about gists#2018-03-1815:32hypirionAlso yes, I'd expect z1 to not be present#2018-03-1815:45Alex Miller (Clojure team)@hypirion I reproduced the simple example in a test case and can’t repro https://github.com/clojure/tools.deps.alpha/blob/master/src/test/clojure/clojure/tools/deps/test_alpha.clj#L100-L113#2018-03-1815:46Alex Miller (Clojure team)which leads me to believe its most likely a path comparison issue with git deps#2018-03-1815:46Alex Miller (Clojure team)the verbose output should be sufficient for me to track that down#2018-03-1815:49hypirionHuh, it seems like it depends on the order of the elements in the map.#2018-03-1815:49hypirionAlmost done with the gists now, one sec#2018-03-1815:54hypirionhttps://gist.github.com/hypirion/b955e63ebb394c216c61fe1b1b6a48fd
If I flip the dependencies around in that repo, it doesn't include z.#2018-03-1815:59hypirionI'm still a bit unsure about how the algorithm works, but I think it should be possible to walk the dependency tree at the end. If you can't reach a dependency by walking the tree, then don't include it as a dependency you need.#2018-03-1816:03Alex Miller (Clojure team)I have to run but will look at it later#2018-03-1816:04Alex Miller (Clojure team)That’s being done but it relies on comparing deps up the path#2018-03-1816:06Alex Miller (Clojure team)If the same dep has two different representations then that path matching will fail. That shouldn’t happen, but that’s my suspicion#2018-03-1816:09hypirion@alexmiller: I flipped the order of the dependencies in your test case and mvn test fails on my machine with that change. https://github.com/hypirion/tools.deps.alpha/commit/5459773c7236e37d0306d1561d5dc5a42243ca70#2018-03-1816:16Alex Miller (Clojure team)Interesting, well I guess I was just lucky :)#2018-03-1816:22hypirionheh, yeah. Dependency algorithms are hard to get right#2018-03-2016:04ghadiDoes anyone use cljfmt via clj?#2018-03-2017:00delaguardoI can use it like that
clj -A:cljfmt -e "(require '[leiningen.cljfmt])" -e "(leiningen.cljfmt/cljfmt {:source-paths [\"src\"] :test-paths [\"test\"]} \"check\")"
with an alias inside on deps.edn
:cljfmt {:extra-deps {leiningen {:mvn/version "2.8.1"}
lein-cljfmt {:mvn/version "0.5.7"}}}
#2018-03-2017:21ghadiHero, thanks#2018-03-2120:45kidpolloTo be able to use :git/url sources the target repo has to also have a deps.edn file right?#2018-03-2120:46Alex Miller (Clojure team)it needs to either not have deps or declare them with an understood manifest format#2018-03-2120:47Alex Miller (Clojure team)currently tools.deps understands deps.edn and pom.xml#2018-03-2120:47Alex Miller (Clojure team)or you can force deps.edn semantics with no deps by using :deps/manifest :deps#2018-03-2120:47kidpolloI see. What about local mvn sources. I am having a hard time getting Artifactory sources working with deps :S#2018-03-2120:48Alex Miller (Clojure team)what does “local mvn sources” mean?#2018-03-2120:48danielcompton@kidpollo note that you won't be able to auth to Artifactory until https://dev.clojure.org/jira/browse/TDEPS-9 is fixed#2018-03-2120:48kidpolloI am looking for a workaround for the “no username password sources supported”#2018-03-2120:50kidpollothanks for pointing that out @danielcompton#2018-03-2120:50Alex Miller (Clojure team)you could mirror to an s3 repo accessed via aws creds#2018-03-2120:50Alex Miller (Clojure team)that’s supported#2018-03-2120:50kidpollook thanks!#2018-03-2121:20danielcomptonI'm eagerly awaiting TDEPS-9 to be fixed, a few https://deps.co customers need it before they can start using it#2018-03-2121:21danielcompton@alexmiller is there anything I can do to help there, or is it just waiting for review?#2018-03-2121:21Alex Miller (Clojure team)it’s just down my queue a bit. We need it for Datomic too.#2018-03-2121:22Alex Miller (Clojure team)I’m on a consulting project these days so only getting a few hours of Clojure time per week atm. My priority right now is the Windows installer which is getting close.#2018-03-2121:22Alex Miller (Clojure team)Any testing or feedback on TDEPS-9 would be helpful.#2018-03-2121:28danielcomptonk, will take a look and see if I can verify that it works#2018-03-2121:48dominicm@alexmiller which patch is it you want testing/feedback on? Or both?#2018-03-2121:52Alex Miller (Clojure team)Don’t know, I haven’t looked at it at all#2018-03-2121:52Alex Miller (Clojure team)my only constraint is - no passwords in deps.edn files :)#2018-03-2121:54dominicmthat seems like it discounts patch a.#2018-03-2121:54Alex Miller (Clojure team)well there you go#2018-03-2122:06dominicm@danielcompton I must say, I don't know if my patch works under java 9. I know reflection changed somehow in java 9, but I haven't taken the time to understand that change.#2018-03-2122:07danielcomptonI'll give that a go too 🙈#2018-03-2122:07dominicmYou've made yourself the expert on Java 9 by blogging about it one time 😛#2018-03-2122:13Alex Miller (Clojure team)Based on what I know, it probably does or can be made to in the context of clj#2018-03-2122:15Alex Miller (Clojure team)The best way to test stuff like this is to hack your active clojure script to prepend your tools.deps.alpha/src/main/clojure dir to your lib path#2018-03-2122:16Alex Miller (Clojure team)Then you pick up all local changes immediately and can interactively test clj#2018-03-2216:03ghadigit-related
- https authentication is not supported yet by clj -- I wrote a patch that is waiting for eyes upstream
- ssh authentication via Java is a usability trashfire. Works easily for some, doesn't work for others. I started rewriting tools.gitlibs (the underlying library) to shell out to git, but I'm not sure it is viable because the solution must work across Windows to be accepted upstream.
- must ensure ssh-keyscan -H <host> >> ~/.ssh/known_hosts because you cannot accept unseen ssh hostkeys (clj is non-interactive)
- esoteric .ssh/config options can break cloning because the underlying java ssh libraries reads the config file poorly
- ed25519 keys supported via ssh-agent, but not via ~/.ssh/config entries
i wrote that internally, thought it would be a useful summary of some usability issues here{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-03-2216:07Alex Miller (Clojure team)If you want to make a troubleshooting page on wiki or site, go for it#2018-03-2216:16ghadiwiki page is a great idea#2018-03-2216:19dominicmReport from a training session we did with clj using the linux installer. The jars were created as root read-only! That makes the scripts broken for most users to our knowledge.#2018-03-2218:38ghadiwhat did you expect to happen?#2018-03-2219:14dominicmThe jars would be readable by everyone.#2018-03-2219:15ghadioh they're only readable by root?#2018-03-2219:15ghadithat's not what I see on my machine#2018-03-2219:17dominicmI think the script changed recently#2018-03-2216:21dominicmLooking at the script, I think it could be different to make these problems go away. Maybe I'll take a stab at tackling them.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-03-2216:21dominicm@alexmiller What is your process for testing the linux installer?#2018-03-2315:28borkdudeI just tested the linux installer in a docker container openjdk:10#2018-03-2315:31borkdudeThat worked. In openjdk:9 I got: Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty#2018-03-2315:32borkdude{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "afddc0c0dbefca9f9ecc9ecc9a9bc9979fc9"}, :content ("[email protected]")}
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "097b66667d493a3b6f6d3e3f306f3b313a3d"}, :content ("[email protected]")}#2018-03-2315:33Alex Miller (Clojure team)@dominicm sorry for non-response above - I saw it but got distracted before answering and then the notifier was gone so forgot about it. :) I don’t test the linux installer very well (just in a vm) and would appreciate any help in making it better.#2018-03-2316:15dominicmOkay, I'll have a think.#2018-03-2315:35Alex Miller (Clojure team)@borkdude if you’re trying to compare times, make sure that you’re comparing cached classpath in both cases by running more than once#2018-03-2315:35Alex Miller (Clojure team)they mentioned that jvm startup times should be expected to be better in java 10 than java 9#2018-03-2315:35timgilbertPossibly the SSL error is this? https://github.com/docker-library/openjdk/issues/145#2018-03-2315:35borkdudeI’m comparing 8 and 9 here with just vanilla clojure after it downloaded the bootstrap dependencies#2018-03-2315:36borkdudeit seems to be a little bit better#2018-03-2315:36borkdudethis is openjdk, no idea if there is any difference with the Oracle JDK#2018-03-2315:36borkdudenot a JVM expert#2018-03-2315:37borkdudeit’s very easy to spin up a Docker: docker run -it openjdk:10 /bin/bash and then do the clj linux installer#2018-03-2316:06tianshuhow to do test with tools.deps?#2018-03-2316:07Alex Miller (Clojure team)What are you trying to test?#2018-03-2316:08tianshuI'm trying to use tools.deps instead of lein. I just want to run test like lein test.#2018-03-2316:09Alex Miller (Clojure team)Check out the test-runner project #2018-03-2316:11tianshuthis is exactly what I'm looking for, thanks!#2018-03-2316:11Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools#2018-03-2316:11Alex Miller (Clojure team)Other useful tools#2018-03-2316:12tianshugreat#2018-03-2316:21tianshuwhen I use git/url, met an error
org.eclipse.jgit.api.errors.TransportException:
I just have get the :sha with git rev-parse HEAD.#2018-03-2316:21tianshuI'm confused what should I do?#2018-03-2316:22Alex Miller (Clojure team)@ghadi had a list of git connection issues in the back chat#2018-03-2316:23Alex Miller (Clojure team)In this case looks like maybe you’re using the ssh url when the https would be better#2018-03-2316:23ghadihttps would work because that repo is public#2018-03-2316:24ghadiotherwise try ssh-keyscan -H >> ~/.ssh/known_hosts (I'm a bit surprised that https://github.com isn't in known_hosts -- perhaps you're containerized?)#2018-03-2316:24Alex Miller (Clojure team)I will update the test-runner read me to use https too#2018-03-2316:25Alex Miller (Clojure team)No reason for an ssh url there#2018-03-2316:27tianshuI wonder where tools.deps put the git dependencies?#2018-03-2316:28tianshumaybe I can find it with clj -Spath#2018-03-2316:29tianshuokay, found it.#2018-03-2316:31tianshuthanks a lot.#2018-03-2317:00dominicmI opened an issue for that @alexmiller#2018-03-2317:58Alex Miller (Clojure team)where?#2018-03-2317:59Alex Miller (Clojure team)nvm, I see it#2018-03-2323:17seancorfield@doglooksgood take a look at the deps.edn file in each of core.cache, core.memoize, and java.jdbc for examples of multiversion testing with clj and test-runner -- I recently converted all of those from Leiningen. #2018-03-2402:01tianshu@seancorfield It's very helpful. these project dose not have a resources folder, is there a resource-path in tools.deps or I should just use paths?#2018-03-2403:02seancorfield@doglooksgood the deps stuff is just about the class path. There's no distinction between the things that are on the class path. #2018-03-2403:04seancorfieldIf you use my boot-tools-deps then it has the idea of Boot resource path -- anything that would go in the uberjar -- and source path -- code that doesn't go in the uberjar. That puts deps paths into resource paths and extra-paths into source paths. #2018-03-2410:16dominicmWhy /usr/local > /usr/? Looking at https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard it suggests that /usr/local is wrong.#2018-03-2410:19dominicmAlso, the deps.edn files should be somewhere in /etc I think too? Reaching out here for some community feedback.#2018-03-2521:27danielcompton@alexmiller did you hear what Java versions people were running that had issues with http://repo.clojars.org?#2018-03-2600:03Alex Miller (Clojure team)No#2018-03-2611:49mfikesNifty. The Planck homebrew build now depends on the Clojure homebrew install: https://github.com/Homebrew/homebrew-core/blob/9d13915487b6c817b0dd551b46243e8e95286774/Formula/planck.rb#L15#2018-03-2611:52mfikesAn interesting aside to the above: Even though Java isn't required to install or use the Planck binary, homebrew doesn't yet make a distinction, so you transitively need Java via Clojure to install the binary. https://github.com/mfikes/planck/issues/695#2018-03-2612:17roman01laHow to specify a dependency on a project within a subfolder in a git repo? Such as this https://github.com/bhauman/lein-figwheel/tree/master/figwheel-core#2018-03-2612:20Alex Miller (Clojure team)you can use :deps/root attribute for the relative subdir in the repo#2018-03-2612:22Alex Miller (Clojure team){:git/url ""
:sha "b3d1d16d4915948a36b341f5648e4c9763628ef3"
:deps/root "figwheel-core"}#2018-03-2612:26roman01lashould it work for Clojure 1.9? or do I need 1.10?#2018-03-2612:26Alex Miller (Clojure team)it declare the deps as needing only clojure 1.8#2018-03-2612:26Alex Miller (Clojure team)but clojurescript 1.10.x#2018-03-2612:41roman01laI’m getting Error building classpath. nil when running clj with the following deps.edn
{:deps {figwheel-core {:deps/root "figwheel-core",
:git/url "",
:sha "b3d1d16d4915948a36b341f5648e4c9763628ef3"}}}
#2018-03-2612:42Alex Miller (Clojure team)I can’t reproduce that#2018-03-2612:42Alex Miller (Clojure team)is there a stack trace?#2018-03-2612:43Alex Miller (Clojure team)from clj -Sverbose, what version are you running?#2018-03-2612:43roman01la1.9.0.358#2018-03-2612:43Alex Miller (Clojure team)that’s latest#2018-03-2612:44Alex Miller (Clojure team)do you have anything in your ~/.clojure/deps.edn?#2018-03-2612:45roman01laThere was deps, but I’ve edited it to be just {} before running#2018-03-2612:45Alex Miller (Clojure team)maybe also use -Sforce to make sure you’re not getting something from cache#2018-03-2612:45roman01lastack trace https://gist.github.com/roman01la/8b7351d7697584a6bcf394adb5f722af#2018-03-2612:45roman01laok#2018-03-2612:46Alex Miller (Clojure team)oh interesting, that’s down in gitlibs. well, you could try cleaning your git cache in ~/.gitlibs#2018-03-2612:46Alex Miller (Clojure team)either selectively or by deleting the whole thing#2018-03-2612:48Alex Miller (Clojure team)that looks like it’s getting a nil url, which is weird#2018-03-2612:55roman01laWorks now. Not sure how exactly, but after cleaning ~/.gitlibs it didn’t work first. Once I ran clj -Sforce it went fine and subsequent clj (without forcing) are ok now#2018-03-2612:55roman01laThanks!#2018-03-2613:07Alex Miller (Clojure team)I suspect something bad in your cpcache#2018-03-2615:47mfikes👍 I managed to get clj to run in the Ubuntu PPA build for Planck. The main trickery was really ensuring it stays in the sandbox, which is a little tighter than Homebrew—you don't even have a home directory, for instance.#2018-03-2622:06richiardiandreaprobably late to the party but this is worth reading : https://hypirion.com/musings/a-resilient-git-dependency-algorithm#2018-03-2701:00ghadiwhat a great article.#2018-03-2703:12richiardiandreaYes#2018-03-2716:20dominicmI'm getting a weird issue. In Heroku, the clone step is failing for a https url with "authentication is required but no credentialsprovider has been registered". I can't think what the cause would be?#2018-03-2716:20dominicmMaybe it's something in ~/.ssh/config 😮 I don't know though.#2018-03-2716:32Alex Miller (Clojure team)this is a git dep?#2018-03-2716:32Alex Miller (Clojure team)with https url shouldn’t be using ssh at all#2018-03-2716:33Alex Miller (Clojure team)are you hitting a private repo via https? that’s not yet supported#2018-03-2716:47dominicmNo, it's a public one.#2018-03-2716:49dominicmOh yes, duh, of course ssh isn't engaged here at all.#2018-03-2716:52dominicm:extra-deps {
io.dominic/krei.alpha {:git/url ""
:sha "778ad4eb6130b7023204bf526b6f49d1429e4132"}}
Works locally, but not in heroku, it's definitely something contextual.#2018-03-2716:55hypirionhttps://github.com/SevereOveridfl0w/krei.alpha is a private repo it seems#2018-03-2716:56dominicmHaha. I see the problem now. There's a typo in my username. I'm so terrible. Sorry for the noise.#2018-03-2718:40jeff.terrellFor what it's worth, I learned recently that you can add custom ssh keys to Heroku, so that it can fetch deps from private git repos via SSH. (I had to do that because we were using a private npm package with github coordinates instead of npm registry coordinates.)
https://github.com/simon0191/custom-ssh-key-buildpack
(I realize you already solved your problem, @dominicm, it's just a useful capability to be aware of, so seemed worth sharing.)#2018-03-2718:41dominicmThat's very useful, thank you{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-03-2721:51richiardiandreaSo spamming a bit more - in #clojurescript Alex gave me this brilliant/weird/maybe odd idea of converting https://github.com/boot-clj/boot/blob/30ba3ce69dcd8b2b3e00677a14db05c8ce49d59c/boot/base/src/main/java/boot/App.java#2018-03-2721:51richiardiandreaso that it can be called directly from clj#2018-03-2721:51richiardiandreaas in boot -m boot.main -param1 -param2#2018-03-2721:51richiardiandreaClojureScript does something similar at the moment and it is definitely a nice pattern#2018-03-2721:53dominicm@richiardiandrea @martinklepsch has already explored some of the issues here, https://clojureverse.org/t/boot-and-deps-edn/1266#2018-03-2721:53richiardiandreaoh ok cool didn't read that thanks#2018-03-2721:54richiardiandreaactually this is more in tune with my idea: https://clojureverse.org/t/boot-as-a-library/1432#2018-03-2721:56dominicmoh, that may be what I meant.#2018-03-2721:56dominicmclojureverse is down for me, so I was stabbing in the dark 😛#2018-03-2804:55semperosWhen specifying :git/url in my deps.edn, I appear to have to use http:// rather than https://. When using https://, the initial message indicates that it's cloning anonymously, but then I get error messages and stack traces showing that it's trying to use {:tag :mailto:gitgithub.comgitgithub.com, :attrs nil, :content nil}. Using http:// does not present this problem and the repo is cloned anonymously as expected.
## clj details
clj -Sdescribe
{:version "1.9.0.358"
:config-files ["/usr/local/Cellar/clojure/1.9.0.358/deps.edn" "/Users/dgregoire/.clojure/deps.edn" "deps.edn" ]
:install-dir "/usr/local/Cellar/clojure/1.9.0.358"
:config-dir "/Users/dgregoire/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
## deps.edn entry for depstar (that fails)
com.healthfinch/depstar {:git/url ""
:sha "4aa7b35189693feebc7d7e4a180b8af0326c9164"}
## Error that occurs when it tries to check out depstar
Cloning:
Error building classpath. #2018-03-2805:06Alex Miller (Clojure team)seems weird. can you use -Sforce too to make sure you’re not seeing something in cache?#2018-03-2805:17semperos-Sforce shows same behavior; deleted ~/.gitlibs and local .cpcache as well, no change#2018-03-2805:18seancorfield@semperos {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "8fe8e6fbcfe8e6fbe7faeda1ece0e2"}, :content ("[email protected]")} does not indicate an https: URL -- that sounds like you've used an SSH URL instead.#2018-03-2805:20semperoscould be that it's late and I'm missing something obvious, but the :git/url in my deps.edn (pasted above) is ""#2018-03-2805:21seancorfieldThis worked for me {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "e39086828da39086828dce9b9390"}, :content ("[email protected]")}#2018-03-2805:22seancorfieldSo try a later SHA.#2018-03-2805:22semperosclj -Sdeps '{:deps {com.healthfinch/depstar {:git/url "" :sha "68f3a04f75c988fa032974e8c600c9086812e9ef"}}}'
Cloning:
Error building classpath. #2018-03-2805:23semperosnot limited to this repo, it's happening for others#2018-03-2805:23seancorfieldAre you on the latest clj / tools.deps ?#2018-03-2805:23semperosI pasted my info above, I believe I am; my $HOME/.clojure/deps.edn is blank and my /usr/local one is:
{
:paths ["src"]
:deps {
org.clojure/clojure {:mvn/version "1.9.0"}
}
:aliases {
:deps {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.5.417"}}}
:test {:extra-paths ["test"]}
}
:mvn/repos {
"central" {:url ""}
"clojars" {:url ""}
}
}
#2018-03-2805:24seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "621107030c221107030c4f1a1211"}, :content ("[email protected]")}#2018-03-2805:24semperosI deleted the cache in the local folder, I'll try clearing out the other cache dirs // no dice#2018-03-2805:26semperosclj -Sverbose
version = 1.9.0.358
install_dir = /usr/local/Cellar/clojure/1.9.0.358
config_dir = /Users/dgregoire/.clojure
config_paths = /usr/local/Cellar/clojure/1.9.0.358/deps.edn /Users/dgregoire/.clojure/deps.edn deps.edn
cache_dir = /Users/dgregoire/.clojure/.cpcache
cp_file = /Users/dgregoire/.clojure/.cpcache/2178403036.cp
Refreshing classpath
Clojure 1.9.0
user=>
#2018-03-2805:29seancorfieldI just tried it on OS X like you Tue Mar 27 22:28:33
(sean)-(jobs:0)-(~/clojure/semperos)
(! 504)-> clj -Sdeps '{:deps {com.healthfinch/depstar {:git/url "" :sha "68f3
04f75c988fa032974e8c600c9086812e9ef"}}}'
Cloning:
Checking out: at 68f3a04f75c988fa032974e8c600c9086812e9ef
Clojure 1.9.0
user=> ^D
Tue Mar 27 22:29:20
(sean)-(jobs:0)-(~/clojure/semperos)
(! 505)->#2018-03-2805:30seancorfield(sorry for the weird line breaks there -- I'm ssh'ing into my Mac from Ubuntu on my Windows 10 laptop!)#2018-03-2805:31semperosthanks for giving it a try; glad to know I'm the exception in this regard 🙂#2018-03-2805:34semperospicard-facepalm I apologize for the spam, I have an entry in my global ~/.gitconfig that was the culprit#2018-03-2805:35seancorfieldOh, interesting... what was it?#2018-03-2805:35semperos[url "
#2018-03-2805:37semperosif I do straight git clone things work as expected#2018-03-2805:37seancorfieldAh... that's good to know...#2018-03-2814:28ghadi@semperos JGit is too clever#2018-03-2814:28semperosmy thoughts too 🙂#2018-03-2818:10stephenmhopperHi, I have two questions. (1) Is there a way to make options required for tools.cli and (2) why doesn't the tools.cli repo have an issues tab? https://github.com/clojure/tools.cli#2018-03-2818:12ghadiClojure projects's issues are managed through JIRA, not Github -- see CONTRIBUTING.md#2018-03-2818:13ghadihttps://dev.clojure.org/jira/browse/TCLI#2018-03-2818:13ghadibug tracker ^#2018-03-2818:14stephenmhopperAh, I guess I didn't realize that JIRA was used for anything not in the core Clojure releases. Also, I think I might have answered my own question: https://dev.clojure.org/jira/browse/TCLI-12{:tag :div, :attrs {:class "message-reaction", :title "ok_hand::skin-tone-4"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-03-2818:35seancorfield@stephenmhopper Is the :missing approach sufficient for you? (and, BTW, tools.cli is not tools.deps / clj -- that seems to be a common confusion nowadays)#2018-03-2818:39stephenmhopper@seancorfield Yeah, :missing seems to be working for me. And yeah, I wasn't sure if this was the right channel or not. #clojure seemed too broad. Perhaps setting the topic for the channel would prevent confusion from folks in the future.#2018-03-2818:42seancorfield(The purpose mentions tools.dep api but that only shows when you have the Channel Details panel open)#2018-03-2818:42stephenmhopperCool. Thank you!#2018-03-2818:43seancorfieldI've also had folks open JIRA issues against tools.cli thinking it's the clj/`clojure` command line tooling 🙂#2018-03-2818:44seancorfieldI maintain tools.cli so I'm happy to field any questions you have about it.#2018-04-0121:50dominicmhttps://github.com/juxt/pack.alpha#auto-add-to-project I came up with an installer for pack this evening. It seems to work, it even preserves whitespace correctly. I'm not certain this is a good idea yet, but it certainly makes things easy.#2018-04-0122:28seancorfield@dominicm What is the final argument there? It looks like a SHA but doesn't match anything else? What am I missing?#2018-04-0205:28dominicmIt's the latest version sha. The sha of injector is newer than the code for it. They can probably match (I could potentially even automatically pick it up using horrible hacks)#2018-04-0202:22ehashmanNot quite a tools.deps question, but I'm not sure of a better place to ask---does anyone have any tips on debugging why an AOT-compiled app would be re-JIT compiling when run?#2018-04-0202:23ehashmanI have not looked inside the jar to check the timestamps, but I don't currently have a reason to believe something modified the source code timestamps to be different than the compiled artifacts#2018-04-0202:27Alex Miller (Clojure team)There is a bug in the maven shade plugin that doesn’t preserve time stamps but not many people use that #2018-04-0203:00mikerod@ehashman https://groups.google.com/forum/m/#!topic/clojure/12BOjg1KsWA#2018-04-0203:01mikerodThat is a related discussion I had on this quite a while back#2018-04-0203:02mikerodWhen I was using maven shade plugin, my solution was to use maven exec plugin to open the shaded jar back up, bump all clj source time stamps back to something in the past and repack the jar. Did that in a script. Hacky#2018-04-0203:03mikerodYou could also do the workaround there about not including source clj files #2018-04-0203:05kidpolloare there any guides to using deps + cider?#2018-04-0203:05kidpolloI am having trouble getting it going with refactor-nrepl#2018-04-0203:50seancorfield@kidpollo Did you try asking in #cider ?#2018-04-0203:50seancorfield(I haven't tried it -- I only recently heard that CIDER supports deps.edn)#2018-04-0204:41kidpolloit does but cant get the refactoring tooling to work#2018-04-0204:53seancorfieldHmm, I'm not sure why deps vs lein/boot would affect the refactoring tool...?#2018-04-0205:41dominicmcider-jack-in, last I checked, hardcoded the deps list. It hadn't yet started using plugins properly.#2018-04-0220:26dominicmhttps://github.com/juxt/pack.alpha#auto-add-to-project now supports updates when an alias for pack exists, regardless of name.
I really like the approach that I'm taking here, I think it has legs. I could even see an alternative version of -Sresolve-tags based on this, which doesn't alter your indentation at all. That would be quite valuable, given how varied the indentation preferences are in the community.#2018-04-0518:08phreedhttps://github.com/RickMoynihan/lein-tools-deps is currently broken, I am trying to fix it.
The problem is in the call to https://github.com/clojure/tools.deps.alpha/blob/master/API.md#resolve-deps .
Expanding org.clojure/clojure {:mvn/version 1.9.0}
make-session nil /home/fred/.m2/repositoryjava.lang.NullPointerException: null
at clojure.tools.deps.alpha.util.maven$make_session.invokeStatic (maven.clj:106)
clojure.tools.deps.alpha.util.maven$make_session.invoke (maven.clj:102)
The system https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj#L74 variable is nil.
As you can see the system variable should be created by https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L84
When I run the relevant stuff from the repl (via clj) it all works fine.
https://github.com/phreed/lein-tools-deps/blob/master/scratch/plugin.clj
But from within the plugin I get the error mentioned above.
https://github.com/phreed/lein-tools-deps/blob/master/src/lein_tools_deps/plugin.clj#L45
Any insight?#2018-04-0518:18Alex Miller (Clojure team)note that lein-tools-deps is using a much older version of tools.deps.alpha than master#2018-04-0518:22Alex Miller (Clojure team)I’m not sure what the problem is that you’re seeing, maybe something to do with loading the right extension namespaces first (some of that is now done for you in current tda but did not used to be)#2018-04-0519:13phreedI am working with a fork of lein-tools-deps which is using 0.5.418-SNAPSHOT
https://github.com/phreed/lein-tools-deps/blob/master/project.clj
https://github.com/phreed/lein-tools-deps/blob/master/deps.edn
Here is the failing line...
https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L84
Here is the function that is failing
(:import
;; maven-resolver-api
[org.eclipse.aether RepositorySystem RepositorySystemSession]
...
(def the-locator
(delay
(doto (MavenRepositorySystemUtils/newServiceLocator)
(.addService RepositoryConnectorFactory BasicRepositoryConnectorFactory)
(.addService TransporterFactory FileTransporterFactory)
(.addService TransporterFactory HttpTransporterFactory)
(.addService TransporterFactory WagonTransporterFactory)
(.setService WagonProvider CustomProvider))))
(defn make-system
^RepositorySystem []
(.getService ^ServiceLocator @the-locator RepositorySystem))#2018-04-0519:16phreedApparently the-locator is nil.#2018-04-0519:23Alex Miller (Clojure team)Well that should never be :)#2018-04-0519:24Alex Miller (Clojure team)What’s up with the other deps - do those match what tda is using?#2018-04-0519:24Alex Miller (Clojure team)Those seem to be the old aether libs not maven-resolver#2018-04-0519:26phreedWhat is tda?#2018-04-0519:26phreed...and what other deps?#2018-04-0519:28phreedtools.deps.alpha#2018-04-0519:30phreedYou mean [org.eclipse.aether/aether-api "0.9.0.M2"] ?#2018-04-0519:30phreedI put those in to try to fix the issue, it did not change anything.#2018-04-0520:16Alex Miller (Clojure team)As you can see locator is a delay instance, so it should never be nil, even if it failed to connect#2018-04-0520:29phreedRight, I moved things around so I could see what was going on better and now I get...
clojure.lang.Compiler$CompilerException: java.lang.IllegalArgumentException: No matching method: newServiceLocator, compiling:(lein_tools_deps/plugin.clj:52:3)
on
(def locator
(MavenRepositorySystemUtils/newServiceLocator
(.addService RepositoryConnectorFactory BasicRepositoryConnectorFactory)
(.addService TransporterFactory FileTransporterFactory)
(.addService TransporterFactory HttpTransporterFactory)
(.addService TransporterFactory WagonTransporterFactory)
(.setService WagonProvider CustomProvider)))#2018-04-0520:33phreedWhich makes no sense
https://maven.apache.org/ref/3.5.2/xref/index.html#2018-04-0520:34phreedWhich is clearly in here https://mvnrepository.com/artifact/org.apache.maven/maven-plugin-api/3.5.2#2018-04-0520:42phreedAnd, when I use clojure.reflect it shows
| :name | :return-type | :declaring-class | :parameter-types | :exception-types | :flags |
|-------------------+---------------------------------------------------+-----------------------------------------------------------------+------------------+------------------+--------------------|
| newServiceLocator | org.eclipse.aether.impl.DefaultServiceLocator | org.apache.maven.repository.internal.MavenRepositorySystemUtils | [] | [] | #{:public :static} |
| newSession | org.eclipse.aether.DefaultRepositorySystemSession | org.apache.maven.repository.internal.MavenRepositorySystemUtils | [] | [] | #{:public :static} |#2018-04-0520:44phreedI am an idiot. I removed the doto as well as the delay.#2018-04-0520:48richiardiandreahello folks, I was wondering if somebody else is working on updates to the linux installer by any chance (e.g., use install) ?#2018-04-0521:35Alex Miller (Clojure team)There are some outstanding patches that I intend to apply for next #2018-04-0522:15dominicmI wrote a patch for install after we had issues on training#2018-04-0523:49richiardiandreacool thanks for sharing, meaning to have a look at that after we had problems too here#2018-04-0520:59phreedIt is actually getService that is returning the nil https://maven.apache.org/resolver/apidocs/org/eclipse/aether/spi/locator/ServiceLocator.html#getService(java.lang.Class)#2018-04-0521:40phreedI upgraded leiningen to 2.8.1 (from 2.7.x) and the problem resolved itself. Maybe a transitive dependency on a maven library. Thanks.#2018-04-0521:54Alex Miller (Clojure team)Yikes #2018-04-0522:55seancorfield@phreed FYI, if you use triple back-ticks around code, it's much easier to read in Slack like this
#2018-04-0806:44seancorfieldFYI [seancorfield/boot-tools-deps "0.4.4"] just went to Clojars: it bumps the tools.deps.alpha dependency to 0.5.417 to address the slf4j logger warning.#2018-04-0816:15mfikesI have a plk script that composes clojure (for the purposes of classpath construction) with planck. This is working out fairly nicely, but there is a hack I needed to do in order to support :main-opts, by parsing the output of -Sverbose 😱
It makes me wonder if, akin to -Spath, a -Smain-opts would be generally useful.
(The fragile hack is at the bottom of https://gist.github.com/mfikes/bbf8a9a5b9ed6ecac15a3bb75d2dc8b4 if curious.)#2018-04-0816:16mfikes(Or alternatively, more info in -Sdescribe could lead to a solution.)#2018-04-0818:07Alex Miller (Clojure team)Happy to consider additions to -Sdescribe - file a ticket if so#2018-04-0819:32mfikesAt the very bottom of /usr/local/bin/clojure, does it make sense to have the script do
exec "$JAVA_CMD" ...
in order to have the shell script launching the Clojure process terminate (or be replaced). I'm asking because TBH I don't fully appreciate the consequences and wondering if anyone familiar with this aspect of Unix might know.#2018-04-0822:25tianshuis it possible to get the paths in program when using tools.deps. I try to write a figwheel.clj build script, but don't want hardcode the paths.#2018-04-0823:24Alex Miller (Clojure team)@doglooksgood could you explain more about what paths you mean?#2018-04-0823:25Alex Miller (Clojure team)@mfikes et al - I have deprecated the INST jira project, please use TDEPS for anything installer / clj related. Since I work on both of these in tandem, it’s just easier for me to manage and prioritize one bucket instead of two. And if you find a link to the INST one, let me know so I can update it.#2018-04-0823:25mfikesWill do!#2018-04-0823:26tianshu@alexmiller I want the :paths in deps.edn.#2018-04-0823:28Alex Miller (Clojure team)generally speaking, I would say yes. but the answer is somewhat subtle.#2018-04-0823:32Alex Miller (Clojure team)there are really three different levels at work here:
1) clj command line script (bash) - fully environment aware with knowledge of multiple sources for deps.edn and caching concerns. The script knows how to find all deps.edn sources and invoke make-classpath
2) make-classpath program (Clojure) - a small program that knows how to read and combine things it is told by #1 to use tools.deps to build a classpath
3) tools.deps resolve-deps / make-classpath - know how to do dependency resolution etc#2018-04-0823:33Alex Miller (Clojure team)to find the same :paths that clj will use, you need at least the top two levels (and possibly the third, if you want to consider the modifications from :extra-paths in aliases)#2018-04-0823:35Alex Miller (Clojure team)but you might be able to take shorter paths to do things that are usually enough - it depends what other constraints/assumptions you can rely on and how close to clj you need to be.#2018-04-0823:37Alex Miller (Clojure team)for example, if you know your local project.clj always has the paths in it and that’s all you want, then you could just literally read the file as edn and extract the :paths key.#2018-04-0823:37Alex Miller (Clojure team)and you don’t even need tools.deps to do that#2018-04-0823:38tianshuI my case, I write a .clj file which is a script file to launch figwheel build, and this require a source-path that contains .cljs files. I want to specify source-path with aliases in tools.deps#2018-04-0823:40Alex Miller (Clojure team)there’s no way to self refer to the paths via a magic variable or anything like that#2018-04-0823:43tianshuonce the clj or clojure started, the paths will never change?#2018-04-0823:47tianshuif it is, provide apis or variables can make something easier, like finding the paths or deps. so some general purpose scripts can be write and reuse. I think this can help people use their own project to customize their workflow.#2018-04-0823:52Alex Miller (Clojure team)as described above, there are a lot of nuances to how clj and tools.deps works#2018-04-0823:53Alex Miller (Clojure team)this tool is designed to launch programs prepared with certain dependencies on the classpath#2018-04-0823:54Alex Miller (Clojure team)perhaps a better way to ask the question is - can you write a program that can be launched by this tool that will output the desired information?#2018-04-0823:55Alex Miller (Clojure team)like clj -e '(System/getProperty "java.class.path")'
#2018-04-0823:55tianshuyes, that is.😶 sorry for my bad question.#2018-04-0823:56Alex Miller (Clojure team)not a bad question, just trying to rephrase to alignment with the goal of the tool#2018-04-0823:57Alex Miller (Clojure team)is the output of the command above the answer you seek? or at least the input? that could easily be filtered to sub paths of the current directory#2018-04-0823:58tianshuyes, I think this is what I seek!#2018-04-0823:59tianshusuch a simple way:joy:#2018-04-0900:00tianshuthanks very much!#2018-04-0916:20Drew VerleeI think it would be really cool to write/blog/livecast someone translating a project.clj to a deps.edn. I mainly think this is cool because im having to do it right now and its very education, by which i mean, i have lots of things I dont yet understand 🙂#2018-04-0916:24dominicmOh, I did it by regex#2018-04-0916:25dominicm:%s/\[\(\S\+\) \(".*"\)\]/\1 {:mvn\/version \2} is how I did it in vim.#2018-04-0916:25dominicmThen I also ran :%s/\[\(\S\+\) \(".*"\)\ \(:exclusions \[.*\]\)]/\1 {:mvn\/version \2\r\3}#2018-04-0916:25Drew Verleeis it that mechanical of a process? Its possible this hinges on my understanding of what deps.edn does vs what lein project.cljs do#2018-04-0916:26dominicm@drewverlee deps.edn only specifies dependencies. So that's the only part that really converts over from project.clj.#2018-04-0916:26Drew VerleeI guess your right, the only thing i have to transfer is the deps#2018-04-0916:26Drew Verleeyep#2018-04-0916:27Drew Verlee@dominicm
out of curiosity, for those trying to use deps.edn, what are they using for
1) profiles
2) build stuff
3) deployment
I’m guessing boot can still handle these responsibilities#2018-04-0916:27dominicmThere are tools, like pack.alpha which add uberjars on top.#2018-04-0916:28dominicmhttps://github.com/juxt/edge has an example of doing most of this.#2018-04-0916:28dominicmEdge doesn't have a pack example yet, bridge does: https://github.com/robert-stuttaford/bridge/#2018-04-0916:28dominicm(although it's out of date, see the pack readme!)#2018-04-0916:34Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools has some tools#2018-04-0916:36Alex Miller (Clojure team)aliases can serve some of the same purposes as profiles. pack.alpha and depstar are two build options.#2018-04-0916:37Alex Miller (Clojure team)if you have consumers use git deps, you can skip build and deploy altogether :) (I am somewhat joking - this is not feasible for a number of cases)#2018-04-0917:13dominicm@alexmiller I've been thinking about krei, there's an API for performing java complication in the jdk, so in theory if I search for an edn file describing how to build from source, it could be feasible.#2018-04-0917:14dominicmHaving git dependencies is a lot like working on a mono repo, which is the pain point krei was designed to solve.#2018-04-0917:15dominicmI think it could work, in theory. Haven't tried that hard yet though.#2018-04-0917:24Alex Miller (Clojure team)What’s krei?#2018-04-0917:32dominicm@alexmiller Krei is a in-repl build tool. It scans the classpath for krei-file.edn files, and then builds sources on the classpath from that.
Currently it supports cljs and sass, because that's what I need for the itch I'm scratching. But I think it could work for java too. Protobuf is probably important to target also.#2018-04-0917:33Alex Miller (Clojure team)gotcha#2018-04-0917:36dominicmMaybe I should split into manifests and try to parse information from pom files? Dunno.#2018-04-0917:43Alex Miller (Clojure team)well I can tell you that parsing info from pom files suuuuucks#2018-04-0917:45dominicmOnly using edn it is!#2018-04-0919:16seancorfield@drewverlee Take a look at this commit where I converted core.cache from using Leiningen to clj/`deps.edn`: https://github.com/clojure/core.cache/commit/1f76bdf7321bce7c5d8c1424943afeccff0fb6d5#2018-04-0919:17seancorfieldAnd here's a slightly more comprehensive example from java.jdbc being switched over https://github.com/clojure/java.jdbc/commit/e8040925beddab6048873625cfe791341082290d#2018-04-0919:19seancorfieldSo :profiles => various :aliases, "build" can either use tooling built on top of deps.edn (for example, I use boot-tools-deps with Boot on top of deps.edn and there's a Leiningen plugin, which is very much a work-in-progress), or it can use the JAR builders that are listed on that wiki page...#2018-04-1115:01bhaumanas folks migrate to tools-deps are people still using a "resources" directory to indicate static resources?#2018-04-1115:02Alex Miller (Clojure team)probably so#2018-04-1115:06bhaumanI'm trying to come up with reasonable initial instructions for where people should place their index.html#2018-04-1115:07bhaumanso I'll keep rolling with resources/public/index.html#2018-04-1115:08bhaumanalso I was surprised that if you place "resources" in :paths that "resources" gets appended to the classpaths of all the :local/root's#2018-04-1115:29dominicmthat sounds like a bug.#2018-04-1115:36Alex Miller (Clojure team)that is a bug that has been mentioned before. possibly no one logged it as I had forgotten about it.#2018-04-1115:37Alex Miller (Clojure team)nope it’s logged https://dev.clojure.org/jira/browse/TDEPS-52#2018-04-1115:38Alex Miller (Clojure team)definitely will fix#2018-04-1116:03bhaumanoh cool, I was worried that it was on purpose#2018-04-1116:09Alex Miller (Clojure team)no, definitely not{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1222:45hagmonkI spent some time trying to make AOT for my app work with tools.deps, finally watched dtrace for a bit and figured out the magic :)#2018-04-1222:45hagmonkIn this case I'm also pulling in rebel-readline and clj-async-profiler:
clj -Sdeps "{:deps {com.bhauman/rebel-readline {:mvn/version \"0.1.2\"} com.clojure-goes-fast/clj-async-profiler {:mvn/version \"0.1.3-SNAPSHOT\"}} :paths [\"classes\" \"src\"]}" -m rebel-readline.main
#2018-04-1222:46hagmonkOh, and one must make the classes folder in advance too#2018-04-1223:09hagmonk@alexmiller is it expected that maven deps with scopes other than "compile" are filtered out? Looking at https://github.com/clojure/tools.deps.alpha/blob/582aee5c7ac43991fb3154a4596910a9d3a0c4f3/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj#L81#2018-04-1223:10hagmonkI found some deps on libs pulled from our corporate artifactory instance were missing, seemingly because they have the scope "runtime" set#2018-04-1223:33Alex Miller (Clojure team)That’s intentional but still thinking about whether to include runtime#2018-04-1300:12seancorfieldWhat does a runtime scope mean for Maven?#2018-04-1300:33Alex Miller (Clojure team)It’s for running things :)#2018-04-1300:34Alex Miller (Clojure team)But not needed to compile#2018-04-1303:21hagmonkHum, well, that's an interesting distinction#2018-04-1303:22hagmonkIs the classpath assembled for a deps.edn based project intended to get it only as far as compilation? I hit this problem through a transient dependency, and I actually had to use lein deps :tree to discover the problem, because tools.deps.alpha completely filtered out the existence of the transient dependency#2018-04-1303:39Alex Miller (Clojure team)yeah, there’s a ticket for this already#2018-04-1303:40Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-26#2018-04-1303:40Alex Miller (Clojure team)feel free to add your vote#2018-04-1303:49hagmonkwill do sir, thanks#2018-04-1306:52seancorfield@richiardiandrea made me do it! clj -Sdeps '{:deps {seancorfield/clj-new {:git/url "" :sha "653441d86834160124a94065c4d92a4a9ad12554"}}}' -m clj-new.create app myapp/clj-test The beginnings of a clj-based new project template generator...{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1313:16johnSo, I can just do clj -m clj-new.create figwheel myapp and it'll bring in the leiningen template and build it?#2018-04-1313:34johnWhen I do clj -Sdeps '{:deps {seancorfield/clj-new {:git/url "" :sha "653441d86834160124a94065c4d92a4a9ad12554"}}}' -m clj-new.create figwheel figfig#2018-04-1313:35johnI get
Failed with: Failed to read artifact descriptor for figwheel:lein-template:jar:RELEASE
Exception in thread "main" clojure.lang.ExceptionInfo: Could not load artifact for template: figwheel
Tried coordinates:
[figwheel/boot-template "RELEASE"]
[figwheel/lein-template "RELEASE"] {}
at clojure.core$ex_info.invokeStatic(core.clj:4754)
at clojure.core$ex_info.invoke(core.clj:4754)
...
#2018-04-1314:50richiardiandreaAll in clj ;)#2018-04-1315:30seancorfieldAs noted, work in progress. Needs more work before it can do more than the built-in templates. But, yes, the plan is that will work. #2018-04-1315:30johnniiiiiice!#2018-04-1315:33seancorfieldI think that fails because I haven't set up the full clj environment for t.d.a. yet (in particular the repos).#2018-04-1320:18seancorfield@john OK, clj -Sdeps '{:deps {seancorfield/clj-new {:git/url "" :sha "a87bdc0347fe1523e8d85283d94b06f5dd68b2cb"}}}' -m clj-new.create figwheel figfig This should work now!{:tag :div, :attrs {:class "message-reaction", :title "parrot"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "parrot", :src "https://emoji.slack-edge.com/T03RZGPFR/parrot/d3b061bf9cff6f2e.gif"}, :content nil})} " 4")}
#2018-04-1320:19johnHahah!#2018-04-1320:20johntworks!#2018-04-1320:20johnthis is newsworthy#2018-04-1320:20johnwe'll have to whip up some clj/cljs.main templates 🙂#2018-04-1320:24seancorfieldI plan to add clj-template support soon, so we can have clj templates that produce deps.edn-based projects.#2018-04-1320:46richiardiandrealet me try macchiato#2018-04-1320:48richiardiandreaworks! awesome 😄#2018-04-1306:53seancorfield(it currently generates boot templates by default -- but it should be able to generate any Leiningen template too)#2018-04-1306:54dominicmLol, that's great.#2018-04-1306:54seancorfieldHe made me itch. I had to scratch it.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1306:55dominicmI spent far too long last night writing an ANSI parser, I feel the sentiment quite hard right now :){:tag :div, :attrs {:class "message-reaction", :title "simple_smile"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "simple_smile", :src "https://a.slack-edge.com/80588/img/emoji_2017_12_06/apple/simple_smile.png"}, :content nil})} " 4")}
#2018-04-1314:52Alex Miller (Clojure team)someone dropped a docker command to spin a linux box for testing clj installs here a couple weeks ago - anyone have that handy?#2018-04-1314:54dominicmI wouldn't use that for testing#2018-04-1314:55dominicmDocker runs as root within the container, you miss a key aspect of testing that way#2018-04-1314:55Alex Miller (Clojure team)I wasn’t only going to only use it#2018-04-1315:50mkvlr> docker run -ti openjdk:8 /bin/bash#2018-04-1315:50hagmonk@dominicm that's true by default but it's trivially disabled#2018-04-1315:54mkvlrapt-get update
apt-get install -y rlwrap curl
export CLOJURE_VERSION=1.9.0.341 && curl -O && /bin/bash linux-install-${CLOJURE_VERSION}.sh#2018-04-1315:55hagmonkalthough for the purposes of installing clojure, it looks like the getting started guide recommends root anyway#2018-04-1315:55mkvlr@alexmiller these two things should do or do you want a built image?#2018-04-1315:57dominicm@hagmonk running it is the difficulty#2018-04-1315:58hagmonkrunning it?#2018-04-1317:04dominicmRunning the script as non-root is the important part, to ensure that OS permissions are correctly set#2018-04-1315:58hagmonkdocker run -ti openjdk:8 bash -c "$(curl ) && clojure"{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1316:08hagmonkNow with @bhauman power up mode: docker run -ti openjdk:8 bash -c "$(curl ) && clojure -Sdeps \"{:deps {com.bhauman/rebel-readline {:mvn/version \\\"0.1.2\\\"}}}\" -m rebel-readline.main"#2018-04-1316:09hagmonkI have a docker container internally that already has a .cpcache for fast rebel readline startup in a container :) Useful for when I need a totally clean slate#2018-04-1316:32Alex Miller (Clojure team)@dominicm curious what you think about https://dev.clojure.org/jira/browse/TDEPS-64#2018-04-1316:33Alex Miller (Clojure team)do you think the prefix dir should be created if it doesn’t exist?#2018-04-1316:38Alex Miller (Clojure team)with the TDEPS-65 patch for the linux install, it will be created if it doesn’t exist#2018-04-1316:38Alex Miller (Clojure team)I think I’m ok with that behavior#2018-04-1317:03dominicmSee no reason not to add a guard if we want to ensure that the prefix exists already. I suppose it's keeping with other tools to not create leading directories by default. I ignored that comment because I was fairly certain that the current behavior didn't follow that.#2018-04-1317:27hagmonkI'd actually argue, just from memory and experience, that directories should be created if it they don't already exist. I can't recall the last time I supplied any directory or prefix argument to a tool and then thought "shoot, I have to manually make that directory"#2018-04-1317:27hagmonkBut that's just my own experience#2018-04-1317:29Alex Miller (Clojure team)this is mostly an issue for when you are supplying the prefix dir#2018-04-1317:29hagmonkright, I'd expect that if I supply the prefix dir, it will be created if it doesn't exist#2018-04-1317:29Alex Miller (Clojure team)yeah#2018-04-1317:30hagmonkI'm trying to think of a case where I've done something similar recently#2018-04-1317:31hagmonkI guess maybe the classic case is ./configure --prefix=/usr/local/whatever#2018-04-1317:31hagmonkwhen using autotools#2018-04-1317:32hagmonkI checked the nearest autotools generated configure script …#2018-04-1317:33hagmonkthey definitely mkdir -p, although they do it in the most gloriously overwrought tradition of GNU autotools:
# as_fn_mkdir_p
# -------------
# Create "$as_dir" as a directory, including parents if necessary.
as_fn_mkdir_p ()
{
case $as_dir in #(
-*) as_dir=./$as_dir;;
esac
test -d "$as_dir" || eval $as_mkdir_p || {
as_dirs=
while :; do
case $as_dir in #(
*\'*) as_qdir=`$as_echo "$as_dir" | sed "s/'/'\\\\\\\\''/g"`;; #'(
*) as_qdir=$as_dir;;
esac
as_dirs="'$as_qdir' $as_dirs"
as_dir=`$as_dirname -- "$as_dir" ||
$as_expr X"$as_dir" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
X"$as_dir" : 'X\(//\)[^/]' \| \
X"$as_dir" : 'X\(//\)$' \| \
X"$as_dir" : 'X\(/\)' \| . 2>/dev/null ||
$as_echo X"$as_dir" |
sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
s//\1/
q
}
/^X\(\/\/\)[^/].*/{
s//\1/
q
}
/^X\(\/\/\)$/{
s//\1/
q
}
/^X\(\/\).*/{
s//\1/
q
}
s/.*/./; q'`
test -d "$as_dir" && break
done
test -z "$as_dirs" || eval "mkdir $as_dirs"
} || test -d "$as_dir" || as_fn_error $? "cannot create directory $as_dir"
} # as_fn_mkdir_p
#2018-04-1317:33hagmonkeww sorry, I thought that would collapse#2018-04-1318:37seancorfieldQuestion about classloaders -- when I run code with clj -m ... should I be able to get at a modifiable classloader?#2018-04-1318:46seancorfieldIt seems that when you're in the REPL, you have a DynamicClassLoader from Clojure, but when running via -m you only have sun.misc.Launcher$AppClassLoader and sun.misc.Launcher$ExtClassLoader neither of which are modifiable...#2018-04-1318:57dominicmThe goal should be to act like a normal Linux program. Regardless of whether we expect those behaviours ourselves.#2018-04-1319:04seancorfieldWell, to dynamically load Leiningen templates at runtime, I need a modifiable classloader... for now I've made one (`clojure.lang.RT/makeClassLoader`) and stuffed it back into the current thread.#2018-04-1319:46seancorfieldAnyone happen to know what package org.sonatype.aether.collection.DependencyGraphTransformer is part of? When I try to load leiningen-core via t.d.a. this doesn't seem to get loaded...#2018-04-1319:56seancorfieldNM, figured it out.#2018-04-1320:18seancorfield@john OK, clj -Sdeps '{:deps {seancorfield/clj-new {:git/url "" :sha "a87bdc0347fe1523e8d85283d94b06f5dd68b2cb"}}}' -m clj-new.create figwheel figfig This should work now!{:tag :div, :attrs {:class "message-reaction", :title "parrot"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "parrot", :src "https://emoji.slack-edge.com/T03RZGPFR/parrot/d3b061bf9cff6f2e.gif"}, :content nil})} " 4")}
#2018-04-1320:23johnour diabolical plans are almost in place#2018-04-1320:23seancorfieldQualified project names do not work (they don't work in boot-new either).#2018-04-1320:23seancorfieldOther known issues https://github.com/seancorfield/clj-new/issues#2018-04-1320:24johnthis is super sweet though, thanks Sean!#2018-04-1320:24seancorfieldI plan to add clj-template support soon, so we can have clj templates that produce deps.edn-based projects.#2018-04-1320:25seancorfieldI'll work on it more over the weekend and make an official announcement on Monday I expect.#2018-04-1320:54richiardiandreaDon't have to say how cool this is, no more lein or boot for template generation is a boon for lumo projects#2018-04-1321:14Alex Miller (Clojure team)might want to add that to https://github.com/clojure/tools.deps.alpha/wiki/Tools#2018-04-1321:15Alex Miller (Clojure team)new tools.deps.alpha 0.5.425 is out. new installers working their way through the meat grinder#2018-04-1321:16Alex Miller (Clojure team)mostly just bug fix releases for both, I’ll do a ggroups announcement when brew is done#2018-04-1321:19seancorfieldI'll add it to the tools page after it's had some polish this weekend. There are a few issues (opened on GH!) that need addressing before it's ready for real world users! 🙂#2018-04-1321:42johns/lein new/clj -A:new/#2018-04-1322:00seancorfieldclj -A:new right? 🙂#2018-04-1322:00seancorfield(or does it support omitting the leading : now?)#2018-04-1322:02richiardiandreaI have put it directly in my :deps in ~/.clojure/deps.edn#2018-04-1322:43johnma bad#2018-04-1322:44johnso what's it look like? the -m ... line goes in extra-args right?#2018-04-1322:44johnah, main-opts#2018-04-1322:55johnYeah, this works for me:
...
:aliases
{:new {:main-opts ["-m" "clj-new.create"]
:extra-deps
{seancorfield/clj-new {:git/url ""
:sha "a87bdc0347fe1523e8d85283d94b06f5dd68b2cb"}}}
...
#2018-04-1617:01timgilbertAdding this magic entry to ~/.clojure/deps.edn, or a project deps.edn, seems like something that could be written as a useful tool, similar to npm install or yarn add or whatnot - typing all of this info is slightly too annoying right now for use in shell scripts / one-liners IMHO#2018-04-1617:04dominicmI have this in pack.alpha#2018-04-1617:04dominicmPack can inject itself for you.#2018-04-1617:04dominicmI'm thinking it might become a general injector eventually which you can then do clj -A:user/inject new to inject into deps.edn{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1322:57johnthen just do clj -A:new figwheel figthing#2018-04-1323:03Alex Miller (Clojure team)I intentionally wrote the code to be a little lax on that leading colon so it’s optional{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-04-1323:26seancorfieldclj-new SHA "87d175c5e49618fffe8ed16b1bf86aa6f0fbc624" -- https://github.com/seancorfield/clj-new/blob/master/README.md -- a whole bunch of clean up, refactoring, and rewriting the default project templates.#2018-04-1323:38seancorfieldThe built in templates now generate projects with deps.edn files $ clj -A:new app myapp
$ cd myapp
$ clj -m myapp.core
Hello World!
$ clj -A:test:runner
...FIXME test failure...
#2018-04-1617:01timgilbertAdding this magic entry to ~/.clojure/deps.edn, or a project deps.edn, seems like something that could be written as a useful tool, similar to npm install or yarn add or whatnot - typing all of this info is slightly too annoying right now for use in shell scripts / one-liners IMHO#2018-04-1400:24seancorfieldOK, something we've all wanted ever since Leiningen did the .core thing... clj-new SHA "f6fcc24bfa5d77167ff826990cd2c9c65eed4fed" prohibits single-segment or unqualified project names!#2018-04-1400:25seancorfieldclj -A:new app foo/bar or clj -A:new app foo.bar#2018-04-1400:25seancorfieldAnd, with that, I'm out for the day...#2018-04-1400:45richiardiandreaThat's great thanks a lot Sean!#2018-04-1407:56seancorfieldclj-new SHA "025f678af79c460e23f762aeb33dbb4af16f7ec7" now supports templates that are :git/url + :sha or :local/root (+ template name) -- https://github.com/seancorfield/clj-new/blob/master/README.md{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1407:56seancorfield(rewriting the Boot generator stuff will probably be tomorrow/Sunday at this point){:tag :div, :attrs {:class "message-reaction", :title "boot-clj"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "boot-clj", :src "https://emoji.slack-edge.com/T03RZGPFR/boot-clj/940f8bbd956fcb1b.png"}, :content nil})} " 4")}
#2018-04-1413:23johnSo what would be the pros and cons of moving some projects off private mavens completely and just moving to private git shas?#2018-04-1413:24johnor private git urls#2018-04-1413:24johnseems like you're duplicating your deps anyway, if you've got your snapshot in github#2018-04-1413:25johnWhile also just copying them to some yaven...#2018-04-1413:28johnin other words, unless I'm providing to some maven consumer that can't use clj, why use maven at all now?#2018-04-1413:33johnhmm, I wonder if a lot of clojure libs somehow depend on things living on maven...#2018-04-1415:04richiardiandrea@john I think that clj deps are fairly independent from maven but I might be wrong, lumo is experimenting with npm packages for instance: https://andrearichiardi.com/blog/posts/lumo-npm-dependencies.html#2018-04-1415:08dominicm@john None I've run into depend on things being from maven.#2018-04-1415:10mfikes@john One small thing, at least for now is that ClojureScript's :aot-cache doesn't yet work on Git deps. But that could change https://dev.clojure.org/jira/browse/CLJS-2651#2018-04-1415:11john@richiardiandrea @dominicm that's what I was thinking#2018-04-1415:12john@mfikes hmm. Does that just mean that some cljs things won't be cached on first pull of the dep?#2018-04-1415:12johnBut they still cache after that, right?#2018-04-1415:13johnor, if someone is using :aot-cache true then it'll break with git deps, currently?#2018-04-1415:14dominicmIt won't break. They just won't be candidates for caching.#2018-04-1415:15mfikes@john It just means that Git deps behave like things did before :aot-cache existed: They do still cache locally on first build.#2018-04-1415:15johnrgr#2018-04-1415:17johnso if you have a project where you're waiting to migrate to clj tool, because of the ongoing work on private maven repos, you might be able to just skip all that and depend on the git dep directly, right?#2018-04-1415:18dominicmAs long as you control the private project you're interested in, yes.#2018-04-1415:18johnObviously I won't fully know until I try migrating a whole project over, but I'm just doing a sanity check first#2018-04-1415:18dominicmTrue for internal projects, false for datomic 🙂#2018-04-1415:19johnright#2018-04-1421:16seancorfieldThis was something that I noticed as I worked on clj-new -- I forked it from boot-new so at first it still had a build.boot, which was the only place where a version number was stored, and was the only place that had anything to do with building artifacts and deploying them (to Clojars). And I was suddenly "Oh, wait! I don't need a Clojars badge on this project, or version numbers, or artifact coordinates... 👀 "#2018-04-1421:19mfikesIn ClojureScript there was a similar issue where it normally produces a version number as part of its build process, which caused a little grief with trying to use ClojureScript as a "buildless" git dep. But that has been resolved (parts of the system have been changed to cope without a version number).#2018-04-1421:24seancorfield@mfikes Some Leiningen templates expect to be able to get at Leiningen's version number, which it assumes is in META-INF/maven/leiningen/leiningen/pom.properties -- which is why both boot-new and clj-new have that tree structure in src.#2018-04-1421:25dominicmWould be interesting to know "your" version (as defined in the deps.edn) somehow. I guess it could be set by clj using system properties when starting a jvm... :thinking_face:#2018-04-1421:25mfikesIn ClojureScript's case the solution we went with was to derive a fake version number from the hashes of all of the code in the source tree#2018-04-1421:26dominicmThat gives me 2 use cases for contextual information now: tools.namespace could use it for reloading, and to grab a common piece of build information: version.#2018-04-1421:27dominicmI suppose it would need to be burned into the resulting UberJARs... but I can take of that 🙂#2018-04-1421:26seancorfieldAnd also the leiningen.release namespace. I tried to incorporate Leiningen 2.8.1 but it has a namespace with a def that slurps one of its own files when you load it(!), so I stuck with 2.7.1. Looks like it'll be "fixed" in 2.8.2.#2018-04-1421:27dominicmI suppose it would need to be burned into the resulting UberJARs... but I can take of that 🙂#2018-04-1421:29mfikesMy takeaway: Anything that needs to be "built" can be a hinderance for just using code directly as a dep.#2018-04-1421:30mfikes(Including something as simple as a version number.)#2018-04-1421:47hagmonkhey this is germane to what I'm working on#2018-04-1421:48hagmonkI started patching depstar to support building thin jars and producing a folder of jar artifacts#2018-04-1421:49hagmonkfor the purposes of building a kafka connector (which requires a thin jar + deps in a dir), and for separating deps into a separate docker file layer for speedier builds#2018-04-1421:50hagmonkquestion I'm facing at the moment: should things like version numbers or aliases that configure how to build the jar be in deps.edn, or kept exclusively as command line arguments?#2018-04-1421:53hagmonkI don't see anyone shoving extra stuff into deps.edn, so I'm somewhat wary about storing build metadata in there#2018-04-1422:11seancorfieldThere are pros and cons to both.#2018-04-1422:12seancorfieldIf you have a project that expects to be built a particular way, it makes sense to have the build stuff in the project (so an alias for using depstar etc sounds reasonable).#2018-04-1422:12seancorfieldIf you have a project that can be used "as-is" but could be built using any tooling available, don't put the build stuff in the project.#2018-04-1422:12seancorfieldThat would be my guidance.#2018-04-1422:14hagmonkthanks @seancorfield … right now I'm patching in clojure.tools.cli support to see how that feels in use#2018-04-1422:15seancorfieldI haven't switched that over to deps.edn yet. It's on my list.#2018-04-1422:15seancorfieldBut I also want to switch it to .cljc for ease of maintenance.#2018-04-1422:16seancorfield(and drop pre-1.7 support)#2018-04-1422:20hagmonkHah, I just realized depstar is @ghadi’s handiwork. That's why this code is so tight :)#2018-04-1423:28hagmonkOk, it's a bit unpolished, but if folks want to give the thin jar / dependency copying support I added, start here:
clojure -Sdeps '{:deps {depstar {:git/url "" :sha "9524dccdb37ee85b79d3a505447d1e5101f1a73d"}}}' -m hf.depstar.core -h
#2018-04-1423:34hagmonkThe TL;DR version is that -m hf.depstar.core jar works like lein jar, and -m hf.depstar.core deps works like lein-libdir#2018-04-1500:57Alex Miller (Clojure team)from my perspective re deps.edn: don’t put arbitrary other gunk in it.#2018-04-1500:58Alex Miller (Clojure team)I make no guarantees that arbitrary stuff will get properly merged across multiple deps files and/or transmitted through the code#2018-04-1501:04hagmonk@alexmiller thanks, that’s a compelling reason to keep things out of there :)#2018-04-1501:05Alex Miller (Clojure team)(although I think it probably will right now)#2018-04-1501:15richiardiandrea@dominicm congrats for the install patch, feels way better. It is also nicer to see the names of the contributors in the log. It was missing before but it goes a long way to "thank" contributors. FWIW, I like this idea.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1502:00Alex Miller (Clojure team)I have done it a couple times in the past, I just forget sometimes.#2018-04-1501:15hagmonkOh cool, TDEPS-26 was fixed too#2018-04-1502:01Alex Miller (Clojure team)for the record… https://groups.google.com/d/msg/clojure/6vEgPw5EW4U/9uamLErFBwAJ#2018-04-1502:01Alex Miller (Clojure team)New releases of tools.deps.alpha and the Clojure tools (clj, clojure) are now available. This is mostly a bug fix release.
To update, use `brew upgrade clojure` on Mac, or the Getting Started installation instructions on Linux.
Windows installers are coming soon(ish)! I have an initial version working and it's about 80% done, so only 80% to go. :)
Updates:
- TDEPS-73 detect invalid deps.edn file and improve error
- TDEPS-61 use Clojars CDN repo
- TDEPS-26 include transitive runtime deps in addition to compile deps
- TDEPS-71 better error if Java not installed - primarily for Linux installer (patch thanks to Mike Fikes)
- TDEPS-65 specify permissions on installed files in Linux installer (patch thanks to Dominic Monroe)
#2018-04-1502:35seancorfieldI ended up not working on clj-new today, but may work on it tomorrow. I still plan to make an announcement about it early next week either way (we have a major infrastructure upgrade on Monday so I suspect it'll be Tuesday when I write up the clj-new stuff).#2018-04-1502:36seancorfieldI did bump the t.d.a. dep to the new release tho' 🙂#2018-04-1502:36seancorfield(locally)#2018-04-1519:47hagmonkso I'm trying to build an uberjar targeting a kafka connector … for the life of me I can't get kafka to load it, it barfs on Could not locate clojure/core__init.class or clojure/core.clj on classpath.#2018-04-1519:47hagmonkcore__init.class is definitely in the uberjar, however#2018-04-1519:48hagmonkbuilding this as an example with depstar, so I'm trying to figure out if the uberjar put together is somehow wrong#2018-04-1519:48hagmonk[2018-04-15 19:47:23,641] INFO Loading plugin from: /connectors/postgres-plugin.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176)
[2018-04-15 19:47:23,646] DEBUG Loading plugin urls: [file:/connectors/postgres-plugin.jar] (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:183)
Exception in thread "main" java.lang.ExceptionInInitializerError
at clojure.lang.Namespace.<init>(Namespace.java:34)
at clojure.lang.Namespace.findOrCreate(Namespace.java:176)
at clojure.lang.Var.internPrivate(Var.java:153)
at thals.pgconnector.SourceConnector.<clinit>(Unknown Source)
#2018-04-1519:49hagmonkhow's this jar managing to get as far as clojure.lang.Namespace but then claiming it's unable to find clojure/core__init.class …#2018-04-1519:54hagmonkoh crap, it's the classloader thing#2018-04-1520:20dominicmCan you link me to the documentation on how kafka loads classes? I'm the author of pack, and I'd like to read about what I'd need to do to support their preferred format.#2018-04-1520:30hagmonksure#2018-04-1520:32hagmonkbut, unfortunately, I don't think it's related to the jar format. I believe they start a new classloader and load all the jars into that, which isn't visible to clojure because use-context-classloader is set to true#2018-04-1520:33hagmonkI'm trying to figure out if there's a way to get around this without resorting to java#2018-04-1520:34hagmonkin terms of how they want stuff packed, it's documented here: https://docs.confluent.io/current/connect/devguide.html#packaging#2018-04-1520:35hagmonkthe tl;dr version is that they can take uberjars, or a directory full of jar dependencies (excluding kafka runtime goop), including obviously the jar containing the connector implementation#2018-04-1520:55dominicmThe zip version would work better, and fits with the
Pack philosophy.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-1702:23cflemingAm I correct in thinking that deps.edn provides no way to build an artifact?#2018-04-1702:24cflemingOr even to specify co-ordinates for one?#2018-04-1702:33richiardiandreaI think you understand correctly. Or put in a better way I understand the same.#2018-04-1702:53dpsuttonHave you seen ghadi's depstar?#2018-04-1702:53dpsuttonhttps://github.com/healthfinch/depstar#2018-04-1702:57cflemingI have, but that just creates a jar#2018-04-1703:05cflemingAlso, is Clojure itself an implicit dependency? The examples don’t add it like lein does.#2018-04-1703:08dpsuttonThat does an Uber jar so I think it gets the deps of the clj runner I think#2018-04-1703:09dpsuttonIt removes itself though#2018-04-1703:09dpsuttonDepstar itself I mean#2018-04-1703:09cflemingOh, I meant in general - if I run a REPL, the Clojure dep comes from the runner?#2018-04-1703:10dpsuttonAs far as I can tell yes. I've never seen deps.edn specify a clojure version#2018-04-1703:11cflemingInteresting. A lot of this is going to be really weird for Cursive support 😐#2018-04-1703:12dpsuttonIt's been rough in CIDER so far as well. It feels a bit like boot. It's not super obvious how to best approach it#2018-04-1703:12cflemingYeah, the loose approach to project identity is tricky.#2018-04-1703:12dpsuttonI think @dominicm has done some work towards that end but it's still a little brittle#2018-04-1703:15dpsuttonKinda like boot you need to know what aliases or tasks to invoke#2018-04-1703:15cflemingYeah, boot is tricky for Cursive too 🙂#2018-04-1703:17seancorfield@cfleming clj -Sdescribe lists the deps.edn files that are read in.#2018-04-1703:18seancorfieldYou can also get that from tools.deps.alpha as clojure-environment (I think).#2018-04-1703:18seancorfieldMy bad: clojure.tools.deps.alpha.reader/clojure-env#2018-04-1703:18cflemingThe difficulty is there’s no consistent identifier for a module#2018-04-1703:18seancorfield(it shells out to clj -Sdescribe tho')#2018-04-1703:19dpsuttonDo you inject your dependencies like cider or do you run another process? I know cursive is big on static analysis#2018-04-1703:19dpsuttonI think we're gonna have to merge edn files from emacs lisp#2018-04-1705:29dominicmNope. -Sdeps. I already have this working in viml, elisp will be a doddle#2018-04-1703:20cfleming@dpsutton I don’t inject any dependencies.#2018-04-1703:20cflemingMost of Cursive’s functionality is static in the editor.#2018-04-1703:23seancorfieldFor clj-new I ended up having to ask clj what deps.edn files it would load, reading them, resolving them, making a classpath, and loading those files... environment (clojure-env)
all-deps (read-deps (:config-files environment))
...
(-> (deps/resolve-deps all-deps resolve-args)
(deps/make-classpath (:paths all-deps) {})
(str/split (re-pattern java.io.File/pathSeparator))
(->> (run! pom/add-classpath))))
#2018-04-1703:24cflemingWhere are git deps cloned to?#2018-04-1703:25cflemingIs there a local .deps directory or something?#2018-04-1703:43cflemingAh, ~/.gitlibs#2018-04-1703:43cflemingIs it possible to use file:/// URLs with :git/url?#2018-04-1703:44seancorfieldYou can use :local/root#2018-04-1703:44cflemingI can’t get it to work.#2018-04-1703:44seancorfield(instead)#2018-04-1703:44cflemingRight, but then I can’t pin a version.#2018-04-1703:45cfleming(although I’m mostly trying to avoid having to publish to github just to play around 🙂 )#2018-04-1703:45seancorfieldSo :local/root deps just stay where they are on disk, with no version. :git/url is cached to ~/.gitlibs per :sha. :mvn/version is cached to ~/.m2 with a real version.#2018-04-1703:56Alex Miller (Clojure team)@cfleming re prior question, the default Clojure version is specified in the install-level deps.edn (which is merged with ~/.clojure/deps.edn and local deps.edn). You can specify it in local deps.edn too if you like.#2018-04-1703:57cflemingI’m thinking about the case when a Cursive user imports a deps project and maybe doesn’t even have the CLI installed.#2018-04-1703:58cflemingWhere is the install-level stuff put? I guess that’s OS/packager dependent?#2018-04-1703:59Alex Miller (Clojure team)yeah, but clj -Sdescribe will tell you that#2018-04-1703:59Alex Miller (Clojure team)you can use the tools.gitlibs library to determine where the gitlibs are being cached https://github.com/clojure/tools.gitlibs/blob/master/src/main/clojure/clojure/tools/gitlibs.clj#L23#2018-04-1703:59cflemingThe problem is that IntelliJ, at least on OSX, doesn’t typically know what the users’s shell path is.#2018-04-1704:00cflemingWill any of this work at all if the user doesn’t have the clj CLI stuff installed?#2018-04-1704:00Alex Miller (Clojure team)no#2018-04-1704:01cflemingThat’s a bummer.#2018-04-1704:01Alex Miller (Clojure team)I mean, some of it can be used directly via the lib but the lib really works in tandem with the scripts#2018-04-1704:01cflemingHmm#2018-04-1704:02Alex Miller (Clojure team)the scripts are the bridge from the user’s environment and config files into the lib#2018-04-1704:02cflemingI’ll have to see if I can detect the install, and consider what to do if it’s not there or I can’t find it.#2018-04-1704:02Alex Miller (Clojure team)the scripts are the only layer that know about (for example) classpath caching#2018-04-1704:03Alex Miller (Clojure team)but they use the lib to build the cache#2018-04-1704:03cflemingSo there’s no way to programmatically determine the classpath from Clojure/Java?#2018-04-1704:04cflemingWithout exec’ing, I mean?#2018-04-1704:04cflemingI guess yes, but with no caching.#2018-04-1704:04cflemingHmm#2018-04-1704:04Alex Miller (Clojure team)there is no simple answer to that#2018-04-1704:04Alex Miller (Clojure team)the lib can compute a classpath given a configuration#2018-04-1704:05Alex Miller (Clojure team)but the configuration is inherently a product of the environment#2018-04-1704:05cflemingThe environment will be different when scripts are invoked by IntelliJ#2018-04-1704:06Alex Miller (Clojure team)to some degree, yes. and I think the question is how much you let the user know that :)#2018-04-1704:06Alex Miller (Clojure team)I think the user expectation will be that they get the same effect as running clj in the root of the project#2018-04-1704:07Alex Miller (Clojure team)I’ve tried to provide a lot of tools to replicate some or all of what’s in the script layer from inside the lib (Sean is using many of those in the boot tooling)#2018-04-1704:08cflemingright, but even running clj in the root of the project will be different when run from IntelliJ.#2018-04-1704:08cflemingWell, the environment will be different, I have no idea if the differences will be significant.#2018-04-1704:09Alex Miller (Clojure team)the only things that are critical are current working directory and path#2018-04-1704:10Alex Miller (Clojure team)there are potentially some environment variables that can affect things but none of them are likely to be in use#2018-04-1704:10cflemingpath will be the thing that’s different.#2018-04-1704:10Alex Miller (Clojure team)can you explicitly add an option to point to the bin dir for clojure?#2018-04-1704:10cflemingIn general OSX doesn’t give apps access to env vars, and doesn’t execute shell init scripts anyway.#2018-04-1704:11cflemingI think I’ll have to.#2018-04-1704:11Alex Miller (Clojure team)on os x, if you install via brew (recommended) clojure will be in /usr/local/bin#2018-04-1704:12Alex Miller (Clojure team)well, on linux too if you follow the script#2018-04-1705:31dominicmMost distro packages won't#2018-04-1704:12Alex Miller (Clojure team)windows is … coming#2018-04-1704:13Alex Miller (Clojure team)and the install level deps.edn will be at /usr/local/lib/clojure/deps.edn#2018-04-1704:14Alex Miller (Clojure team)that last is linux, mac will vary per brew install#2018-04-1704:15Alex Miller (Clojure team)mac will be at something like /usr/local/Cellar/clojure/1.9.0.375/deps.edn when installed via brew. maybe there’s a symlink that#2018-04-1704:15Alex Miller (Clojure team)’s version independent, not sure#2018-04-1704:18cflemingOk, thanks.#2018-04-1704:22Alex Miller (Clojure team)I have tried to do what I can to make things discoverable and computable. if there’s things I can do to help more, I’m open to suggestions#2018-04-1704:27seancorfieldWith boot-tools-deps I actually pull the install-level deps.edn from the Clojure brew-install GitHub repo, into the JAR of the library. Looking back, that was probably a bad idea 🙂#2018-04-1704:28seancorfieldI probably ought to change it to use clojure-env instead!#2018-04-1704:31Alex Miller (Clojure team)it’s a bad idea :)#2018-04-1711:42mfikes@cfleming If it helps, I've been using Cursive for deps.edn projects by first doing clj -Spom and then using the resulting pom.xml#2018-04-1711:45mfikesI also imagine Cursive might be able to use the output of clojure -Spath, as it gives you the classpath, including paths for any :local/roots or :git/url / :sha deps#2018-04-1717:03richiardiandreaSo...say I have a colleague who does not trust brew install and would like to use the linux script for installing clojure...is it possible to do that?#2018-04-1717:03richiardiandrea(answer is no, we tried and install -D does not unfortunately work on mac)#2018-04-1717:13dominicmThe install really only does 5 steps, should be easy enough to do them manually.#2018-04-1717:28richiardiandrea@dominicm yep he probably will do that#2018-04-1717:40Alex Miller (Clojure team)you’ll notice the brew install just runs an install.sh in the tar#2018-04-1717:40Alex Miller (Clojure team)so you can manually do basically the same thing#2018-04-1717:42Alex Miller (Clojure team)with some effort and messiness, install.sh for mac and linux-install.sh could become the same script. I have so far not been able to convince myself that gives me any great value though and there are some useful reasons for them to be different.#2018-04-1718:08richiardiandrea@alexmiller thanks for answering, was wondering about that as well, good to know#2018-04-1718:32richiardiandreaso the difference about install on Macosx is only with -d Create directories. Missing parent directories are created as required. No -D (capital).#2018-04-1718:46dominicmBSD vs GNU install 😔 if we want to work on BSD we should support both really.#2018-04-1718:53dominicmI've seen some applications probe by using --help and grep#2018-04-1719:12dominicmhttps://www.freebsd.org/cgi/man.cgi?query=install&apropos=0&sektion=0&manpath=FreeBSD+11.1-RELEASE+and+Ports&arch=default&format=html for reference#2018-04-1719:21richiardiandreaI think for mac is even a bit different from BSD, I did not see -D at all#2018-04-1720:13dominicmHum, weird, annoying, standards, yay.{:tag :div, :attrs {:class "message-reaction", :title "disappointed"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😞")} " 4")}
#2018-04-1802:39mfikesPlanck now joins the deps.edn fray with a new plk script which is literally a rip-off of clj#2018-04-1802:42Alex Miller (Clojure team)something something flattery{:tag :div, :attrs {:class "message-reaction", :title "upside_down_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙃")} " 4")}
#2018-04-1900:04zentropeIf I have :main-opts ["-e" "(println 'hello)"] I get an EOF while reading error. Is there something I’m missing with the “-e” option?#2018-04-1900:08gfredericksdo you get that with (println :hello) as well?#2018-04-1900:08zentropeHm. If there’s any whitespace in the expression at all, the error happens?#2018-04-1900:09zentropeYeah, :hello causes the issue, too.#2018-04-1900:09zentropeHowever, :main-opts ["-e" "(println,'hello)"] succeeds (note the , for white space).#2018-04-1900:10gfredericksah ha#2018-04-1900:10gfredericksI would call that a bug but I Am Not A Clojure Lawyer#2018-04-1900:10zentropeIt’s certainly principle of greatest surprise. ;)#2018-04-1900:14zentropeI can’t find it in Jira, if it already exists. Not that I have any skill searching Jira.#2018-04-1900:27mfikes@zentrope https://dev.clojure.org/jira/browse/TDEPS-56#2018-04-1900:30zentropeYes, it seems major to me, too.#2018-04-1900:31mfikesSweet , whitespace hack though. Awesome.#2018-04-1900:33zentropeIf found that in a @seancorfield example for deps.edn somewhere. I wondered why he had that in there. ;)#2018-04-1900:34seancorfieldYeah, I spent a while bashing my head on the whitespace errors before thinking of , in a drunken moment of defiance, late at night! 🍻#2018-04-1900:35gfredericksI have also used , to avoid quoting issues at some point in my life#2018-04-2003:43mfikesThat , is curiously useful. Here is another use: https://github.com/cljsrn/cljsrn-org/blob/master/script/build#L1#2018-04-2003:47mfikesTry as I might, I can't find another way to embed deps EDN in a shebang without the Corfield Comma{:tag :div, :attrs {:class "message-reaction", :title "face_with_hand_over_mouth"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 8")}
#2018-04-2004:40seancorfieldI'm honored -- I have punctuation named after me :rolling_on_the_floor_laughing:{:tag :div, :attrs {:class "message-reaction", :title "comet"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☄️")} " 4")}
#2018-04-2014:23johnCorfield comma 😂#2018-04-2014:26john@mfikes this is exactly the simple static site generator I was looking for! thanks#2018-04-2014:27johnthat's really simple#2018-04-2014:39dominicm@mfikes just a caution, using the shebang that way won't work on Linux#2018-04-2014:40mfikes@dominicm Yep. I know. I know.#2018-04-2014:41mfikesIt is a nice pattern for when you want a script to specify its own deps, but alas.#2018-04-2014:50dominicmoh god#2018-04-2014:50dominicmI came up with something that works#2018-04-2014:52dominicm#!/bin/sh
"exec" "clj" "-Sdeps" "{:deps,{markdown-clj,{:mvn/version,\"0.9.85\"}}}" "$0" "
Run as
❯ ./foo.clj '# Hello, world'
I don't know whether to share this or burn it{:tag :div, :attrs {:class "message-reaction", :title "scream"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😱")} " 16")}
#2018-04-2014:53dominicmYou can kill the commas too, just tested#2018-04-2014:54mfikesNice, so the extra stuff there is just interpreted as strings by the compiler that otherwise do no harm 🙂#2018-04-2014:54dominicmBingo#2018-04-2014:55dominicmI had issues with your md->html, the docs said it was md-to-html? Dunno if in cljs it's different, but thought I would warn you#2018-04-2015:49timgilbertThat is a cool hack (also slightly horrifying)#2018-04-2015:50timgilbertI'm wondering if various bits of clojure syntax farther down in the file would break bash, but I guess it really does just read line-by-line?#2018-04-2015:53mfikesMy guess is that exec seals the deal, replacing the script with the clj process, thus making it so no further bash interpreting is required{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-2016:05Alex Miller (Clojure team)this still pales in comparison to boot’s hack I think - it embeds a jar file in the bash#2018-04-2016:06mfikes😱{:tag :div, :attrs {:class "message-reaction", :title "scream"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😱")} " 4")}
#2018-04-2016:06mfikesLike, the binary image?#2018-04-2016:07Alex Miller (Clojure team)yes#2018-04-2016:07Alex Miller (Clojure team)just go edit your boot script :)#2018-04-2016:07mfikesHackers gonna hack#2018-04-2016:07dominicmfun fact, zip readers scan the file until they find the header.#2018-04-2016:07dominicmCapsule also has "really executable jars"#2018-04-2016:07Alex Miller (Clojure team)yep#2018-04-2016:19mfikesI like your solution @dominicm; it works on macOS as well. I’m thinking of putting it in the Planck online docs given it is so useful. https://github.com/planck-repl/planck/issues/720#issuecomment-383147716#2018-04-2016:19dominicm@mfikes Go ahead! I'm also honoured you took the time to figure out my gh username 🙂#2018-04-2016:20dominicmI tested it with a posix-only (ish) shell, so it should work everywhere#2018-04-2016:24mfikesI forgot about the fact that it needs to be well-formed Clojure code and used single quotes above. Hrm. It still worked.#2018-04-2016:27dominicmI guess that is well-formed clojure code, you are doing (quote "{:deps …") (quote "$0") 😂#2018-04-2016:28dominicmyou're not doing what it looks like you're doing, but it works#2018-04-2016:29mfikesYep. Wow#2018-04-2016:29mfikesI think I’ll use \" as it is a little less mystifying#2018-04-2016:34dominicmIt's also way more annoying 😉#2018-04-2016:34pesterhazySo much progress in one day! Corfield Comma... Monroe's Shebang{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 12")}
#2018-04-2016:35pesterhazyI think I'd use clojure instead of clj. The latter is wrapped in rlwrap, which seems unnecessary for scripts{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-04-2016:51pesterhazyTrying to code-golf: #!/usr/bin/env bash
{ "exec" "clojure" "-Sdeps" "{:deps {markdown-clj {:mvn/version \"0.9.85\"}}}" "$0" "#2018-04-2016:51pesterhazynot sure that's any better though#2018-04-2016:53dominicmIsn't it exactly the same? 😂 Although I like the hack#2018-04-2017:10pesterhazyYeah, the real insight is in the different interpretation of strings by sh and clojure#2018-04-2017:24mfikesOddly, it works with ClojureScript, which I would have expected to take issue with some strings appearing before an ns form
#!/usr/bin/env bash
"exec" "clojure" "-Sdeps" "{:deps {org.clojure/clojurescript {:mvn/version \"1.10.238\"} funcool/tubax {:mvn/version \"0.2.0\"}}}" "-m" "cljs.main" "-re" "node" "$0" "#2018-04-2018:45ghadiupgraded clj today, from 359 to 375, and I ran into the error Error building classpath. Destination path "my-private-lib" already exists and is not an empty directory
Seems like this popped up recently.#2018-04-2018:45Alex Miller (Clojure team)can you explain more about the context?#2018-04-2018:55Alex Miller (Clojure team)was this on a linux install or using clj?#2018-04-2021:24ghadilinux install, using clj#2018-04-2101:25Alex Miller (Clojure team)Using prefix dir?#2018-04-2314:34niquolaHi i have monorepo with submodules - looks like deps does not follow second level of relative paths? Should it?#2018-04-2314:38niquolacleo/cleo ../backend
fhir/fhir ./sansara/fhir
#2018-04-2314:38niquolafhir is not found in path#2018-04-2314:41Alex Miller (Clojure team)Can you explain your setup?#2018-04-2314:54niquolaui/
deps.end
backend/
deps.edn
libs/
lib1/deps.edn
lib2/deps.edn
# in ui
backend {:local/root "../backend"}
# in backend
lib1 {:local/root "./libs/lib1"}#2018-04-2314:54niquolaso lib1 is not visible from ui#2018-04-2314:55niquolaAnd -Spaths is wrong $ui-dir/./backend/libs/lib1/src
$ui-dir/../backend/src
#2018-04-2314:55niquolalooks like it just ignores relative paths#2018-04-2314:57Alex Miller (Clojure team)you had “bacend” above - just checking that was a typo here#2018-04-2314:57Alex Miller (Clojure team)relative paths should work#2018-04-2314:58Alex Miller (Clojure team)and “.libs/lib1” should be “libs/lib1" or “./libs/lib1”#2018-04-2314:59niquolaI’ve tried both 🙂 - will check one more time#2018-04-2315:01Alex Miller (Clojure team)clj -Stree might be easier to read too#2018-04-2315:02niquolaYes i read it#2018-04-2315:02Alex Miller (Clojure team)there might be an issue with relative local deps of relative local deps?#2018-04-2315:02niquolaAnd wrong path is generated 😞#2018-04-2315:02niquolait lost ..#2018-04-2315:03niquolaso i’ve got ui/./libs/lib1#2018-04-2315:03niquolainstead of ui/../backend/./libs/lib1#2018-04-2315:05Alex Miller (Clojure team)I can imagine how that would not work in the code#2018-04-2315:10niquolaShould i switch to absolute paths or try to pull request 🙂?#2018-04-2315:10Alex Miller (Clojure team)well you can definitely use absolute paths#2018-04-2315:10niquolado you have some reader macro to read env in deps.edn?#2018-04-2315:10Alex Miller (Clojure team)we don’t take PRs for this project, but you can definitely file an issue at https://dev.clojure.org/jira/browse/TDEPS if you like#2018-04-2315:15niquolaYes, that’s my case - are you planning to support this situation?#2018-04-2315:20Alex Miller (Clojure team)off the top of my head, I don’t see why this shouldn’t work#2018-04-2315:20Alex Miller (Clojure team)shouldn’t be made to work that is#2018-04-2315:20Alex Miller (Clojure team)so yes#2018-04-2315:20Alex Miller (Clojure team)I think I know where the problem is#2018-04-2315:21Alex Miller (Clojure team)but would be great to have a ticket#2018-04-2315:24niquolayep, will do - thank you. Done #TDEPS-74 + repo to reproduce https://github.com/niquola/local-deps-bug#2018-04-2316:29niquolaIs there any way to inject env variable into deps.edn?#2018-04-2316:57delaguardoclj -Sdeps '$(cat deps.edn | echo)' mb? But I never tried that before troll#2018-04-2317:04dominicmI mentioned this before, it's because the paths are not resolved relative to dependency manifest, but are relative to where the JVM started.
Is it now the case that you're interested in supporting this? 😯#2018-04-2317:14Alex Miller (Clojure team)Sorry, I don’t remember that conversation #2018-04-2317:28dominicmif tdeps-74 is likely to be fixed, I'm happy. It suits my mono-repo intentions 🙂 ¯\(ツ)/¯#2018-04-2407:08delaguardohttps://store.docker.com/images/clojure?tab=description
Official tools-deps based clojure docker image is released. docker run -ti --rm clojure:tools-deps clojure{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-04-2409:42pesterhazyVery interesting!#2018-04-2412:13Alex Miller (Clojure team)In what way is it official?#2018-04-2412:48bronsahttps://docs.docker.com/docker-hub/official_repos/#2018-04-2412:48bronsaofficial by docker#2018-04-2412:50mfikesHrm. Seems odd Docker would create an official thing using clojure as its name. (I admittedly don't understand the bigger picture.)#2018-04-2412:50gfredericksI think it's existed for a while as a leiningen image?{:tag :div, :attrs {:class "message-reaction", :title "white_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✅")} " 4")}
#2018-04-2413:08delaguardoyes, it is. And another for boot as well. And official means that this image is served from the official docker registry and curated by docker maintainers how suppose to take care about upgrading software inside of the image#2018-04-2620:39phreedI am wanting to create an uberjar and upload it to a local nexus repository. I am planning on using https://github.com/healthfinch/depstar or#2018-04-2712:45niclasnilssonQuick question: Transitive git deps in deps.edn? I.e. a/deps.edn points to b on github and b/deps.edn points to c on github. Doesn’t seem to work without including c in a’s deps.edn. Not implemented or am I doing it wrong?#2018-04-2712:59hypirionThat should work, I've used that myself a couple of times#2018-04-2713:08Alex Miller (Clojure team)should work#2018-04-2713:10Alex Miller (Clojure team)if you want more information, try clj -Sdeps "{:aliases {:verbose true}}" -R:verbose -Stree, I can probably interpret that a bit more to see what’s up#2018-04-2715:47rickmoynihannew snapshot of lein-tools-deps deployed btw… Should feature first stages of git & local/root support etc.
YMMV: https://github.com/RickMoynihan/lein-tools-deps#2018-04-2801:00mfikes^ If you'd like to use lein-tools-deps from Cursive, it is now usable, and I've documented some workarounds for some path issues here https://gist.github.com/mfikes/f803fef3013927c376063a3d72b69d60 (cc @rickmoynihan)#2018-04-2801:20cflemingThanks @mfikes, I’m hoping to add deps support to Cursive soon, that’s helpful.#2018-04-2801:20mfikesSweet!#2018-04-2801:22mfikes@cfleming Of interest to you will perhaps be the call (reader/clojure-env) and how it fails to find /usr/local/bin/clojure, if you end up going down a similar path https://github.com/RickMoynihan/lein-tools-deps/blob/master/src/lein_tools_deps/plugin.clj#L19#2018-04-2801:23cflemingThanks - I haven’t read about any of that yet, but I assumed that Cursive would have trouble with the CLI stuff since IntelliJ doesn’t see shell env vars like PATH.#2018-04-2801:25mfikesYeah, in the above, the workaround is to instead launch the IDE from a terminal via open -a "IntelliJ IDEA"#2018-04-2810:19rickmoynihanThanks @mfikes I might link to those docs from the README. I’m an Emacs/cider user myself, though am familiar with that problem because of how OSX’s GUI environment doesn’t fork from a shell with your sourced env, like it does under classical Unix systems.
The problems in resolving those file locations were actually the rationale behind letting users provide their own chain of files instead of [:system :home :project], so I’m glad you’ve found that useful; though it’d be good to find a better solution. It’s a shame that tools.deps has to shell out to get that env & I’m definitely open to other solutions; though it’s really more of a problem for tools.deps to solve (if it can).#2018-04-2813:36mfikesYeah, I feel like lein-tools-deps is at a point where early adopter community members can actually make use of it and provide feedback related to the various use cases that may arise out there.#2018-04-2817:23potetmfwiw I’ve gotten by thus far by generating a pom (via clj -Spom) — you don’t get auto-importing, but it’s a very clear execution model#2018-04-2817:25mfikesYeah, I think one new thing that lein-tools-deps gets you is Git deps (`clj -Spom` doesn't seem to support that)#2018-04-2817:26potetmah yeh, makes perfect sense#2018-04-3001:17hagmonkhey so can someone riddle me this … I'm trying to replicate https://github.com/stuartsierra/log.dev/blob/master/project.clj in tools.deps, but finding it quite irritating#2018-04-3001:19hagmonkthe problem is that lein takes the top level list of :exclusions in project.clj and applies those to all dependencies, and seemingly to all transitive dependencies#2018-04-3001:20hagmonkit doesn't appear that deps.edn has a facility to achieve the same thing, meaning that one has to manually review the dependency graph, and if any transient dependencies include an unwanted dependency, one must include that dependency and put a specific exclusion in the maven coordinate#2018-04-3001:34Alex Miller (Clojure team)That’s true#2018-04-3001:37hagmonkI put a demo repo here: https://github.com/hagmonk/log.dev.deps#2018-04-3001:38hagmonkIs there a more "tools.deps-ish" way to solve this problem?#2018-04-3001:56Alex Miller (Clojure team)No#2018-04-3002:00Alex Miller (Clojure team)This is not a common issue so I don’t see it very high on the priority list#2018-04-3002:01hagmonkHmm, it's an issue I often run into when trying to "rationalize" the logging situation in a project that has lots of Java deps#2018-04-3002:01Alex Miller (Clojure team)One thing that may exist at some point is the means to specify a lib removal. Given that, this could be done with override-deps or possibly a root dep exclusion#2018-04-3002:04hagmonkIt's kind of gross that the presence of certain deps on the classpath causes side effects for logging frameworks. Perhaps this is best handled by a small tool that rewrites deps.edn to incorporate the necessary inclusions / exclusions#2018-04-3013:14pesterhazy@hagmonk wow log.dev looks great, I never knew that existed#2018-04-3016:32hagmonk@pesterhazy yeah! since I end up importing a lot of Java deps I frequently start projects with this set of inclusions / exclusions just so I know that Java logging will be sane#2018-05-0314:35mfikeshttps://twitter.com/mfikes/status/992049711069704192#2018-05-0314:35mfikes^ demo of using some latest WIP for lein-tools-deps within Cursive#2018-05-0314:37Alex Miller (Clojure team)what’s that’s :system :home :project from?#2018-05-0314:37Alex Miller (Clojure team)because I would call those :install :user :project instead#2018-05-0314:38Alex Miller (Clojure team)is that lein-tools-deps?#2018-05-0314:38Alex Miller (Clojure team)cc @rickmoynihan ^^#2018-05-0314:43dominicm@alexmiller is lein-tools-deps relying on implementation behaviour in order to label those things, based on positional information?#2018-05-0314:43Alex Miller (Clojure team)presumably#2018-05-0314:44mfikesYes @alexmiller https://github.com/RickMoynihan/lein-tools-deps/blob/master/src/lein_tools_deps/plugin.clj#L19#2018-05-0315:09rickmoynihan@alexmiller: Yes, it’s lein tools deps.#2018-05-0315:10rickmoynihanYes, good idea those are better names I’ll create a ticket to rename them.#2018-05-0315:11rickmoynihan@dominicm: It does indeed. Do you know of a better way right now?#2018-05-0315:11dominicmNo, there's a ticket open on JIRA. I'm waiting on it#2018-05-0315:12rickmoynihando you have the ticket number? I’ll create an issue to update lein-tools-deps when tools deps improve it.#2018-05-0315:12dominicmhttps://dev.clojure.org/jira/browse/TDEPS-57#2018-05-0315:12rickmoynihanta#2018-05-0315:14dominicmIn my case, I want to allow users of pack to run pack from outside their main deps.edn if they like, which means replacing the project-deps.#2018-05-0315:16Alex Miller (Clojure team)we’re still kicking around this stuff#2018-05-0315:17Alex Miller (Clojure team)from one perspective there are fixed things and from another tools.deps doesn’t ultimately care where they come from. Right now the scripts know about specific locations, but tools.deps doesn’t.#2018-05-0315:18Alex Miller (Clojure team)but there are some places where it would be useful to know the context, for example when resolving deps.edn files in local transitive deps, you want to interpret that in the proper context (with install and user but without the initial dep project deps.edn)#2018-05-0315:18Alex Miller (Clojure team)and probably without -Sdeps#2018-05-0315:20rickmoynihan@alexmiller yeah I agree entirely.#2018-05-0315:21Alex Miller (Clojure team)I am taking the long hammock on the best resolution to this#2018-05-0315:21rickmoynihanhaha very wise 🙂#2018-05-0315:21Alex Miller (Clojure team)we’ve got some datomic tooling that wraps around all this and has some of the same tensions#2018-05-0315:21Alex Miller (Clojure team)build tooling that is#2018-05-0315:23rickmoynihanI’ve been meaning to ask on macos with the brew installer, it looks like /usr/local/bin/clj etc are sym linked to the latest version; would it not be possible for /usr/local/etc/deps.edn to also symlink to the latest system deps.edn?
It looked link the linux installer puts system in a well known location.#2018-05-0315:23dominicmthat's a bit exciting, datomic + tools.deps seemed unlkkely to me as I expected java to be involved.#2018-05-0315:24Alex Miller (Clojure team)on macos, some of that is dependent on how you use brew I think. while it will usually be there, you can change your prefix dir for brew#2018-05-0315:24rickmoynihan@mfikes: sorry I haven’t got around to looking at your PRs yet btw… I’ll try and get around to it.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0315:24dominicm@rickmoynihan don't depend on where the linux installer puts it, some distro packaging requirements put it elsewhere.#2018-05-0315:24Alex Miller (Clojure team)and similarly we now accept a prefix dir for linux#2018-05-0315:25Alex Miller (Clojure team)@dominicm this is related to how datomic cloud is built#2018-05-0315:25Alex Miller (Clojure team)wrt dependency fetching#2018-05-0315:26dominicmNixOS and GoboLinux certainly don't use /usr at all.#2018-05-0315:26Alex Miller (Clojure team)eventually some of that whole system may percolate into something public and generally useful but not sure#2018-05-0315:26rickmoynihan@dominicm: Yeah I know… just thinking an acceptable approach for lein tools deps, might be to do some default checks (in a fallback chain) with no explicit config - but let users override.#2018-05-0315:27dominicm@rickmoynihan I would expect most linux users to not be aware of where their distro chooses to put these files.#2018-05-0315:27Alex Miller (Clojure team)certainly, the install-specific location is a pain#2018-05-0315:27rickmoynihanwould obviously prefer not to do that#2018-05-0315:28Alex Miller (Clojure team)one option would be to get rid of the install file and move it into the code#2018-05-0315:28rickmoynihanI’d expect most linux users would; or would be able to find out. They can always use clj -Sdescribe, which can be documented in an FAQ for lein-tools-deps#2018-05-0315:29Alex Miller (Clojure team)if the base is loaded as a resource, then the scripts no longer care{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-05-0315:29rickmoynihanthat’s a great idea#2018-05-0315:30Alex Miller (Clojure team)likewise, you could restrict ~/.clojure/deps.edn to aliases only#2018-05-0315:30rickmoynihanresolving user is a lot easier too so install is the main problem#2018-05-0315:30Alex Miller (Clojure team)well, not only aliases, but not :deps/:paths#2018-05-0315:31rickmoynihan@alexmiller: sorry not sure why that’s an issue…#2018-05-0315:32Alex Miller (Clojure team)it goes to what you need to consider when reading transitive deps.edn, and also to what -Srepro means#2018-05-0315:32Alex Miller (Clojure team)there are a bunch of these things that interrelate#2018-05-0315:33rickmoynihanok I see what you mean#2018-05-0315:43dominicmThat would solve some issues I have in pack with reproducibility, I don't look at the system deps.edn at all.#2018-05-0316:23seancorfieldboot-tools-deps includes a version of the system deps.edn that gets pulled down from GitHub and baked in at build time. While that means you can technically use boot-tools-deps without needing clj etc installed, it's also a very bad idea: if you have clj installed, it could have a different system deps file and you could get different behavior.#2018-05-0316:24dominicm@alexmiller I am a little confused by what you were saying about building datomic cloud, are you able to elaborate.#2018-05-0316:28Alex Miller (Clojure team)no#2018-05-0316:29Alex Miller (Clojure team)it’s all internal and I didn’t work on it#2018-05-0316:24seancorfieldGiven that I often tell folks boot-tools-deps is only intended to add Boot-like functionality to an existing clj project, I should start relying on clj -Sdescribe (via the API call in tools.deps.alpha).#2018-05-0316:26mfikesOne problem with the API call is that clojure needs to be in your PATH.#2018-05-0316:27mfikesSo, this causes an issue if you happen to run things from Cursive on macOS, which ends up with a degenerate path that doesn't include /usr/local/bin#2018-05-0316:28mfikesThis essentially hacks things so that it looks in there: https://github.com/RickMoynihan/lein-tools-deps/pull/31/files#2018-05-0316:30Alex Miller (Clojure team)I think given the changes I described above you may be able to avoid all of those#2018-05-0316:30Alex Miller (Clojure team)in that tools.deps.alpha would already include the necessary stuff#2018-05-0316:30mfikes(In case it is useful, here is the degenerate path that Cursive gets: /usr/bin:/bin:/usr/sbin:/sbin)#2018-05-0316:31mfikesYeah, TL;DR is stuff is baked in, no need to really run clojure, etc.#2018-05-0316:31Alex Miller (Clojure team)the tradeoff is that the install deps.edn then becomes “magical” in that it’s hard to see what’s in it#2018-05-0316:32mfikesDefinitely calls for some hammock time#2018-05-0316:32dominicmI imagine in enterprisey environments you would want to have a system override of maven/repos#2018-05-0316:32Alex Miller (Clojure team)💤#2018-05-0316:34seancorfieldChanging the subject slightly... at work we have a mono-repo with multiple subprojects and we're starting to convert those over to deps.edn from Boot (with external dependencies in EDN files, using a different/simpler format). One of the problems we have to tackle is that, currently, we use a version.properties file to "pin" version numbers for libraries (each line is of the form: group/artifact=x.y.z to specify the version).#2018-05-0316:35seancorfieldSince clj requires that :override-deps be in an alias (yes?), I created a versions/deps.edn containing a :defaults alias with :override-deps for everything we need, but then it's a little clunky to run: clj -Sdeps "`cat ../versions/deps.edn`" -A:defaults:test:runner
#2018-05-0316:36seancorfieldIf clj allowed us to specify extra config files, that would be nicer. Thoughts?#2018-05-0316:40dominicmAlias only files would solve that issue too I guess, if they were searched for in parent directories.#2018-05-0317:14Alex Miller (Clojure team)so the :default-deps is really about this problem of dep mgmt in the large#2018-05-0317:14Alex Miller (Clojure team)but it is not a fully formed (or working) idea#2018-05-0317:15Alex Miller (Clojure team)so I guess I’m saying - we see this problem and think it’s important.#2018-05-0317:16Alex Miller (Clojure team)but there’s not a complete answer there yet.#2018-05-0317:24seancorfieldGood to know. Thanks @alexmiller!#2018-05-0317:25seancorfieldOur transition from Boot to clj / deps.edn is likely to be very slow/gradual because of stumbling blocks like this.#2018-05-0317:28seancorfieldOur monorepo has about 30 subprojects with all sorts of cross-dependencies. Right now we have a Boot task that uses tools.namespace to deduce dependencies based on namespace usage, so we don't have to explicitly manage those cross-subproject dependencies, but as we're restructuring code, I'd like to see that become more explicit and streamlined.#2018-05-0317:29seancorfield(we built the namespace-based dependency analyzer because we initially had some strict rules about how code in a few of those subprojects could use code in other subprojects, to maintain separation via APIs etc -- but we've changed our position on that now)#2018-05-0317:31dominicmI had been excited to see what you would invent for managing the deps.edn files using the automatic dependency analysis. I'm glad to hear someone else is going down the same path as us.#2018-05-0317:36seancorfieldYeah, we've talked about using the existing analyzer code to generate the cross-dependencies as deps.edn files but I think I want those human-editable with comments etc and I don't want to deal with any Clojure source reader complex enough to retain whitespace/comments 🙂#2018-05-0317:39dominicmIf you did want to go down the whitespace/comment retention, I had surprisingly good success writing the injector for pack with rewrite-clj: https://github.com/juxt/pack.alpha/blob/master/src/mach/pack/alpha/inject.clj 140 lines gets an entirely whitespace-safe modification of the deps.edn. It even does the funky vertical alignment of parameters if you do that in the same map at any point.#2018-05-0318:06seancorfieldYeah, I looked at that code -- and that convinced me I don't want to do that 🙂#2018-05-0321:26rickmoynihan@seancorfield / @dominicm can you describe those dependency analysis systems at all? What exactly are they doing, and what problems are they solving? Are you building multiple applications? If so is a limitation not that all applications in the same mono-repo need to use the same version of any given namespace?
If that’s the case what is the motivation for doing the analysis at all? i.e. why not just package everything into the same jar, and rely on the entry point determining the loaded namespaces? Is it because some of your libraries/apps are opensource and others aren’t?
Curious because we’re planning to move towards more of a mono-repo approach too.#2018-05-0321:32seancorfieldAs an example, we have subprojects for low-level systems-y stuff, data mapping (our wrappers for JDBC), a core domain subproject, a web utilities subproject, and so on. Then we have subprojects for each actual app (api, auth, login, billing, etc). So any given app subproject relies on a particular combination of other subprojects (most of them) but we also have background processes that don't need any of the web stuff etc. The tools.namespace analysis looks at the project you want to build, finds all the namespaces in it (across all subprojects) and the list of namespaces they all require, transitively (the basic graph analysis, treating everything as "one big app"), then it takes the project you want to build and starts from those namespaces to identify which subprojects contain namespaces that are (transitively) required, then the dependencies EDN file in each of those subprojects are read to generate the full list of external libraries needed.#2018-05-0321:33seancorfieldDoes that explain it well enough?#2018-05-0321:35seancorfieldThe two main goals were so that we didn't have to explicitly write down the subproject dependencies (they change automatically as code requires a different set of namespaces in other subprojects) and we wanted to enforce certain code path exclusions (e.g., no apps are supposed to depend on billing directly, they're only supposed to use the billing-sdk, billing apps shouldn't use the code domain library except via a specific "DMZ"-style namespace that defines the API).#2018-05-0321:37seancorfieldThat second goal is no longer important as we've substantially changed our deployment / infrastructure plans since mid-2017. And, without that, the benefit of auto-tracking dependencies is questionable -- I'm leaning toward explicit dependencies more now.#2018-05-0321:40rickmoynihanok so every app uses the same version of everything in the monorepo?#2018-05-0321:41rickmoynihanWhy not to just put all the src paths into one big classpath, uberjar that, and rely on the require-tree to just shake out what’s not used?
Is it because you also wanted the rules to be enforced?#2018-05-0321:42seancorfieldYes, the rules were important when we started down this path.#2018-05-0321:42seancorfieldWe can't quite make an uberjar of the whole thing because there are some subprojects that do rely on different versions of external libraries.#2018-05-0321:43seancorfieldBut we could uberjar most of it into one big JAR and start it with -m ws.whatever.namespace.#2018-05-0321:43rickmoynihansub libraries, or sub apps?#2018-05-0321:44seancorfieldOur core domain subproject has several -main entry points already for background processes and cron tasks.#2018-05-0321:46seancorfieldThe auto-tracking of dependencies makes it a lot easier to move code into new subprojects (keeping the ns the same) which we've done quite a bit as we've teased apart some of the legacy code from the newer code. But overall it wouldn't be much work to just manually update deps.edn in each subproject as/when we do that.#2018-05-0321:47rickmoynihanand each of those -main entry points has different external library versions?
Sorry trying to understand whether you consider multiple -main’s the same app, or is it what makes a different app that it has the same library versions / classpath, or something else?#2018-05-0321:48rickmoynihani.e we have some apps with multiple -main entry points… but they’re really different aspects/modes of the same app.
We also have different apps altogether#2018-05-0321:49seancorfieldFor the most part, each web app is its own separate subproject, since we deploy different apps on different servers. A lot of the background stuff runs a different server (well, two servers), so we want it separate from the web apps, but mostly all together for the server deployment.#2018-05-0321:50seancorfieldWe have some different dependencies in our web apps: one is heavy on websockets, one uses Bidi, the rest don't. We have two web apps that use netty/socket-io but have to use different versions because ... history ...#2018-05-0321:51seancorfieldWe also have a couple of subprojects that are only used from our legacy code base, so we want to keep that code physically separate so we can just delete it once we retire the legacy apps.#2018-05-0321:52rickmoynihanyeah, that’s an issue for us doing what I was suggesting… i.e. the transitive dependencies start to conflict and then you’re in a world of pain#2018-05-0321:53seancorfieldRight, we're in the situation where the dependencies would only conflict in the top-of-tree subprojects, not the lower-level stuff.#2018-05-0321:54seancorfield(and that's because we've historically "pinned" the versions of most of our external libraries)#2018-05-0321:46dominicmWe can't do as you propose due to different deployment environments meaning that certain code paths completely don't work.
It's also nice to be able to actually remove one of the two hundred dependencies every now and then, which is easier when they're limited to certain boundaries.
Our deployable files are massive now.#2018-05-0321:46seancorfieldOur JAR files are about 30-40MB each.#2018-05-0321:47dominicmThere's a lot of substituting in our projects. So running A might have it's iProtocol that's incompatible with B.#2018-05-0321:48dominicmOur files are 5x that#2018-05-0321:53rickmoynihan@dominicm: can you explain what you mean by deployment environments influencing the code paths working?#2018-05-0321:59rickmoynihanI’m hoping to manage our apps as independent projects in the same mono repo; with some shared support libraries alongside; where all apps will effectively share the same version of the supporting code… but apps may have different 3rd party deps.#2018-05-0322:04rickmoynihanThinking out loud; I’ve been wondering how this stuff fits with something like integrant… though it could well be a terrible idea#2018-05-0322:10rickmoynihanactually it is a terrible idea#2018-05-0405:30dominicmIntegrant with aero is basically how we solve some of this. We built something that looks like Integrant using component.#2018-05-0408:13rickmoynihanYeah, we use integrant too; in one project via duct (so weavejester can write all the middleware config layer for us) and in another older project we retro-fitted integrant with aero… which in many ways I prefer to duct; but there are pros and cons.
What I was really getting at though with the integrant thought was that you could possibly imagine combining integrant and deps.edn into a hybrid that would give you dependency resolution on components and configure/wire them together at the same time.
I find it interesting how tools like integrant/component essentially make your require tree composable and configurable, and wonder if there’s a way to use integrant for packaging too. i.e. to turn integrant components (modules) into individually resolvable artifacts that something like tools.deps could then fetch and resolve for you.
It’s currently a very half baked notion, and most likely a terrible idea though 🙂#2018-05-0405:33dominicmOur problem is pretty much that a namespace can blow up if deployed where it can't read certain environment variables and look at certain files. These files change per library, with the same path. I guess there's some convention for the path. This is particularly noticeable in localization.#2018-05-0407:23sveri@rickmoynihan In our team using gradle we define every dependency explicitly, no matter if its an in-team-project dependency or an external one. So naturally I would try the same with leiningen / tools-deps / boot.#2018-05-0408:17rickmoynihan@sveri: Not sure exactly what you mean by defining every dependency explicitly? Do you just mean that you produce mvn artifacts for each subproject?#2018-05-0409:15sveri@rickmoynihan gradle has a concept of project dependencies. In the root build file you can define your projects which then can depend on another.#2018-05-0414:51Alex Miller (Clojure team)this is kind of a wip / sneak-peek but I’ve implemented a dynamic add-lib that you can use at the repl (that also will download and add transitive deps)#2018-05-0414:52Alex Miller (Clojure team)$ clj -Sdeps "{:deps {org.clojure/tools.deps.alpha {:git/url \"" :sha \"d492e97259c013ba401c5238842cd3445839d020\"}}}"
Cloning:
Checking out: at d492e97259c013ba401c5238842cd3445839d020
Clojure 1.9.0
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'org.clojure/core.memoize {:mvn/version "0.7.1"})
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.pom from
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.pom from
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.pom from
Downloading: org/clojure/core.memoize/0.7.1/core.memoize-0.7.1.jar from
Downloading: org/clojure/core.cache/0.7.1/core.cache-0.7.1.jar from
Downloading: org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar from
true
user=> (require 'clojure.core.memoize)
nil
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
true
user=> (add-lib 'clj-time {:mvn/version "0.14.3"})
false{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0414:52Alex Miller (Clojure team)^^ you can use that command at the top to try it yourself off the branch#2018-05-0414:53Alex Miller (Clojure team)everything about this is subject to change as Rich and I are still discussing many aspects of naming and use#2018-05-0414:56Alex Miller (Clojure team)there is also a new libmap namespace that tracks the libs currently in use (these can be loaded via a property that points to a file which is set by clj, or by a property set to edn data - hook for tools, or by just inspecting the classpath and trying to understand it)#2018-05-0414:57Alex Miller (Clojure team)add-lib will add new libs in the context of the existing libs, so it won’t (for example) download and add libs of transitive deps that have already been loaded. when you add libs, those are added to the runtime lib map and considered in subsequent adds#2018-05-0414:59dominicmThis is very cool, and useful!#2018-05-0414:59Alex Miller (Clojure team)the primary use here is to add a capability to the repl environment to dynamically add libs. this does not give you the isolation of something like pods, but it’s a small useful step into a more live repl experience.#2018-05-0415:00dominicm@alexmiller this uses java properties you say?#2018-05-0415:01Alex Miller (Clojure team)yes - clojure.libfile (will now be set by clj) and clojure.libmap#2018-05-0415:02Alex Miller (Clojure team)although the fallback to classpath introspection actually does a reasonable job even if that’s not set in my experience so far#2018-05-0415:03dominicmPods may conflict with that, as java properties aren't localized to the pod. I don't have an alternative suggestion, but just a data point that occurs to me.#2018-05-0415:04Alex Miller (Clojure team)if you’re using pods, you don’t need this#2018-05-0415:05Alex Miller (Clojure team)classpath introspection:#2018-05-0415:05dominicmI'm thinking of inspection of clojure.libfile for other purposes, e.g. finding gitlibs to ignore during a (reset)#2018-05-0415:05dominicmThat's a bad example in the context of pods, but I think it conveys my intent.#2018-05-0415:06Alex Miller (Clojure team)the clojure.libfile points to the libfile corresponding to the classpath used when starting the jvm. that has a fixed answer.#2018-05-0415:07Alex Miller (Clojure team)if you’re using pods, you’re in a bubble under that.#2018-05-0415:07dominicmlibmap then, rather than libfile?#2018-05-0415:07dominicmhttps://github.com/RutledgePaulV/clj-embed wouldn't be able to convey clojure.libmap to it's runtimes?#2018-05-0415:08Alex Miller (Clojure team)libmap is still jvm-wide so like libfile#2018-05-0415:08Alex Miller (Clojure team)but I think it’s interesting to consider how to allow classloaders to build their own env#2018-05-0415:08dominicmMaybe I don't understand what libmap is, what is it?#2018-05-0415:08Alex Miller (Clojure team)it’s just libfile in edn form#2018-05-0415:09Alex Miller (Clojure team)so a tool could start the jvm, build the classpath via means other than clj and pass the libmap via data rather than file#2018-05-0415:10dominicmGiven the definition of "jvm-wide", then java properties are always right. My thoughts are that it's really closer to being about the classloader though, what libmap generated the current classloader.#2018-05-0415:11Alex Miller (Clojure team)the runtime libmap is basically stored in a var right now. maybe interesting to consider whether that should be dynamic to allow per-thread context or something per-classloader to handle pod/clj-embed type cases#2018-05-0415:12dominicmSorry, this probably sounds very pedantic. This is really exciting, I'm happy about the feature even as it currently is.#2018-05-0415:12dominicmIt's definitely notable that due to how Clojure works, embedding always carries a new copy of Clojure. So "clojure local" is perhaps sufficient?#2018-05-0415:33Alex Miller (Clojure team)no, I appreciate the good pokes on this. we’ve thought a fair bit about the add-lib stuff but less about the runtime management of the libmap and so I’m still trying to work out the ideal shape of that. and we have other deeper plans for the libmap stuff so I want to get it right.#2018-05-0415:33Alex Miller (Clojure team)it’s reasonably likely that part will actually float into Clojure itself#2018-05-0415:34dominicmWhich part, sorry?#2018-05-0415:35Alex Miller (Clojure team)the runtime libmap management (not add-lib)#2018-05-0415:37dominicmSo clojure.core will be aware of the deps.edn format? :thinking_face:#2018-05-0415:37Alex Miller (Clojure team)it’s just a map#2018-05-0415:37Alex Miller (Clojure team)so, yes it will be aware of maps :)#2018-05-0415:39Alex Miller (Clojure team)in terms of doing semantic things - that doesn’t have to be in core. core’s job is to ensure that it’s loaded and available in a way that can be replied upon#2018-05-0415:39Alex Miller (Clojure team)I wouldn’t rule out core being aware of it but I don’t see a need for that yet#2018-05-0415:40Alex Miller (Clojure team)well, maybe. we’ll see where things go. :)#2018-05-0415:42Alex Miller (Clojure team)maybe I should push this stuff into a blog post#2018-05-0415:47dominicmI'm a little confused. Will it be a state storage, which libmap will sit in as a key?#2018-05-0415:48dominicmBlog posts are always exciting, even if marked as "these ideas are alpha"#2018-05-0415:48Alex Miller (Clojure team)I don’t think I understand the question#2018-05-0415:48Alex Miller (Clojure team)right now it’s a var holding an atom holding the libmap#2018-05-0415:49Alex Miller (Clojure team)you can look at the code on the branch, it’s not that long#2018-05-0415:49Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/compare/add-lib#2018-05-0415:50dominicmWhat will Clojure know about managing the runtime libmap though? I'm really confused by what it means for that to be in Clojure, sorry.#2018-05-0416:29Alex Miller (Clojure team)it means that anything using Clojure could look at the libmap (not just things using tools.deps.alpha)#2018-05-0416:42dominicmI see! So there would be a var which was mutated or something?#2018-05-0416:34Alex Miller (Clojure team)http://insideclojure.org/2018/05/04/add-lib/#2018-05-0416:37seancorfieldThe channel name is #tools-deps not #tools.deps BTW @alexmiller (last line of the blog post)#2018-05-0416:37Alex Miller (Clojure team)gosh dang it{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0416:38Alex Miller (Clojure team)fixed#2018-05-0416:42dominicmI see! So there would be a var which was mutated or something?#2018-05-0416:56Alex Miller (Clojure team)it’s a var holding an atom, so would just swap! the atom#2018-05-0416:57Alex Miller (Clojure team)but I think there are interesting potential uses that are merely read-only#2018-05-0417:09dominicmI understand now. I was interpreting your words too broadly I think. This would solve the problem of pods.
If I'm thinking right, this would complicate Clojure becoming aware of class loaders in the future. Admittedly, this is something I am a minority who longs for this. #2018-05-0417:25Alex Miller (Clojure team)Clojure only works because of custom classloaders so I don’t think I agree at all with that#2018-05-0417:25Alex Miller (Clojure team)I also don’t think this would solve the isolation aspect of pods#2018-05-0419:43dominicmI mean that you can't require multiple versions of a namespace from different class loaders.
No? The pod could be controlled to update the atom on start couldn't it?#2018-05-0420:02Alex Miller (Clojure team)I think you’re misunderstanding what is happening here#2018-05-0420:02Alex Miller (Clojure team)but I’m not sure that I understand what you think it’s doing enough to clarify#2018-05-0420:03Alex Miller (Clojure team)the libmap is not affecting anything about how the Clojure runtime currently works#2018-05-0420:05Alex Miller (Clojure team)it is assisting in opening a path to additively add jars to the DynamicClassLoader in use (this actually has existed in Clojure for a very long time with clojure.core/add-classpath which is deprecated and I think has not worked for a very long time)#2018-05-0420:08Alex Miller (Clojure team)pods in Boot provide isolation (different Clojure runtimes) and already do allow you to load different versions of a Clojure namespace at the same time in different pods as far as I understand them#2018-05-0420:12dominicmMy use of from there was poor. You can't have multiple versions of a library loaded into a single Clojure. You have to load a whole new Clojure.#2018-05-0420:14Alex Miller (Clojure team)yes and nothing is changing about that#2018-05-0420:14Alex Miller (Clojure team)I’d say that problem is way down the interest list#2018-05-0420:16dominicmI know. I was thinking about it that were ever to be "fixed". Yeah, I'm one of the few it bothers because I have interest in a constrained deployment environment, where dependencies aren't really possible.#2018-05-0420:22Alex Miller (Clojure team)I totally get why and when isolation is useful, but Clojure was never designed with that intent and there seem to be more important problems to solve#2018-05-0421:24mfikeshttps://twitter.com/mfikes/status/992514389042454529{:tag :div, :attrs {:class "message-reaction", :title "the_horns"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 8")}
#2018-05-0508:22slipsetAs asked on the twitters:
https://twitter.com/slipset/status/992662387600887808#2018-05-0508:25slipsetReason I’m asking is sort of at least twofold:
1) If it’s sort of the same, is there anything to be learned by the clj-refactor impl
2) If it’s sort of the same, can add-lib be used as a replacement for the clj-refactor impel so we get consistent behavior.
3) most likely just an off by one error#2018-05-0508:26dominicmI'm certain it's almost entirely the same. I expect that in a few months this implementation will find it's way into refactor-nrepl.#2018-05-0519:11hagmonk@alexmiller Riddle me this: how does one add a maven dep and the corresponding sources and javadoc jars to the classpath? (needed for cider javadoc to work)#2018-05-0519:12dominicm@hagmonk https://dev.clojure.org/jira/browse/TDEPS-12{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0519:13Alex Miller (Clojure team)Not supported #2018-05-0519:13Alex Miller (Clojure team)Currently you can only have one artifact per library#2018-05-0519:14Alex Miller (Clojure team)I’m not opposed to supporting it, just trying to figure out how#2018-05-0519:15hagmonkGot it#2018-05-0519:25dominicm@alexmiller would treating it as a unique library be insufficient?#2018-05-0519:28hagmonkI'd propose clojure.tools.deps.alpha/merge-alias-rules be extended with some way to delegate merge strategy to the extension type#2018-05-0519:29hagmonkUh, maybe not, that is only used to merge the same map key between different deps.edn sources#2018-05-0519:31hagmonkmaybe clojure.tools.deps.alpha/resolve-deps instead - when merging :extra-deps into :deps#2018-05-0519:34hagmonkyou still end up spitting out a map though, perhaps the key of that map needs to change from being the dependency path, to an opaque composite key that represents "a unique artifact". The dep path itself could move into the value (during resolution, not in deps.edn itself)#2018-05-0519:36Alex Miller (Clojure team)That’s only used for alias types, which are not extensible #2018-05-0519:37Alex Miller (Clojure team)The merge-alias-rules that is#2018-05-0519:38Alex Miller (Clojure team)Treating it as a unique library isn’t possible as the group/artifact is used for resolution#2018-05-0519:38Alex Miller (Clojure team)This needs more thought and I consider it a lower priority than other stuff on the plate{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0619:32hagmonkOk, another riddle for you guys#2018-05-0619:33hagmonkIf I include :mvn/repos in deps.edn for project A, because one of the coordinates has to come from a private maven server, then I depend on project A from project B via :local/root, clojure -Sdeps dies saying it can't find the artifact in the private repo#2018-05-0619:35hagmonkI keep running into these instances that are encouraging me to go back to producing a maven artifact, rather than using :local/root or :git/url#2018-05-0621:36Alex Miller (Clojure team)This is a known bug, haven’t had a chance to fix it yet{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-0622:20hagmonkI created a little tool to help me convert troublesome project.clj files into deps.edn. There are probably a dozen of these but I couldn't find one that actually worked for me :)#2018-05-0622:20hagmonkhttps://github.com/hagmonk/depify#2018-05-0623:07mfikes@hagmonk That's cool. I tried seeing what would happen if you made it run directly as a git dep, but the command line gets quite large https://github.com/mfikes/depify#usage#2018-05-0623:10hagmonk@mfikes yah, I thought about doing that too, and wondered if you could specify the git coordinate + -Sresolve on the command line and make it work ... haven't tried yet#2018-05-0623:10hagmonkgit coordinate as in the repo location + ref#2018-05-0623:24hagmonkah: -Sresolve-tags, and no that doesn't work, since it wants to mutate the deps.edn file ¯\(ツ)/¯#2018-05-0818:24richiardiandreaI have a question, with this deps.edn:
{:paths ["src"]
:aliases {:dev {:extra-paths ["test"]}
:self-host {:extra-deps
{andare {:mvn/version "0.9.0" :exclusions [org.clojure/clojure]}
mount {:mvn/version "0.1.13-SNAPSHOT"}}}}}
and command:
clojure -Srepro -A:dev:self-host -Stree
shouldn't be org.clojure completly excluded from the tree?#2018-05-0818:24richiardiandrea(`mount` does not include it)#2018-05-0818:25Alex Miller (Clojure team)-Srepro is kind of in an in between state right now - it’s actually including the install deps.edn, but not ~/.clojure/deps.edn.#2018-05-0818:25Alex Miller (Clojure team)and the install one includes clojure#2018-05-0818:25richiardiandreauhm...#2018-05-0818:25Alex Miller (Clojure team)I do not expect this to always be the case#2018-05-0818:25richiardiandreaok that answers it#2018-05-0818:25Alex Miller (Clojure team)but the meaning and use of -Srepro is kind of a topic of internal debate atm#2018-05-0818:26richiardiandreaso at the moment is there a way to work in isolation? without any interference with the global env?#2018-05-0818:26Alex Miller (Clojure team)there is no way to completely exclude the install-level clojure dep#2018-05-0818:26Alex Miller (Clojure team)(although clojure doesn’t run without clojure so I didn’t expect that to be an issue)#2018-05-0818:28Alex Miller (Clojure team)what are you trying to do?#2018-05-0818:29richiardiandreaI am working with self-host ClojureScript so I don't need Clojure for running, but having it on the classpath might cause issues#2018-05-0818:29richiardiandrealet me debug and see if that's the issue though#2018-05-0818:29richiardiandreabut in general you probably might want to consider the use case#2018-05-0818:52Alex Miller (Clojure team)clj runs clojure.main (always) which requires a class on the runtime. How does self hosted avoid that?#2018-05-0818:53Alex Miller (Clojure team)That is, the entire purpose of clj is to run a class on the JVM #2018-05-0818:54dominicm@alexmiller I'm guess -Spath is being used only#2018-05-0820:36richiardiandreaYes I am using it only for building the classpath#2018-05-0820:36richiardiandreaAnd I am using clojure for it#2018-05-0820:42Alex Miller (Clojure team)well, you could avoid this by invoking tools.deps directly too#2018-05-0820:43Alex Miller (Clojure team)that is, as a library, rather than via the script#2018-05-0820:45Alex Miller (Clojure team)if you use -A:deps that will add tools.deps.alpha and its dependencies to your classpath. you could even invoke the clojure.tools.deps.alpha.script.make-classpath script directly#2018-05-0820:54Alex Miller (Clojure team)clj -A:deps -m clojure.tools.deps.alpha.script.make-classpath --config-files /path/to/whatever.edn --libs-file /path/to/out.lib --cp-file /path/to/out.cp --jvm-file /path/to/out.jvm --main-file /path/to/out.main -R :self-host#2018-05-0820:55Alex Miller (Clojure team)something like that - I think that script requires all of those and that they are absolute paths#2018-05-0821:06richiardiandreaOk that is a possible solution thanks#2018-05-0821:23Alex Miller (Clojure team)if you want to write code instead, you could drop down to the tools.deps.alpha layer and write a few lines of code instead.#2018-05-0900:34richiardiandreayeah I would really like to explore tools.deps.alpha a bit more, expecially the resolver part#2018-05-0900:35richiardiandreaone day...#2018-05-0913:05dominicm@alexmiller Can you think of a case where clj on a deps.edn file wouldn't resolve a dependency that tools.deps.alpha would resolve against a deps.edn file?
In particular, https://github.com/juxt/pack.alpha/issues/7#issue-321346647 something is going off to another repo. pack only considers the deps.edn, and doesn't merge the system one. For some reason it works if I add maven central, but what does that have to do with glassfish?#2018-05-0913:05Alex Miller (Clojure team)the install deps.edn includes the central and clojars maven repos#2018-05-0913:06Alex Miller (Clojure team)https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/deps.edn#2018-05-0913:07Alex Miller (Clojure team)other than that, shouldn’t be any different#2018-05-0913:09dominicm@alexmiller I guess another way of describing this would be:
- I run clj against a deps.edn file - so everything should end up in .m2
- I run tools.deps.alpha against the same deps.edn and it fails as more needing more resolving#2018-05-0913:09Alex Miller (Clojure team)doesn’t make sense to me#2018-05-0913:10dominicmIf I add the :mvn/repos to deps.edn, somehow it allows it to fetch from glassfish repo#2018-05-0913:10dominicmah, they commented https://github.com/juxt/pack.alpha/issues/7#issuecomment-387732402#2018-05-0913:11dominicmI'm not really sure what is going on here, and why this either needs to run every time (uncacheable) or doesn't get picked up by clj#2018-05-0913:12Alex Miller (Clojure team)I guess because once it’s in central they want you to get it from there?#2018-05-0913:12Alex Miller (Clojure team)dunno, weird#2018-05-0913:12dominicmyeah, real weird.#2018-05-0913:13dominicmThis definitely highlights that I need a good way to do -Srepro without the local deps.edn file ever being picked up though 😉#2018-05-0913:13dominicmOr rather, I need a way to pick up only the system deps.edn#2018-05-1020:59rickmoynihan@alexmiller: I’m bike shedding a little but what would you recommend calling the key for lein-tools-deps config. Currently it’s called :tools/deps which feels a little wrong, should it be :lein.tools/deps or :lein.tools.deps, or just :lein-tools-deps?#2018-05-1021:02rickmoynihanThis is the key for using in a project.clj for configuring how the lein-tools-deps plugin should work with tools.deps#2018-05-1021:03Alex Miller (Clojure team)should be namespaced#2018-05-1021:03Alex Miller (Clojure team)wait, this is in project.clj?#2018-05-1021:04Alex Miller (Clojure team)I guess most stuff there is not namespaced#2018-05-1021:04Alex Miller (Clojure team)saying “lein” seems redundant#2018-05-1021:04Alex Miller (Clojure team)I guess I’d say :lein-tools-deps or :tools-deps#2018-05-1021:05dominicmNo harm in setting a new precedent.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-1021:06rickmoynihanyeah its in project.clj#2018-05-1021:06rickmoynihanbut thought it might be nice to set a precedent#2018-05-1021:06Alex Miller (Clojure team):lein-tools-deps/config#2018-05-1021:09rickmoynihanyeah I had thought of that one too… I like that it nests config under it, but it gets wordy… though that’s not really an issue#2018-05-1021:09rickmoynihanit also matches the actual project name on github, and root namespace name#2018-05-1021:10rickmoynihanI think that’s definitely the most correct one#2018-05-1021:10rickmoynihanthanks#2018-05-1114:28ghadiI'm having trouble configuring an S3 private repo. To debug I've set -Srepro so that mavencentral & clojars are not seen by t.d.a, and I've set :mvn/repos to my own. For some reason I still get the error: cannot fetch the $dep from "central"#2018-05-1114:33ghadinever mind -- it's a bad error message. when all of the configured repos fail, it throws with one of them.#2018-05-1114:43ghadiTwo other misc issues:
cannot use override-deps to override from a git/sha to a local/root
:mvn/repos doesn't work transitively when pointing to a :local/root#2018-05-1115:04Alex Miller (Clojure team)-Srepro does actually include the install deps.edn btw#2018-05-1115:05Alex Miller (Clojure team)at the moment that is#2018-05-1115:07Alex Miller (Clojure team)re “cannot use override-deps to override from a git/sha to a local/root” - with same lib name? prob worth a ticket if so#2018-05-1115:07ghadiYes, same lib name#2018-05-1115:07ghadi👌:skin-tone-4:#2018-05-1115:07Alex Miller (Clojure team)re “:mvn/repos doesn’t work transitively when pointing to a :local/root” - I don’t understand what that means#2018-05-1115:08ghadiA => :local/root B => B has different :mvn/repos => C#2018-05-1115:09Alex Miller (Clojure team)ok. that’s related to some of the other local dep issues that are already logged#2018-05-1115:12Alex Miller (Clojure team)@robert-stuttaford re twitter convo… current hack is:
{:deps {com.datomic/datomic.pro {:local/root "/path/to/datomic-pro-whatev.jar"}
com.datomic/datomic.pro.deps {:local/root "/path/to/dir-with-datomic-pom"}}#2018-05-1115:12Alex Miller (Clojure team)where the first pulls the jar and second treats the directory holding the pom file (which has to be named “pom.xml” as a local project to pick up the transitive deps)#2018-05-1115:13Alex Miller (Clojure team)that’s the hack#2018-05-1115:13Alex Miller (Clojure team)however, I just landed a commit to make local jar deps use their embedded pom file to handle this without the hack#2018-05-1115:14Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/commit/de844ebcfe7787abe89f4f02b27abdbfdd5c7769#2018-05-1115:15Alex Miller (Clojure team)will be in the next release#2018-05-1115:13ghadi(I don't think the transitive :mvn/repo thing should be supported, fwiw. Just logging the papercut)#2018-05-1115:14Alex Miller (Clojure team)well then don’t log it :)#2018-05-1115:14Alex Miller (Clojure team)I’m kind of on the fence though - the equivalent will work if you’re using maven deps I think#2018-05-1115:14ghadiI'll file a ticket for the git -> mvn override-dep#2018-05-1117:42Alex Miller (Clojure team)@ghadi https://dev.clojure.org/jira/browse/TDEPS-51 is maybe sufficient for that#2018-05-1117:44ghadiyup, perfect#2018-05-1211:06robert-stuttafordthank you, @alexmiller! this is rad!#2018-05-1212:08mhuebertis there a simple way to specify a particular version of tools.deps.alpha itself (eg. to use last night’s release before it shows up on brew, or to point to a particular commit)?#2018-05-1212:10Alex Miller (Clojure team)No, but I just did a release last night and there is a pending brew pr which should get merged today{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-1212:11dominicmtime to bump packages#2018-05-1212:11Alex Miller (Clojure team)New tools.deps should be available#2018-05-1212:13Alex Miller (Clojure team)0.5.435{:tag :div, :attrs {:class "message-reaction", :title "heart_eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😍")} " 4")}
#2018-05-1213:08mhuebertah, brew has a nice way of pulling a PR. brew pull 27778 && brew upgrade clojure#2018-05-1217:32rickmoynihanI’ve made some changes to the lein-tools-deps README…
Any thoughts on how to improve it would be appreciated:
https://github.com/RickMoynihan/lein-tools-deps/tree/0.4.x#2018-05-1217:34dominicmhttps://github.com/RickMoynihan/lein-tools-deps/tree/0.4.x#why-not-use-boot-instead I don't know that you need to make this argument 🙂#2018-05-1217:39rickmoynihanI did consider not making it… as you’re probably a lein user if you’re reading it… But I figured new users might have been advised to use just one of the build tools; and be seeing this page, so thought it worth mentioning there are other choices too#2018-05-1217:39rickmoynihanor are you not a fan of boot? And would prefer everyone chose leiningen? 🙂#2018-05-1217:40dominicmThe opposite 😉 I'm the boot advocate at JUXT.#2018-05-1217:40rickmoynihanlol#2018-05-1217:41rickmoynihanthen is my comment fair?#2018-05-1217:48dominicmI think it is, other than perhaps:
> Leiningen projects are harder to turn into unique snowflakes, which might be better or worse for you.
For me the "snowflake" term has gained a disparaging connotation. But I understand what you're getting at, and I think it's mostly fair. We had issues with lein because it couldn't build sass & cljs at the same time, which made it a pain for users jacking in, but maybe we're in snowflake territory there?#2018-05-1218:06rickmoynihanYeah it is a slightly disparaging term; but I think it’s fair here - if you have a boot project it will often be pretty different to the next boot project; right?
More so than with lein projects.
I take your point that building sass/cljs simultaneously shouldn’t be such a unique a thing; but currently I think it is 😞 And yes if you want to do that boot is your best option; unless you want to build a lein plugin that essentially copies two other plugins entirely to compose them together on two futures 😞#2018-05-1314:52Alex Miller (Clojure team)Now available:
- tools.deps.alpha 0.5.435
- clj 1.9.0.381
- Support for Maven deps in authenticated repositories (TDEPS-9 - thanks Dominic Monroe!)
- In local jar dependencies like {:local/root “path/to.jar”}, if the jar has an embedded pom, it will be used to traverse and include transitive deps
- Use exec for final Java invocation in scripts (TDEPS-76 - thanks Mikhail Gusarov!)
- Convey lib map via Java system property - a first step towards add-lib and other future functionality
For Maven deps in authenticated repositories, existing Maven infrastructure is used to convey credentials via the ~/.m2/settings.xml:
<servers>
…
<server>
<id>my-auth-repo</id>
<username>zango</username>
<password>123</password>
</server>
…
</servers>
Then in your deps.edn include a repo with a name matching the <id>:
{:deps
{authenticated/dep {:mvn/version "1.2.3"}}
:mvn/repos
{"my-auth-repo" {:url ""}}}{:tag :div, :attrs {:class "message-reaction", :title "tada"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🎉")} " 12")}
#2018-05-1322:11zentropeAre these release-notes/change-logs recorded anywhere? I see new homebrew versions every now and then and wonder what the changes are.#2018-05-1322:11zentropeIn other words, release notes of the whole thing CLI utility, not just individual libs?#2018-05-1323:31seancorfieldThe shell script etc's commit history is here https://github.com/clojure/brew-install/commits/1.9.0#2018-05-1323:32seancorfieldAnd the library is everything else https://github.com/clojure/tools.deps.alpha/commits/master#2018-05-1323:34seancorfield@zentrope ^ does that help?#2018-05-1323:40zentrope@seancorfield Yep! Thanks.#2018-05-1323:41zentropeI wish brew packages came with a change log of some sort built-in. Or brew info would show the latest changes. Mostly I don’t care, but sometimes it’s interesting.#2018-05-1400:25dottedmag@zentrope brew log -p clojure#2018-05-1400:26Alex Miller (Clojure team)Both of those GitHub projects also have a CHANGELOG file too#2018-05-1400:28Alex Miller (Clojure team)The scripts do install a man page for clj and clojure - I could probably add something there but I don’t know if that’s weird#2018-05-1400:28zentropeYes, I saw those. I glanced right over brew-install as a project before asking.#2018-05-1400:29zentropeYeah, easy change logs aren’t really part of the brew thing, it seems, outside of that @dottedmag just showed.#2018-05-1400:55Alex Miller (Clojure team)I added links from the man page to the changelogs for the next release{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-1509:05shakdwipeeaHi guys, cross posting from #clojure
I am getting a dependency mismatch with s3. In my project I have specified dependency on com.amazonaws/aws-java-sdk-s3 {:mvn/version "1.11.261"}.
Also, I have a library whose transitive dependency include a different version of sdk . I have specified a exclusion for s3 as:
libB {:mvn/version "2.0.0-SNAPSHOT"
:exclusions [com.amazonaws/aws-java-sdk-s3]}
Now, when I do clj -Stree. I can see that only the correct version is being included.
But when I tried to find the location of sdk being included using
System.out.println(SDKGlobalConfiguration.class.getProtectionDomain().getCodeSource().getLocation()); as per https://github.com/aws/aws-sdk-java/issues/1071
I see that it is referring to the location of the jar of the transitive dependency which is the older version of aws sdk.
FWIW I am using the clj tool and deps.edn and the conflicting libraries are java based.
I am able to use the correct version from boot for the same tree.#2018-05-1522:29kennyWhere is the code for the clj cli tool located?#2018-05-1522:30ghadiIt's a bit oblique why, but it's here https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/clojure#2018-05-1522:31ghadiclj == rlwrap clojure basically#2018-05-1522:39kennyThanks#2018-05-1617:45kennyIs there a way to print the final deps.edn as data to stdout?#2018-05-1618:14Alex Miller (Clojure team)currently, no#2018-05-1618:19Alex Miller (Clojure team)you could use the output of -Sdescribe + a few calls in the api to replicate it#2018-05-1618:23kennyCool, figured it out. That option would be useful 🙂#2018-05-1618:27kennyFor those interested... Some psuedo code:
(defn full-deps-edn
[]
(let [config-files (:config-files (exec-sync-edn "clj -Sdescribe"))
deps "'{:deps {org.clojure/tools.deps.alpha {:mvn/version \"0.5.435\"}}}'"
code (str/join " " ['(require '[clojure.tools.deps.alpha.reader :as reader])
(list 'reader/read-deps config-files)])
cmd (str "clj -Sdeps " deps " -e '" code "'")]
(exec-sync-edn cmd)))
#2018-05-1722:06kennyHow do git deps work in a monolith type repo? It looks like I can only specify the root git URL.#2018-05-1722:16Alex Miller (Clojure team)You can also use :deps/root to specify a relative dir path from the root#2018-05-1722:16kennyPerfect! I was looking here to better understand all the options I can pass but it looks like it's not quite up to date 🙂 https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/specs.clj#2018-05-1722:17Alex Miller (Clojure team)Yeah those are a bit lagging#2018-05-1820:53kennyIs there a reason you shouldn't always use the -A CLI option?#2018-05-1820:55dominicmif the test alias uses test-runner, you may not want the main aspect of it 🙂#2018-05-1900:14seancorfieldThis is why I think it's better practice to have separate test and runner aliases 🙂#2018-05-1820:57kennyCould you clarify what you mean? What is test-runner? And what is the main aspect of it?#2018-05-1820:58dominicmSorry, I should be clearer, you may not want the :main-opts option to kick in.#2018-05-1820:59kennyAh that makes sense. Is that the only reason?#2018-05-1822:01hagmonkFun! clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.6.0"} gorilla-repl {:mvn/version "0.4.0"}}}' -m gorilla-repl.core#2018-05-1900:14seancorfieldThis is why I think it's better practice to have separate test and runner aliases 🙂#2018-05-2013:11raymcdermottI’m trying to make use of add-lib using clojure 1.10-alpha4 with the prepl …#2018-05-2013:12raymcdermott@alexmiller is that combo possible?#2018-05-2013:12dominicm@raymcdermott what happened when you tried?#2018-05-2013:16raymcdermott:cause “Context classloader is not a DynamicClassLoader”#2018-05-2013:17raymcdermottbit weird though cos it partly works#2018-05-2013:20raymcdermottso it ‘works’ when evaluating something that is in the path … but that’s not so useful 😉#2018-05-2013:24raymcdermottbut I’m stabbing and poking around, so asking here to get a steer on whether it’s meant to be viable#2018-05-2013:44dominicm@raymcdermott My suspicion is that however the socket repl starts, doesn't cause a DynamicClassLoader to be run, I think you can insert one, but can't remember from the top of my head how that's done.#2018-05-2013:46Alex Miller (Clojure team)That would be a good guess but I haven’t tried it#2018-05-2013:52dominicm@raymcdermott How are you running that clj file?#2018-05-2013:52raymcdermottlein run#2018-05-2013:53dominicmI think that uses aot-ish stuff, but I'm not certain.#2018-05-2013:55raymcdermottI’ll run it with clj and see if it makes any difference#2018-05-2013:56mfikes@raymcdermott when I hacked add-lib into ClojureScript, the hack needed to deal with the same issue: https://github.com/mfikes/clojurescript/commit/d68c9397599366777d9b322ec586fdd398302f25#2018-05-2013:57mfikesSee that bit where it forced the issue so that the classloader is a clojure.lang.DynamicClassLoader. This might not be the "correct" solution, but it worked.#2018-05-2013:58raymcdermottnice - I’ll check it out#2018-05-2014:00mfikesIf you end up needing to run that code for any reason to suss out what it is doing, https://gist.github.com/mfikes/a81ae3d006830dc34663f976aaf2291b#2018-05-2014:23Alex Miller (Clojure team)Setting the context classloader to a DCL should work#2018-05-2014:24Alex Miller (Clojure team)Although that is a per-thread solution#2018-05-2014:26raymcdermottok, so it’s the same whether I run it via CLJ or not … just thought I would check that out first#2018-05-2014:26raymcdermottDCL next - one thread is fine for now ( who needs more! 😉 )#2018-05-2014:40mfikesI'd cram it in early @raymcdermott (in a main if you can like in that ClojureScript hack), so any threads thereafter inherit it.
"When a Thread is created, it’s context classloader is set from the parent Thread." https://puredanger.github.io/tech.puredanger.com/2007/06/15/classloaders/#2018-05-2016:01Alex Miller (Clojure team)wow, old school reference :)#2018-05-2014:41raymcdermottparty-corgi#2018-05-2014:41raymcdermottwith DCL all works#2018-05-2014:41raymcdermottgood call @mfikes, will do - had it in the REPL code but will move it further out#2018-05-2015:49raymcdermottand - bonus - my code is now all set up with deps.edn and running with CLJ 😎#2018-05-2016:30robert-stuttafordwhat’s the correct way to force a SNAPSHOT download with clj / tools.deps?#2018-05-2017:10Alex Miller (Clojure team)There isn’t a way at the moment - I have a ticket for that. You can just delete the version from your .m2 though and use -Sforce#2018-05-2018:04robert-stuttafordthanks Alex!#2018-05-2110:31borkdudeStandalone script using tools.deps: https://github.com/borkdude/balcony#2018-05-2110:34robert-stuttaford@alexmiller @dominicm thanks for TDEPS-9 - using Datomic PRO with clj is rad#2018-05-2110:35dominicm@robert-stuttaford I setup the JUXT website last night, was nice 🙂#2018-05-2110:35robert-stuttafordi recognise this may be a daft question, but is there perhaps a way to run two ‘mains’ at once from clj ? i see the docs say that it’ll run the last one encountered, but i’m hoping to get a boot-like thing going where i can run both figwheel-main and my 10-liner CIDER repl script in one process#2018-05-2110:36dominicm@robert-stuttaford Update to cider 0.17 and use -e before your normal -m#2018-05-2110:36robert-stuttaford-reads about -e-#2018-05-2110:36dominicmhttps://github.com/clojure-vim/vim-jack-in/blob/master/autoload/jack_in.vim#L56-L70 -e is eval#2018-05-2110:37dominicm^^ this might be useful to see what cider generates when it does jack-in, and hard-code it in 🙂#2018-05-2110:37dominicmbut you could implement cider-nrepl.main/init yourself.#2018-05-2110:37robert-stuttafordi’ve got a working cider script:
(require '[clojure.tools.nrepl.server :refer [start-server]]
'[refactor-nrepl.middleware :refer [wrap-refactor]])
(defn nrepl-handler []
(require 'cider.nrepl)
(ns-resolve 'cider.nrepl 'cider-nrepl-handler))
(def port 7800)
(spit ".nrepl-port" port)
(start-server :port port :handler (wrap-refactor (nrepl-handler)))
(println "Started nREPL on port" port)
(.addShutdownHook (Runtime/getRuntime)
(Thread. #( ".nrepl-port")))
#2018-05-2110:38robert-stuttafordi want to run it and figwheel-main together, in one process. right now i use two#2018-05-2110:38dominicmAre you using cljs? That doesn't look like it will work with cljs. Unless you have a separate nrepl for figwheel?#2018-05-2110:39robert-stuttafordhmm, i guess there’s no way around it; i’ll have to write a clj script that starts the nrepl-server and invokes the figwheel-main stuff afterwards#2018-05-2110:40dominicmYou can replace that whole file with:
clj -e "(require 'cider-nrepl.main) (cider-nrepl.main/init '[refactor-nrepl.middleware/wrap-refactor])"
#2018-05-2110:40dominicmI'm presuming that can go into the deps.edn somewhere.#2018-05-2110:45dominicm@robert-stuttaford Apparently not, but mains can run other mains.
(defn -main [& args]
(start-nrepl)
(apply clojure.repl/main args))
#2018-05-2112:07robert-stuttafordturns out the figwheel-main api is super simple, this was a total non-issue#2018-05-2112:07robert-stuttaford;; nREPL
(require '[clojure.tools.nrepl.server :refer [start-server]]
'[refactor-nrepl.middleware :refer [wrap-refactor]])
(defn nrepl-handler []
(require 'cider.nrepl)
(ns-resolve 'cider.nrepl 'cider-nrepl-handler))
(def port 7800)
(spit ".nrepl-port" port)
(start-server :port port :handler (wrap-refactor (nrepl-handler)))
(println "Started nREPL on port" port)
(.addShutdownHook (Runtime/getRuntime)
(Thread. #( ".nrepl-port")))
;; Figwheel
(require '[figwheel.main :refer [start]])
(start "dev")
#2018-05-2119:14jeff.terrellWith the new clojure CLI tool, can I have a standalone .clj script, i.e. apart from a src/ directory and without a corresponding deps.edn file, ideally invokable via a simple shebang line (`#!`) at the top of the file? If so, what would that shebang line be?#2018-05-2119:16dominicmYes#2018-05-2119:16dominicmDigging it up now#2018-05-2119:20dominicm@jeff.terrell
#!/bin/sh
"exec" "clojure" "-Sdeps" '{:deps {funcool/tubax {:mvn/version "0.2.0"}}}' "$0" "#2018-05-2119:20jeff.terrellImpressive. Thanks, @dominicm! I'm assuming the :deps are optional?#2018-05-2119:22dominicm@jeff.terrell what do you mean optional?#2018-05-2119:22jeff.terrellAs in, I don't need any deps. I can just leave that part out, right?#2018-05-2119:23dominicmOh, sure. In that case you can just do.
/usr/bin/env clojure
...
#2018-05-2119:24dominicmNo args = shebang. Args = exec#2018-05-2119:25jeff.terrellAh perfect. I did that initially, but I was getting confused because I expected clojure to automatically call my -main function. But I see now that it was working fine. Thanks for the help!#2018-05-2119:31dominicm@jeff.terrell I came up with it.#2018-05-2119:47jeff.terrellNice trick! Thanks again.#2018-05-2119:59jeff.terrellI added a PR for a FAQ entry about this on http://clojure.com:
https://github.com/clojure/clojure-site/pull/270#2018-05-2121:33jeff.terrellOK, followup question: how do I access command-line arguments to a #!/usr/bin/env clojure standalone script?#2018-05-2121:46mfikes@jeff.terrell They will be in *command-line-args*#2018-05-2121:47jeff.terrellThat's it. Thanks!#2018-05-2121:48mfikes@jeff.terrell If useful, some similar material to your FAQ is at http://planck-repl.org/dependencies.html under Shebang Deps{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2214:03borkdude@jeff.terrell @mfikes I used that approach extensively here: https://github.com/borkdude/balcony/blob/master/balcony.clj{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2214:04borkdudeI credited Planck for it.#2018-05-2215:02jeff.terrellThat is one freaky shell/Clojure chimera there, @borkdude. simple_smile Well done.#2018-05-2215:05jeff.terrellI think you should be able to replace the outer double-quotes after the semi-colons with single-quotes, so that you don't have to escape the inner ones.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2215:06jeff.terrell(Bash deals with single quotes, and Clojure isn't going to care about them after the ;. That's my reasoning, anyway, but maybe I'm missing something.)#2018-05-2215:12Alex Miller (Clojure team)boot embeds a jar inside the shell script. I think that wins the chimera game for me.{:tag :div, :attrs {:class "message-reaction", :title "boot-clj"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "boot-clj", :src "https://emoji.slack-edge.com/T03RZGPFR/boot-clj/940f8bbd956fcb1b.png"}, :content nil})} " 4")}
#2018-05-2215:13borkdude@jeff.terrell thanks, made the change#2018-05-2215:13mfikesThat one wins the "Hello my Baby" frog jumping out of the box award{:tag :div, :attrs {:class "message-reaction", :title "laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😆")} " 4")}
#2018-05-2307:16martinklepschThe error message when deps.edn’s parens aren’t balanced could be friendlier:
Error building classpath. EOF while reading
java.lang.RuntimeException: EOF while reading
#2018-05-2309:41Alex Miller (Clojure team)File a ticket#2018-05-2314:49kennylein-tools-deps does not seem to be marking my src directory as a source root. I am using [lein-tools-deps "0.3.0-SNAPSHOT"] with :tools/deps set to [:system :home "/path/to/deps.edn"]. Has anyone else ran into this issue?#2018-05-2315:09mhuebert@kenny I have hit this just in the last couple of days, setting up a monorepo with a bunch of directories which each have their own deps.edn. I ended up adding "src" to :paths in each deps.edn#2018-05-2315:12kenny@mhuebert That is my exact situation and that solution did not work for me 😞#2018-05-2315:16mhueberthmm, I am trying to set up the repo on another machine now to do some more testing and getting a completely different error with lein-tools-deps
mattpro:chia MattPro$ lein pom
java.lang.NullPointerException: null
at clojure.tools.deps.alpha.util.maven$make_session.invokeStatic (maven.clj:105)
clojure.tools.deps.alpha.util.maven$make_session.invoke (maven.clj:102)
clojure.tools.deps.alpha.extensions.maven$eval851$fn__853.invoke (maven.clj:74)
clojure.lang.MultiFn.invoke (MultiFn.java:243)
clojure.tools.deps.alpha$expand_deps.invokeStatic (alpha.clj:170)
clojure.tools.deps.alpha$expand_deps.invoke (alpha.clj:152)
clojure.tools.deps.alpha$resolve_deps.invokeStatic (alpha.clj:215)
clojure.tools.deps.alpha$resolve_deps.invoke (alpha.clj:197)
lein_tools_deps.plugin$resolve_deps.invokeStatic (plugin.clj:78)
lein_tools_deps.plugin$resolve_deps.invoke (plugin.clj:69)
lein_tools_deps.plugin$middleware.invokeStatic (plugin.clj:93)
lein_tools_deps.plugin$middleware.invoke (plugin.clj:85)
#2018-05-2316:05mhuebert*figured out this was due to a too-old Leiningen version#2018-05-2315:19mhuebertthe line numbers on make_session make me think it is somehow not a recent version of tools.deps, though i just ran brew upgrade clojure#2018-05-2315:22Alex Miller (Clojure team)this is a known issue with interpreting the paths for dependent repos#2018-05-2315:23Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-74#2018-05-2315:23Alex Miller (Clojure team)or at least that’s my guess as to what you’re seeing#2018-05-2315:23Alex Miller (Clojure team)setting :paths explicitly in each deps.edn should workaround it#2018-05-2315:25mhuebertthat workaround worked in my case#2018-05-2315:26mhuebertare local paths skipped intentionally during pom-generation or is that just WIP?#2018-05-2315:27mhuebertam doing a workaround currently, generating a project.clj for lein to use lein-tools-deps to generate a full pom which includes everything necessary for IntelliJ to figure out the project#2018-05-2315:34Alex Miller (Clojure team)the first local path is added as the build source path, but I think that actually needs to be changed so it’s a resource path. I don’t think there’s actually a ticket right now about that#2018-05-2315:35Alex Miller (Clojure team)if you can file a ticket to remind me (or a patch!) that would be helpful#2018-05-2315:42mhueberthmm. why would it be a resource path?#2018-05-2315:42mhuebertI am comparing to what lein generates, it also makes <build><sourceDirectory>src</sourceDirectory> ... </build>#2018-05-2315:43mhuebertthat part seems to play well with intellij#2018-05-2315:46mhuebertSkipping paths: _build
Skipping coordinate: #:local{:root reactive}
Skipping coordinate: #:local{:root triple-db}
Skipping coordinate: #:local{:root view}
Skipping coordinate: #:local{:root gesture}
Skipping coordinate: #:local{:root style}
#2018-05-2315:48mhuebertI am wondering what can/should done about ^^these sorts of paths & local deps#2018-05-2315:50Alex Miller (Clojure team)oh, there is nothing that can be done about them - Maven has no concept of local deps#2018-05-2315:51Alex Miller (Clojure team)or git deps#2018-05-2315:51Alex Miller (Clojure team)so that’s never going to work#2018-05-2315:52Alex Miller (Clojure team)I guess you could write a custom Maven wagon and use that, but I’m not going to do that#2018-05-2315:57mhuebertit looks like lein pom uses a plugin (`build-helper-maven-plugin`) to add the local sources for those:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>_build</source>
<source>reactive/src</source>
<source>triple-db/src</source>
<source>view/src</source>
<source>view/test</source>
<source>gesture/src</source>
<source>style/src</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
#2018-05-2315:58mhueberthttps://github.com/technomancy/leiningen/blob/master/src/leiningen/pom.clj#L258#2018-05-2316:00mhueberthttps://www.mojohaus.org/build-helper-maven-plugin/usage.html#2018-05-2316:01Alex Miller (Clojure team)yeah, I’m not going to use that#2018-05-2316:01Alex Miller (Clojure team)you can add these to your pom manually if you like. -Spom will sync an existing pom, just updating the rest of it#2018-05-2316:03mhuebertok. at some point maybe intelliJ/cursive will support deps.edn directly#2018-05-2316:47Alex Miller (Clojure team)I think he’s started on that#2018-05-2319:46hagmonkANN: I just updated depify so that it can work from a deps.edn alias, if anyone is interested: https://github.com/hagmonk/depify#2018-05-2408:59mhuebert@kenny I just found something that may be related to your issues yesterday. In my mono-repo-of-`:local/root`-projects, if I change one of the deps.edn files inside of a :local/root directory, tools.deps does not recognize the change, and uses its previously cached classpath. I have to make some minor modification to the root deps.edn file (like, add a space somewhere) and then it invalidates the cache. So you may have configured everything correctly and clj was just using an old classpath. Can test for this by running clj -Spath before/after changes to see when it uses the cache vs. when it generates a fresh one.#2018-05-2409:08mhuebert@alexmiller would it be worth creating an issue for this? I am not sure exactly where/how cache invalidation is handled#2018-05-2412:12Alex Miller (Clojure team)I have an issue for it#2018-05-2412:15Alex Miller (Clojure team)Actually looks like maybe I never filed a public issue but I have it in my internal list{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-2412:15Alex Miller (Clojure team)Workaround is to use -Sforce#2018-05-2415:54borkdude-e, --eval string Eval exprs in string; print non-nil values
is it possible to turn off the printing?#2018-05-2416:02borkdudenever mind#2018-05-2418:47raymcdermottwhen using add-lib I sometimes get errors when adding via a git SHA …. Manifest type not detected when finding deps#2018-05-2418:47raymcdermotthere is the longer stack trace#2018-05-2418:48ghadiraymcdermott you may be add-lib'bing a git repo that has no deps.edn in it#2018-05-2418:48raymcdermotta ha!#2018-05-2418:48raymcdermott@ghadi I did not realize that was a pre-requisite#2018-05-2418:49raymcdermottpretty sure that is the case then#2018-05-2418:49ghadiYeah. tools.deps cannot infer third-party dependencies without some sort of manifest file{:tag :div, :attrs {:class "message-reaction", :title "man-bowing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-2418:50ghadiright now deps.edn and pom.xml are supported#2018-05-2418:50raymcdermottok, cool so that’s why Maven deps are safer#2018-05-2418:50raymcdermottfor the time being at least 😉#2018-05-2418:52raymcdermottat least I know which repos I can demo now#2018-05-2418:55raymcdermottoff the wall question … has tools.deps been tried out on graal?#2018-05-2418:58Alex Miller (Clojure team)Not to my knowledge#2018-05-2418:59Alex Miller (Clojure team)I suspect it’s bounded by network io usually#2018-05-2419:00raymcdermottI was thinking that it probably couldn’t change the classpath#2018-05-2419:01ghadiyeah, it cannot. add-lib will not work over there.
Does the REPL work in Graal?#2018-05-2419:01raymcdermottI’m guessing that an app can still expose a socket-server#2018-05-2419:02ghadi(I'm assuming you mean the Graal AOT binary part, not just running Graal through JVMCI)#2018-05-2419:02ghadithe repl (whether socket server or otherwise) generates classes, which is incompatible with Graal's AOT native images.#2018-05-2419:03ghadiBut all that should work properly through Graal non-AOT (the JVMCI)#2018-05-2419:03raymcdermottthat sounds legit…#2018-05-2419:04raymcdermottso that means - in theory at least - that you can call another language via the REPL#2018-05-2419:07raymcdermottthey have a polyglot shell so I’m guessing yes at this stage#2018-05-2419:08raymcdermottalthough it seems to switch languages rather than offer FFI#2018-05-2419:09raymcdermotthmmm this seems to be the thing#2018-05-2419:09raymcdermotthttp://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Context.html#2018-05-2419:10raymcdermottok anyway … that’s a different channel#2018-05-2419:11raymcdermottthanks for the help @ghadi#2018-05-2419:51raymcdermottbtw I made a small demo app with the pre-reqs in place for add-lib#2018-05-2419:51raymcdermotthttps://github.com/raymcdermott/add-lib-demo-app#2018-05-2419:52raymcdermottI have proven it with my REPL so it might work elsewhere 🙂#2018-05-2518:12kennyIs there a reason the -Spom option does not offer any way to configure the generated pom (i.e. project name and group id are the directory name and the version is hardcoded to 0.1.0)?#2018-05-2518:15Alex Miller (Clojure team)-Spom syncs an existing pom, so run it, then modify the pom. subsequent runs will keep that stuff.{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-2519:46kennyIf I run this
clj -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.9.0"}}}' -Spath
In a directory containing a deps.edn that looks like this:
{:deps {org.clojure/clojure {:mvn/version "1.8.0"}
org.clojure/core.async {:mvn/version "0.4.474"}}}
I get a classpath containing core.async. Why does that happen? I thought -Sdeps was supposed to be the final deps.edn file.#2018-05-2520:03hiredmanhttps://clojure.org/reference/deps_and_cli#_classpath_construction#2018-05-2520:04hiredmanit looks like dep.edns are merged, and -Sdeps is the final value merged#2018-05-2520:05kennyOh, you're right. Hmm, ok. It'd be nice to have a way to easily reproduce issues by explicitly passing in the one and only deps edn.#2018-05-2521:41seancorfieldThere's an argument with -Srepro that it should ignore everything except the current directory's deps.edn -- on the grounds that it would make a project "reproducible". As it is, ignoring the install directory with -Srepro means that you need extra stuff in your project deps.edn that you wouldn't normally need (`:paths`, repos?) which means you can't just take an arbitrary project's deps.edn file and run it with -Srepro unless that's how the author expected it to be run.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2521:47kennyI was asking for a directory independent way of running clj. That topic is somewhat related but still different.#2018-05-2521:56seancorfieldYeah, having more control over which of the four deps locations were consulted would be very nice.#2018-05-2521:57seancorfieldAnd your scenario -- ignoring the current directory's deps.edn -- is a good use case that it would be nice to see supported...#2018-05-2615:41renewdoitIs there any idea to download deps in parallel to speed up?#2018-05-2616:59Alex Miller (Clojure team)It’s something I have thought about. It’s a little tricky the way things are set up right now but def doable#2018-05-2809:01borkdudewithout Srepro deps aren’t downloaded but with it they are?
clojure -Srepro -Spath
I need the path but also the effect of deps being downloaded, but nothing else. What’s the best option for this?#2018-05-2809:02borkdudethis is excellent for lumo/planck btw, very fast#2018-05-2813:43Alex Miller (Clojure team)repro should not affect downloading. Are you just seeing m2 caching?#2018-05-2813:58borkdude@alexmiller when I e.g. delete tools.cli from my .m2 repo, only -Spath does not force downloading, but it works with -Srepro. What should the behavior be?#2018-05-2814:07Alex Miller (Clojure team)Oh, you’re misinterpreting stuff due to classpath caching#2018-05-2814:08Alex Miller (Clojure team)Normally, if there is a cached classpath, it’s used (and the actual paths aren’t checked, so it won’t notice you deleted from m2). But if there is no cached classpath it will download#2018-05-2814:08borkdudeif I can set M2_HOME somehow, I could try to reproduce. Don’t know how to do that with clj#2018-05-2814:09borkdudeah, that explains it#2018-05-2814:09Alex Miller (Clojure team)With repro, I think it does not use the cached classpath#2018-05-2814:09Alex Miller (Clojure team)So forces download #2018-05-2814:09Alex Miller (Clojure team)You can use -Spath -Sforce to do what you want#2018-05-2814:09borkdudeso in normal circumstances ( I don’t delete something from .m2 manually ) all should work according to plan#2018-05-2814:10Alex Miller (Clojure team)Yes, in the normal circumstances things work normally#2018-05-2814:11Alex Miller (Clojure team);)#2018-05-2814:11borkdudeyeah, but -Sforce takes longer, that’s not preferred for planck or lumo, so I’ll go for just -Spath#2018-05-2814:11Alex Miller (Clojure team)We actually have a ticket open for noticing broken paths #2018-05-2814:11borkdudehehe, yes, I mean, there is no logic to actually check for the existence of jars after classpath caching#2018-05-2814:12borkdudeso, I understand now, thanks#2018-05-2814:12borkdudeside question, is there a way to change which .m2/repository to use with clj?#2018-05-2814:12Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-70#2018-05-2916:55rickmoynihanMerely as an extra data point I ran into this issue yesterday because I’d blown an .m2 dep away whilst deving/testing some CI stuff. I realised I also needed to blow .cpcache away too to get the effect I wanted; but either a good message, or re-resolving the missing deps would’ve been super nice. It could easily have been a hard error to diagnose.#2018-05-2916:56Alex Miller (Clojure team)yeah, I’ve been there (even knowing how it works)#2018-05-2814:13Alex Miller (Clojure team)Yes, you can set :deps/local-repo in deps.edn map#2018-05-2814:13borkdudeah, that’s actually the issue yes. well, if it takes extra time it would be worth considering not doing it… or with another option#2018-05-2814:13borkdudeif it’s a couple of ms, it would be ok#2018-05-2814:14Alex Miller (Clojure team)Yeah I haven’t done any eval on it yet#2018-05-2814:14Alex Miller (Clojure team)TBD #2018-05-2814:15Alex Miller (Clojure team)There are also cases where you might expect to have a missing path dir#2018-05-2814:15Alex Miller (Clojure team)And would want that to be ok#2018-05-2814:16borkdudeI have this in deps.edn:
{:deps/local-repo "/tmp/repro"}
and do this from /tmp/repro:
clj -Sforce -Sdeps '{:deps {org.clojure/tools.cli {:mvn/version "0.3.7"}}}' -e "(require '[clojure.tools.cli])"
it doesn’t seem to use the local dir#2018-05-2814:28borkdudejust upgraded to deps.edn file#2018-05-2814:31borkdudeah, it was :mvn/local-repo#2018-05-2816:44Alex Miller (Clojure team)Oh sorry, yeah that :)#2018-05-2901:00hagmonkDoes anyone have a deps.edn snippet or advice on how to get a browser based CLJS app running, with an nREPL that Cider is able to reach? Basically like the new CLJS quickstart guide, except with the REPL in emacs instead.#2018-05-2903:39richiardiandrea@hagmonk the new cider should inject all the necessary deps as long as you have a deps.edn in the root folder#2018-05-2903:39richiardiandreaAlso better ask this in #cider#2018-05-2903:54hagmonk@richiardiandrea to rephrase my question so that it's not being confused with something cider specific: does anyone have a deps.edn snippet or advice on how to get a browser based CLJS app running, using cljs.main, such that it starts an nREPL server rather than an interactive terminal REPL?#2018-05-2904:04richiardiandreaDo not have that, but it would be a very small variation of this gist: https://gist.github.com/arichiardi/8ad47f5edf185876e82015739f038a22
Still believe some folks can help you in #cider simple_smile{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2907:52delaguardoHi there! I have a question about :default-deps. Does anybody have a good example how to use it? Official documentation tells it provides a set of default coordinate versions to use if no coordinate is specified. But how I can not specify coordinate versions for the :deps or :extra-deps?#2018-05-2911:54Alex Miller (Clojure team)This area is a little incomplete and there is not really a great story around how to use it yet#2018-05-2912:18delaguardoThere is my case if you are interested: I’m trying to create “monorepo” with root deps.edn and a few “sub-projects” with their own local deps.edn. I would like to have a list of all used dependencies with their coordinates as a :default-deps in root deps.edn. All local deps.edn files should have only a list of dependencies without coordinates specified.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2913:22Alex Miller (Clojure team)yeah, this is in line with what we intended but there needs to be another round of thinking on how to make this a reality{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2915:06delaguardo:thumbsup:#2018-05-2910:53shakdwipeeaHi guys, One of my transitive deps is trying to pull javax.activation and running into this error:
Error building classpath. Could not find artifact activation:activation:jar:1.0.2 in central ()
org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact activation:activation:jar:1.0.2 in central ()
Turns out the jar is not available in maven but it is available in one of the adobe public repos.
So I added
{:mvn/repos {"adobe-public" {:url ""}}}
Now this only works if javax.activation is a direct dependency but not when it is a transitive dep.
Also, the jar is available in maven local but I am still running into this.#2018-05-2910:58dominicmI've seen some funky poms where the pom actually refers to other repositories and those are transitively picked up.#2018-05-2911:56Alex Miller (Clojure team)@shakdwipeea are you getting to these transitive deps through a maven dep or a local/git dep?#2018-05-2912:00shakdwipeea@alexmiller It is coming through a git dep. I also tried with https://jitpack.io to use it as a mvn dep but that also didn't work. The funny thing is that the required dep is present in my maven local but still this fails.#2018-05-2912:01shakdwipeeaFWIW the dep is a java project and uses maven.#2018-05-2912:14Alex Miller (Clojure team)Yeah this is a known issue with git and local deps not picking up the repos in the pom#2018-05-2912:17Alex Miller (Clojure team)Or at least I thought it was, I don’t see a ticket for it. I’d be happy to have a ticket with more specific details so I can take a look#2018-05-2915:23shakdwipeeaI'll try to create a minimal repro and file a ticket.#2018-05-2915:44Alex Miller (Clojure team)thx#2018-05-3007:05shakdwipeeaI tried to reproduce the issue on a new repo. https://github.com/shakdwipeea/tdeps-transitive-repo-test Now when you clj -A:test-git it fails with the message that
`
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact activation:activation:jar:1.0.2 in central ()
The same deps works with the boot file that is present in the repo. The interesting thing is that
when I do boot show -d, I get
[com.github.shakdwipeea/smtp "0.3.1"]
└── [javax.javaee/javaee "6.0-alpha-1"]
├── [javax.activation/activation "1.1-rev-1"]
...
This uses the version 1.1-rev-1 which is available in the maven repo.
The 1.0.2 version should not even be fetched.
I am not sure if I am doing sth wrong, so wanted to consult before opening a ticket.#2018-05-2918:13hagmonk@alexmiller how does one contribute to the tools.deps.alpha github wiki? I wanted to put a link to depify, and @seancorfield's clj-new.#2018-05-2918:14Alex Miller (Clojure team)it’s a wiki - anyone should be able to edit#2018-05-2918:15hagmonkCool! I did see the edit button but didn't want to boldly assume :)#2018-05-2918:17seancorfield@hagmonk Thank you in advance for adding a link to clj-new 🙂{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-05-2918:21hagmonkDone! I'm sure there are more that could be added ...#2018-05-3007:05shakdwipeeaI tried to reproduce the issue on a new repo. https://github.com/shakdwipeea/tdeps-transitive-repo-test Now when you clj -A:test-git it fails with the message that
`
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact activation:activation:jar:1.0.2 in central ()
The same deps works with the boot file that is present in the repo. The interesting thing is that
when I do boot show -d, I get
[com.github.shakdwipeea/smtp "0.3.1"]
└── [javax.javaee/javaee "6.0-alpha-1"]
├── [javax.activation/activation "1.1-rev-1"]
...
This uses the version 1.1-rev-1 which is available in the maven repo.
The 1.0.2 version should not even be fetched.
I am not sure if I am doing sth wrong, so wanted to consult before opening a ticket.#2018-05-3011:28borkdude@shakdwipeea what if you move the deps to a toplevel :deps?#2018-05-3011:29shakdwipeeaI had tried that.. Same result.. Made aliases to test out the two conditions separately#2018-05-3113:05rickmoynihanFYI, we’ve just merged :resolve-aliases support in to lein-tools-deps. Currently available only on the 0.4.x branch:
https://github.com/RickMoynihan/lein-tools-deps/blob/0.4.x/example/project.clj#L20{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-3116:03kennyHow do I load a Clojure file on the classpath from the CLI? I tried doing clj -i my.ns and I get a FileNotFound exception.#2018-05-3116:05kennyI guess I could just call load in -e. Is there an easier way though?#2018-05-3116:07ghadiclj -i foo.clj should work -i expects a file name, not a namespace{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-05-3116:22kennySay I have a monolith type git repo that looks something like this:
mylib1
mylib2
mylib2 has a deps.edn files that looks like this:
{:deps {mylib1 {:local/root "../mylib1"}}}
Is it possible to use mylib2 as a git dep (`{:git/url "https://github.com/..." :sha "..." :deps/root "mylib2"}`) from the CLI and have mylib1 dependency be resolved using the :local/root path and the same :sha used previously?#2018-05-3116:28kennyOr perhaps even changing mylib1 coordinate to be {:git/url "" :deps/root "mylib1"} and the :sha would be automatically set to the same :sha used for the the git URL ... that is already on the classpath.#2018-05-3116:32Alex Miller (Clojure team)In short, no - git libs are only resolved from git#2018-05-3119:54kennyIf I have foo.clj:
(ns foo)
(defn bar
[& args]
(prn *command-line-args*))
And run:
clj -i foo.clj -e '(foo/bar)' asd
I get a FileNotFoundException.#2018-05-3119:55kennyI assume this is because it is expecting the last param to be a path. Why does it require that?#2018-05-3119:58ghadi@kenny https://github.com/clojure/clojure/blob/master/src/clj/clojure/main.clj#L388-L417
hope that helps#2018-05-3119:58ghadiyes the last param is a path#2018-05-3119:59kennyRight. Why does it require that?#2018-05-3120:02zentropeAre relative paths wonky for deps? For instance, if you local/root a deps.end which in turn local/root’s a dep (something like “../../../whatever”) is the “../../..” going to mess things up if the top level isn’t in a “../../../” relationship to that project?#2018-05-3120:08zentropehttps://dev.clojure.org/jira/browse/TDEPS-74#2018-05-3120:03seancorfield@kenny Use - to introduce the arguments: clj -i foo.clj -e '(foo/bar)' - asd#2018-05-3120:03zentropeHm. Overriding it in the “current” deps.edn solves it.#2018-05-3120:04kenny@seancorfield The program doesn't exit then.#2018-05-3120:05ghadi@kenny what are you trying to achieve?#2018-05-3120:05seancorfieldAh, yes, you'd have to send it empty input 🙂#2018-05-3120:05kennyTo evaluate the function in -e.#2018-05-3120:05ghadithere may be a more straighforward path.#2018-05-3120:05kenny(foo/bar)#2018-05-3120:06seancorfieldYou can only get *command-line-args* if you have main-opts#2018-05-3120:06ghadiyou can -e "(require 'foo)" -e "(foo/bar)"#2018-05-3120:07kennyI also want *command-line-args* 🙂#2018-05-3120:07ghadioh -- yeah you can't.#2018-05-3120:08ghadiOr, this (cue eye roll):
clj - <<EOM
(require 'foo)
(foo/bar)
EOM
#2018-05-3120:08ghadi(In bash or zsh)#2018-05-3120:08seancorfieldRename bar to -main and then clj -i foo.clj -m foo asd#2018-05-3120:08ghadi^^^#2018-05-3120:08seancorfield(! 633)-> cat foo.clj
(ns foo)
(defn -main
[& args]
(prn *command-line-args*))
Thu May 31 13:08:39
(sean)-(jobs:0)-(~/clojure/kenny)
(! 634)-> clj -i foo.clj -m foo asd
("asd")
#2018-05-3120:08kennyBleh. And I can't rename because -main is used for something else.#2018-05-3120:09kennyI also may want to call different functions besides -main.#2018-05-3120:09seancorfieldAdd another namespace for this entry point?#2018-05-3120:10kennyPerhaps. But I may have 50 different functions that I'd like to call. I don't want 50 namespaces to simply call a function.#2018-05-3120:11seancorfieldOK, how about this: (! 638)-> cat foo.clj
(ns foo)
(defn -main [& args] (println "I am foo's -main"))
(ns foo.main)
(defn -main
[& args]
(prn *command-line-args*))
Thu May 31 13:10:56
(sean)-(jobs:0)-(~/clojure/kenny)
(! 639)-> clj -i foo.clj -m foo
I am foo's -main
Thu May 31 13:11:04
(sean)-(jobs:0)-(~/clojure/kenny)
(! 640)-> clj -i foo.clj -m foo.main asd
("asd")
#2018-05-3120:11seancorfield(multiple namespaces in one file 🙂 )#2018-05-3120:12kennyAh, interesting.#2018-05-3120:12kennyStill feels like a bit of a hack but that could work. It seems like it'd be useful to have a CLI option to execute a function besides -main.#2018-05-3120:13seancorfieldWe have a job script at work whose first argument is a namespace-qualified symbol and it calls that from -main. So you'd say clj -i job.clj -m job job/foo asd#2018-05-3120:14seancorfieldRelying on dynamic require and resolve#2018-05-3120:14kennyYeah, I think I like that option more.#2018-05-3121:43kennyShould :override-deps work to change an artifact's coordinate from Git to local?#2018-05-3121:53kennyIt doesn't seem to work. Example a/deps.edn:
{:deps {org.clojure/tools.cli {:mvn/version "0.3.7"}}}
b/deps.edn:
{:deps {a {:git/url ""
:sha "7b8e5c33588a94c1318e225dc9fe655599216370"}}
:aliases {:dev {:override-deps {a {:local/root "../a"}}}}}
Result of clj -A:dev -Sforce -Spath:
src:/home/kenny/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/home/kenny/.gitlibs/libs/a/a/7b8e5c33588a94c1318e225dc9fe655599216370/src/main/clojure:/home/kenny/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/home/kenny/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar
That classpath does not contain tools.cli as required by the overridden a dependency.#2018-05-3122:14Alex Miller (Clojure team)No, there is a ticket for this right now#2018-05-3122:17kennyhttps://dev.clojure.org/jira/browse/TDEPS-51. Is the plan to allow this behavior or throw?#2018-05-3122:38Alex Miller (Clojure team)To allow{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-06-0115:29kennyAny way to use aliases defined in a library that is in your :deps?#2018-06-0115:46Alex Miller (Clojure team)no#2018-06-0115:34rickmoynihanFYI :aliases, :resolve-aliases and :classpath-aliases are implemented on 0.4.x branch of lein-tools-deps… just need to kick the tyres a bit and add some documentation before merging to master{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-0120:10hypirionIs there any way to compile or refer to Java files with deps.edn, or do I have to do that in a separate project?#2018-06-0120:11dominicm@hypirion Separate#2018-06-0120:14hypirionOkay, I guess that is by design? It's a bit iffy if I just want to improve the performance of a component in the system by converting it to Java.#2018-06-0120:31Alex Miller (Clojure team)in scope: a) make classpaths and b) launch Clojure programs#2018-06-0120:31Alex Miller (Clojure team)there are a vast array of other “build” activities you may wish to do#2018-06-0120:31dominicmI have thought about writing something which scans for .java rules in directories on the classpath and builds them.#2018-06-0120:32Alex Miller (Clojure team)well there are about 10 programs that do that already#2018-06-0120:32dominicmThere are?#2018-06-0120:33Alex Miller (Clojure team)Sure, Ant, Maven, Leiningen, Boot, Ivy, Grape,#2018-06-0120:33Alex Miller (Clojure team)and on and on#2018-06-0120:34dominicmSure. They're a bit different. By default they don't expect to compile dependencies, eg local and root dependencies#2018-06-0120:34Alex Miller (Clojure team)some yes, some no#2018-06-0120:34Alex Miller (Clojure team)you can easily coax Ant or Ivy into this#2018-06-0408:22borkdudeI’m looking into the nix package manager right now and wonder if classpath caching was inspired by this maybe?#2018-06-0409:40mkvlr@borkdude as someone without fear of parens guix might also be of interest to you. It’s basically nix written in scheme.#2018-06-0409:41borkdudecool#2018-06-0410:31robert-stuttafordwhat are the rules for downloading snapshots? i have two extra :mvn/repos defined (Datomic’s, and our own S3 one), and for some reason, tools.deps is failing on figwheel.main’s dep because it’s not attempting it on the one repo that has it - http://clojars.org.#2018-06-0410:32robert-stuttafordgosh, nevermind. i had a non-standard space after the dep name, which caused it to be included in the dep name 🙈#2018-06-0410:39dominicmthat's horribly awesome.{:tag :div, :attrs {:class "message-reaction", :title "japanese_goblin"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👺")} " 8")}
#2018-06-0411:13borkdudemaybe write a patch for Clojure to treat that like the other whitespace 😛#2018-06-0411:13dominicmHow will I drive my co-workers crazy then?#2018-06-0415:51dominicmI'm having an issue I don't really understand, I'm trying to depend on jnr-posix, with lein all is happy, and with clojure tools the same snippet doesn't work.
com.github.jnr/jnr-posix {:mvn/version "3.0.42"} is the version.
user=> (import
#_=> (jnr.posix POSIXFactory POSIXHandler))
jnr.posix.POSIXHandler
user=> (POSIXFactory/getNativePOSIX)
#object[jnr.posix.LinuxPOSIX 0x6dd5e5ec "
Is the snippet that does work in lein, but not in clojure cli tools.
I even tried doing lein deps :tree and using those dependencies in deps.edn:
{com.github.jnr/jnr-posix {:mvn/version "3.0.42"}
com.github.jnr/jnr-constants {:mvn/version "0.9.9"}
com.github.jnr/jnr-ffi {:mvn/version "2.1.7"}
com.github.jnr/jffi {:mvn/version "1.2.16"}
com.github.jnr/jnr-x86asm {:mvn/version "1.0.2"}
org.ow2.asm/asm-analysis {:mvn/version "5.0.3"}
org.ow2.asm/asm-commons {:mvn/version "5.0.3"}
org.ow2.asm/asm-tree {:mvn/version "5.0.3"}
org.ow2.asm/asm-util {:mvn/version "5.0.3"}
org.ow2.asm/asm {:mvn/version "5.0.3"}}
But that hasn't resolved the issue either.#2018-06-0415:55dominicmLooks like in the deps.edn version, ( "jni/x86_64-Linux/libjffi-1.2.so") doesn't resolve.#2018-06-0415:55dominicmOH, I see. It's because jffi has a native classifier.#2018-06-0415:57dominicmTDEPS-50 and 12. No fun being here 😛#2018-06-0415:57dominicmIs there any known workarounds?#2018-06-0416:10Alex Miller (Clojure team)you could pull the other classifiers in as local jars#2018-06-0416:11Alex Miller (Clojure team){:deps {madeup/dep {:local/root "path/to/the.jar"}}}#2018-06-0416:19dominicm@alexmiller this is in pack, I'd like to keep the repo small (no jars in the repo). But I've done that into my maven repo for now, only I need to run the tests, right?#2018-06-0416:22dominicmTrying to write some tests for pack, which involves changing the jvm directory in order to have :local/root "../blah" work correctly.#2018-06-0416:52Alex Miller (Clojure team)I don’t understand what you’re saying, sorry#2018-06-0417:17dominicm@alexmiller I'm changing the cwd of the process so that relative paths work in deps.edn files.#2018-06-0417:18dominicmSo that I can write tests for pack#2018-06-0417:42dominicmhttps://www.engineyard.com/blog/bats-test-command-line-tools maybe this would have been less effort...#2018-06-0422:31kennyI'd like to make a script to run tests and launch that script with clj. Something like clj -A:test -m test-runner. In order to run tests, you need to know which namespaces you want to test. To determine the namespaces to test you need to find all namespaces given some file system paths. I'd like to be able to figure out the file system paths from the super set of :paths and :extra-paths in your deps config files given the aliases you passed to the clj command. I can't see a way to get the aliases you used to launch the clj command. Any ideas?#2018-06-0422:39seancorfield@kenny Have you looked at Cognitect Labs' test-runner project?#2018-06-0422:39seancorfieldhttps://github.com/cognitect-labs/test-runner#2018-06-0422:42kennyOh cool! Seems like that project takes a different approach by defaulting to test directory otherwise you need to explicitly specify the directory for your tests. I suppose that'd work.#2018-06-0422:44kennyI still think it would be useful to introspect the clj props used to launch the JVM.#2018-06-0422:45seancorfieldSee https://github.com/seancorfield/dot-clojure for an example of using test-runner where the tests could be set by an alias (I specifically run tests with clj -A:test:runner -- where :test can vary by project)#2018-06-0422:46seancorfieldHere's an example of that test-runner used with multi-version testing https://github.com/clojure/java.jdbc/blob/master/deps.edn#2018-06-0422:47seancorfieldhttps://github.com/clojure/java.jdbc/blob/master/run-tests.sh#L47#2018-06-0422:48seancorfield> I can't see a way to get the aliases you used to launch the clj command. Any ideas?
I'm not sure why you'd need/want this?#2018-06-0422:49seancorfieldYou can't do anything with those aliases...#2018-06-0422:49kennyMakes sense.#2018-06-0422:50kennyThere are lots of things I can do with the aliases. I can use them to determine information about the deps.edn file. I can launch sub-processes with the same classpath as the parent process.#2018-06-0422:52seancorfieldSo this isn't a real world need, just a "what if"?#2018-06-0422:52kennyIt is real world because that's the exact problem I'm running into right now 🙂#2018-06-0422:53seancorfieldCan you elaborate? What, specifically, are you trying to do where knowing the actual aliases used would be a solution?#2018-06-0422:55kennyA simple example would be to get a set of all :paths and :extra-paths used to launch the JVM via clj.#2018-06-0422:56seancorfieldYou can already get the classpath in any JVM program... and then you can walk that to see which are directories and which are files...#2018-06-0422:56kennyDoesn't work with local deps 😉#2018-06-0422:56kenny:local/root#2018-06-0422:57kennyThe set of all directories on the classpath may be a super set of :paths, :extra-paths, and other directories with no way to differentiate the three.#2018-06-0422:58seancorfieldAnd you need to tell them apart because... why?#2018-06-0422:58kennyDifferentiate files in your current project from dependencies.#2018-06-0422:58kennyI simply want a list of files in :paths and :extra-paths.#2018-06-0422:59seancorfieldSorry, I'm still not understanding the actual problem you're trying to solve here. I'll just give up and go away.#2018-06-0423:01kennySorry, I should have started with the most precise, minimal problem.
I'd like to get a list of files within the :paths and :extra-paths for the clj launched JVM within the JVM.#2018-06-0423:02seancorfieldYes, you keep saying that, but I still don't understand what problem you are trying to solve.#2018-06-0423:07kennyIt could be anything - this will be used for general operations over your project files. For example, I may want to run a formatter over all my project files. As done in the test-runner you sent, I could require the user to explicitly pass the directory and default to something like ["src" "test"]. It seems preferable, however, to default to the current set of files within :paths and :extra-paths because these are the files you are currently working with.#2018-06-0423:13seancorfieldWhy not use the current working directory where the <insert general tooling> was invoked, and walk the classpath for paths that include that directory as a prefix? Then you have something that is general and could be invoked via clj, lein, or boot.#2018-06-0423:14seancorfield(with an option to specify paths as -main arguments)#2018-06-0423:16kennyBut why require the user to specify a prefix when they already included the directories they want via aliases?#2018-06-0423:16seancorfieldRe-read what I suggested.#2018-06-0423:16seancorfieldThe user would not have to specify a prefix.#2018-06-0423:17kennyOh I see.#2018-06-0423:17seancorfieldThen the tooling you produce really is general rather than being hardwired to some implementation artifact of how clj happens to work today.#2018-06-0423:17kennyI don't believe that is general though. You can specify a path via an alias that is not in the parent path.#2018-06-0423:19seancorfieldIf :paths or :extra-paths end up pointing to directories outside the project where you invoked clj (and your imaginary tooling), I think most people would be surprised that it performed operations on those files, rather than what is local inside the current project folder...?#2018-06-0423:21kennyPerhaps. But they also explicitly included that as a :path instead of a local dependency so they probably have a reason for doing.#2018-06-0500:36Alex Miller (Clojure team)this should be considered highly unusual and possibly even a bug#2018-06-0423:22seancorfieldI think you're trying to solve a non-problem and making your tooling set up more complicated (and more brittle) than it needs to be.#2018-06-0423:23seancorfieldSolve the simple, obvious use case first -- using just the current working directory and the classpath, which is the general tool-agnostic case -- and then see if any of your users really do things differently.#2018-06-0423:24kennyThat's fair 🙂#2018-06-0423:24seancorfield(I've learned this the hard way with boot-tools-deps -- with hindsight, I would have done that very differently)#2018-06-0423:25kennyWould you prefer the test-runner project default to directories on the classpath that are prefixed with your current directory or simply the "test" directory?#2018-06-0423:27seancorfieldFor a test runner, it makes sense to provide an external, tool-agnostic way to specify where to find tests -- since you already know the source code is going to be on the classpath (as well as the test code).#2018-06-0423:29seancorfieldFor example, in simple Clojure projects, src and test are the two top-level folders. In mixed clj/cljs projects, you often have src/clj, src/cljc, src/cljs (and similarly for test) so specifying just what should be run on the JVM is already more complex. And in Maven-style projects, tests are under src/test -- where source code is under src/main (and those tend to have <language> as the next level and then the actual code tree).#2018-06-0423:33kennyMakes sense. Thanks for the direction.#2018-06-0604:05cflemingI’m looking into integrating deps support into Cursive. I have some questions about how deps is expected to be invoked for this use case. I’m assuming for the moment that the user must have the command line tools installed, and will have to configure the path to the clojure command.#2018-06-0604:06cflemingOnce I have that, I can then execute with either -Sverbose or -Sdescribe, and get some information about the environment, and I can get a classpath.#2018-06-0604:06cflemingBut I can’t see a way with the command line tool to get path information like source and test paths.#2018-06-0604:08cflemingI think I’ll have to be using tools.deps.alpha to read e.g. deps.edn files, but I also can’t see a good way to find either the uberjar from the local install, or get the tools.deps.alpha version that I would need to download it myself and use it as a library.#2018-06-0604:09cflemingIs anything provided for this kind of use case?#2018-06-0604:10cflemingAlso, I’m assuming that deps doesn’t provide any help for downloading source artifacts of dependencies?#2018-06-0604:19cflemingI just tried clojure -Sdeps '{joda-time/joda-time {:mvn/version "2.9.7" :classifer "sources"}}' -Spath but no luck.#2018-06-0605:29seancorfield@cfleming Not sure whether you can rely on it "forever" but clojure -A:deps brings in the tools.deps.alpha version...#2018-06-0605:33seancorfield(and, yes, the tools.deps.alpha library will resolve and download all your dependencies)#2018-06-0611:28Alex Miller (Clojure team)On src/test paths - deps.edn doesn’t make this distinction. Normally I’d expects src to be part of the default :paths and test to be provided via an :extra-paths in an alias #2018-06-0611:29Alex Miller (Clojure team)On tda version, I could probably add that version to verbose or describe#2018-06-0611:30cfleming@alexmiller Right, the issue is that I need t.d.a to get that info because I really have to parse deps.edn to get that level of detail.#2018-06-0611:30cflemingThat would be great.#2018-06-0611:31Alex Miller (Clojure team)On source artifacts, nothing in clj, but you can easily use stuff in tda directly to do that. Surely IntelliJ already has something though? I guess maybe it relies on a pom?#2018-06-0611:31cflemingRight, that’s all in the Maven plugin which is an optional dependency.#2018-06-0611:32cflemingI’ve had problems with users disabling that and then wondering why e.g. autocomplete in project.clj doesn’t work.#2018-06-0611:33cfleming@alexmiller Is the deps alias going to stick around?#2018-06-0611:34cflemingIf so then I might be able to use clojure -A:deps -Spath, although it’s a little hacky.#2018-06-0611:36Alex Miller (Clojure team)Not planning on removing it but I hate to have you rely on that#2018-06-0611:37cflemingHaving some canonical way to build a classpath with the installed version of tda would be nice.#2018-06-0611:37Alex Miller (Clojure team)Yeah#2018-06-0611:37cflemingEven putting the version in -Sdescribe punts to the client and forces it to support Maven resolution.#2018-06-0611:38cflemingHaving the classpath just include the uberjar would be fine too.#2018-06-0611:39Alex Miller (Clojure team)The uberjar makes me feel dirty :)#2018-06-0611:39cflemingActually, perhaps just add the version and the uberjar path to -Sdescribe? Seems a bit internal… yeah#2018-06-0611:40cflemingBut, you know, it’s already there…#2018-06-0611:41Alex Miller (Clojure team)Give me a day or two to think about it#2018-06-0611:41cflemingSure#2018-06-0611:42cfleming@alexmiller Another feature request for the API would be to provide a download progress callback.#2018-06-0611:43cflemingThe Maven stuff supports that, it’s probably trickier for git.#2018-06-0611:43cflemingDo you use JGit or shell out?#2018-06-0611:43cflemingJGit might provide something.#2018-06-0611:44Alex Miller (Clojure team)Jgit atm. They do have some callback stuff.#2018-06-0611:44cflemingIt would be great to expose that - downloading the deps of something like immutant can take forever.#2018-06-0611:45Alex Miller (Clojure team)Expose like via printing or by letting a caller get notification ?#2018-06-0611:46Alex Miller (Clojure team)Would be great to get tickets for this kinda stuff#2018-06-0611:47cflemingI can file some tomorrow.#2018-06-0611:48cflemingBy passing in a function which gets called back.#2018-06-0611:48cflemingThe pomegranate version looks like this: https://github.com/cemerick/pomegranate/blob/27b4ff3418bd989a52a6b0bd9df16c4954a92d8f/src/main/clojure/cemerick/pomegranate/aether.clj#L69-L110#2018-06-0611:49cflemingThat works nicely.#2018-06-0611:50cflemingAnd this:#2018-06-0611:50cfleminghttps://github.com/cemerick/pomegranate/blob/27b4ff3418bd989a52a6b0bd9df16c4954a92d8f/src/main/clojure/cemerick/pomegranate/aether.clj#L126-L137#2018-06-0611:51cflemingRight, bedtime. I’ll file some tickets tomorrow.#2018-06-0616:01Hugh PowellHi folks,
I'm working on a project to get leiningen to source deps.edn for dependency information (lein-tools-deps).
We currently shell out to clojure -Sdescribe to get the vector of :config-files.
This command also returns other information which requires resolution of all a projects dependencies (specifically :resolve-aliases, I think).
Is there any way to get just the :config-files (or all the information that doesn't need full resolution of all dependencies to be completed)?#2018-06-0616:49seancorfield@hugh336 Do you mean https://github.com/RickMoynihan/lein-tools-deps ?#2018-06-0708:03Hugh PowellYep, that's the one#2018-06-0715:09rickmoynihan@hugh336: Yeah it would be really useful if there was a variant on -Sdescribe that didn’t attempt to resolve your dependencies etc, and only returned the information that is known locally about the installation.
As for us in lein-tools-deps this means that we end up resolving aliases twice once when we don’t really want to#2018-06-0715:10rickmoynihanI don’t think there’s a good way around this without changing tools deps / or the clj/clojure command line project.#2018-06-0715:14Alex Miller (Clojure team)there is actually a flag buried in the underlying clojure program to tell it not to resolve#2018-06-0715:15Alex Miller (Clojure team)but it’s not surfaced out to the top#2018-06-0715:15Alex Miller (Clojure team)I’d be happy to have a ticket about the problem in TDEPS so I can think about a solution#2018-06-0715:16rickmoynihanwhat is the flag?#2018-06-0715:17Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj#L30#2018-06-0715:18Alex Miller (Clojure team)if set, that avoids doing the resolution, making a classpath, and saving those off#2018-06-0715:20Alex Miller (Clojure team)was added for some stuff Stu was doing (in the guts of ion impl) where similar need arose#2018-06-0715:21rickmoynihanInteresting… I thought the issue was here as we shell out to run -Sdescribe
https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/clojure#L320-L332
though I could be misunderstanding the flow#2018-06-0715:22Alex Miller (Clojure team)yes, this flag is at a lower level and not exposed at the script level as I mentioned above#2018-06-0715:23rickmoynihancool… we can look at registering an issue#2018-06-0715:24Alex Miller (Clojure team)wait, -Sdescribe doesn’t resolve anything#2018-06-0715:24Alex Miller (Clojure team)so I will go all the way back to the top of this discussion and disagree with the premise#2018-06-0715:25rickmoynihanlol ok 🙂#2018-06-0715:25Alex Miller (Clojure team)so this: “Yeah it would be really useful if there was a variant on -Sdescribe that didn’t attempt to resolve your dependencies etc, and only returned the information that is known locally about the installation.” is not correct#2018-06-0715:26Alex Miller (Clojure team)that’s exactly what -Sdescribe does#2018-06-0715:27rickmoynihanok it’s not what we’re seeing… maybe there’s something else causing us to misdiagnose the problem though#2018-06-0715:27Alex Miller (Clojure team)well maybe there’s a bug there#2018-06-0715:27Alex Miller (Clojure team)looking at the script, could definitely be but it’s just a matter of another flag check before resolving#2018-06-0715:28Alex Miller (Clojure team)so, I’d say that’s the ticket to file - ensure -Sdescribe doesn’t trigger resolution{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-06-0715:30Hugh Powell@rickmoynihan @alexmiller Thanks for the discussion, I'll file a ticket{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-0715:30Alex Miller (Clojure team)sorry for my densitude#2018-06-0715:31rickmoynihannot at all, you must be pulled every which way#2018-06-0715:35rickmoynihan@alexmiller One other thing it looks like when using tools.deps as an API we don’t get the caching behaviour because it’s trapped inside the shell script here:
https://github.com/clojure/brew-install/blob/e706dbedbe91446ae0a09ff32fd36620cc966c55/src/main/resources/clojure#L238
I understand it’s probably there because it avoids you having to start another VM, but it seems to mean we don’t get the libs caching in our plugin. Is there a clj implementation of this somewhere, or do we need to write one?#2018-06-0715:56rcustodioHey… I’m using this org.imgscalr/imgscalr-lib {:local/root "libs/imgscalr-lib-4.3.jar"}
But when I do uber with boot, using seancorfield/boot-tools-deps, this jar is not extracted and added to the standalone jar, how do I do it, this is my boot command
BOOT_CLOJURE_VERSION=1.9.0 boot -d seancorfield/boot-tools-deps:0.4.5 \
boot-tools-deps.core/deps -B \
pom -p "zzk/catalog-image-svc" -v "0.0.1" -d "ZZK image processing" -u "" -l "X:X" \
uber \
jar -m clojure.main \
target
#2018-06-0716:16Hugh Powell@rickmoynihan @alexmiller Ticket created https://dev.clojure.org/jira/browse/TDEPS-82{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-0716:18Alex Miller (Clojure team)thx#2018-06-0716:36seancorfield@rcustodio Does it work (run) if you use just clj, not boot? Can you build a JAR with depstar or pack?#2018-06-0716:36rcustodioYes, it works with run#2018-06-0716:36rcustodioclj run#2018-06-0716:37rcustodiodepstar pack, i dunno what that is#2018-06-0716:37seancorfieldCan you put the project up on GitHub so I can clone it and try it out?#2018-06-0716:37seancorfielddepstar and pack are clj-based utilities to create uberjars.#2018-06-0716:38rcustodiohmm.. I can’t because its a in-company project#2018-06-0716:38rcustodioThat is my deps
{:aliases
{:deps {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.5.435"}}},
:test {:extra-paths ["test"]}},
:deps
{br.com.vikingmakt/augustus {:mvn/version "0.1.12"},
br.com.vikingmakt/njord {:mvn/version "0.3.0"},
cheshire {:mvn/version "5.8.0"},
clj-time {:mvn/version "0.14.4"},
com.novemberain/langohr {:mvn/version "5.0.0"},
com.novemberain/monger {:mvn/version "3.1.0"},
net.mikera/imagez {:mvn/version "0.12.0"
:exclusions [org.imgscalr/imgscalr-lib]},
org.apache.logging.log4j/log4j-core {:mvn/version "2.11.0"},
org.apache.logging.log4j/log4j-slf4j-impl {:mvn/version "2.11.0"},
org.clojure/clojure {:mvn/version "1.9.0"},
org.clojure/tools.logging {:mvn/version "0.4.1"},
org.imgscalr/imgscalr-lib {:local/root "libs/imgscalr-lib-4.3.jar"}},
:mvn/repos
{"central" {:url ""}, "clojars" {:url ""}},
:paths ["resources" "src/main/clojure"]}
#2018-06-0716:39seancorfieldAnd that imgscalr-lib is your own custom in-house version?#2018-06-0716:39rcustodionope, I just got the last (master) from the github repository#2018-06-0716:39rcustodioBecause I’m using something that is not in the latest version from http://maven.org#2018-06-0716:42rcustodioI will try it out with depster#2018-06-0716:42seancorfield@rcustodio can you add -vv after -B on the deps task and send me the output via DM here? No point in clogging up the channel with it.#2018-06-0716:43rcustodiookey#2018-06-0716:43seancorfieldYou can find links to Depstar, Pack, and other tools.deps tooling here: https://github.com/clojure/tools.deps.alpha/wiki/Tools#2018-06-0719:34kennyNot sure if this is a tools-deps problem but if I don't have AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY set as env vars, I get a access denied from a Maven repo. I am specifically trying to access the Datomic cloud repo "" which, I assume, does not require authenticated access.#2018-06-0721:34Alex Miller (Clojure team)Yeah, same is under discussion in #datomic - I’m going to dig into it a little more#2018-06-0803:51cflemingAm I correct in thinking that aliases are not transitive? So if I have a git dep, there’s no way for me to use test code from it?#2018-06-0803:56cflemingIn fact, the same also seems to be true of local deps - is this intentional, or am I missing a way to work around this?#2018-06-0803:59cflemingI’m also confused by the fact that local deps don’t seem to be transitive. I have a test project, and it has a subproject which the root one depends on using :local/root. The subproject has a git dep, but that dep doesn’t appear on the classpath for the root project.#2018-06-0804:35ghadialiases being non transitive is intentional#2018-06-0804:36ghadiAs for the other comment, local deps should be transitive (provided a pom.xml or deps.edn)#2018-06-0902:51cfleming@ghadi Local deps are definitely not transitive at the moment - I haven’t tested git deps.#2018-06-0902:51ghadiBug#2018-06-0903:02cflemingActually, I might be going nuts - I can’t reproduce what I was seeing yesterday.#2018-06-0903:04cflemingI wonder if this was a caching problem:#2018-06-0903:05cfleming~/d/t/test-project (master)> tree .
.
├── deps.edn
└── sub-project
├── deps.edn
└── sub-sub-project
└── deps.edn
2 directories, 3 files
~/d/t/test-project (master)> cat deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}
test/sub-project {:local/root "sub-project"}}}
~/d/t/test-project (master)> cat sub-project/deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "0f06ca5eb08726e74b2dab8654923894b7ace6af"}
test/sub-sub-project {:local/root "sub-sub-project"}}}
~/d/t/test-project (master)> cat sub-project/sub-sub-project/deps.edn
{:deps {}}
~/d/t/test-project (master)> clojure -Spath
Error building classpath. Manifest type not detected when finding deps for test/sub-sub-project in coordinate #:local{:root "sub-sub-project"}
~/d/t/test-project (master)> clojure -Spath -C:test
test:src:/Users/colin/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/Users/colin/.gitlibs/libs/org.clojure/data.json/13e9d244678be7b235bb24a10310f9d147ea088d/src/main/clojure:/Users/colin/dev/tools.deps.alpha/test-project/sub-project/src:/Users/colin/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/Users/colin/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar
#2018-06-0903:05cflemingHere the first classpath with no alias gives an error that the manifest is not supplied.#2018-06-0903:06cflemingThe second doesn’t give that error, but returns an incorrect classpath.#2018-06-0903:06cflemingI add the manifests:#2018-06-0903:06cfleming~/d/t/test-project (master)> tree .
.
├── deps.edn
└── sub-project
├── deps.edn
└── sub-sub-project
└── deps.edn
2 directories, 3 files
~/d/t/test-project (master)> cat deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}
test/sub-project {:local/root "sub-project"
:deps/manifest :deps}}}
~/d/t/test-project (master)> cat sub-project/deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "0f06ca5eb08726e74b2dab8654923894b7ace6af"}
test/sub-sub-project {:local/root "sub-sub-project"
:deps/manifest :deps}}}
~/d/t/test-project (master)> cat sub-project/sub-sub-project/deps.edn
{:deps {}}
#2018-06-0903:07cflemingAnd then both work:#2018-06-0903:07cfleming~/d/t/test-project (master)> clojure -Spath
src:/Users/colin/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/Users/colin/.gitlibs/libs/org.clojure/data.json/13e9d244678be7b235bb24a10310f9d147ea088d/src/main/clojure:/Users/colin/dev/tools.deps.alpha/test-project/sub-project/src:/Users/colin/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/Users/colin/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar:/Users/colin/dev/tools.deps.alpha/test-project/sub-sub-project/src
~/d/t/test-project (master)> clojure -Spath -C:test
test:src:/Users/colin/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/Users/colin/.gitlibs/libs/org.clojure/data.json/13e9d244678be7b235bb24a10310f9d147ea088d/src/main/clojure:/Users/colin/dev/tools.deps.alpha/test-project/sub-project/src:/Users/colin/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/Users/colin/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar:/Users/colin/dev/tools.deps.alpha/test-project/sub-sub-project/src
#2018-06-0903:08cflemingThen if I remove the manifest, I get the error with both:#2018-06-0903:08cfleming~/d/t/test-project (master)> cat deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}
test/sub-project {:local/root "sub-project"}}}
~/d/t/test-project (master)> cat sub-project/deps.edn
{:deps {org.clojure/data.json {:git/url ""
:sha "0f06ca5eb08726e74b2dab8654923894b7ace6af"}
test/sub-sub-project {:local/root "sub-sub-project"}}}
~/d/t/test-project (master)> clojure -Spath
Error building classpath. Manifest type not detected when finding deps for test/sub-sub-project in coordinate #:local{:root "sub-sub-project"}
~/d/t/test-project (master)> clojure -Spath -C:test
Error building classpath. Manifest type not detected when finding deps for test/sub-sub-project in coordinate #:local{:root "sub-sub-project"}
#2018-06-0904:08Alex Miller (Clojure team)too tired to read all that atm but there is a known issue that the caching will not notice when local dep project classpaths change, so you need to -Sforce to recompute. not sure if that’s what you’re seeing. I have a plan to fix that just haven’t had time to work on it yet.#2018-06-0904:09Alex Miller (Clojure team)if you think there’s a bug, please feel free to drop it in a ticket so I can look at it later as this will get lost#2018-06-0919:49gfrederickswhat's the right way to call clojure with a script going to stdin?
it works okay with no args, but prints a bunch of repl prompts I don't want#2018-06-0921:24Alex Miller (Clojure team)We don’t have that level of repl control with clojure.main#2018-06-0921:25Alex Miller (Clojure team)Well, maybe the repls not involved there?#2018-06-0921:27gfredericks$ echo '(println 42)' | clojure
Clojure 1.9.0
user=> 42
nil
user=>
is what I'm referring to#2018-06-0921:28Alex Miller (Clojure team)Isn’t there a - option you can pass?#2018-06-0921:29gfredericksthere is, that's probably all I need#2018-06-0921:42richiardiandreaOh awesome didn't know that either 😃#2018-06-1015:39borkdudeCool, I needed this once too#2018-06-1016:07ghadiI do that all the time. I like datomic ion's take on it too.#2018-06-1016:12ghadiSending in an edn map through an arg or stdin#2018-06-1017:00gfredericksironically I quickly realized that I couldn't do that in this case, because the program also took its input from stdin#2018-06-1017:01gfredericksI guess in theory I could have arranged for the script to be prepended to the input and make sure the last form is one that does the work and then System/exits, but that sounds pretty hacky#2018-06-1017:05borkdude@gfredericks I once asked for an option in here to provide a program with the -e option, without printing expressions.#2018-06-1017:05borkdudeWould that make sense in your case?#2018-06-1017:05gfredericksI could've but it was a ~20 line script so I felt weird putting that in an arg#2018-06-1017:06borkdudeput it in a variable.#2018-06-1017:06dominicmclojure -e <<EOF
(println "ta-da")
EOF
#2018-06-1017:06dominicmheredoc can be pretty useful#2018-06-1017:07dominicm(apparently that doesn't work)#2018-06-1017:07borkdude{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "571538253c33223332173538253c33223332"}, :content ("[email protected]")}#2018-06-1017:08borkdudeBut then again, you could just do clj hello.clj 🙂#2018-06-1017:13gfredericksI'm not a bash lawyer yet, but I got the distinct impression yesterday that <<EOF goes to stdin#2018-06-1017:23dominicmclojure -e "$(cat << 'FOO'
(println "ta-da")
FOO
)"
You're right, I've just got on a journey learning that 🙂#2018-06-1017:24gfredericksI've been learning some bash recently.
Weirdest thing so far is the <> redirection operator.#2018-06-1017:26dominicmwhat does that do?#2018-06-1020:08gfrederickssame as < except it makes stdin writable#2018-06-1112:02delaguardoIs it possible to specify init-opt within deps.edn file?#2018-06-1112:19Alex Miller (Clojure team)No, not right now#2018-06-1117:11bhaumanshould I be surprised that clojure forms in main opts are not output as quoted args?#2018-06-1117:11bhaumanI.E. :main-opts ["-e" (list 1 2 3)]#2018-06-1117:12bhaumanthe list expression gets broken up into separate args#2018-06-1117:13bhaumanI guess escaping gets complicated#2018-06-1117:14bhaumanor there is a depth of features here I don't understand#2018-06-1117:21hiredmanI think the thing you might be missing is :main-opts is meant to be a list of strings passed as command line arguments, not a list of arbitrary datastructures#2018-06-1117:25seancorfieldAlso remember that , is whitespace so you can often get around shell-level tokenization by using , in strings instead of spaces. But, yeah, :main-opts are strings @bhauman.#2018-06-1117:26seancorfieldSee https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L33-L37 for example...#2018-06-1117:26bhaumanthanks guys#2018-06-1117:27bhaumananother question, is this not a feature that we might want?#2018-06-1117:28bhaumanto take advantage of the fact that we are in a clojure form to begin with and script our way to euphoria?#2018-06-1117:29seancorfieldBut :main-opts are what is passed as a sequence of strings to the -main function via clojure.main -- it's all strings. You can easily pass an EDN string if you want a data structure.#2018-06-1117:29seancorfieldThere's no difference between command-line invocation and :main-opts.#2018-06-1117:30seancorfieldand it is passed by a script, not by Clojure code.#2018-06-1117:31bhaumanthe script doesn't parse the deps.edn#2018-06-1117:31seancorfieldThe :main-opts are cached to a text file and then piped into the (string) options passed to clojure.main. Just like the :jvm-opts.#2018-06-1117:31bhaumandoes it?#2018-06-1117:31bhaumanI get this but I'm sure clojure code prepares the opts#2018-06-1117:31bhaumanbefore caching#2018-06-1117:33seancorfieldIt's just strings in the cached file. And it's just strings that are passed to clojure.main.#2018-06-1117:34bhaumanI must be missing something, are you saying that the feature isn't possible?#2018-06-1117:35bhaumanor that it messes up the semantics?#2018-06-1117:35seancorfieldTake a look at the clojure shell script. Here's an example of what gets cached: (! 763)-> ls -l .cpcache/*.main
-rw-r--r-- 1 sean staff 90 Mar 26 15:38 .cpcache/1770942858.main
-rw-r--r-- 1 sean staff 17 Apr 14 00:52 .cpcache/2142975460.main
-rw-r--r-- 1 sean staff 17 Apr 14 18:18 .cpcache/4269137912.main
-rw-r--r-- 1 sean staff 22 May 2 14:38 .cpcache/84686696.main
(sean)-(jobs:0)-(~/clojure)
(! 764)-> cat .cpcache/84686696.main
-m rebel-readline.main
(sean)-(jobs:0)-(~/clojure)
(! 765)-> cat .cpcache/1770942858.main
-e (require,'[clojure.tools.nrepl.server,:refer,[start-server]]),(start-server,:port,5555)
(sean)-(jobs:0)-(~/clojure)
(! 766)->
#2018-06-1117:37bhaumanyes but we call the tools.aplha to resolve the tags#2018-06-1117:38bhaumanwhich is parsing main opts and therefore could quote clojure forms before caching them?#2018-06-1119:07hagmonkI just made a fairly big update to https://github.com/hagmonk/depify/ … would appreciate any folks here who could kick the tires a bit before I post to the mailing list#2018-06-1119:14seancorfieldThis includes adding some test runner boilerplate from clj-new. -- not sure what you mean there @hagmonk?#2018-06-1119:15seancorfieldShould that link to Cognitect's test-runner repo perhaps?#2018-06-1119:17hagmonkOh I pinched that from a clj-new generated app - specifically the separate test and runner aliases #2018-06-1119:18hagmonkI didn’t see the same style being used on the test-runner repo, but maybe I missed it #2018-06-1119:19seancorfieldAh, I see. Linking to my dot-clojure repo probably makes more sense then for that https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L9-L19 (at least by way of explanation).#2018-06-1119:19hagmonkCool! I will make that change#2018-06-1119:22hagmonkI’m exploring the idea of adding more aliases to provide parity with project.clj, in those cases where someone has a tools.deps based tool we can use #2018-06-1119:22hagmonkCLJS and uberjars for instance #2018-06-1119:25hagmonkso now I'm looking at your deps.edn file and getting lots of ideas :) but I need to work on my real projects today at some point, LOL#2018-06-1120:21Alex Miller (Clojure team)@bhauman I didn’t understand your earlier comments re quoting. if it’s something I should pay attention to, please try again :)#2018-06-1120:21bhaumanhappy to 🙂#2018-06-1120:21Alex Miller (Clojure team)the ground truth is that the jvm takes a classpath, a main class and string args to the main class. so we want to expose that, first.#2018-06-1120:22Alex Miller (Clojure team)other things on top of that may also be useful#2018-06-1120:22bhaumanabsolutely#2018-06-1120:22Alex Miller (Clojure team)but I didn’t understand what problem you were trying to solve or thing you were trying to make better or avoid#2018-06-1120:23bhaumanit just seems there is an opportunity when providing clojure forms (not strings) inline in :main-opts to output a single string arg that will make it through to the jvm process as a single arg#2018-06-1120:25Alex Miller (Clojure team)the ultimate call needs individual string args though - how do you decide how to parse and tokenize?#2018-06-1120:28bhauman:main-opts ["-e" {:build 1}] => clojure -e "{:build 1}"#2018-06-1120:28bhaumanis what I'm imagining#2018-06-1120:29Alex Miller (Clojure team)you seem to be suggesting a read/print cycle but using different kinds of printing#2018-06-1120:30Alex Miller (Clojure team)“-e” would be read as a string and pr’ed as “-e” (perhaps escaped in the invocation as clojure -e '"-e"`#2018-06-1120:32bhaumanit seems that string typs can go though untouched and that forms would be dispatched as such#2018-06-1120:32bhaumanother than string forms 🙂#2018-06-1120:32bhaumanits a common need, and it seems like a shame to be in a clojure form and then have to translate and escape a clojrue form correctly#2018-06-1120:33Alex Miller (Clojure team)there are a couple layers of transformation here and I intend to sit down and sort through them methodically. I’m well aware it’s broken in various ways at the moment.#2018-06-1120:33Alex Miller (Clojure team)I’m a little concerned about implying evaluation in the deps.edn file#2018-06-1120:34Alex Miller (Clojure team)the args there are read (as edn) and never evaluated and I don’t want to imply that (or do it)#2018-06-1120:34Alex Miller (Clojure team)it’s important to me that it remains plain edn data#2018-06-1120:35bhaumanabsolutely#2018-06-1120:36bhaumannot what I'm thinking or wanting to imply at all#2018-06-1120:36Alex Miller (Clojure team)I didn’t think so, but it’s easy to get that impression from that example#2018-06-1120:36Alex Miller (Clojure team)I’m not a fan of “special” arg handling#2018-06-1120:36bhaumanwhere I really want it is for cljs.main and figwheel.main to supply inline config#2018-06-1120:37Alex Miller (Clojure team)yeah, that’s really same need as say, server repl#2018-06-1120:37Alex Miller (Clojure team)passing arbitrary edn as an arg#2018-06-1120:38bhauman:main-opts ["-m" "cljs.main" "-co" {:output-to "out/main.js"} ]#2018-06-1120:38bhaumanyeah#2018-06-1120:38Alex Miller (Clojure team)in a way that is not a nightmare, and that works :)#2018-06-1120:38bhaumanwhat works?#2018-06-1120:38Alex Miller (Clojure team)I’m just saying that a requirement is that the end result works#2018-06-1120:38Alex Miller (Clojure team)(as it’s broken now)#2018-06-1120:39bhaumangotcha 🙂#2018-06-1120:40bhaumanit is interesting though that I'd expected this to work#2018-06-1120:40bhaumanI intuited that it might me a likely feature#2018-06-1120:40Alex Miller (Clojure team)there might be a higher level solution to this problem too - I know Rich has mentioned various ideas that are in this ballpark but a somewhat different take on it#2018-06-1120:41Alex Miller (Clojure team)for example, you could expose functions directly and hide the main goo entirely#2018-06-1120:42bhaumanthat is interesting#2018-06-1120:42Alex Miller (Clojure team)it’s come up in several contexts and I know he’s noodling about it#2018-06-1120:42Alex Miller (Clojure team)but sometimes he noodles for many years before it lands somewhere :)#2018-06-1120:43Alex Miller (Clojure team)so I make no suggestion that a deux ex machina will save us all here#2018-06-1120:44Alex Miller (Clojure team)clojure.main itself is both very flexible and frustratingly limited#2018-06-1120:44Alex Miller (Clojure team)like I think it sucks that rebel-readline couldn’t just be hooked directly#2018-06-1120:44bhaumanI had some thoughts about that#2018-06-1120:45bhaumanI could potentially graalvm it, and provide a rlwrap equivalent#2018-06-1120:45Alex Miller (Clojure team)well, I need to roll back to other work, but I would like to hear them somewhere :)#2018-06-1120:46bhaumanthanks Alex! later#2018-06-1120:46Alex Miller (Clojure team)later…#2018-06-1210:54borkdudehmm, native rebel-readline… interesting#2018-06-1220:58hagmonkIs there some escaping trick for supplying edn inside tools.deps aliases?#2018-06-1220:58hagmonk:cljs {:main-opts ["-m" "cljs.main" "--compile-opts" "{:asset-path \"out\" :output-dir \"resources/public/out\"}" "--compile" "access-manager.client" ]}}
#2018-06-1220:58mfikesUse a lot of non-whitespace whitespace (aka ,#2018-06-1220:59bhaumanreplace the whitespace with commas#2018-06-1223:50richiardiandreawouldn't go around telling colleagues about this though, we might be mistaken for a bunch of JS devs 😄 😄 🙂#2018-06-1223:56zentropeWith deps.edn + cljs.main + figwheel.main, the tooling is so simple and streamlined that I just can't see how JS devs put up with that they do.#2018-06-1300:01hagmonkwhere's the cutting edge example for figwheel.main? I was just about to go looking for that#2018-06-1300:02hagmonkI am thinking of adding it to https://github.com/hagmonk/depify, so that existing figwheel projects can be more easily converted over#2018-06-1300:02bhauman@hagmonk working on a template right now{:tag :div, :attrs {:class "message-reaction", :title "star-struck"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-06-1300:03bhaumanfigwheel-main will be a tough conversion though#2018-06-1300:03bhaumanconfig names are different#2018-06-1300:04bhaumanbut actually it should be pretty straightforward for most projects#2018-06-1300:07hagmonkI'm sure there are some leiningen-isms that are not possible to bring across, or considered an anathema in the eyes of deps.edn, but I'd like to bootstrap people to the greatest extent possible#2018-06-1300:07hagmonkI'm already generating Makefiles for my internal projects, so that particular pandora's box is open#2018-06-1300:19bhaumanYeah just created a Makefile myself, I forgot how dang effective they are{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-1300:19hagmonkRight?? Quick and painless … it had been at least 10 years since I wrote one#2018-06-1300:20bhaumanyeah and it can get pretty sophisticated with relatively little syntax#2018-06-1300:20bhaumanits like it was built for this particular job#2018-06-1300:21hagmonkJust remember that literal $ signs need to be $$ to stop make from interpreting them :) Got caught by that one#2018-06-1300:22bhaumanI have noticed something interesting, if you use lein without plugins and only use the profiles and aliases, the performance is pretty similar#2018-06-1300:22bhaumanor significantly improved#2018-06-1300:23hagmonkfascinating … perhaps plugin dependencies are in serial after everything else#2018-06-1300:23bhaumanthey are dynamicly loaded#2018-06-1300:24bhaumanwhich is what causes the hit#2018-06-1300:27hagmonkinherent to changing the classpath dynamically, or something specific with clojure like having to compile?#2018-06-1300:27bhaumanclojure needing to compile#2018-06-1300:30hagmonkIf the jar is in the classpath beforehand, is compilation skipped or just faster because the JVM gets to batch it all up at once?#2018-06-1300:32bhaumanlein is aot compiled#2018-06-1300:33bhaumanso... fast by itself#2018-06-1300:33zentropeIf someone made a super quick class-path-maveny thing in non-Java, make would sing.#2018-06-1301:17Alex Miller (Clojure team)that would be very tedious#2018-06-1301:17Alex Miller (Clojure team)however, you could graalvm the classpath maker#2018-06-1302:59hagmonkhmm, @bhauman - java.lang.RuntimeException: No such var: cljs.repl/*repl-env*#2018-06-1303:00bhauman@hagmonk that would happen if you were somehow getting an earlier version of CLJS than 1.10#2018-06-1303:01bhaumanwe can talk about this in #figwheel-main#2018-06-1303:01hagmonkah, didn't know that existed :)#2018-06-1317:15bhaumantools-deps does a silent failure when an alias doesn't exist#2018-06-1317:17dominicms/silent failure/ignores#2018-06-1317:37Alex Miller (Clojure team)ticket welcome#2018-06-1317:37Alex Miller (Clojure team)I thought I actually had a check for that, but maybe not#2018-06-1317:39dominicmI don't know it's strictly a problem tbh#2018-06-1317:47Alex Miller (Clojure team)seems like it’s more likely to be a typo than cleverness#2018-06-1319:10hagmonk13th law of software development: itches are to be scratched by the one who itches.#2018-06-1319:10Alex Miller (Clojure team)I’ve used the lein stuff for this a bit and it’s SUPER slow#2018-06-1319:11Alex Miller (Clojure team)so slow that I just go use the equivalent web pages instead#2018-06-1319:11Alex Miller (Clojure team)I have no idea why#2018-06-1319:11Alex Miller (Clojure team)but maybe worth understanding#2018-06-1319:12Alex Miller (Clojure team)some next level stuff would be to think about bridging the gap between ns and dep#2018-06-1319:12dominicm@alexmiller what are the important features of the search? I've got a week of open source and JUXT are interested in deps.edn.#2018-06-1319:13Alex Miller (Clojure team)lein search does approximately what I want, I just find it too slow to use#2018-06-1319:13Alex Miller (Clojure team)I want google for deps#2018-06-1319:13dominicmStartup or runtime? It will almost certainly be called find, because 4 letters.#2018-06-1319:13Alex Miller (Clojure team)runtime is the part I’m referring to#2018-06-1319:14dominicmBut mavens is fast?#2018-06-1319:14Alex Miller (Clojure team)I think there might be some giant index file that gets downloaded periodically?#2018-06-1319:14Alex Miller (Clojure team)I don’t remember what maven has for this#2018-06-1319:14Alex Miller (Clojure team)maybe I’m wrong that they do#2018-06-1319:15Alex Miller (Clojure team)I just tried lein search again and it was pretty fast#2018-06-1319:15Alex Miller (Clojure team)been a while#2018-06-1319:16Alex Miller (Clojure team)how about this - I’d like to do lein search functionality without being tied to lein#2018-06-1319:16dominicmOr tools.deps.alpha!#2018-06-1319:16dominicmJust a main#2018-06-1319:17Alex Miller (Clojure team)no reason it needs to be there#2018-06-1319:18dominicmI agree.#2018-06-1319:18Alex Miller (Clojure team)I don’t think maven actually has apis for this - it’s site-specific functionality iirc#2018-06-1319:18Alex Miller (Clojure team)(which also means they are different :)#2018-06-1319:18dominicmClojars has an index which mentions nexus.#2018-06-1319:19Alex Miller (Clojure team)well, maybe they’re the same then, not sure#2018-06-1319:19dominicmMaybe. No idea. Just a data point.#2018-06-1319:19Alex Miller (Clojure team)the next level and far more interesting problem is how to connect namespaces and artifacts#2018-06-1319:20dominicmI'm super interested in this.#2018-06-1319:20dominicmSolving this makes writing plugins easier.#2018-06-1319:20dominicmBoth for different kinds of artefact, and for build tools.#2018-06-1319:20Alex Miller (Clojure team)Rich and I have talked about it a lot but it’s a place where Maven doesn’t really help you much#2018-06-1319:23dominicmWhat would a straw man solution look like? I'm struggling to picture something.#2018-06-1319:30Alex Miller (Clojure team)I don’t have one :)#2018-06-1319:30Alex Miller (Clojure team)sorry, other things I need to focus on atm#2018-06-1322:32danielcomptonLein now uses the Clojars api iirc. It used to download some kind of massive index which was very slow#2018-06-1400:03Alex Miller (Clojure team)Glad I’m not imagining that, and that it doesn’t now! It did seem plenty fast#2018-06-1323:20bhauman@seancorfield trying to use clj-new with a locally installed lein template and not getting very far#2018-06-1323:32seancorfield@bhauman Can you be a bit more specific?#2018-06-1323:32seancorfield"locally installed"?#2018-06-1323:33bhaumanI did a lein install on a template I'm working on locally#2018-06-1323:34bhaumanshould it be able to resolve it?#2018-06-1323:34bhaumanas it dies in clj_new.helpers$resolve_remote_template.invokeStatic(helpers.clj:64)#2018-06-1323:34seancorfieldHmm, my normal process for testing templates is to install them locally and then try to use them via new...#2018-06-1323:35bhaumanhere is the message#2018-06-1323:35bhaumanException in thread "main" clojure.lang.ExceptionInfo: Unable to locate Clojure's edn files {:exit 1, :out "Invalid option: -Sdescribe\n", :err ""}#2018-06-1323:35seancorfieldAh, that's where it shells out (via tools.deps.alpha) to run clojure -Sdescribe...#2018-06-1323:36seancorfieldSo the error is coming from tools.deps.alpha not being able to run that command.#2018-06-1323:36bhaumanand so there needs to be a deps.edn in the current dir?#2018-06-1323:36seancorfieldIf you type clojure -Sdescribe in the folder where you are running clj-new what do you get?#2018-06-1323:36seancorfieldNo, this isn't about deps.edn#2018-06-1323:37bhaumaninvalid option#2018-06-1323:37bhaumanI have to update#2018-06-1323:37seancorfieldHow old is your version of clojure?#2018-06-1323:37seancorfieldHere's what I see in a folder without deps.edn: (! 665)-> clojure -Sdescribe
{:version "1.9.0.375"
:config-files ["/usr/local/Cellar/clojure/1.9.0.375/deps.edn" "/Users/sean/.clojure/deps.edn" ]
:install-dir "/usr/local/Cellar/clojure/1.9.0.375"
:config-dir "/Users/sean/.clojure"
:cache-dir "/Users/sean/.clojure/.cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
#2018-06-1323:38seancorfieldI thought -Sdescribe had been in there for ages... 👀#2018-06-1323:38bhaumanor not#2018-06-1323:39bhaumanyeah its not in my help string#2018-06-1323:40bhaumanso upgrading#2018-06-1323:41bhauman@seancorfield and you are enforcing that project names are multisegment and qualified?#2018-06-1323:41bhaumanshouldn't that be up to the template?#2018-06-1323:42seancorfieldBy popular demand, I am disallowing single-segment project names, yes. You can either use a multi-segment name or a qualified name (or both).#2018-06-1323:42seancorfieldSome templates don't accept a qualified name -- which I consider a bug in those templates 🙂#2018-06-1323:42bhauman@seancorfield it worked 🙂 but I'd like to be able to specify a single segement name like lein allows#2018-06-1323:43seancorfieldIf they don't accept a qualified name, they should accept a multi-segment name.#2018-06-1323:43bhaumanthe lein app template morphs a single segment name into a name.core#2018-06-1323:44seancorfieldYes, which is a terrible idea.#2018-06-1323:44seancorfieldEven Phil says so.#2018-06-1323:44seancorfieldYou can always do this: (! 666)-> clj -A:new app bhauman.core
Generating a project called bhauman.core based on the 'app' template.
Wed Jun 13 16:43:52
(sean)-(jobs:0)-(~/clojure/scratch)
(! 667)-> tree bhauman.core/
|____
| |____.gitignore
| |____.hgignore
| |____CHANGELOG.md
| |____deps.edn
| |____doc
| | |____intro.md
| |____LICENSE
| |____README.md
| |____resources
| | |____.keep
| |____src
| | |____bhauman
| | | |____core.clj
| |____test
| | |____bhauman
| | | |____core_test.clj
#2018-06-1323:46gfredericksis (! 666)-> your bash prompt#2018-06-1323:47gfrederickser; I guess the last of three lines of it#2018-06-1323:47seancorfieldAh, yes, my crazy bash prompt... just a sec, I'll get the code for that...#2018-06-1323:48seancorfield#
PS1="\n\d \t\n\[\e[32;1m\](\[\e[37;1m\]\u\[\e[32;1m\])-(\[\e[37;1m\]jobs:\j\[\e[32;1m\])-(\[\e[37;1m\]\w\[\e[32;1m\])\n(\[\[\e[37;1m\]! \!\[\e[32;1m\])-> \[\e[0m\]"
#2018-06-1323:48bhauman@seancorfield OK well I don't see the disturbing problem with using a single segment when you are developing a webapp#2018-06-1323:48bhaumanthats going to be deployed to heroky#2018-06-1323:48bhaumanI should call it heroky from now on#2018-06-1323:49seancorfieldThen use <whatever>.core as your project name and you'll get what Leiningen does.#2018-06-1323:49bhaumanthanks I understand that, was just wanting to keep things familiar for folks who are used to doing lein new figwheel hello-world#2018-06-1323:50bhaumanbut not a big deal#2018-06-1323:50bhaumanclj -Anew figwheel-main hello.core works#2018-06-1323:53seancorfieldhttps://github.com/bbatsov/clojure-style-guide/pull/100#2018-06-1323:53seancorfieldSee comments about ClojureScript in that thread, as well as Alex's comment.#2018-06-1323:54bhaumanyou do realize I'm not advocating single segment namespaces?#2018-06-1323:54seancorfieldYou seemed to be when you complained that clj-new disallowed them 🙂#2018-06-1323:55bhaumana single segment argument atumatically gets a "core" segment appended#2018-06-1323:55seancorfieldI haven't -- yet -- disallowed them in boot-new but I may well propose that.#2018-06-1323:56bhaumansingle segments don't work I get that#2018-06-1323:56seancorfieldYes, and having thousands of projects with spurious .core namespaces, just because we were all too lazy to teach beginners to name things properly, is not a great situation to be in.#2018-06-1323:59bhaumanOK so I'm not asking for single-segment namespaces, and defaulting to core is a bad idea for an initial template#2018-06-1400:00bhaumanI'm also betting that defaulting to "app" rather than core doesn't work for you either#2018-06-1400:02seancorfieldPeople should be explicit about their project namespace.#2018-06-1400:05bhaumanand my point of view is that the name in the template can be more general than a project name, and it can be helpful to let folks who write templates do what they want.#2018-06-1400:06bhaumanthe clj generators come in handy there#2018-06-1403:11hagmonkJust a heads up, I pushed a new https://github.com/hagmonk/depify that now includes the ability to read project.clj from stdin, as well as interpreting output from lein pprint#2018-06-1420:57hagmonkHey so riddle me this ... has any thought been given to using deps.edn in the context of a monorepo? #2018-06-1420:58hagmonkI’m not sure it’s currently possible to supply a path to a directory inside a larger git repo - perhaps I just missed that feature? #2018-06-1421:04seancorfieldWe're working with a monorepo at work and slowly figuring that out. Not a priority tho' so progress is a bit slow. #2018-06-1421:06Alex Miller (Clojure team)as a git dep or a local dep?#2018-06-1421:07Alex Miller (Clojure team)there are some ways with :deps/root etc to do some of this but also some known issues#2018-06-1421:10seancorfieldI'm working with local deps so far. #2018-06-1421:14hagmonkI guess in the context of the monorepo itself, you would use :deps/root to point at other resources in the monorepo ... but clients outside the monorepo have no way to point at a subfolder’s deps.edn #2018-06-1421:15hagmonkA real world example would be figwheel-main’s deps.edn: https://github.com/bhauman/lein-figwheel/blob/master/figwheel-main/deps.edn#2018-06-1421:15hagmonkAFAICT, there’s no way I can depend on that deps.edn file, because it’s not at the root of the repo #2018-06-1421:28kenny@hagmonk Setting :deps/root to "figwheel-main" doesn't work?#2018-06-1421:28hagmonkOh, perhaps that is the thing I’m missing! #2018-06-1421:29kenny{:git/url ""
:sha "6368c0e49108fc7a8ec99df05831ff289123f1a5"
:deps/root "figwheel-main"}
🙂{:tag :div, :attrs {:class "message-reaction", :title "heart_eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😍")} " 4")}
#2018-06-1421:29hagmonkThanks @kenny! Had no idea that existed#2018-06-1421:30kennyYeah I had the same problem. I don't think it's documented yet.#2018-06-1422:23Alex Miller (Clojure team)it’s not#2018-06-1422:24bhaumangood to know!#2018-06-1515:38richiardiandrea@michal interesting work you have shared there thank you!#2018-06-1515:46michal@richiardiandrea you're welcome. this is something I had on my mind since I saw JUXT's edge project first time. I hope it will be useful at least for some people here 🙂#2018-06-1601:16hagmonk@richiardiandrea @michal what are you guys referring to? I think it might have scrolled out of the channel history …#2018-06-1601:24richiardiandrea@hagmonk this -> https://github.com/mbuczko/revolt#2018-06-1601:26hagmonkThat's cool!#2018-06-1601:27hagmonkhow did I not know about Capsule before this …#2018-06-1604:06dominicmIt took me so long to find#2018-06-1604:06dominicmAnd I was really looking#2018-06-1700:31hagmonkAlright @alexmiller, I've almost finished that "google for deps without lein" tool you were wanting :) Just putting the finishing touches on the readme …#2018-06-1701:08hagmonkAlright, would love if folks here could kick the tires on this and tell me what you think: https://github.com/hagmonk/find-deps#2018-06-1701:09hagmonkI think this is one tool I actually started using while I was in the middle of writing it. Didn't realize how badly I wanted it :) No more going to google to find dep coords#2018-06-1702:30mfikes^ Very nice 🙂{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 12")}
#2018-06-1721:09hagmonkOk, who here knows something about stdin/stdout flushing? I'm adding the ability to pipe find-deps commands together, and having a devil of a time getting that to work#2018-06-1721:38hagmonkfor posterity:
1) shopt -u lastpipe is evil, and I had to disable that (`shopt -s lastpipe`)
2) (print "") (flush) at the top of -main will prime the stdout stream such that other processes connected via pipes can test for this condition with (pos? (.available System/in))#2018-06-1722:44hagmonkLatest find-deps was just pushed, it now has the ability to write updates to your deps.edn file. I posted an announcement to the Clojure list too#2018-06-1722:45hagmonkE.g. adding metosin/jsonista is now as easy as: clj -A:find-deps -F:save jsonista#2018-06-1722:45hagmonkI haven't really found any cases yet where the fuzzy matching has picked totally the wrong lib - that's probably the number one thing you can bring to my attention if you see it fail#2018-06-1801:41cfleming@hagmonk That looks lovely!{:tag :div, :attrs {:class "message-reaction", :title "smile"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😄")} " 4")}
#2018-06-1801:42cflemingSearching is by artifact/group name, right? AFAIK you can’t use the Maven APIs to search for namespace names?#2018-06-1801:43hagmonkyes, just the group/artifact string#2018-06-1801:44cflemingWe really need something like Maven’s search by classname for namespaces and vars, but the indexing required for that is non-trivial.#2018-06-1801:45hagmonkright, I did some poking around the crossclj sources, seems like the fellow maintaining that has taken a step back from it#2018-06-1801:45cflemingYes, he has#2018-06-1801:46cflemingI’m not sure he maintained it much past the initial implementation, but I might be wrong there.#2018-06-1801:46hagmonkIt would be great to have a CLJ/CLJS source code index that also tracked the source repo and any built artifacts spun out of that#2018-06-1801:47cflemingI have a question about deps support for Cursive. As I envision it, each deps project (i.e. a deps.edn, basically) would be a module in IntelliJ, so modules will be created for local deps etc, and libraries created for git deps.#2018-06-1801:48cflemingMy problem is that modules have to be given a name, and it has to be something that I can uniquely determine from the deps.edn file, which AFAIK doesn’t provide anything like that.#2018-06-1801:48cflemingThe only thing that I can think of is to use the name of the containing directory. Does that seem like a reasonable convention?#2018-06-1803:12Alex Miller (Clojure team)We have talked about this a lot in the context of git deps. I don’t have time to write it all now will try to hit this tomorrow#2018-06-1801:49cflemingI could also allow a name to be provided in a :cursive/name my-module entry in the deps map, assuming that sort of thing is allowed.#2018-06-1801:50cflemingOh, and one restriction is that modules have to have unique names.#2018-06-1801:51hagmonkI think the general sentiment has been to keep deps.edn uncluttered with anything not related to building a classpath or running the program. Whether unrelated keys are preserved seems to be by chance rather than design#2018-06-1801:51cflemingOk#2018-06-1801:52hagmonkbut you just need a name for the module, it doesn't need to be the same as, say, the official maven artifact name#2018-06-1801:52cflemingHowever, I doubt Cursive is the only system that would find it useful to assign a name to a project.#2018-06-1801:52cflemingTrue, but there’s literally nothing there to base it off right now.#2018-06-1801:52cflemingI mean, I could use the sha of the path to the project, but no-one is likely to be happy with that 🙂#2018-06-1801:53hagmonkhaha, what? who here doesn't love a full length SHA? :)#2018-06-1801:54cflemingHey, it’s (almost) guaranteed unique!#2018-06-1801:55hagmonkI'm just looking through the tools on https://github.com/clojure/tools.deps.alpha/wiki/Tools (which I have to update with find-deps and mbuczko/revolt) to see what conventions exist for project name, in those cases where one is needed#2018-06-1801:56cflemingAs I understand it, if there are two projects which both refer to a common library using a local ref, there’s no requirement even for the name they assign to it to be unique.#2018-06-1801:56cflemingi.e. project1: foo/bar {:local/root "../lib"} and project2: baz/bup {:local/root "../lib"}#2018-06-1801:57cflemingIn fact, I have no idea where those names are used, if anywhere#2018-06-1801:58hagmonkI suppose the resolved path to the dep is what actually matters there, rather than the symbol. I'm not aware of any use for it either#2018-06-1801:58hagmonkI guess one could also look at any git metadata in the project folder - perhaps prefer that, then fall back to directory name#2018-06-1801:59cflemingI guess I could use the path from the main project root as the module name, but that will get weird if the local root is outside the main project.#2018-06-1801:59cflemingI don’t know if IntelliJ has restrictions on what modules can be called, actually.#2018-06-1802:00cflemingThe problem with git metadata is that Cursive then needs git capabilities as well as maven capabilities#2018-06-1802:00hagmonkplus it won't always exist#2018-06-1802:00cflemingRight.#2018-06-1802:01hagmonkperhaps the basename of the resolved path of the dep is good enough for most cases? If there is a name clash it might be worth foisting that on the user anyhow#2018-06-1802:01cflemingThat’s basically the name of the containing directory, right?#2018-06-1802:02hagmonkRight - or even "basename-1/basename" ... most of my projects would be sensibly named with that convention#2018-06-1802:02cflemingI guess I’m worried about cases like server/api/deps.edn and client/api/deps.edn#2018-06-1802:02cflemingYeah#2018-06-1802:02cflemingI’ll see if I can find out whether there are restrictions on module names.#2018-06-1802:03cflemingI think I’ll just have to implement things until people stop complaining.{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 4")}
#2018-06-1802:04hagmonkThat is the zen of software development right there!#2018-06-1802:04cflemingThe problem with foisting it on the user is that there’s no way to store a customisation for the name within the IntelliJ project - I have to return a project structure based on a list of deps.edn files, no more.#2018-06-1802:04cflemingIndeed - I think of it as a dynamic search algorithm.#2018-06-1802:05cfleming@alexmiller I’m interested if there’s an official position on keys like :cursive/name my-module in deps files.#2018-06-1803:01seancorfield@cfleming I don't know how "official" it is but whenever the subject has come up, Alex has always discouraged people adding "random" keys to the file...#2018-06-1803:05hagmonkAlso anecdotal, but listening to Rich on the cognitect podcast (when will they fix the audio?) I heard him say how nice it was to look at a deps.edn file and see “only deps” in there … #2018-06-1803:05cflemingI guess I can always have a cursive.properties file alongside the deps.edn, but that seems way uglier to me.#2018-06-1803:05cflemingWell, except it’s not only deps.#2018-06-1803:05cflemingI mean, there are tasks with aliases etc etc as well.#2018-06-1803:05cflemingAnyway, I’ll wait and see what Alex says.#2018-06-1803:07hagmonkThat was my thought too, I think it’s more accurate to say it contains “keys required to build a classpath and invoke the JVM with that classpath”#2018-06-1803:11seancorfieldThere's an argument that Leiningen's project.clj sets a precedent for allowing additional tool-specific keys...#2018-06-1803:15hagmonkI think tools.deps is a deliberate move away from that - where is the sweet spot though? Should the canonical name for the project be something in the deps file? It’s not really a tool-specific thing at all. Ironically, the deps file is a list of canonical names for other projects, or at least the names they go by in their repos #2018-06-1803:21seancorfieldI would say that deps.edn goes a long way toward removing the importance of names -- you can give arbitrary names to local and Git deps. The names of Maven deps really only exist to match up with group/artifact names that are published. When you depend on, say, a Gist, or some code in a random directory, the name is unimportant and synthetic for the most part...#2018-06-1803:22cflemingSadly, my users are not going to want shas as their module names.#2018-06-1803:25cflemingThey’re fine in a gist link, but not for a descriptive name for someone who wants to get an overview of their project.#2018-06-1803:33hagmonkThese are the kind of highfalutin customers @cfleming's software attracts :)#2018-06-1803:36cflemingThey’re so fussy.#2018-06-1803:39hagmonkFor some reason I'm reminded of Ghostbusters: "I've worked in the private sector … they expect results"#2018-06-1803:41cflemingIt does sometimes help to imagine myself as Bill Murray when I have to do this stuff.#2018-06-1904:26cflemingIs there a list of likely locations I can try to look for the clojure command, so I can default it for the user if it exists?#2018-06-1904:26cflemingCurrently my list only contains /usr/local/bin/clojure from OSX.#2018-06-1904:27cflemingWhere is it on linux or WSL?#2018-06-1904:31richiardiandrea@cfleming not sure about windows but I guess on Linux it really depends, I actually added a prefix option to make install location customizable. Usually you use something like which or env depending on the use case. This answer might help you: https://unix.stackexchange.com/questions/12736/how-does-usr-bin-env-know-which-program-to-use#12749#2018-06-1904:31cfleming@richiardiandrea Thanks. The problem is that when I execute from within IntelliJ, the user’s path is generally not set correctly.#2018-06-1904:32cflemingSo the user will have to configure it, but I was hoping for a list of likely absolute paths I could check.#2018-06-1904:33richiardiandreaThe one you mentioned is most likely one. I use $HOME/.local/bin/clojure#2018-06-1904:33richiardiandreaSome distro might install in /opt#2018-06-1904:34richiardiandreaSome directly in /usr/bin maybe? I guess these are the most common if I were to answer that#2018-06-1904:42cflemingHow many installers for linux distros are out there at the moment?#2018-06-1905:17seancorfield@cfleming Why is the user's path not set for IntelliJ? (genuinely curious -- I'm not used to programs running without the standard user shell environment being available... I think?)#2018-06-1906:43dottedmag@seancorfield I have stumbled upon it on Mac: Dock and Finder are launched directly by user's launchd, which is launched directly by system launchd, so no shell is executed or profile file is read, so their environment is pretty bare.#2018-06-1906:44dottedmag@cfleming /opt/local/bin/clojure for MacPorts#2018-06-1906:46dottedmag@cfleming /usr/bin/clojure on Arch Linux (and I suppose in every other distro that bothered to package Clojure)#2018-06-1906:48dottedmag@cfleming /usr/local/bin/clojure for Linux installer from http://clojure.org#2018-06-1907:45cfleming@dottedmag Thanks!#2018-06-1907:46dominicmvoid places it in /usr/bin/clojure too#2018-06-1909:27cfleming@dominicm Is that on linux?#2018-06-1909:39dominicm@cfleming yes#2018-06-1911:38mfikes@cfleming FWIW, lein-tools-deps went with a mixture of overridable defaults along with searching https://github.com/RickMoynihan/lein-tools-deps/blob/master/src/lein_tools_deps/env.clj#L14-L22#2018-06-1911:40mfikesGiven the above, it would ultimately end up having defaults like this on non-Windows platforms
["/usr/local/bin/clojure" "/usr/bin/clojure" "/opt/local/bin/clojure"]
#2018-06-1912:12rickmoynihanYeah I’d much rather of not relied on the executable at all (and just used the library); but then you don’t know where the find the :install deps.edn file#2018-06-1912:43Alex Miller (Clojure team)I’m considering moving that into tools.deps#2018-06-1916:43rickmoynihanso it would be on the classpath?#2018-06-1916:45dominicmThe risk with that I guess is that organisations cannot specify repos globally. Maybe the root directory should go into /etc/clojure.conf or something. That's what apache, nginx, etc. do#2018-06-1917:02Alex Miller (Clojure team)it would be on the classpath or perhaps it would be automatically included such that it isn’t even a distinct deps.edn. still tbd.#2018-06-1917:03Alex Miller (Clojure team)@dominicm I think that would be the right solution to that problem, if we wanted to do something about it (which I do not at this time)#2018-06-1923:02cfleming@rickmoynihan Yes, I would much prefer that too - then the user could just specify a tools.deps version.#2018-06-1923:02cfleming@alexmiller Moving the install file into tools.deps would be great.#2018-06-1923:03cflemingOne possible source of confusion with that is that the versioning is different for tools.deps and the cli stuff.#2018-06-1923:20cfleming@mfikes Thanks - I looked through the lein-tools-deps issues when researching this but hadn’t found the code.#2018-06-2016:09gfredericksI just encountered my first instance of a clojure library only being available via github#2018-06-2016:11nwjsmith?#2018-06-2016:14gfredericksi.e., it's not on clojars#2018-06-2016:14gfredericksyou are expected to use it by specifying the github repo and a sha#2018-06-2016:53seancorfieldThe Cognitect Labs test-runner is like that -- but I assume you mean "aside from clj-specific tooling"?#2018-06-2016:54seancorfield(also my clj-new is not on Clojars but, again, clj-specific tooling)#2018-06-2018:32kennyCan I specify a username and password for a Maven repo?#2018-06-2018:57seancorfield@kenny In settings.xml I believe...#2018-06-2018:58kennyNot in a deps.edn?#2018-06-2018:58seancorfieldYou certainly don't want your Maven creds in your project's deps file under version control.#2018-06-2018:58kennyVia environment variables.#2018-06-2018:59seancorfielddeps.edn is static EDN, no variables.#2018-06-2018:59kennySo there is no way to use private Maven repositories? Or does tools-deps read in settings.xml?#2018-06-2019:01seancorfield@kenny Discussed yesterday in #clojure https://clojurians.slack.com/archives/C03S1KBA2/p1529458093000094#2018-06-2019:01seancorfieldI haven't tried it myself (and haven't seen specific documentation about it).#2018-06-2019:13seancorfield@kenny I haven't read through all of the details but this is the JIRA issue that covers this https://dev.clojure.org/jira/browse/TDEPS-9#2018-06-2019:14seancorfieldLooks like it relies on Maven's standard settings.xml file according to the patch in that ticket.#2018-06-2019:15kennyGot it, thanks!#2018-06-2019:54Alex Miller (Clojure team)yes, that is all correct#2018-06-2019:54seancorfield(it was a TIL moment for me so thank you for asking @kenny!)#2018-06-2019:54Alex Miller (Clojure team)For Maven deps in authenticated repositories, existing Maven infrastructure is used to convey credentials via the ~/.m2/settings.xml:
<servers>
…
<server>
<id>my-auth-repo</id>
<username>zango</username>
<password>123</password>
</server>
…
</servers>
Then in your deps.edn include a repo with a name matching the <id>:
{:deps
{authenticated/dep {:mvn/version "1.2.3"}}
:mvn/repos
{"my-auth-repo" {:url ""}}}
{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-06-2020:04Alex Miller (Clojure team)I added this info to docs: https://clojure.org/reference/deps_and_cli#_maven_authenticated_repos#2018-06-2020:28seancorfieldThanks. I'd searched for settings.xml earlier and couldn't find anything in the guides or reference.#2018-06-2020:58Alex Miller (Clojure team)I forgot to add it#2018-06-2100:44kennyThanks!#2018-06-2019:55hagmonkis it not possible to specify relative paths in :local/root?#2018-06-2019:56Alex Miller (Clojure team)it is possible#2018-06-2019:56hagmonkI'm trying to use lein-tools-deps to point to two different deps.edn, one of which points at the other via relative path#2018-06-2019:56hagmonkcould be a lein-tools-deps bug …#2018-06-2019:57Alex Miller (Clojure team)there are some known issues with local to local and relative paths#2018-06-2019:57Alex Miller (Clojure team)particularly around :paths#2018-06-2019:58cfleming@alexmiller Just a gentle reminder about the module naming issue, you said you had had discussions about this in the context of git deps.#2018-06-2020:01Alex Miller (Clojure team)yeah, haven’t had time, and won’t today. but I haven’t completely forgotten#2018-06-2020:01cflemingOk, thanks#2018-06-2020:02hagmonkchanging :local/root to be an absolute path resolved things for lein-tools-deps. Hum.#2018-06-2020:09hagmonkOh cute, I got it working#2018-06-2020:10hagmonkSolution is to not specify the deps.edn files directly in project.clj, just leave it as :project. Then, make an uber deps.edn that points at the deps.edn files. Things resolve as expected#2018-06-2122:28rickmoynihanGlad you got this working; if you think it’s a bug though please file an issue with lein-tools-deps
As Alex says, there’s some open bugs on tools.deps itself to do with paths, so could be that; but could also be due to us resolving paths relative to :project… would need to have a test case to recreate#2018-06-2115:21timgilbertSay, is there a tools.deps equivalent of lein deps, meaning "download everything you need to so I can cache it on my CI server"?#2018-06-2115:26bronsaclj -Sforce < /dev/null should do it{:tag :div, :attrs {:class "message-reaction", :title "pray"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙏")} " 8")}
#2018-06-2115:29sparkofreasonI'm running most of my dev CLI stuff under WSL, but Cursive runs under Windows. So a tool like lein-tools-deps won't work correctly in the Cursive context. Anybody have any luck working around this?#2018-06-2115:44richiardiandreaOh. I hoped that would work for a couple of colleagues of mine. I have exactly the same setup for macos users#2018-06-2116:02hagmonkI think lein-tools-deps works by invoking the clj command? If so you’d need to wait for Windows port. #2018-06-2116:06sparkofreasonI can get clj to execute under WSL using a batch script, but not sure that will be sufficient.#2018-06-2116:15seancorfield@U06HHF230 would have more insight but I would have expected lein-tools-deps to work on Windows if you explicitly specify which deps.edn files you need to use in project.clj so it didn't have to rely on tools.deps.alpha's ability to shell out to the clojure command?#2018-06-2116:43richiardiandreaI think lein-tools-deps calls clojure under the hood, but I saw that you can configure the location of it as well.#2018-06-2117:13seancorfieldWell, if it asks tools.deps.alpha for the (clojure-env), the shell out to clojure happens there (inside tools.deps.alpha itself). When I switch development machines (fairly soon, I suspect), I'll also be dealing with this since I use Atom/ProtoREPL on the host O/S. Right now, I'm on a Mac so the host for Atom is the same as the host for lein/`boot`/`clj` but I've already run into this on (Windows) laptop and my workaround was to start a REPL under WSL and then connect to it from Atom -- but there are still gotchas like file paths differing between Windows and WSL: when you "load file", ProtoREPL assumes that the REPL can be asked to load the file using the editor host path which fails unless you've been very careful setting up symlinks in WSL etc.#2018-06-2122:45rickmoynihanHi, sorry I don’t use Windows or have access to a machine I can test this setup on. Am I right that the problem is that you’re effectively running clojure in a (pseudo) linux environment, but Cursive in Windows; so the paths are resolving incorrectly?
@U04V70XH6 is right that we call clojure-env and rely on tools.deps shelling out to clojure to get the env for us. Unfortunately it’s a slightly tricky problem just now and the current design is a bit brittle.
I’d be open to disabling the call to clojure-env; though if we do that you’d also need to set the paths to your :install deps.edn and user one by hand.
If this issue is effecting you though, please open an issue where we can discuss more.#2018-06-2200:53cflemingI was actually planning to ask about this. It is possible to access WSL from Windows? If I try to open /usr/local/bin/clojure or whatever from Windows, does that work or not?#2018-06-2200:54cflemingMy currently plan for Cursive is to bypass the CLI stuff altogether, and just use the library. There are some outstanding questions about how to do that, such as how to get the system deps.edn - for an initial beta I’m going to bundle it with Cursive, but I’m hoping that it will eventually be available in the lib - Alex mentioned he was thinking about that.#2018-06-2200:55cflemingAssuming I just use the lib, then it should all work on Windows.#2018-06-2200:56cflemingAnother complication is that the user will have to choose the version of t.d.a to use, and those versions don’t correspond to CLI versions, so there’s some potential for confusion.#2018-06-2200:57cflemingAnd I guess if people are using the CLI in WSL and Cursive can’t access that, then that will also be confusing since you’ll end up with two parallel systems.#2018-06-2201:00cflemingThis is old, but it has big red letters: https://blogs.msdn.microsoft.com/commandline/2016/11/17/do-not-change-linux-files-using-windows-apps-and-tools/#2018-06-2201:01cflemingThe red letters are:
> DO NOT, under ANY circumstances, create and/or modify Linux files using Windows apps, tools, scripts, consoles, etc.
And some following large black ones:
> Creating/changing Linux files from Windows will likely result in data corruption and/or damage your Linux environment requiring you to uninstall & reinstall your distro!#2018-06-2201:02cflemingSo, I read that to mean that using the CLI from WSL will be totally inaccessible to Cursive anyway.#2018-06-2201:02cfleming@U066LQXPZ ^^#2018-06-2201:10sparkofreasonI think that guidance is somewhat out of date, and the interop between WSL and Windows file systems is much improved. I actually keep all the clojure project stuff on the Windows side anyway, but and do CLI stuff from WSL. #2018-06-2201:13cflemingRight, but I think that WSL is aware of Windows but not the other way around.#2018-06-2201:14cflemingAlthough I haven’t really followed it recently.#2018-06-2201:14sparkofreasonIt is possible to execute clojure under WSL from Windows, bash -c "clojure". I haven't been able to make that work with lein-tools-deps yet, was planning on making a fork and investigating locally. #2018-06-2201:15cflemingI’d be selfishly interested to know if you could execute it via ProcessBuilder from a Java app running on Windows.#2018-06-2201:16cflemingIf that is possible, then it should be possible to access it from Cursive. I’m still probably going to go lib-only unless there’s a really good reason not to because it’s much simpler, but it would be interesting to know.#2018-06-2201:18cfleming@U066LQXPZ BTW that guidance is still in the WSL FAQ: https://docs.microsoft.com/en-us/windows/wsl/faq
> One of the main limitations of using WSL is that changing Linux files with a Windows app or tool is not allowed.#2018-06-2201:22cflemingComment here: https://superuser.com/a/1111092
> Every file that resides on a Volfs folder (like /home) has Extended Attributes that store the Linux permissions of that file. If you edit that file on a windows editor then those attributes are lost and file will disappear from bash.#2018-06-2201:28cflemingLooks like CLion works around this by running development over ssh: https://blog.jetbrains.com/clion/2018/01/clion-and-linux-toolchain-on-windows-are-now-friends/#2018-06-2201:31cfleming@U066LQXPZ If you’re running your dev in WSL, doesn’t that mean that your source files are also under WSL? How does IntelliJ access them? Or do you treat WSL as a server you deploy to?#2018-06-2201:41sparkofreasonAll of the files are in the windows FS. The tooling (apart from the IDE) executes under WSL, and can work with the windows FS no problem. #2018-06-2201:41cflemingOh right, of course. So your Maven local repo etc is all under Windows? That makes sense.#2018-06-2201:42cflemingSo I guess the only issue would be accessing the installed deps.edn files (system and user). Do you use the CLJ_CONFIG env var to keep the user one under Windows too?#2018-06-2201:51sparkofreasonNo, that is on the todo list. #2018-06-2201:52sparkofreasonIt's how I have both lein and boot set up. #2018-06-2202:19sparkofreason@U0567Q30W (clojure.java.shell/sh "clojure.bat" "-Sdescribe") works. I assume ProcessBuilder would as well.#2018-06-2202:20cfleming@U066LQXPZ Ok, interesting, thanks.#2018-06-2202:21cflemingI guess the hard thing is that the paths that -Sdescribe will return are all in linux-land. So even if you configure CLJ_CONFIG to /mnt/c/Users/dave/.clojure or something, that path is still invalid in Windows.#2018-06-2202:25sparkofreasonRight, you'd definitely want to move it to be under windows. You can configure WSL to behave as though Windows drives are mounted under root, so you'd get /c/Users/dave/.clojure, but obviously that still can't be used as-is.#2018-06-2202:33cflemingI think it would be better to leave them under /mnt, since Cursive has to be aware it’s talking to WSL anyway.#2018-06-2202:33cflemingIt would make it easier to detect that it’s getting a Windows path back from Linux#2018-06-2203:17sparkofreasonYeah, but other things like docker benefit from this (so they tell me, I haven't run into this myself). Maybe some sort of config option to indicate WSL? #2018-06-2203:37seancorfieldMicrosoft Windows [Version 10.0.17692.1000]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\Users\seanc>wsl.exe clojure
Clojure 1.9.0
user=>
C:\Users\seanc>wsl.exe clojure -Sdescribe
{:version "1.9.0.381"
:config-files ["/usr/local/lib/clojure/deps.edn" "/home/seanc/.clojure/deps.edn" ]
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/seanc/.clojure"
:cache-dir "/home/seanc/.clojure/.cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
C:\Users\seanc>#2018-06-2203:37seancorfieldUse wsl.exe to invoke Linux commands 🙂#2018-06-2203:39seancorfieldAnd just to be clear, when you do that, the Linux command executes in the context of Linux, e.g., user.home: C:\Users\seanc>wsl.exe clojure
Clojure 1.9.0
user=> (System/getProperty "user.home")
"/home/seanc"
user=>
#2018-06-2204:01seancorfield(of course this totally doesn't help in terms of actually accessing those files from Windows, nor in starting/running a REPL that can accept "load file" commands from Windows)#2018-06-2204:02cfleming@U04V70XH6 Thanks. So I’d need to make Cursive WSL-aware.#2018-06-2204:02cflemingThe problem is also still accessing the deps files from Windows.#2018-06-2204:02cfleming(or, a problem, rather)#2018-06-2204:03seancorfieldYeah, it's a tough nut to crack. I've dinked around a bit with Windows-Atom/ProtoREPL and WSL-REPL but not really gotten anywhere.#2018-06-2204:03cflemingI think just using the lib and keeping everything in Windows-land looks like the cleanest solution.#2018-06-2204:04seancorfieldWell, once the lib knows how to -Sdescribe the environment...#2018-06-2204:05cflemingRight, I’d have to hack that for the moment, probably bundle the system deps.edn in Cursive, and perhaps allow the user to edit it somehow.#2018-06-2204:06cflemingAnd allow them to configure where their user one is, defaulting as required.#2018-06-2204:06seancorfieldMy fallback with Boot has been to have Boot installed on both Windows and WSL and let Atom start a REPL via Boot and keep all the files in a project structure under the C: drive that I can easily map to Linux (symlink to make the same file paths without /mnt/c).#2018-06-2204:09seancorfieldWhen the PowerShell port of clojure hits, we'll all be a lot better off -- but we'll still have the weird "duplication" of the "home" deps.edn because that will be in a different place on Windows and WSL. I can't see a workaround for that since the user home folder has to be different.#2018-06-2204:16cflemingYou could use CLJ_CONFIG in WSL?#2018-06-2204:17cflemingAnd point the linux one to where it is on Windows?#2018-06-2204:23seancorfieldI haven't tried CLJ_CONFIG...#2018-06-2204:26seancorfieldOh right, so on WSL, CLJ_CONFIG could point to /mnt/c/Users/<whatever>/... yeah, sounds like a good approach once we have PowerShell clojure.#2018-06-2204:38cflemingRight.#2018-06-2204:39cflemingAlthough generally I think the best option for clients like Cursive and lein-tools-deps would be bundling the system deps.edn in the lib, and perhaps providing a method to locate the user deps.edn.#2018-06-2204:54seancorfieldWhen I first built boot-tools-deps I baked in the deps.edn from the brew-install GitHub repo so I didn't need clojure installed on the system. Alex regularly told me that was a bad idea 🙂 So I changed it to call (clojure-env) once t.d.a had that API. And now we're talking about the system deps.edn being baked into t.d.a anyway! :rolling_on_the_floor_laughing:#2018-06-2210:53rickmoynihanInteresting discussion - I think I’d like to remove the call to clojure-env from lein-tools-deps and follow the bundled system approach; and just let people override the :system location if they need to.#2018-06-2219:52cfleming@U06HHF230 Yes, I agree - this would make Cursive much easier too#2018-06-2219:53cflemingThe only issue is the need to specify the version to use.#2018-06-2115:33Alex Miller (Clojure team)@timgilbert additionally, you could set :mvn/local-repo to a fresh directory, then capture that directory to know what you need.#2018-06-2117:12timgilbertAm I crazy, or is tools.deps super slow at downloading versus lein deps? I'm seeing my download step go from about 40 secs to about 4 minutes on the same hardware#2018-06-2117:43Alex Miller (Clojure team)I haven’t heard that before now#2018-06-2117:44Alex Miller (Clojure team)Are you comparing with cold m2 repo in both cases?#2018-06-2117:47timgilbertYeah, they are both running from nothing on circleci servers. I do wonder if it might have to do with invalid artifacts in my tree, combined with a private S3 repo.#2018-06-2117:48timgilbertI will do some poking around and file a ticket if I can come up with some numbers and a reproducible case#2018-06-2117:50Alex Miller (Clojure team)Yeah, would be curious#2018-06-2117:50Alex Miller (Clojure team)Since most people have stuff in cache, most people wouldn’t notice#2018-06-2117:52timgilbertYeah. Does tools.deps cache misses? Somehow or another I've wound up with com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml in my deps tree, which doesn't exist, and it seems to be causing some of the latency along with several other similar artifacts#2018-06-2117:53timgilbert(Obviously, these are things I am going to exclude from the tree once I've tracked them down)#2018-06-2118:09Alex Miller (Clojure team)tools.deps uses the same Maven apis that lein (via pomegranate) does to download metadata and deps#2018-06-2118:11Alex Miller (Clojure team)so I would expect it to generally do pretty similar things. there are some differences, notably that tools.deps uses its own transitive dep walk and can potentially choose different versions and may walk versions that lein deps doesn’t.#2018-06-2122:48rickmoynihanI’ve been meaning to ask are the differences in resolution documented anywhere, or formally known? Asking because it’s relevant to leiningen projects moving deps into deps.edn and lein-tools-deps.#2018-06-2200:54Alex Miller (Clojure team)Well it’s known to me but I haven’t gotten around to writing it up yet#2018-06-2200:55Alex Miller (Clojure team)It’s somewhat similar to the new golang stuff #2018-06-2118:11Alex Miller (Clojure team)tools.deps itself doesn’t explicitly do any caching of metadata or libs at all - Maven handles all of that#2018-06-2118:13Alex Miller (Clojure team)several possible things to investigate there. if you file a ticket with a deps.edn to repro, I would eventually take a look at it#2018-06-2118:13timgilbertOk, cool. I'll see what I can do to come up with a reproducible case. Thanks!#2018-06-2121:45kennyIf both :sha and :tag are provided for a Git dep but :tag points to a different :sha, which SHA is used?#2018-06-2121:56seancorfield@kenny I thought only :sha was used for dependencies and the :tag was there for use with the option that adds :sha based on :tag...?#2018-06-2121:56seancorfield(it's easy enough to test the behavior!)#2018-06-2121:57kennyThis part from https://clojure.org/reference/deps_and_cli#_dependencies makes it a bit unclear
> The :sha attribute should indicate the full commit sha
> The :tag attribute is optional but can be used to indicate the semantics of the sha#2018-06-2121:58kennyAlso makes it sound like :sha is always required.#2018-06-2121:59kennyYes. Will test it out later today. Definitely could use some clarification in the docs though.#2018-06-2122:04seancorfieldThat reads that :sha is used for dependencies and :tag is purely descriptive -- although there's an option to update deps.edn adding/replacing SHAs with the values that correspond to tags: clj -Sresolve-tags#2018-06-2122:06seancorfieldMy interpretation of that is that if you have :sha and :tag and the latter doesn't match the former, it would not affect resolution. And if you ran clj -Sresolve-tags it would update the :sha to match what the :tag pointed at. @alexmiller can you confirm or deny?#2018-06-2122:06Alex Miller (Clojure team)Correct#2018-06-2122:07kennyThat interpretation makes sense to me. I'd still suggest adding that information into the docs 🙂#2018-06-2122:11Alex Miller (Clojure team)Still considering whether to remove :tag and -Sresolve-tags entirely :)#2018-06-2122:19kennyThe reason I was thinking about using :tag is to convey breakage. With only :sha, determining breakage is a lot harder.#2018-06-2201:11Alex Miller (Clojure team)Can you explain?#2018-06-2201:11Alex Miller (Clojure team)If you mean like api breakage, just don’t do that :)#2018-06-2201:12Alex Miller (Clojure team)We have sha logic comparison in there - it can tell which one is a parent of the other#2018-06-2201:12Alex Miller (Clojure team)If more than one available in the tree, it will pick the descendant #2018-06-2215:30val_waeselynckIt seems to me it's not possible to have a spaces in a :main-opts entry (e.g :main-opts ["--space-separated-arg" "Joe Jack Williams Averell"]), is this is known issue?#2018-06-2215:32Alex Miller (Clojure team)Yes, use commas for now#2018-06-2215:33Alex Miller (Clojure team)Or in a case like this, not going to work for now#2018-06-2215:38val_waeselynck@alexmiller thanks, currently my workaround is simply to pass the args form the command line#2018-06-2217:16seancorfield@val_waeselynck https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L32-L37 commas-as-spaces#2018-06-2404:40hagmonkI've updated https://github.com/hagmonk/find-deps, it now allows you to save matches even if the deps.edn file doesn't exist (handy for new projects). It also has a new -F:cli output format that prints the deps on a single line appropriate for clojure -Sdeps usage{:tag :div, :attrs {:class "message-reaction", :title "clap"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👏")} " 8")}
#2018-06-2408:58pesterhazy@hagmonk just tried it, it doesn't seem to work for me. After following the instructions, clj -A:find-deps -h gives me the normal "deps" help screen#2018-06-2409:00pesterhazyis the sha in the readme up to date?#2018-06-2413:03Alex Miller (Clojure team)Do you have the latest clj? That -h thing was a bug I fixed a while back#2018-06-2414:29pesterhazy@alexmiller I have version = 1.9.0.381 from homebrew#2018-06-2414:31Alex Miller (Clojure team)That should be latest#2018-06-2414:33Alex Miller (Clojure team)Maybe you’d see that if it’s missing the find-deps alias#2018-06-2415:26pesterhazy@alexmiller that was it. I followed the readme but not correctly. PR to make the docs more explicit for busy readers: https://github.com/hagmonk/find-deps/pull/1{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 4")}
#2018-06-2517:44hagmonkI've pushed a new find-deps with @pesterhazy's PR merged and a fix for no longer returning SNAPSHOT versions from clojars (hard coded for the time being)#2018-06-2517:48pesterhazy@hagmonk just tried it, works really well#2018-06-2517:48pesterhazyI think hardcoding the hide-SNAPSHOT feature is a much better default#2018-06-2517:49hagmonkYes definitely, I think the bug should not crop up in maven searches, I need to spend time finding an example there to validate that#2018-06-2517:49hagmonkI believe maven doesn't offer snapshot results by default#2018-06-2517:53pesterhazygood job on the library btw!#2018-06-2517:59hagmonkThanks! Let me know if any of the search results don't look good :)#2018-06-2518:54dominicm@hagmonk one piece of feedback is that it will modify the deps.edn formatting. #2018-06-2518:56hagmonk@dominicm that's worth an issue on github … appreciate any pointers you might have for how to avoid that#2018-06-2518:57dominicmPack avoids it (my library). Happy to try share a library for solving this problem. I'm secretly hopeful it can go into tools deps itself. #2018-06-2519:27hagmonkThanks @dominicm, I'll have a look after work#2018-06-2519:28hagmonkI'm still enjoying how much time this is saving me, occasionally I'm surprised it works as well as it does 😂#2018-06-2519:28hagmonkclj -A:find-deps "clojure cache"
{:deps {org.clojure/core.cache {:mvn/version "0.7.1"}}}
#2018-06-2519:31pesterhazyone possibility would be to run the result through zprint, a great library for re-formatting clojure code or edn#2018-06-2519:32pesterhazyzprint is an under-appreciated gem#2018-06-2519:55dominicm@pesterhazy the problem is that everybody does formatting a little differently. I took the approach of trying to match existing indentation at most levels. #2018-06-2520:12mkvlrit would be so great if we could agree on a canonical way to format clojure code#2018-06-2520:13mkvlrhaving worked with elm-format and exfmt I found it so nice to just not have to worry about this at all anymore#2018-06-2520:14pesterhazyzprint gets you 90% there#2018-06-2520:16dominicmZprint puts my defn all on one line, which feels wrong#2018-06-2520:23pesterhazyThey are too short :)#2018-06-2520:24dominicmDo you happen to know of a way to change that behaviour? 80 chars isn't a goal, so much as a limit.#2018-06-2520:27pesterhazyNot off the top of my head#2018-06-2520:29pesterhazyBut if you get consistency, convenience and the end of style wars in return, wouldn’t oneliners be a small price to pay?#2018-06-2520:30dominicmI want consistency, very much. But I first have to convince important people that they will like the formatting, and that it's worth pioneering.#2018-06-2607:44kommenthe :respect-nl config should be able to help you with “80 chars isn’t a goal”: https://github.com/kkinnear/zprint/#respect-nl-false#2018-06-2607:50kommenit says it is meant for vectors, but might work for lists as well#2018-06-2607:51dominicmI don't necessarily want to have it retain where I place newlines. I just find it weird to see:
(defn foo [args] body)
over
(defn foo
[args]
body)
#2018-06-2607:59kommenah, I see. haven’t tried to configure that, but pretty sure it is possible#2018-06-2608:04dominicmIt would be nice to have a style which is "do what most libraries do for formatting"#2018-06-2608:07kommenI think the :community style is meant for that. kkinnear has been open to adapt that style to fit best practices, especially if those things would be documented in the clojure community style guide#2018-06-2608:50pesterhazyNOT respecting newlines is one of the best things about elm-format#2018-06-2608:51pesterhazyyou can totally write your code in a single line and elm-format will expand it out#2018-06-2608:51pesterhazyit helped me a lot when I was learning the language#2018-06-2608:55dominicmI particularly don't want my tool to look at the input beyond raw clojure forms tbh. I'd even be happy if it made decisions about #:foo {:blah 10} into {:foo/blah 10} (but that's because I don't like the syntax)#2018-06-2520:14pesterhazyAgree about elm-format, it’s a pleasure to use #2018-06-2520:14mkvlrwe’re using zprint but there’s so many knobs. I think an autoformatter should ideally have 0 options. (or maybe one for the line width)#2018-06-2520:15mkvlrsorry for going a bit offtopic here…#2018-06-2521:57lwhortoni’m not sure if this is the right place to ask… but does the :git/url coordinate require a deps.edn file at that coordinate?#2018-06-2521:57lwhortoni.e. it doesn’t try to read a project.clj or boot file… it only plays nicely with another deps.edn (or maybe a pom) file?#2018-06-2521:58ghadiThat's right @lwhorton #2018-06-2521:58ghadiIt does read deps.edn and pom.xml transitively{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2522:08Alex Miller (Clojure team)correct (for now)#2018-06-2522:08Alex Miller (Clojure team)it’s also tolerant of finding none of those for the case of a dep with no transitive deps#2018-06-2522:56cfleming> I think an autoformatter should ideally have 0 options. (or maybe one for the line width)
Preach it! If I ever design a language, it will have a gofmt-style formatter from day 1.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 16")}
#2018-06-2522:57cflemingIndentation turns out to be one of those things that everyone thinks they care about, until they can’t control it, at which point it turns out they don’t care much as long as the results are reasonable.{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 24")}
#2018-06-2522:59Alex Miller (Clojure team)+1{:tag :div, :attrs {:class "message-reaction", :title "heavy_plus_sign"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("➕")} " 12")}
#2018-06-2602:12hagmonkWould love for us to swallow that frog and just settle on one format to rule them all. I'd gladly give up my personal formatting preferences in exchange for not having to care about this ever again :)#2018-06-2606:44pesterhazyIf @bozhidar and @cfleming band together and settle on a zformat formatting style, that’s a significant portion of the market#2018-06-2606:45pesterhazyMake it opt-in and apply on save#2018-06-2606:51orestisWe’re considering introducing Clojure to an existing team – not having to argue about formatting would be huge — zprint looks fantastic, with native-image support so could even be run on a github pre-commit hook.#2018-06-2606:54hagmonkI think people quite quickly get the whole paredit/parinfer concept, where the editor prevents you from creating an improperly delimited program … it doesn't seem a fair stretch to also prevent you from ever creating an improperly indented program#2018-07-0208:46rickmoynihanI’m definitely all for structured editing; a classic example of a really good one was Diva Scheme for PLT Scheme (now Racket):
https://www.youtube.com/watch?v=GnQV4je9wTQ
You need a mode to disable them temporarily though e.g. when resolving a merge.#2018-06-2606:56hagmonkI'm trying out parinfer and am irritated that I seem able to hit "tab" and move blocks of code around to create valid but aesthetically awful code (probably some config thing I borked). I'd love for it to be impossible to write untidy code, like making it physically impossible to create:#2018-06-2606:57hagmonkdangling paren:
(comment
(some-stuff "test")
)
or have two newlines between def:
(def this "thing")
(def other "thing")
… etc#2018-06-2607:00dominicmParinfer + zprint could be quite great for beginners... #2018-06-2607:16orestisOTOH, I tried Parinfer via Nightcode — I knew how to write Clojure code in spacemacs via Cider, which doesn’t try to do too much — but parinfer messed me up quite a lot — it tried to be helpful but got in my way.#2018-06-2607:35dominicmhave you tried it since smart mode was added?#2018-06-2607:35dominicmhttps://shaunlebron.github.io/parinfer/demo.html#2018-06-2609:16orestisI need to find how to integrate with spacemacs and evil-mode, but will try again…#2018-06-2607:17orestisIt helps a beginner to be able to balance forms manually, esp. if you have a linter that will point out possible problems, and a formatter that would take stylistic decisions away from you. Parinfer might be useful once you make relatively few mistakes when coding Clojure.#2018-06-2607:24hagmonkParinfer is a little more loose, I agree. I think we could go a step further and have editors that totally precluded interim mistakes at all. Like: trying to add a third argument to a 2-arity function. Why do we let you type that in? Why do we let you type in something that is highly likely knowable at the reader level to be invalid, such as transposing the arguments to map – i.e. putting something that doesn't support IFn as the first argument?#2018-06-2612:09Alex Miller (Clojure team)There’s an interesting talk in this area at Strange Loop this year. https://thestrangeloop.com/2018/hazel-a-live-functional-programming-environment-with-typed-holes.html#2018-06-2607:26hagmonkI realize the dynamic nature makes some things hard to guarantee, but even though I'd say I'm a journeyman Clojure programmer, I would still appreciate the help. It's like type checking but while you type, not when you compile.#2018-06-2607:27orestisThe UX for that is extremely hard to get right though — when I type a character, I expect it to show up in my buffer. If the editor puts a squiggly line later, then I know something’s wrong and I can learn something from it.#2018-06-2607:31hagmonkIt need not be "can't type the key" levels of strictness, more like how we flag unbalanced parens today … "this is for sure going to barf". Ideally unbalanced parens become physically impossible, while messing up arguments graduates to being detected in the editor buffer rather than runtime or at compilation#2018-06-2607:35hagmonkI think we also have opportunity (I realize this is drifting off tools.deps but ya'll are an erudite bunch) for hinting, for instance, when I type (let [[foo bar] some-thing] …) I occasionally pass in something non-seqable as some-thing and invariably it takes me a few minutes to reverse engineer the cryptic error message and wth I did to cause it. Just flagging in the editor "reminder: some-thing now has to be seqable!" could be quite interesting …#2018-06-2607:41cfleming@hagmonk This is what you’re looking for: https://www.facebook.com/notes/kent-beck/prune-a-code-editor-that-is-not-a-text-editor/1012061842160013#2018-06-2607:42cflemingPlus some static analysis#2018-06-2607:42cflemingThis should probably get moved over to #editors though.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2607:45hagmonkYes, I recall this whizzing past at some point. I'll take another look#2018-06-2610:45mgrbytehaving trouble figuring out what the correct coordinate would be to use a local jar (datomic-pro), anyone doing this?#2018-06-2610:47dominicm@mgrbyte :local/root pointed at the jar#2018-06-2610:53mgrbytethat's what I've tried. doesn't work for me. I have:
:deps {
com.datomic/datomic.api {:local/root "/home/matt/datomic-pro/datomic-pro-0.9.5697/datomic-pro-0.9.5697.jar"}
}
#2018-06-2610:54dominicm@mgrbyte I may have crossed wires, just to check, can you try :deps/root instead?#2018-06-2610:54mgrbyteIt's definitely :local/root in the map in the docs#2018-06-2610:55dominicmhttps://clojure.org/guides/deps_and_cli#local_jar nope, I remembered correctly#2018-06-2610:55dominicmah, you're having an issue with that working 😄#2018-06-2610:55dominicm@mgrbyte I'm not sure I understand, what is the problem? 🙂#2018-06-2610:55mgrbyteclj works, but requiring the dep doesn't#2018-06-2610:55mgrbyteclj
Clojure 1.9.0
user=> (require '[datomic.api])
ClassNotFoundException org.fressian.handlers.IWriteHandlerLookup java.net.URLClassLoader.findClass (URLClassLoader.java:381)
user=>
#2018-06-2610:56mgrbyteI'm using the very same jar in a lein project, which works fine.#2018-06-2610:57mgrbyteinstalled locally via bin/maven-install in the datomic-pro distro (same path as the jar in the deps I listed above)#2018-06-2610:58mgrbytenm, think I see what's going on here#2018-06-2610:59mgrbytedatomic pro has a pom.xml with a bunch of deps in it. can't just use that jar, need all the deps in "./lib" from the distro too.#2018-06-2612:11Alex Miller (Clojure team)The latest tools.deps should actually read the pom inside the jar and load those for you#2018-06-2612:12Alex Miller (Clojure team)Make sure you are on latest clj tool#2018-06-2612:54mgrbytethanks, will try that now#2018-06-2612:54mgrbyteIs there any reason to prefer to doing that than using bin/maven-install and using the mvn/version w/datomic-pro?#2018-06-2612:55Alex Miller (Clojure team)you can do either. you can also use the clj support to load the pro jar from your authenticated s3 datomic repo{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2612:57mgrbytety#2018-06-2611:05dominicmI thought it might be that.#2018-06-2611:06dominicmYou should be able to set :local/root to the dir with the pom.xml in, but not certain.#2018-06-2611:45mgrbytenearly, fails with some dep. Not a problem tho, can just use :mvn/version of the local install#2018-06-2611:54hkuptyQuick question.. does tools.deps expose a function to add to classpath? I couldn't find anything from the documents but I would also like to avoid adding pomegranate or lucidity.package..#2018-06-2611:56borkdude@ingvij I think it’s called add-lib#2018-06-2611:56borkdude@ingvij http://insideclojure.org/2018/05/04/add-lib/#2018-06-2612:01hkuptyNot there as of latest published version..#2018-06-2612:01Alex Miller (Clojure team)Still experimental#2018-06-2612:01hkuptyI see.. So that can change until I actually can rely upon..#2018-06-2612:01hkuptyI'll wait then.. not in a hurry for that though#2018-06-2619:11hagmonk@alexmiller I was going to start thinking about add-lib in the context of find-deps. I could have users do this: (-> "clojure cache" find-deps.core/deps clojure.tools.deps.alpha.repl/add-lib)#2018-06-2619:12hagmonkor I could provide a fn inside find-deps.core that invokes add-lib for you. right now I'd have to depend on your add-lib branch, so I was hesitant to bake that in quite yet#2018-06-2619:12Alex Miller (Clojure team)yeah, I wouldn’t do that yet#2018-06-2619:14hagmonkcool, I might limit integration to some notes in the readme for the time being. I can't wait until we have a workflow where I can seamlessly add deps without restarting anything :)#2018-06-2620:41cfleming@hagmonk Have you ever tried alembic?#2018-06-2620:56hagmonk@cfleming I have not ... Googling ...#2018-06-2620:57hagmonkfascinating, this would have been nice to have :)#2018-06-2621:42Alex Miller (Clojure team)note that the tools.deps add-lib is a little different than the other existing things out there#2018-06-2621:43Alex Miller (Clojure team)it’s not just blindly adding a lib (and/or it’s transitive deps), it’s actually detecting which things you don’t already have, resolving lib versions, and updating an internal map of loaded libs and versions#2018-06-2621:44hagmonkand obviously it can work on more than just jars#2018-06-2621:45Alex Miller (Clojure team)that too, all the standard tools.deps differences#2018-06-2622:09cflemingAlembic also does the detection of existing things you have installed, although it doesn’t use a map, it uses the list of things in your existing classloader.#2018-06-2622:48Alex Miller (Clojure team)The tools.deps impl falls back to that#2018-06-2622:31hagmonkI have some crazy ideas in this space - I feel like we can go a lot further than just adding the libs. For find-deps I want to analyze clj code on github to build "recommended deps" based on the one you're importing. For instance, reagent and re-frame often go together. Intelligently figuring out exclusions could be another one.#2018-06-2622:46sparkofreasonHow would I do a local build and deploy of tools.deps.alpha?#2018-06-2622:46Alex Miller (Clojure team)mvn clean install#2018-06-2622:47Alex Miller (Clojure team)That will use the current snapshot version in the pom{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2623:26sparkofreasonDebugging some issues using tools.deps.alpha on Windows, and pretty much a noob with aether. Why would clojure.tools.deps.alpha.util.maven/make-system return nil?#2018-06-2623:26Alex Miller (Clojure team)Dunno, not sure why windows would be special#2018-06-2623:27Alex Miller (Clojure team)I’ve got ports of clj using tdeps on Windows that work#2018-06-2623:29sparkofreasonHmmm, that's what I would have guessed. It's being called by lein-tools-deps, so maybe the problem originates farther up the call stack.#2018-06-2623:41seancorfield@dave.dixon Perhaps because lein-tools-deps calls (clojure-env) in t.d.a. and that shells out to call clojure to get the list of deps.edn files to consume -- and I would expect that command to fail on Windows (until we have a Windows port of the clj/`clojure` shell scripts).#2018-06-2623:41seancorfield(so maybe it doesn't have the environment that make-system requires)#2018-06-2623:42sparkofreason@seancorfield That I have working, albeit locally hacked. I'll double-check that the right info is getting scraped.#2018-06-2623:42seancorfieldAh, OK. Cool.#2018-06-2623:43seancorfieldI nearly said "...unless you have a mocked up version of clojure on Windows!" 🙂#2018-06-2701:13sparkofreasonOkay, wacky problem 1 solved, somehow I managed to have two versions of lein installed.#2018-06-2716:22sparkofreasonThanks to everyone's help, I was able to get a working setup using lein-tools-deps and Cursive on Windows, with clj installed on WSL. It's pretty hacky, but I think will tide me over until the tooling solves this problem in the right way. See fork on lein-tools-deps at https://github.com/ComputeSoftware/lein-tools-deps. Use at your own risk 😋#2018-06-2810:04darnokI'd like to use :local/root to tie up my dependent projects in the monolith repo, but I have an issue with one of the dependencies. It has one Java source file. Can you recommend, how to handle this?#2018-06-2811:33Alex Miller (Clojure team)Well one solution is to move it into its own project and make an artifact#2018-06-2811:34darnokok#2018-06-2811:37darnokI thought that it would need to be resolved like this. Thanks 🙂.#2018-06-2814:00dottedmag#2018-06-2814:01dottedmag@alexmiller ^ is this an intentional behaviour?#2018-06-2814:20Alex Miller (Clojure team)there’s a ticket, send me a patch :)#2018-06-2814:21Alex Miller (Clojure team)I just haven’t gotten to it yet#2018-06-2814:22Alex Miller (Clojure team)actually, maybe nobody made a ticket so feel free to make one in https://dev.clojure.org/jira/browse/TDEPS#2018-06-2814:42pesterhazyyeah, the "problem" I reported a few days back was because the alias wasn't properly configured; a simple name check would caught this#2018-06-2814:49Alex Miller (Clojure team)happy to fix#2018-06-2814:50dottedmag@bhauman ^#2018-06-2814:53bhaumanI think my use of the word extremely was extremely extreme#2018-06-2815:41seancorfieldIt's bitten me a couple of times so I'll vote that ticket 🙂 but it hasn't bitten me hard/often enough for me to summon up the energy to create such a ticket...#2018-06-2816:20hagmonktoday in #tools-deps: extremely extreme bugs! do they bite hard and often enough for a ticket? do they come from the Australian mainland? only jira can tell!{:tag :div, :attrs {:class "message-reaction", :title "rolling_on_the_floor_laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-06-2816:20hagmonk(clearly this channel is where all the action is at)#2018-06-2816:36Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-85 created#2018-06-2816:36Alex Miller (Clojure team)vote away{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 4")}
#2018-06-2816:36Alex Miller (Clojure team)if no else patches I’ll get to it when I next work on TDEPS#2018-06-2816:40bhaumanThanks Alex!#2018-06-2816:46seancorfieldYes, thank you, Alex!#2018-06-2821:07kennyWhy is the clojure CLI written in bash instead of Python or Node?#2018-06-2821:07dominicm@kenny because shell is universally available and python/node isn't.#2018-06-2821:08dominicmHopefully it can be ported to POSIX shell without any bash, that would be a nice improvement 🙂#2018-06-2821:09kennyWhy do you say Python and Node aren't universally available?#2018-06-2821:11dottedmag{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "86f4e9e9f2c6e7e4e3ea"}, :content ("[email protected]")}{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-06-2821:11dominicm@kenny because they're less likely to be installed than bash.#2018-06-2821:12dottedmagEven if they were, what'd be the point? clojure(1) is trivial.#2018-06-2821:15kennyInstalling Python/Node is trivial. Point is you get a script that is portable to any OS.#2018-06-2821:17dottedmag@dominicm clojure uses arrays from Bash. Not trivially convertable.#2018-06-2821:18dottedmagAlso that means testing the script under several implementations (busybox, dash and bash at least). Not worth it.#2018-06-2821:20dominicm@dottedmag that's unfortunate. Although I did stumble onto https://github.com/Ventto/libshlist recently.#2018-06-2821:22dominicmI personally am ready to remove bash for all but clojure on my system. I use dash & fish primarily.#2018-06-2821:22Alex Miller (Clojure team)because I wrote it and I don’t like Python or JS :)#2018-06-2821:22dominicm@alexmiller So 💎 ruby?#2018-06-2821:22Alex Miller (Clojure team)geez man{:tag :div, :attrs {:class "message-reaction", :title "wink"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😉")} " 4")}
#2018-06-2821:23Alex Miller (Clojure team)for whatever reason, I just don’t get Ruby.#2018-06-2821:24dominicmOh really? Oh I was not expecting that response. I suppose you're not from the ruby shop that's associated with Clojure 😛. You strike me as a perl guy? 😉#2018-06-2821:24dominicm(I'll just keep guessing acting like I knew the whole time...)#2018-06-2821:24dottedmagI feel this is the exactly the kind of conversations that cause Codes of Conduct written which precludes snarky remarks and personal attacks.#2018-06-2821:24Alex Miller (Clojure team)I have used all of these languages profesionally#2018-06-2821:25dottedmag@dominicm The point of clojure is to just work. Anywhere. Without installing anything.#2018-06-2821:25Alex Miller (Clojure team)and I respect people that like and use all of them#2018-06-2821:25dottedmagThe script itself is trivial. There is no gain from using any other language.#2018-06-2821:25dominicm@dottedmag just to be clear, I actually hold no contempt for any of these languages. I feel like I have a fairly good grasp of where each one is pretty good.
I agree. I don't actually think ruby is suited here.#2018-06-2821:26Alex Miller (Clojure team)bash is about as lingua franca as it gets these days{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2821:26hagmonkbash is far and away the lowest common denominator on Unix-ish systems, with the possible exception of sh, which on many distros just links to bash.#2018-06-2821:26hagmonkit absolutely sucks, but if your goal is maximum multi-year portability, there's no other viable alternative#2018-06-2821:27dominicmbash isn't so bad.#2018-06-2821:27Alex Miller (Clojure team)I’ve grown to almost like it#2018-06-2821:27dominicmI wrote a whole panel in bash.#2018-06-2821:27dottedmagThis discussion already amounts to 30% of the size of clojure(1).#2018-06-2821:27Alex Miller (Clojure team)having done the windows port to powershell, that wasn’t bad either#2018-06-2821:28dottedmag@alexmiller Oh, there is PowerShell one? In the same repo?#2018-06-2821:28Alex Miller (Clojure team)it’s on a branch I think#2018-06-2821:28Alex Miller (Clojure team)if I pushed it#2018-06-2821:28hagmonkthere's a team at work that has >100k written in bash, with strict coding guidelines and practices it can work. You can grow to like many things if you immerse yourself in their idiosyncrasies :)#2018-06-2821:28Alex Miller (Clojure team)it’s not done yet#2018-06-2821:29dottedmagFound it, thanks#2018-06-2821:29dottedmagYeat, it beats cmd#2018-06-2821:30Alex Miller (Clojure team)fer sure#2018-06-2821:30Alex Miller (Clojure team)it’s kind of like if you took bash and applied sane engineering to it#2018-06-2821:30dottedmagAnd isane amount of syntax, but oh well.#2018-06-2821:31hagmonkheck, powershell does autocomplete of function arguments on the command line. It's kinda impressive#2018-06-2821:31kennyWhy not use ClojureScript compiled to Node?#2018-06-2821:31dominicm@kenny because node isn't always installed!#2018-06-2821:31kennyIt is trivial to install.#2018-06-2821:32dottedmagAmazing.#2018-06-2821:32hagmonk@kenny the goal is not to install anything except what is required to bootstrap clojure#2018-06-2821:32Alex Miller (Clojure team)stop trying to oppress me man#2018-06-2821:33hagmonkthis coming from a man comfortable writing all day in bash :){:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2821:33Alex Miller (Clojure team)the water’s fine, come on in#2018-06-2821:33dominicm@hagmonk (and any others) I just pushed up 857bbc85c33b23b5f8e51d68eb724776564d9c16 to juxt/pack.alpha which adds skinny jar support, it spits it out in multiple forms based on the parameters. Any feedback welcome!{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-06-2821:50dominicm@hagmonk let me know if this works for you btw.#2018-06-2821:50dominicmI don't really have a concrete use case myself yet, so I'm guessing at things a little. #2018-06-2821:57hagmonkyeah for sure, I'll poke at it when I get a chance!#2018-06-2915:18dominicmI had to do a little bump, having : in filenames doesn't go well with classpaths, 800afc066907e31fe90ce4df4f519e6224116867 is the latest commit#2018-07-0315:21dominicm@hagmonk Did you get any time to see if this looks good to you? 😄#2018-07-0323:50hagmonkSorry @dominicm, haven't forgotten, but I've been crushed at work lately :/#2018-07-0405:29dominicmNot at all. Entirely when you get some spare time. 😊 #2018-06-2821:34dominicmIt's in alpha-alpha, as it's not in master yet, I'll be rebasing at some point, but I won't force push over those commits.#2018-06-2821:34hagmonkI can't, I didn't quote that string properly and so I dived into an empty pool#2018-06-2821:35dottedmag@hagmonk rm -rf "${PREFIX}/${SHAREDIR}". Guess what happened? True story.{:tag :div, :attrs {:class "message-reaction", :title "scream"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😱")} " 4")}
#2018-06-2821:35kennyMake Node required to bootstrap Clojure.{:tag :div, :attrs {:class "message-reaction", :title "upside_down_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙃")} " 4")}
#2018-06-2821:41gfrederickson the off-chance that you're not trolling, that would be way too much friction#2018-06-2821:35dottedmag@hagmonk Postinstall script in a Linux package. Root privileges.{:tag :div, :attrs {:class "message-reaction", :title "scream"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😱")} " 8")}
#2018-06-2821:36hagmonkTruly a trap for young players, right there … a mistake you never forget :)#2018-06-2821:44hagmonkSo quick story time about the clojure installer being written in bash: the other day I was mucking with Homebridge on a raspberry pi, and needed to do some interrogation of a web API. In five minutes I had a Clojure REPL up with rebel readline, it was great. Had the installer required ruby, python, perl, etc, I would have had to dive into the Pi's package management and pull a ton of unrelated crap down, just to get a REPL. Had clojure needed a version of nodejs that was in any way incompatible with homebridge, I would have immediately faced the problem of juggling multiple npm versions. An example of the leverage you get from picking the unsexy, boring, lowest common denominator for bootstrapping tools{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 16")}
#2018-06-2821:44dottedmagDoes PowerShell come preinstalled on Windows nowadays?#2018-06-2821:44hagmonkI do believe so#2018-06-2821:46Alex Miller (Clojure team)yeah#2018-06-2910:48cflemingOf particular interest is the git lib attached to library (“Deps: org.clojure/tools.gitlibs:cab6ab”)#2018-06-2910:49dominicmNice 🙂#2018-06-2910:50cflemingThere’s still lots of trickiness to work out, but the first 90% is done. Now there’s just the remaining 90%.{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 8")}
#2018-06-2911:06Andreas LiljeqvistLooking forward to it, my temporary solution isn't exactly scalable...#2018-06-2913:49pesterhazy@cfleming this will be a godsend for our team#2018-06-2914:59mfikesWow! This will be extremely useful.#2018-06-2922:08cfleming@pesterhazy Great! I know a lot of people are waiting on it.#2018-06-3003:46dorabIf I try lein ancient after incorporating lein-tools-deps in my project.clj it always tells me that the artifacts are up-to-date. What do I need to do to get lein ancient work with lein-tools-deps? Thanks.#2018-06-3005:15hagmonk@dorab because with lein-tools-deps you're storing your deps in deps.edn, I'd recommend you use a tool that targets deps.edn directly. So far I only know of https://github.com/Olical/depot, but it works great#2018-06-3015:41dorabThanks!#2018-06-3015:48dorabDid exactly what I wanted.#2018-06-3018:04stathissiderishow alpha is add-lib? It fails for me like:
eggshell.controller> (deps/add-lib 'spec-provider {:mvn/version "0.4.14"})
true
eggshell.controller> (require 'spec-provider.provider)
FileNotFoundException Could not locate spec_provider/provider__init.class or spec_provider/provider.clj on classpath. Please check that namespaces with dashes use underscores in the Clojure file name. clojure.lang.RT.load (RT.java:463)
#2018-06-3018:05stathissiderisIf I try to add-lib again, it returns false#2018-06-3018:05stathissideriswhich means that it didn’t have to add it again#2018-06-3018:11dominicm@stathissideris don't know why that's happening, buut, you need to be under a DynamicClassLoader for it to work.#2018-06-3018:11stathissiderisI’m not even sure what that means 🙂#2018-06-3018:27stathissiderisI’m in a REPL, is that ok?#2018-06-3018:27hiredmanadd-lib is very alpha#2018-06-3018:29stathissideris@hiredman sounds like I have to wait for a bit longer before I integrate it into my app#2018-06-3020:50dominicm@stathissideris it is possible to change the class loader of a repl, I believe it's in the recent history.#2018-06-3020:53Alex Miller (Clojure team)add-lib works by finding the highest dynamic class loader in the current classloader chain and altering its loader sources #2018-06-3020:54Alex Miller (Clojure team)Some repl contexts do not have a shared dcl above all repl invocations#2018-06-3020:54Alex Miller (Clojure team)So changing it would not necessarily have any effect, I think nrepl creates a new context on every evaluation#2018-06-3020:55Alex Miller (Clojure team)Using clj repl works for me but nrepl things I’ve tried do not#2018-06-3020:56dominicmwe've been talking about how to fix this in nrepl.#2018-06-3020:57Alex Miller (Clojure team)Just need a dcl shared by all evals#2018-06-3020:58stathissideris@alexmiller I’m trying to integrate add-lib to my Swing app (and yes, I do realize it’s 2018 😄 ), the REPL entered the discussion because I naively thought that calling add-lib from the app would also make it available to the REPL I’m developing with#2018-06-3020:58Alex Miller (Clojure team)Are you using nrepl?#2018-06-3020:58stathissiderisyes#2018-06-3020:59stathissiderisbut having the new libs available in the repl is not essential to me#2018-06-3021:02stathissiderisI may need to read up on classloaders, I call add-lib from within a future, would that make a difference?#2018-06-3021:21stathissideris@alexmiller this is the classloader chain within the future:
(#object[sun.misc.Launcher$AppClassLoader 0x42a57993 "
Starting with (.getContextClassLoader (Thread/currentThread)). So no DCL there, I guess that explains it#2018-06-3021:22dominicmYou can do a set @stathissideris #2018-06-3021:23stathissiderishm?#2018-06-3021:23stathissiderisoh, setContextClassLoader!#2018-06-3021:25mfikes@stathissideris If it helps, I had to do something similar here https://github.com/mfikes/clojurescript/commit/d68c9397599366777d9b322ec586fdd398302f25#diff-2c521d7ba435fbef7d128f4295f86785R612#2018-06-3021:27stathissideris@mfikes thanks, that looks like it may work 🙂#2018-06-3021:37stathissideris@mfikes @dominicm some progress! it seems that setting the classloader as @mfikes demonstrated allows add-lib to work, but the added lib is still not available to the part of the app that needs to use it, presumably because it doesn’t share the same DCL… but now at least I understand how it works and I may be able to fix it somehow#2018-06-3021:38stathissiderismany thanks to all#2018-06-3021:39mfikes@stathissideris Cool. IIRC, in the change linked above, I shoved that DCL in at the earliest possible moment.#2018-06-3021:40mfikes(It is the very first bit of code executed in main, and my hope was that any threads spawned afterward would inherit it.)#2018-06-3021:42Alex Miller (Clojure team)that should be the case#2018-06-3023:58cflemingJust to check my understanding of this - is it the case that alembic no longer works under nREPL, because of the dynapath change for Java 9? Sounds like nREPL used to actually modify the URLClassLoader but can no longer do so?#2018-07-0105:04cflemingActually, I’ll ask this over in #nrepl#2018-07-0103:11lincpa@stathissideris I encountered a similar failure during the use of Lein::
1. Add missing dependencies in project.clj (prompt in error message) 2.lein clean#2018-07-0200:39gfredericksthe +6 can be made less hacky using some sort of fancier "read up until you encounter this magic string" approach#2018-07-0200:41gfredericksstartup is like 7sec though O_O
I guess -Sdeps subverts caching?#2018-07-0212:56Alex Miller (Clojure team)It used to but I think fixed in the current release#2018-07-0215:41gfredericksoooh#2018-07-0215:41gfredericksfaaancy#2018-07-0215:41gfredericksI will try it#2018-07-0215:43gfredericksyep, seems better#2018-07-0220:18martinklepschI’m running a script with clojure and have a println at the end. After that line is printed the script takes another 30+ seconds to exit. I’m probably just doing something weird but anyone an idea how to debug this?#2018-07-0220:19manutter51I don’t know about debugging, but that looks like the kind of problem that is often solved by adding a call to shutdown-agents somewhere at the end.#2018-07-0220:24martinklepsch@manutter51 that indeed resolved the issue, thanks!{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-07-0308:31pesterhazy@martinklepsch probably not related to the println itself, right?#2018-07-0308:33martinklepschNo, println was just literal string, but the shot down agents thing fixed it for whatever reason :man-shrugging: #2018-07-0400:47hagmonkyou can try hitting it with jstack next time to see where it's hung up{:tag :div, :attrs {:class "message-reaction", :title "coffee"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☕")} " 4")}
#2018-07-0917:28andy.fingerhutLikely there is something in your program that is creating a future, which causes that behavior, e.g. clojure.java.shell/sh -- http://clojuredocs.org/clojure.core/future#2018-07-1313:21martinklepsch@U0CMVHBL2 thanks that was most likely the issue. Is there any way to properly shutdown an individual future so that I don’t have to clean up using shutdown-agents later?#2018-07-1314:56martinklepschThis issue came up pretty frequently again, tried jstack now which returns the follow (besides lots of other stuff):
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f89a7016800 nid=0x5103 in Object.wait() [0x000070000957b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c000e778> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c000e778> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f89a7013800 nid=0x2e03 in Object.wait() [0x0000700009478000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c000e948> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c000e948> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#2018-07-1315:01martinklepschduh, found it 🙂 didn’t stop a scheduler facepalm#2018-07-1315:04martinklepschSeems shutdown-agents is still required in some capacity though#2018-07-0522:03richiardiandreais there a known issue in clojure exit codes not forwarded with -m ?#2018-07-0523:01ghadiI haven't seen one @richiardiandrea can you make a reproduction case?#2018-07-0523:01richiardiandreabefore that I'll make sure it is not on my side 😄#2018-07-0611:48gfredericksdoes customizing a repl consist of having an alternate (different from clj) executable for starting the repl, or is there a facility in tools.deps to customize it? e.g., customize the printing function, or ensure that certain code is run before the repl starts#2018-07-0612:09Alex Miller (Clojure team)Once you have a repl, you can start a nested repl within it with whatever customizations you want. But I think this an area ripe for better options.#2018-07-0612:26gfrederickslooks like I can't put a system-global user.clj file on the classpath automatically, is that correct?#2018-07-0612:27gfredericks(due to "only the last :paths is kept and others are dropped")#2018-07-0614:13ghadiyou can certainly start a different main method rather than clojure.main and give yourself a souped up repl @gfredericks#2018-07-0614:34gfredericks@ghadi but that requires passing args each time, right?#2018-07-0614:43ghadisi#2018-07-0614:43bronsasi`* :P#2018-07-0614:47ghadialias repl='java $(cat jopts) -cp $(clojure -Spath) clojure.main -m gfredericks.fave.repl'#2018-07-0707:13cflemingAs I understand it, aliases are never transitive. So this means that when you depend on a project using, say, :local/root then you will only ever see deps/paths from your dependency which appear in the top level of the deps.edn file, correct?#2018-07-0707:14dominicm@cfleming No transitive aliases, correct.#2018-07-0721:58martinklepschHey folks 👋 i have two questions: a) (how) does tools.deps support maven scopes? & b) is there a way to include dependencies with “provided” scope in the classpath? Thanks! 🙏#2018-07-0722:15Alex Miller (Clojure team)a) not supported and b) that’s not a thing you should do anyways#2018-07-0810:26cflemingb) is absolutely useful. I use it to compile Clojure code against the IntelliJ classes, but not have them included when I run.#2018-07-0810:27cflemingWhat would be the deps solution to that - two separate classpaths using aliases? I guess I’d need an alias for the IntelliJ classes and would have to use that when compiling but not running?#2018-07-0810:53dominicmaliases is how I've solved it up until now.#2018-07-0815:22Alex Miller (Clojure team)Yes#2018-07-0815:24Alex Miller (Clojure team)You seem to be describing using provided scope aliases in your own project (which is useful, but handled by aliases). The question I read above though sounded like how to include transitive provided-scope deps, which does not seem like a thing you should do#2018-07-0816:38martinklepschYeah, I need b) for a similar tooling related concern #2018-07-0816:53martinklepschI want to load code from Artifact A which has a “provided” dependency B. Now I’d like to make sure all these dependencies are loaded without manually listing those “provided” deps. Hope that makes sense#2018-07-0816:56Alex Miller (Clojure team)Well, the whole point of provided scope is: you provide them#2018-07-0816:57Alex Miller (Clojure team)So, not going to make something to do this#2018-07-0817:31dorabIs resolve-deps supposed to download the relevant dependencies if I call it myself?#2018-07-0817:31dorab{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "5f3b302d3e3d1f3b302d3e3d72102f2b360f333a2772666f6d6f"}, :content ("[email protected]")}#2018-07-0817:36dorabIf it is supposed to work, what am I doing wrong above?#2018-07-0817:37dorabBTW, this is in a directory (my home dir) that does not have a deps.edn in it.#2018-07-0821:24Alex Miller (Clojure team)You need to provide the :mvn/repos to look in (these are in the install deps.edn with clj)#2018-07-0821:53dorabAh. I thought those would automatically be included. So, I should create the full deps.edn map myself and then pass that to resolve-deps. Thanks.#2018-07-0901:40dnolenseems like there’s a bug with git deps where top-level :paths applies instead of the defaults or whatever is supplied in the git deps.edn#2018-07-0903:22Alex Miller (Clojure team)Yes, there is a ticket for it#2018-07-0911:07dnolenah k#2018-07-0913:04dominicmWorse than the silently ignoring problem, I seem to have encountered a case where all aliases are ignored if one of them is not present.#2018-07-0913:06Alex Miller (Clojure team)I’m planning to look at this today#2018-07-0913:12dominicmfwiw, they weren't all ignored. Just no fetching took place. I haven't looked hard at this, but it confused me.#2018-07-0920:40kennyI seem to be running into an issue with local deps. If I have this directory structure
app
lib1
lib2
with app/deps.edn as
{:deps {lib2 {:local/root "lib2"}}}
and lib1/deps.edn as
{:deps {}}
and lib2/deps.edn as
{:deps {lib1 {:local/root "../lib1"}}}
and I run clj -Spath in the app directory, I get this message:
Error building classpath. Manifest type not detected when finding deps for lib2/lib2 in coordinate #:local{:root "lib2"}
Are nested local dependencies supposed to work?#2018-07-0922:24Alex Miller (Clojure team)Shouldn’t your app one have ../lib2 ?#2018-07-0922:47kennyYou're absolutely right. That's what I get for working while hungry...#2018-07-0922:51hagmonkOh wow, I came here to report the same thing, except I've had lunch so I know it's a bug :)#2018-07-0922:53kennyI think I'm hitting the same problem with a slightly different directory structure. Except this time it is correct.
lib1
lib2
samples/sample1
with lib1/deps.edn as
{:deps {}}
and lib2/deps.edn as
{:deps {lib1 {:local/root "../lib1"}}}
and samples/sample1/deps.edn as
{:deps {lib2 {:local/root "../../lib2"}}}
and I run clj -Spath in the samples/sample1 directory, I get this message:
Error building classpath. Manifest type not detected when finding deps for lib1/lib1 in coordinate #:local{:root "../lib1"}
#2018-07-0922:53hagmonkYeah same issue:
./libs/dep-b
{:deps {org.clojure/clojure {:mvn/version "1.9.0"}}}
./libs/dep-a
{:deps {dep-b {:local/root "../dep-b"}}}
./apps/foo-app
{:deps {dep-a {:local/root "../../libs/dep-a"}}}
#2018-07-0922:55hagmonkI can work around it by having libs/dep-a refer to ../../libs/dep-b instead, but that is only a coincidence in this case, if apps was one level deeper it wouldn't work#2018-07-0922:57hagmonklooks like it's already reported here: https://dev.clojure.org/jira/browse/TDEPS-74#2018-07-0922:59kennyOh ok. That's a nasty one.#2018-07-0922:59hagmonkpush the "vote" button :){:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-07-0923:03kennyIt looks like the user in that issue is able to get output from clj -Spath though.#2018-07-0923:03kennyBut it sounds identical. Perhaps an older version of tools-deps didn't have this error message.#2018-07-0923:07hagmonkI thought :deps/root might be another way to dig oneself out of this hole, but perhaps I'm misunderstanding how that works#2018-07-1012:48pesterhazyI know that you can specify local exclusion in deps.edn like so "reagent {:mvn/version "0.7.0", :exclusions [cljsjs/react-dom cljsjs/react cljsjs/create-react-class cljsjs/react-dom-server]}" Is there a way to specify global exclusions as well?#2018-07-1014:41Alex Miller (Clojure team)No#2018-07-1204:18cflemingI’m working on some tools.deps testing, and I can’t figure this out. I’m calling resolve-deps like this:
(resolve-deps
{:deps {'local {:local/root "/Users/colin/dev/tools-deps-test/library"}}}
nil)
#2018-07-1204:18cflemingThat deps.edn looks like this:
~/d/cursive (deps)> cat /Users/colin/dev/tools-deps-test/library/deps.edn
{:deps {org.clojure/tools.gitlibs {:git/url ""
:sha "cab6abf0122cffb9e6b947c838458c69a0992031"}}}
#2018-07-1204:19cflemingWhat I’m getting back in my lib map includes this:
local {:local/root "/Users/colin/dev/tools-deps-test/library",
:deps/manifest :deps,
:deps/root "/Users/colin/dev/tools-deps-test/library",
:paths []},
#2018-07-1204:19cflemingI can’t figure out why the :paths vector is empty here.#2018-07-1204:19cfleming/Users/colin/dev/tools-deps-test/library/src exists.#2018-07-1204:41cflemingUgh, never mind me, I’m not merging in my config maps.#2018-07-1204:41cflemingThis works:
(let [deps (reader/merge-deps [(reader/slurp-deps "/usr/local/Cellar/clojure/1.9.0.381/deps.edn")
(reader/slurp-deps "/Users/colin/.clojure/deps.edn")
(-> {:deps {'local {:local/root "/Users/colin/dev/tools-deps-test/library"
:deps/manifest :deps}}}
(#'reader/canonicalize-all-syms))])]
(resolve-deps deps nil))
#2018-07-1215:38kennyIt seems tools-deps suffixes all Git deps with each path in my :paths vector. If I have ["src" "foo1" "foo2"] for my :paths, the output of clj -Spath will suffix each Git dependency with each path. i.e. /Users/kenny/.gitlibs/libs/lib1/lib1/f31e19ffe371a60f0474759451dba9b88bf7ef04/src:/Users/kenny/.gitlibs/libs/lib1/lib1/f31e19ffe371a60f0474759451dba9b88bf7ef04/foo1:/Users/kenny/.gitlibs/libs/lib1/lib1/f31e19ffe371a60f0474759451dba9b88bf7ef04/foo2. Is this intentional?#2018-07-1215:39dominicmknown bug#2018-07-1215:40kennyIs there an issue?#2018-07-1215:42dominicmThere is, although I'm not sure of the ref for it#2018-07-1215:48kennyI believe it's https://dev.clojure.org/jira/browse/TDEPS-52#2018-07-1219:24aviI got bit by that yesterday, working with a local lib. My app has main test-runner etc under src but the lib doesn’t have main so none of the clj files could be found by require/load#2018-07-1219:25aviAh no, my problem was slightly different#2018-07-1219:25aviAlmost the inverse#2018-07-1219:25avi(Just read the issue more closely)#2018-07-1219:25aviI’ll try to check whether an issue for my experience already exists.#2018-07-1302:22dnolenare :extra-deps actually merged? like if in an alias I want :exclusions but not otherwise?#2018-07-1302:32Alex Miller (Clojure team)I’m not sure I understand what you’re asking but it sounds like :override-deps#2018-07-1302:33dnolenoh! cool 🙂#2018-07-1302:35dnolenthanks#2018-07-1312:21martinklepschNot sure if a bug but clj -Sdeps '{:deps {org.asciidoctor/asciidoctorj {:mvn/version "1.5.6"}}}' -Spath doesn’t pull in the dependencies below (lein/boot do)
[org.asciidoctor/asciidoctorj "1.5.6"]
├── [com.beust/jcommander "1.35" :scope "runtime"]
└── [org.jruby/jruby-complete "1.7.26" :scope "runtime"]
#2018-07-1312:26dominicmRuntime scope dependencies aren't or weren't brought in. I think they are now. #2018-07-1611:45martinklepschDo I need to manually upgrade tools-deps for that to take effect somehow?#2018-07-1611:45martinklepschsorry noob q 😄#2018-07-1612:08dominicmhttps://dev.clojure.org/jira/browse/TDEPS-26#2018-07-1612:10dominicmI guess you bump your version? Not sure when the last release was#2018-07-1321:35dnolenI guess there’s no error on non-existent alias?#2018-07-1321:36Alex Miller (Clojure team)Ha, I just committed that within the last hour :)#2018-07-1321:36Alex Miller (Clojure team)Will be in next release{:tag :div, :attrs {:class "message-reaction", :title "bananadance"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "bananadance", :src "https://emoji.slack-edge.com/T03RZGPFR/bananadance/5394a2df1be70a15.gif"}, :content nil})} " 4")}
#2018-07-1322:23rbergerIf one’s ssh key for github is not id_rsa or any of the default ssh names, do you need to specify the name somewhere in deps.edn? I can git clone a private dependency repo with my keys in the ssh-agent, but the clojure command always ends up with:#2018-07-1322:23rbergerError building classpath.
#2018-07-1322:41rbergerThis is on macos High Sierra and Clojure 1.9.0.381#2018-07-1323:29ghadiThis is an awful answer, and it's a bit of a mess, and I'd really like to fix it once and for all by shelling out to git directly (golang does this):
@rberger
for now, make sure the environment SSH_AUTH_SOCK is set to your ssh-agent's socket path
move any extraneous Identity keys in your .ssh/config out of the way#2018-07-1323:29ghadiJGit / JSch tries to read them and gets confused#2018-07-1323:29ghadi(I use ed25519 keys, which are common these days...only supported via the ssh agent)#2018-07-1323:39rbergerThanks! It looks like I had some things in the ~/.ssh/config that was causing the problem. Might be good to at least mention this in the docs (https://clojure.org/reference/deps_and_cli)#2018-07-1323:41rbergerThough the info in the config file is the correct path to the IdentifyFile.
Host
User git
Port 22
HostName
IdentityFile ~/.ssh/github.rsa
AddKeysToAgent yes
TCPKeepAlive yes
IdentitiesOnly yes
#2018-07-1514:01alvinathere is command line tool for finding latest version of the software#2018-07-1514:01alvinacan some body remind me the name of that command?#2018-07-1516:28dorabWill https://github.com/Olical/depot work for you?#2018-07-1520:11pesterhazy@rumaysaalvina2 @hagmonk's https://github.com/hagmonk/find-deps{:tag :div, :attrs {:class "message-reaction", :title "smile"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😄")} " 4")}
#2018-07-1600:37alvinathanks a lot#2018-07-1608:14levitanongHi all, trying to get the clojure:tools-deps docker image to work with gitlab-ci. Does anyone have experience in the matter?#2018-07-1609:16Andreas LiljeqvistI have never used gitlab-ci, but here is my .yml for running on bitbucket.
I imagine it would look quite similar.
https://pastebin.com/SE1shJE3#2018-07-1610:42delaguardoWhat kind of problems you are seeing?#2018-07-1611:07levitanong@U04V4KLKC clj: command not found. Same with clojure. I’ve tried purposefully misspelling the docker image to see if there’s a distinct error when the docker image isn’t found, but it appears gitlab-ci just silently fails, and proceeds to run as if from a blank image.#2018-07-1611:08levitanong@U7YG6TEKW Yeah, mine looks similar, except gitlab complains that the clojure command isn’t found.#2018-07-1611:15delaguardoclj should is not available cause readline is not installed. But clojure should be there.#2018-07-1611:16delaguardoCould you show your config file?#2018-07-1611:21delaguardoAlso I can suggest to use clojure:tools-deps as your image so it is hard to misspell the name#2018-07-1611:28levitanongtest:tools-deps:
image: clojure:tools-deps-alpine
script:
- clojure -A:test
#2018-07-1611:29levitanongSkipping Git submodules setup
$ clojure -A:test
bash: line 52: clojure: command not found
ERROR: Job failed: exit status 1
#2018-07-1611:33delaguardoThat is odd… My config file looks a bit different, but it should not be a problem#2018-07-1611:34levitanongOne of the first things I tried was:
image: clojure:tools-deps
test:
script:
- clojure -A:test
#2018-07-1611:34levitanongsame error 😮#2018-07-1611:37delaguardoWhich version of gitlab-runner you are using?#2018-07-1611:38levitanongRunning with gitlab-ci-multi-runner 9.5.1 (96b34cc)#2018-07-1611:41delaguardoI’m using 11.0. Let me downgrade, it can be a problem#2018-07-1611:43levitanongooh#2018-07-1612:00delaguardoUnfortunately, I can not downgrade to 9.5.1 on my mac. Looks like a bunch of dependencies conflicts there. Is it possible for you to upgrade yours and try again?#2018-07-1612:02levitanongHmm. I’m working off of my company’s on-prem installation of gitlab. I’ll ask the dev-ops guy if he can upgrade it. I’ll keep you posted!#2018-07-1614:12levitanongApparently our devops guy needs to backup all our repos over the weekend to upgrade gitlab. I’ll prolly have to get back to you in a week. Good thing lein still works!#2018-07-2403:43levitanonghello, tried again on gitlab-runner 11.1.0. Still says clojure: command not found. Do you think it’s because we’re using the on-prem community version of gitlab?#2018-07-2406:28levitanongOop, scratch that. we had a faulty config somewhere else. Works now, thanks!#2018-07-2406:44delaguardoNice! parrot #2018-07-1609:58darnokHi. I have a problem w tools-deps. I'm using :local/root quite extensively and I noticed a problem. If I have a project project1 that points to another library with {:local/root "../../libs/abc"} and then that library uses another library {:local/root "../other-lib" it doesn't resolve the abc library classpath correctly.#2018-07-1610:00darnokIt resolves the second local/root according to the project1 root.#2018-07-1610:44dominicmyeah. There is a JIRA for that.#2018-07-1610:45dominicmI've been able to "flatten" everything up until now though.#2018-07-1611:31darnokI think it should be fixed https://dev.clojure.org/jira/browse/TDEPS-74. Probably, it's not that easy to fix.#2018-07-1611:45martinklepsch# clj
Please install rlwrap for command editing or use "clojure" instead.
# echo $?
0
has changing the status code here to some non-zero value been discussed before?#2018-07-1612:08delaguardoI remember such discussion but can’t find an issue. But I’m definitely want to vote for it#2018-07-1612:44Alex Miller (Clojure team)no ticket, but feel free to log on and/or provide a patch#2018-07-1613:44martinklepsch@U064X3EF3 can you point me to the source of the clj clojure scripts maybe? thanks 🙂#2018-07-1613:45Alex Miller (Clojure team)Tickets go to https://dev.clojure.org/jira/browse/TDEPS, patches for that stuff will be on https://github.com/clojure/brew-install (scripts are at src/main/resources)#2018-07-1613:55martinklepsch@U064X3EF3 https://dev.clojure.org/jira/browse/TDEPS-86#2018-07-1612:04stathissiderishello, is there some special magic to make git deps from private repos work? I’m using the {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "03646a7743646a776b76612d606c6e"}, :content ("[email protected]")} URL and auth fails#2018-07-1612:06delaguardoYesterday was discussion about it - https://clojurians.slack.com/archives/C6QH853H8/p1531520620000066#2018-07-1612:07stathissideristhanks! I’ll have a look#2018-07-1713:16martinklepschIs there a way to specify an entirely different set of :deps for an alias? (i.e. no merge, no override, just different value)#2018-07-1713:21dominicm@martinklepsch in what context?#2018-07-1713:23martinklepsch@dominicm not sure I understand what you mean by context? 😄#2018-07-1713:32martinklepschI’d like to swap out the value of :deps in the top level map by another value via an alias#2018-07-1713:54dominicmSo from the clj command line use? What's the use case?#2018-07-1714:11ghadi:override-deps should do it, but I think there is an acknowledged bug when the override is of a different "type" (mvn/git)#2018-07-1714:24ghadihttps://dev.clojure.org/jira/browse/TDEPS-51#2018-07-1714:58dominicm@ghadi I think @martinklepsch means that he wants to have:
{:deps {A {:mvn/version "1.1.0"}}
:aliases {:B {:the-new-deps {B {:mvn/version "5.4.1"}}}}}
And when doing clj -A:B that should result in only B being on the classpath#2018-07-1714:59ghadioh#2018-07-1714:59martinklepschyeah, that’s it @dominicm#2018-07-1715:00ghadiI think this has come up before -- not supported by aliases, but perhaps by a different deps.edn file#2018-07-1715:00dominicmI don't understand why you'd practically need that though 🙂#2018-07-1715:00dominicmI suppose you could actually have:
{:deps {}
:aliases {:A {:extra-deps {A …}} :B {:extra-deps {B …}}}}
#2018-07-1715:01martinklepschyeah aliases would work I think#2018-07-1715:02martinklepschI’m not 100% sure I need it either but I come up with some approaches and have no idea what tools-deps allows for so just asking 😄 sorry for the noise 🙂#2018-07-1715:13ghadino problem @martinklepsch not noise at all#2018-07-1715:13ghadiIt does seem a bit "off-label prescription"#2018-07-1715:13ghadi(the use case)#2018-07-1715:31martinklepsch@ghadi cool, thanks 🙂#2018-07-1715:32martinklepschIf I specify a deps.edn with -Sdeps any local deps.edn in the working directory should be ignored — is that correct?#2018-07-1715:32dominicmNo, though others have tripped over that wording too.#2018-07-1715:33martinklepschuh, yeah, especially that “final” part: “Deps data to use as the final deps file”#2018-07-1715:37martinklepschis there another way to achieve it then? specifying a file would also be fine, just looking for something that ignores an existing deps.edn#2018-07-1716:53dominicmFinal in the context of merging together deps.edn files 🙂#2018-07-1717:45martinklepschHm, the word “final” creates a different expectation for me 😄
So is the only way to run a clj process with a deps.edn that’s different than the one in the current working directory to move files around?#2018-07-1718:01dominicmI think the proposal was to change it to last.#2018-07-1718:02dominicmNot that I know of. I think pack is unique in utilizing tools.deps.alpha in that way, and I'm not sure it really works due to the cwd being internal to t.d.a rather than passed in. #2018-07-1718:11martinklepschMaybe another option could be added#2018-07-1718:20martinklepschhttps://dev.clojure.org/jira/browse/TDEPS-87#2018-07-1718:57Alex Miller (Clojure team)-Sdeps!#2018-07-1719:23dominicmHow well does that work in shells? #2018-07-1722:38Alex Miller (Clojure team)I was actually kidding :)#2018-07-1805:14dominicmVim wasn't (for its internal commands). #2018-07-1808:13dominicmhttps://github.com/nrepl/nREPL/issues/8 ah, that explains why add-lib doesn't work with nrepl. I was hoping to patch nrepl, but it seems harder now.#2018-07-1808:17dominicmOh actually, when add-lib refers to the "highest" class loader, I guess that means it would find the parent one if I set it, so I could fix this in nrepl#2018-07-1809:53dominicmhttps://github.com/nrepl/nREPL/pull/35 resolved! (pr merged)#2018-07-1808:40martinklepschUnder what conditions is it expected that a user has to manually delete .cpcache?#2018-07-1809:54dominicmbugs? 😄 What happened?#2018-07-1809:54dominicmI wouldn't expect to ever have to from what I know, fixes are being made to handle cases where it isn't being updated correctly#2018-07-1810:17pesterhazy@U050TNB9F I think this is an interesting question#2018-07-1812:24Alex Miller (Clojure team)Generally using -Sforce to force recomputing a single classpath makes more sense. One known case where this is required is when local transitive deps change their dep structure - right now that won’t be automatically discovered#2018-07-1813:00martinklepsch> local transitive deps change their dep structure
that might have been it then#2018-07-1814:46Alex Miller (Clojure team)I have a plan to fix this, but it’s kind of involved#2018-07-1809:53dominicmhttps://github.com/nrepl/nREPL/pull/35 resolved! (pr merged)#2018-07-1810:22martinklepschWith git-libs is there a way to specify another “root” path to look for a deps.edn in?#2018-07-1810:22martinklepschI remember this being discussed previously but can’t recall#2018-07-1810:22dominicmYep! I think :deps/root is the key#2018-07-1810:50martinklepschError building classpath. Manifest type not detected when finding deps for codox/codox in coordinate {:git/url "", :sha "ac6c83a4c7d334bbc27a1bad4240ed7e5fa04ff2", :deps/root "codox/"}
Hrm, I must be missing something. For some reason manifest here seems to be nil: https://github.com/clojure/tools.deps.alpha/blob/84826fef06c725dac0c3d63cfbfd0916553b7cf9/src/main/clojure/clojure/tools/deps/alpha/extensions/git.clj#L45-L47#2018-07-1810:51dominicmdo you have a deps.edn file?#2018-07-1810:51dominicmor pom.xml#2018-07-1810:51cflemingDoesn’t :deps/manifest only support :deps at the moment?#2018-07-1810:52cflemingi.e. deps only knows how to extract path info from other deps projects?#2018-07-1810:53dominicmI think there's a :deps/manifest :mvn implementation#2018-07-1810:55martinklepschduh, I thought I commited the deps.edn file but didn’t 🙈#2018-07-1822:49zentropeI’ve got a github repo with mname/foo with a clojure project inside it, mname/foo/clojure-project/: is it possible to use that for a deps.edn git dependency?#2018-07-1822:50zentropeAnswer seems to be “no” as I can’t reference a github URL + subdir.#2018-07-1822:51zentropeOh, is that the :deps/root thing I see up above there? Hm.#2018-07-1822:55zentropeGetting a “don’t know how to create ISeq” from the -m namespace part of the CLI, but at least it seems to find the right subdirectory.#2018-07-1822:55zentropeOops, typo. Nevermind.#2018-07-1822:55zentrope👍#2018-07-1903:24cflemingCursive support for deps is out!: https://groups.google.com/d/topic/cursive/32tfBegzfVc/discussion#2018-07-1903:25cflemingStill lots to do and lots of potential changes.{:tag :div, :attrs {:class "message-reaction", :title "clap"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👏")} " 52")}
#2018-07-1908:31claudiuHi, is there any tools-deps alternative for https://github.com/luminus-framework/lein-uberwar ?#2018-07-1908:32dominicm@claudiu I'd be interested in adding uberwar support to https://github.com/juxt/pack.alpha I'm not familiar with the format really though and what would be needed. Is it just a jar with a particular manifest?#2018-07-1909:11claudiunot really sure to be honest. 🙂 I am deploying my test-app to google-appengine, and that's the format there.#2018-07-1909:12claudiuAdded lein-tools-deps to my project, seems to work for development but the uberwar seems to compile multiple times the same file, and when I run it I get some strange errors.#2018-07-1909:19dominicmI can try find some time to look into it for you.#2018-07-1909:23dominicmLooks like a jar with a specific fs layout, shouldn't be too difficult to get working with pack, I'm a little worried about git dependencies, that might need some testing.#2018-07-1910:13claudiuCool 🙂 Could help with testing git & normal, and see how it matches the lein one.#2018-07-1911:24claudiuHaving the uberwar option would be really awesome.#2018-07-1917:28dominicmIs there an easy way to test an uberwar locally?#2018-07-2009:15claudiuI have only used uberwar in appengine context. The simplest way to test the war I can think of is : https://gist.github.com/claudiuapetrei/540c8aca1ef53d86f75f123b319a3703#2018-07-2010:06dominicmThanks, I'll take a look#2018-07-1915:44Alex Miller (Clojure team)released tools.deps.alpha 0.5.442, clj 1.9.0.391{:tag :div, :attrs {:class "message-reaction", :title "pray"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙏")} " 28")}
#2018-07-1915:45Alex Miller (Clojure team)- Throw error if unknown aliases are used (TDEPS-85)
- Fix bad break character in rlwrap (TDEPS-77)
- Use non-0 exit code in clj if rlwrap doesn't exist (TDEPS-86 - thanks Martin Klepsch!)
- Change wording for -Sdeps in help and man for clarity (TDEPS-87)
#2018-07-2000:09cflemingI’m testing deps with Cursive, and I’m finding that resolving several large projects at once is really slow. This is a project with several apps in it, all of which use a series of libraries transitively via ‘:local/root’. Is it possible that each project refresh works out a complete list of what to download up front and then downloads it, such that each project will download everything even if another project has concurrently downloaded that dep in the meantime?#2018-07-2000:14cflemingIn total there are 15 projects (including apps + libs), it uses kafka so who knows how much each one will download. But I’m on a 1 Gb fibre connection, so it’s generally pretty quick. It’s taking around 15-20 minutes to resolve them all.#2018-07-2000:32Alex Miller (Clojure team)It uses Maven to download as it goes, which uses the Maven cache#2018-07-2000:33Alex Miller (Clojure team)That said, I have spent no time optimizing any of that and none of it is parallel (but most likely some could be)#2018-07-2000:34cflemingRight, my case is probably unusual in that if you set up a large project in Cursive (which I just did based on data a user sent me) then IntelliJ will go off and resolve all the projects in parallel.#2018-07-2000:40cflemingDoes the maven lib separate the resolution into a resolution step and a download step? I’m wondering if each project does the resolution step, decides what to download and then downloads it without checking before each download if something else has done so in the meantime?#2018-07-2012:59mpingHi, is there a recommended way to integrate lein with tools-deps? I want to use git dependencies#2018-07-2013:09andre.stylianosThe only one I know: https://github.com/RickMoynihan/lein-tools-deps#2018-07-2013:09mgrbyte@mping Not sure about "recommended", but I'm using lein-tools-deps - works pretty well so far. Only things that don't work are lein deps :tree, lein search AFAICT#2018-07-2013:09mping@mgrbyte 👍#2018-07-2015:31hlolliis there a function in tools.deps that I could use to generate a vector of maps containing all remote dependencies for a project's dependencies, plus, if possible, the sha-sums of the .jar files. I'm wiring a deriviation for nix(Os), and it seems that there's no tool for generating nix expression for clojure dependencies available. There's one for maven, called mvn2nix. I guess I could use that, but I'd have to make some hack for mirrors, adding clojars etc..#2018-07-2015:32dominicmyeah, I think that would be possible via the api#2018-07-2015:33dominicmthe api is small, call both of the documented functions#2018-07-2015:33hlollimake-classpath and resolve-deps? I will try!#2018-07-2015:35dominicmmaybe not make-classpath actually#2018-07-2015:35dominicmjust resolve-deps & vals#2018-07-2015:53rickmoynihan@alexmiller: should the tools.deps library be including slf4j-nop?
https://github.com/clojure/tools.deps.alpha/blob/744c9c027b1b0d71081a7903b517abbd960a9849/deps.edn#L15{:tag :div, :attrs {:class "message-reaction", :title "eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👀")} " 4")}
#2018-07-2016:49hlolli@dominicm thanks for the tip, I'm getting all the deps printed, which is good. But that which is missing, is the url's to the .jar files
fipp
{:versions
{#:mvn{:version "0.6.12"}
{:mvn/version "0.6.12", :deps/manifest :mvn}},
:paths {#:mvn{:version "0.6.12"} #{[]}},
:select #:mvn{:version "0.6.12"},
:pin true},
probably straightforward to do, but how would I get the download urls from this information?#2018-07-2016:51dominicm@hlolli I think there's something missing here, you've called resolve-deps?#2018-07-2016:51hlolliyes I did this
(prn
(pprint (resolve-deps '{:deps
{org.clojure/clojure {:mvn/version "1.9.0"}
org.clojure/clojurescript {:mvn/version "1.10.238"}
org.clojure/tools.reader {:mvn/version "1.3.0-alpha3"}
com.cognitect/transit-cljs {:mvn/version "0.8.256"}
malabarba/lazy-map {:mvn/version "1.3"}
fipp {:mvn/version "0.6.12"}}}
{:verbose true}
)))
#2018-07-2016:55dominicmSomething looks off about your result to me. The value of :paths for me is different, I have jars in there.#2018-07-2016:55dominicmor dirs, context dependent.#2018-07-2016:57hlolliah yes, if the jar is found, it appears#2018-07-2016:57hlollibut only locally#2018-07-2016:57hlollicom.google.protobuf/protobuf-java
{:mvn/version "3.0.2",
:deps/manifest :mvn,
:paths
["/home/hlolli/.m2/repository/com/google/protobuf/protobuf-java/3.0.2/protobuf-java-3.0.2.jar"],
:dependents [com.google.javascript/closure-compiler-unshaded]},
#2018-07-2017:01dominicmYou want the remote values?#2018-07-2017:03hlolliyes, I need that, and the sha sums, but I could just get them after all is downloaded#2018-07-2017:07hlollithe pain with nix, is that nothing can be downloaded without being declared, all http fetches need to be done on nix's side#2018-07-2017:08dominicmI don't think you can get the remote urls, I think it's blackboxed into a maven library#2018-07-2017:09hlollimaybe, you can always download jars directly from clojars, maybe maven is more blackboxed#2018-07-2017:10dominicmI meant maven library which handles clojars & maven#2018-07-2017:10dominicmt.d.a has no concept of "get-url-for-dep"#2018-07-2017:10dominicmYou'd probably have to utilize the same approach that mvn2nix uses, and also write equivalents for git & local deps.#2018-07-2017:12hlollipff, hope its easy 🙂 will research.#2018-07-2022:31noprompthas anyone run in to problems with using krypton and git deps?#2018-07-2022:32nopromptclj complains it can’t find the ~/.ssh/id_krypton private key which it shouldn’t because, as i understand it, krypton has that key.#2018-07-2105:32dominicm@noprompt I imagine jgit doesn't support krypton#2018-07-2109:39dominicm@alexmiller @ghadi curious whether you've looked into libgit and whether it could be used in place of jgit to solve this problem? Maybe before falling back on jgit on any unsupported platforms?#2018-07-2109:40dominicm> Linux, FreeBSD, OpenBSD, Mac OS X, iOS, Amiga, MinGW and fully native Windows.
is pretty good though, "Amiga", like, come on!#2018-07-2109:40dominicmThere's a library, "jagged" for doing libgit w/ java, but it's not on maven.#2018-07-2113:32pesterhazyif it's fast enough, shelling out to /usr/bin/git seems optimal#2018-07-2113:36dominicmHaving tried to wrap libgit2 this morning, I'm thinking that shelling out is increasingly valuable. I'm wondering about environments which may not have the git binary available though.#2018-07-2113:36dominicmHaving libgit2 embedded for that situation is very valuable.#2018-07-2113:37dominicmOf course, /usr/bin/git falling back onto jgit or something would work for that pretty well#2018-07-2113:48pesterhazytrue, I didn't think of the case where the binary is not available (not unrealistic in this world of Docker)#2018-07-2118:10athomasoriginalDoes anyone know of examples of clj cli tool + deps.edn monorepo setups? I have been studying Juxt's Edge repo, but this does not yet include any other modules. My assumption is that one would create a :dev alias where, instead of referencing the artifact, it references the local path to your module...curious as to how other's approach this :thinking_face:#2018-07-2119:12ghadiWe need to shell out to git to solve all the ssh related issues#2018-07-2119:13ghadiAll the other git based deps tools do that, like golangs#2018-07-2119:24athomasoriginal@ghadi sorry, was this in reference to my question?#2018-07-2119:25ghadiSorry, earlier Q about libgit#2018-07-2119:40dominicm@ghadi libgit doesn't do the same thing as git? #2018-07-2120:59cflemingIntelliJ also shells out to Git for its support.#2018-07-2120:59cflemingThere’s probably a bunch of useful support code there if we need it (Apache licensed)#2018-07-2219:13uwoare :exclusions only for mvn deps, or do they work for git deps as well? Running clj -Stree made me think they were, because the exclusion list appeared to be applied to the dep tree#2018-07-2220:21Alex Miller (Clojure team)Should work for both#2018-07-2221:30uwoawesome thanks!#2018-07-2220:20hlolliSo I succeeded with using tools.deps for generating nix-expressions, https://github.com/hlolli/clj2nix now it should be easy for nix users to start bombarding clojure-based application into nixpkgs, or use nix to build/deploy clojure applications. If you're a nix user, please give it a try 🙂#2018-07-2221:57dominicm@hlolli how did you do it? #2018-07-2222:02hlolli@dominicm after looking at a nix function fetchMaven, I realized that all maven2 remote url's follow a pattern, from there I discovered fetchMavenArtifact, which I use, for both clojar and maven central. Just try both for each dep, and both clojar and maven central work with same url format.#2018-07-2222:03dominicm@hlolli do you handle git and local? #2018-07-2222:08hlolligood question. I don't handle it automatically. And it's possible to do it.#2018-07-2222:09hlolliAnother thing I need to add is classifier. Which I don't handle, or the funtcion fetchMavenArtifact doesn't handle it.#2018-07-2222:20dominicmI'm not certain if t.d.a handles classifier yet, I might be wrong#2018-07-2222:20hlolliI'm about to test it 🙂, curious.#2018-07-2222:57Alex Miller (Clojure team)It does{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-07-2222:59Alex Miller (Clojure team)With the caveat that classifiers do not distinguish in a lib (they’re part of the coord, not the lib key), so you can’t use multiple classifiers (or no classifier) for the same lib at the same time#2018-07-2223:00Alex Miller (Clojure team)It also only handles jars and ignores other extension types#2018-07-2310:42cflemingIs there a rough ETA on Windows/Powershell support?#2018-07-2314:21Alex Miller (Clojure team)Sorry, I just haven’t had time to pull it over the line yet. I expect it will be a couple weeks before I get back to it#2018-07-2321:14gfredericksI'm thinking about how to best replace all the tools that in leiningen mostly consisted of :injections, without creating a combinatorial explosion of mains
I had the idea that maybe that could be what a single user.clj could be for#2018-07-2321:15gfredericksI'm assuming tools.deps lets you add common source paths in the user's deps.edn, but I might have made that up#2018-07-2400:16gfredericksit works, at least:
:paths ["/home/memyself/.clojure/user-classpath"]#2018-07-2415:15timgilbertBest of luck with the Windows stuff. I've recently been running into this fiendish error with lein: https://stackoverflow.com/questions/201816/how-to-set-a-long-java-classpath-in-windows/201969#2018-07-2423:38souenzzoreal funny the kind of problem and solutions that windows users do in 2018.
WLS still not a thing?#2018-07-2513:37timgilbertWSL work pretty well, and you can run the ubuntu version of tools.deps on it with no problems. The pain comes when you are trying to interoperate with native Windows stuff - eg, IntelliJ runs natively, so it has its own notion of an m2 repository, classpath, etc{:tag :div, :attrs {:class "message-reaction", :title "confused"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😕")} " 4")}
#2018-07-2416:04ghadi@argument files
one or more argument files containing options
Java now accepts a file to read arguments ^{:tag :div, :attrs {:class "message-reaction", :title "thinking_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-07-2423:23gfredericksthe worst part is having to hard-code my home directory in the deps.edn; I'm not sure if there's a way to avoid that#2018-07-2500:25aviRelative paths? Paths that include things like ~ ? (Not sure what kind of thing ~ is. Alias?)#2018-07-2500:42gfredericksit's a bashism I believe#2018-07-2500:42gfredericks. and .. are a lot more first-class than ~#2018-07-2500:42gfredericksI can't use ~ because it's just bash, and I can't use . and .. because I assume that'd refer to the working directory of the java process, which could be anywhere#2018-07-2519:38pesterhazyAn equivalent of expanduser is easy to implement: https://stackoverflow.com/questions/6803913/java-analogous-to-python-os-path-expanduser-os-path-expandvars but not sure it’s a good idea#2018-07-2711:02gfredericksbut this is in a deps.edn file, where, unless I'm mistaken, there's no opportunity to execute code#2018-07-2809:32pesterhazyright, I meant as a potential feature for tools.deps. Many tools parsing yaml/json/ini configuration files process filenames with expanduser#2018-07-2812:35gfredericksah right#2018-07-2613:31mitchelkuijpersIs anyone here using this on circleci? I think it is a good idea to cache the .cpcache folder? Because I saw this in the reference guide:
The cached classpath file is used when:
* It exists
* It is newer than all deps.edn files
I would guess if i checkout the project and then restore a cache the .cpcache folder is always newer#2018-07-2613:31mitchelkuijpersHmm this is exactly what you want if I base the cache on deps.edn.. While I typed this I figured it out lol#2018-07-2614:04mitchelkuijpersIs there some place where it caches git dependencies? I am currently caching the .m2 folder and the .cpcache folder#2018-07-2614:26mfikes@mitchelkuijpers by default ~/.gitlibs#2018-07-2614:28mitchelkuijpersAh nice thnx#2018-07-2619:07Peter Wilkinshi, anyone know how to add a local maven repository to a deps.edn file?#2018-07-2619:11mfikes:mvn/local-repo#2018-07-2619:13mfikesSee https://github.com/clojure/tools.deps.alpha/blob/4f24e4ea6911e9a30a14979386f2b2cc3ad2ecc8/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj#L31-L35#2018-07-2619:16mfikesSo, for example, this will create and stash stuff in ./my-local-repo
clj -Sdeps '{:mvn/local-repo "my-local-repo"}'
{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-07-2711:09gfredericksis add-lib likely to get officialized anytime soon?#2018-07-2714:25Alex Miller (Clojure team)Define soon#2018-07-2715:09mikerodI’m seeing 2 definitions:
1) in or after a short time.
2) used to indicate one’s preference in a particular matter.
but more interesting, the origin:
Old English sōna ‘immediately.’
😛#2018-07-2715:58Alex Miller (Clojure team)Then I’ll go with no :){:tag :div, :attrs {:class "message-reaction", :title "laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😆")} " 4")}
#2018-07-2721:01ghadinotes on git deps with
https with authentication is not supported yet by clj (https is 65% of github's traffic)
ssh authentication via Java is a usability trashfire (some KEX methods unsupported)
The terminal is not interactive in clojure, so you cannot accept unseen public host keys - this might be ok, but surprising.
Esoteric .ssh/config options can break cloning because the underlying java ssh libraries misread the .ssh/config file
ed25519 keys supported via ssh-agent, but not via ~/.ssh/config entries
Porting to shell out to git would help all of these issues, if remains compatible with Windows support{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-07-2721:01ghadinotes I've previously pasted here ^#2018-07-2812:36gfrederickswould support for ~ in paths in tools.deps (or parts of it? e.g. just the user file) be a welcome feature? I'd be happy to work on it#2018-07-2812:47Alex Miller (Clojure team)My first reaction is no#2018-07-2812:47Alex Miller (Clojure team)But feel free to file a jira and I’ll think about it#2018-07-2814:02gfredericksroger thanks#2018-07-2815:26gfrederickscan git deps specify a subdirectory of the repo?#2018-07-2815:39gfredericksI'm making a library called user.clj and all it does is contain a user.clj file that loads other user files#2018-07-2815:39gfrederickse.g., ~/.clojure/user.clj, and it could probably search the classpath for user-<n>.clj#2018-07-2815:39gfredericksand you could disable it with env variables#2018-07-2815:42mfikesGit deps can point at a subdirectory. I can’t recall the key. Maybe :deps/root or some such {:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-07-2914:38gfredericksWelp, I did it.
Nobody told me not to.
https://github.com/gfredericks/user.clj#2018-07-2915:01dominicmAre they merged? Or something else? #2018-07-2915:13gfrederickswell right now it only supports a single file, so that's not an issue#2018-07-2915:14gfredericksif I can think of a performant way of searching the classpath for other files, then (once defined what's in scope (`user.autoload.*`?)) I'd probably just load them in alphabetical order#2018-07-2915:14gfredericksI don't think merely saying "all user.clj files will be loaded!" is possible because AFAIK clojure just loads an arbitrary one, so there's no guarantee it would be mine#2018-07-2915:15gfredericksso the whole approach assumes that the rest of your user files are named something different#2018-07-2915:15gfredericks(note that ~/.clojure/user.clj would not, in general, be on the classpath)#2018-07-2915:19dominicmWhat does the library do then? I'm a bit confused! #2018-07-2915:19dominicmI thought this was a new version of ./util#2018-07-2915:20gfredericksoh no, not at all#2018-07-2915:20gfredericks> The immediate problem it's meant to solve is that the user-level deps.edn file can't reference $HOME-relative paths as a way of providing your own user.clj directly.#2018-07-2916:39dominicmI see... So this is an alternative to having an alias in your own ~/.clojure/deps.edn pointing at a dir?#2018-07-2916:57gfredericksyes, especially because doing it that way means you need an absolute path#2018-07-2916:57gfredericksso there seems to be no way to make it relative to $HOME#2018-07-3002:37cflemingWhy does t.d.a have the org.clojure/clojure dep specified as provided?#2018-07-3002:57seancorfieldThat seems correct to me: it's always going to be run in a context that already has some version of Clojure loaded. #2018-07-3002:58seancorfield(shouldn't most Clojure projects assume Clojure is "provided"?)#2018-07-3004:09richiardiandreaThe only exception IMHO is cljs self-host which would not require any dependency on clojure#2018-07-3004:45cfleming> shouldn’t most Clojure projects assume Clojure is “provided”?
Maybe, but none of them do.#2018-07-3004:48cflemingIt means that if I get the classpath for t.d.a from Maven or whatever, it won’t contain Clojure, which it clearly needs.#2018-07-3005:02seancorfieldYeah, I've tried to drum up support for a movement to change the Clojure dep to "provided" in projects but... inertia... 🙂#2018-07-3005:20cflemingThinking about it though, none of those libs work without Clojure so it really is a dependency for them.#2018-07-3005:31seancorfieldIt can be argued both ways. Clojure itself is kind of a weird "dependency" because in most projects, the choice of Clojure version comes first and then all the other dependencies are added. So, in that sense, Clojure is "provided". But, yeah, you can also argue Clojure "really is a dependency" -- but it's often going to be a different version for each of the other dependencies...?#2018-07-3005:33Alex Miller (Clojure team)I think you have to separate app and lib contexts here too#2018-07-3005:33Alex Miller (Clojure team)Apps should def declare#2018-07-3005:34Alex Miller (Clojure team)Generally you don’t want your libs deciding the Clojure version of the context they’re used in#2018-07-3005:34seancorfieldYeah, I was kinda coming at this from the app p.o.v.: it chooses the Clojure version and the libs just deal with it 🙂#2018-07-3005:35Alex Miller (Clojure team)Which is effectively what happens#2018-07-3005:35Alex Miller (Clojure team)As top deps have priority#2018-07-3005:36Alex Miller (Clojure team)So in app context, their choice is most important#2018-07-3005:36seancorfieldPretty much all the libs out there should use "provided" with Clojure and apps then drive it. But that doesn't seem to get much traction 🙂#2018-07-3005:37Alex Miller (Clojure team)Yeah, I don’t think that’s a battle worth fighting#2018-07-3005:58cflemingBut t.d.a is an app, right?#2018-07-3006:01seancorfieldI've only ever used it as a library...#2018-07-3006:06seancorfieldThe clj script invokes one of the -main functions in t.d.a. tho' (resolve tags, generate manifest, make classpath) but I'm not sure I'd consider tho' an app tho'?#2018-07-3006:42Alex Miller (Clojure team)It’s both in this case I think#2018-07-3006:44Alex Miller (Clojure team)But brew-install builds an uberjar so it’s really the app-maker#2018-07-3016:39kennyHow are you guys debugging dependency conflicts?#2018-07-3016:40ghadiclojure -Stree for a start @kenny#2018-07-3016:41kennyDoesn't show the decision made in selecting the dependency.#2018-07-3016:45ghadifrom what I've seen: the chosen dependency should only appear once in that tree, and its only parent is the driver of version selection#2018-07-3016:47kennyCorrect. But I'd like to know the other parent dependencies that also have the conflicting child dependency.#2018-07-3019:36Alex Miller (Clojure team)There is a hidden verbose mode that sheds some additional light#2018-07-3019:38Alex Miller (Clojure team)clj -Sdeps ‘{:aliases {:v {:verbose true}}}' -A:v#2018-07-3019:39Alex Miller (Clojure team)Likely to be confusing to some degree as it’s mostly designed to let me debug stuff right now#2018-07-3019:40Alex Miller (Clojure team)I make zero promises about it’s existence or operation :)#2018-07-3019:44j0nihehe do I detect a tiny hint of revelation regret 😄#2018-07-3020:13Alex Miller (Clojure team)Just making it clear that is a not even alpha feature :)#2018-07-3022:56gfredericksI think for that we have to drop down to the hebrew alphabet and call it an aleph feature#2018-07-3023:17cflemingIf we require people to do clj -א:v I’m pretty sure not many will find it.{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 12")}
#2018-07-3023:19gfredericksperfect#2018-07-3023:43cflemingSeveral Cursive users have requested that Cursive show alongside the available aliases whether they’re system, user or project aliases. I can’t do that at the moment because there’s no way to tell which config file is which (short of something awful like guessing based on which is under the user’s home or something).#2018-07-3023:44cflemingOne thing that would fix that would be if -Sdescribe returned:
:config-files { :system "/usr/local/Cellar/clojure/1.9.0.381/deps.edn" :user "/Users/colin/.clojure/deps.edn" }
{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-07-3023:44cflemingAny love for that idea?#2018-07-3023:49seancorfieldChanging -Sdescribe at this point will break a bunch of other tooling that relies on the format of the (clojure-env) call I think?#2018-07-3023:49gfredericksstill alpha!#2018-07-3023:50cflemingNo doubt.#2018-07-3023:50seancorfieldAlso, there's no reason that t.d.a. is limited to those three specific EDN files -- you can merge any number of EDN files to produce your environment...#2018-07-3023:50cflemingI guess following Rich’s latest talk that should be :config-files2#2018-07-3023:50seancorfield🙂#2018-07-3023:50cflemingYou could put any number of keys in that map too.#2018-07-3023:51gfredericks-Sdescribe2#2018-07-3023:51cflemingI love it.#2018-07-3023:51ghadiit's not a breaking change at all#2018-07-3023:51ghadiyou cannot change :config-files but you could enrich the map{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-07-3023:52seancorfield:config-file-map 🙂#2018-07-3023:53ghadiI ran into only being able to do -Sdeps once per invocation today#2018-07-3023:53cflemingHehe, or :config-file-type indexed the other way around, so the file path goes to a keyword describing its type.#2018-07-3023:53ghadithis is related to the "how many config files are being merged?" question#2018-07-3106:21dominicm@cfleming I've opened a ticket for this. It was a map at some point. #2018-07-3106:50Alex Miller (Clojure team)Currently the config files are treated generically by the code as a list of dep sources. We now have several places that need to know the semantic meaning of different config files so they can be included or not, located, etc. (-Srepro, local dep path resolution, env, etc). This also interacts with whether the install level config is an explicit file or embedded as a resource inside tda. Something definitely needs to change but I need a clear space to sit down and think through all the details, so it’s coming.{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 20")}
#2018-08-0216:09martinklepschAre there any restrictions to where tools.deps will look for the :sha provided?#2018-08-0216:10ghadiwhat do you mean?#2018-08-0216:11Alex Miller (Clojure team)where else would it look other than the git repo?#2018-08-0216:13martinklepschclj -Sdeps '{:deps {org.clojure-grimoire/lib-grimoire {:git/url "" :sha "767d68f605e5ec2ab740ae42cfc7b90a2948713c"}}}'
This fails with Manifest type not detected when finding deps for org.clojure-grimoire/lib-grimoire in coordinate {:git/url "", :sha "767d68f605e5ec2ab740ae42cfc7b90a2948713c"}
In a more involved setup it fails with the following error/stacktrace :#2018-08-0216:14martinklepsch#2018-08-0216:16ghadithe stack trace is a bit revealing... did you make edits to the file in .gitlibs/libs/... ?#2018-08-0216:16Alex Miller (Clojure team)767d68f605e5ec2ab740ae42cfc7b90a2948713c doesn’t look like a commit in https://github.com/martinklepsch/lib-grimoire ?#2018-08-0216:17ghadiI cloned and found that commit#2018-08-0216:17Alex Miller (Clojure team)oh yeah, weird - github didn’t find it#2018-08-0216:18ghadi(We should consider making the checkout dirs readonly)#2018-08-0216:18dominicm@ghadi how do you make dirs read only? 😄#2018-08-0216:18dominicmI guess you can chmod on linux, to prevent self-harm#2018-08-0216:19Alex Miller (Clojure team)I got your second error trying to run your first command#2018-08-0216:20martinklepschhaha#2018-08-0216:21martinklepschI didn’t do any manual edits in .gitlibs (as far as I’m aware at least 😛)#2018-08-0216:21ghadiI found it super easy to accidentally jump to source in Emacs to a gitlibs SHA dir.#2018-08-0216:22ghadimake edits, hit save, break Content-Addressing guarantees#2018-08-0216:23dominicmI've done the same thing to real jars#2018-08-0216:23dominicmI had to uninstall zip so vim wouldn't let me :w inside a jar.{:tag :div, :attrs {:class "message-reaction", :title "upside_down_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙃")} " 4")}
#2018-08-0216:23dominicmI should fix it properly#2018-08-0216:22Alex Miller (Clojure team)I think the second error is the real error (the MissingObjectEx)#2018-08-0216:22Alex Miller (Clojure team)and then after that, it looks like you have a checkout directory, but it didn’t finish so it’s missing files (like the manifest files)#2018-08-0216:22martinklepschI don’t even know how to jump to source so guess I’m safe 😄#2018-08-0216:22martinklepschshould I try nuking .gitlibs?#2018-08-0216:22Alex Miller (Clojure team)I think you’ll see the same problem again#2018-08-0216:23ghadino, not if Alex repro'ed immediately.#2018-08-0216:23Alex Miller (Clojure team)^^#2018-08-0216:23martinklepschright, fair enough 🙂#2018-08-0216:24bronsai can confirm reproing w/o any particular setup#2018-08-0216:24martinklepschInterestingly there are some files in that SHA’s gitlib dir (LICENSE and README)#2018-08-0216:25Alex Miller (Clojure team)yeah, it got part way done#2018-08-0216:25ghadi➜ lib-grimoire git:(develop) git cat-file -t c7cd4f50871b20a580a66e77a34bfa9d336fe430
fatal: git cat-file: could not get object info
#2018-08-0216:25ghadirepush#2018-08-0216:25Alex Miller (Clojure team)I guess a) there is either something weird in this repo or a bug in jgit#2018-08-0216:26Alex Miller (Clojure team)and b) in this kind of a failure, we should nuke the checkout dir#2018-08-0216:27Alex Miller (Clojure team)ticket for b) in https://dev.clojure.org/jira/browse/TDEPS welcome#2018-08-0216:28Alex Miller (Clojure team)a) is worth a bit more investigation#2018-08-0216:28Alex Miller (Clojure team)maybe there’s a force push or something weird there?#2018-08-0216:28ghadiThere is something weird in this repo.#2018-08-0216:28martinklepschWould love a tarball deps procurer btw 😛#2018-08-0216:28ghadi➜ lib-grimoire git:(develop) git cat-file -p HEAD^{tree}
100644 blob 3bc48030cba1b0f250f8b3d2cb0a2635f5195339 .gitmodules
100644 blob 7689f30efd6dcaadfb4033a444fa1c3f1848dba3 LICENSE
100644 blob c87d64a78fcaebbdfe02f188168a3270962869a1 README.md
040000 tree 4c51b4bd0c9980a782b29286cdeefdb49e7bd899 bin #### THIS
100644 blob 54f9aeba51d4b4a61c9ad3de046e770859a217c1 deps.edn
100644 blob f60a2b75223a9c8739d525ea62ba48aeeab878b0 project.clj
040000 tree 97a002c7db97cd89dc74606b77de343d2dbc878d resources
040000 tree 28ffc92968bebaf4930b08ff59d9ff5a3a5c297b src
040000 tree f4581b6f130115e5f0035945fc4ef6036076ba88 test
➜ lib-grimoire git:(develop) git cat-file -p 4c51b4bd0c9980a782b29286cdeefdb49e7bd899 ## Drill in
160000 commit c7cd4f50871b20a580a66e77a34bfa9d336fe430 hooks
100644 blob f3acf21ed723011f2586416e842ce41683bf3f29 install-hooks.sh
#2018-08-0216:29ghadiThe second command cats out the SHA for the tree representing the bin directory, and it's pointing to a commit blob mysteriously#2018-08-0216:29ghadiI expect only blobs and trees to be children of trees#2018-08-0216:30martinklepschI did force push to that branch, and it might have even been after I tried to use another revision from that repo — but seems weird that this would still replicate on a system that hasn’t had a clone yet?#2018-08-0216:32Alex Miller (Clojure team)I agree with Ghadi that that looks like a bad git structure#2018-08-0216:33ghaditry to overwrite bin/hooks and recommit it#2018-08-0216:36martinklepschDeleted bin/ entirely & commited as 9810f7b20c1a676060f939cfd8f2f785f4dea3c7 but still same issue#2018-08-0216:36ghadifetching....#2018-08-0216:36martinklepschdid you have something else in mind when you wrote “overwrite”?#2018-08-0216:37ghadirepo looks good to me, you updated to 9810f7b20c1a676060f939cfd8f2f785f4dea3c7?#2018-08-0216:37martinklepschoh no wait#2018-08-0216:38martinklepschseems to work now!#2018-08-0216:38ghadicool#2018-08-0216:38ghadiI don't know how that commit object got into the repo!#2018-08-0216:39martinklepschjust saw an exception and was assuming it’s the same but it was a different one 🙂#2018-08-0216:39ghadigit fsck didn't complain at all...#2018-08-0216:39ghadibut that might not be looking for this prob#2018-08-0216:40Alex Miller (Clojure team)I was just doing that myself :)#2018-08-0216:41martinklepschgotta say, I’m impressed by your git-fu 😄#2018-08-0216:41Alex Miller (Clojure team)you’ve impressively horked this repo, kudos :)#2018-08-0216:42martinklepschdon’t look at me, I’m pretty sure it was @arrdem 🙂#2018-08-0216:42Alex Miller (Clojure team)fair enough :)#2018-08-0216:42Alex Miller (Clojure team)just change my “you” ref via force-push#2018-08-0216:44martinklepschan aside to this: are there any plans to add more procurers? A tarball procurer would have been cool in this situation 🙂#2018-08-0216:45dominicmI've been hoping that a procurer extension mechanism would appear, off the back of the work to locate the owner of a namespace.#2018-08-0216:45Alex Miller (Clojure team)seems pretty far down the priority list#2018-08-0216:46Alex Miller (Clojure team)why not just the existing jar support?#2018-08-0216:46dominicmI imagine because github provides a tar.gz/zip on the releases page#2018-08-0216:46martinklepsch👆 this#2018-08-0216:47Alex Miller (Clojure team)well, feel free to file an enhancement, preferably with a use case#2018-08-0216:51ghadiseems like a weird use-case#2018-08-0216:53Alex Miller (Clojure team)I can envision how to get a repo into this state via the plumbing api but I find it harder to envision how it happened via the porcelain#2018-08-0216:53Alex Miller (Clojure team)I assume some kind of bad rebase#2018-08-0216:53Alex Miller (Clojure team)but geez#2018-08-0216:55Alex Miller (Clojure team)and ghadi, it does seem like Git handles this better than jgit so maybe you’d never even see it shelling out#2018-08-0217:35lilactownI’m trying to patch juxt/pack to allow me to pass in an alias to use while generating a jar#2018-08-0217:36lilactownthis is the relevant bit of code that is catching fire atm:
(let [deps-map (tools.deps.reader/slurp-deps (io/file deps))
resolve-args (tools.deps/combine-aliases
deps-map
(get-in parsed-opts [:options :alias]))]
(tools.deps/resolve-deps deps-map resolve-args)
#2018-08-0217:37lilactownI’m currently getting: org.apache.maven.model.resolution.UnresolvableModelException: Could not find artifact org.apache:apache:pom:1
3
#2018-08-0217:37dominicm@lilactown try adding :mvn/repos to your deps.edn#2018-08-0217:37dominicmSee pack issues, just eating dinner else I'd give the full snippet #2018-08-0217:39lilactownthat seemed to fix the error 🙂 thanks @dominicm#2018-08-0217:41dominicm@alexmiller to solve this, I'm considering inlining the system deps.edn into pack, do you think that would be a problem? #2018-08-0217:41lilactown(and it works!)#2018-08-0217:49Alex Miller (Clojure team)Not as long as you’re willing to roll with future changes#2018-08-0218:20dominicmHappy enough for now. #2018-08-0218:41seancorfieldI originally had the system deps baked into boot-tools-deps then took it out in favor of calling (clojure-env) to get the list of deps.edn files to include -- which takes care of the :mvn/repos issue.#2018-08-0218:41seancorfieldDoes pack not use the system/user deps.edn files?#2018-08-0218:42dominicm@U04V70XH6 I don't want to do that in pack, I explicitly want to ignore the user deps.edn file, and I can't do that without (clojure-env) returning a map for the deps.edn files it considers.#2018-08-0218:43seancorfieldYou could at least assume the first entry in that vector was the system deps tho'...#2018-08-0218:44dominicmBut that isn't part of the api, it could break at any point. I'd feel like a bad citizen depending on a coincidence rather than a contract.#2018-08-0218:48seancorfieldI thought we were "guaranteed" that the vector of deps files was in order? Or do you mean that whole -Sdescribe thing isn't documented/guaranteed? /cc @alexmiller#2018-08-0218:49dominicmI recall asking and being told the order wasn't guaranteed. I asked when lein-tools-deps did this.#2018-08-0218:53seancorfieldI must admit, I find that very surprising since order does matter ...#2018-08-0218:54dominicm@U04V70XH6 but being the first isn't attributed to system in anyway. A least-important override could be added at any time.#2018-08-0220:17Alex Miller (Clojure team)you can be guaranteed that they are in order: install, user, project, -Sdeps#2018-08-0220:18Alex Miller (Clojure team)however, the last 3 could all independently be either missing or present#2018-08-0220:18Alex Miller (Clojure team)and in future the first may possibly not exist#2018-08-0220:31seancorfieldThank you!#2018-08-0221:06dominicmIf this guarantee is given, then I think the open ticket to convert it to a map is not needed.#2018-08-0221:50seancorfieldWell, you still couldn't tell the difference between system + user and system + project (except, perhaps, by inspection of the actual paths?) and that was something @U0567Q30W was after I believe.#2018-08-0221:51seancorfieldAnd you also wouldn't necessarily be able to tell the difference between system + user and user + project at the point where the system deps moved into the library itself (depending on how that is handled).#2018-08-0221:52Alex Miller (Clojure team)I think it’s useful to differentiate#2018-08-0221:52Alex Miller (Clojure team)for other needs too#2018-08-0221:55seancorfieldAdding a new key to the result of -Sdescribe with some sort of map explaining where the deps files came from seems useful regardless of other concerns. I think if the system deps file moves into the codebase, then indicating that somehow in the list of config files, with a way to get the raw data structure, will also become important for tooling.#2018-08-0302:06cflemingThe latest version of Cursive also bundles the system deps.edn BTW#2018-08-0218:49dominicm@alexmiller I might be being silly, but I don't see a way in the public api to slurp a dep which is a java.net.URL, e.g. io/resource. slurp-edn depends on being a file, and canonicalize-all-syms is a private function, so I cannot utilize it.#2018-08-0220:24Alex Miller (Clojure team)could probably break slurp-edn into something smaller that takes a reader (although the file-ness is being used to construct a good error message there)#2018-08-0220:24Alex Miller (Clojure team)canonicalize-all-syms is not public as it may go away#2018-08-0220:25Alex Miller (Clojure team)there’s now a canoncialization step built into the extension apis#2018-08-0220:25Alex Miller (Clojure team)you can of course invoke it via the private var if you want to call it for now#2018-08-0321:10emergence@alexmiller (or anyone else in the know) - I'm pulling down a company Clojure library from a private github repo via deps.edn -- had to tweak a few things to get auth to play nice, but now I have another issue: is there a way to indicate which jar file it will use?#2018-08-0321:10emergenceI have one that will include test dependencies and am looking to use that version instead#2018-08-0321:20ghadiwhat did you tweak for Auth @emergence?#2018-08-0321:21emergenceI had to comment out my IdentityFile line in .ssh/config#2018-08-0321:23ghadiyeah this is a known issue#2018-08-0321:23ghadiwe have a potential resolution for a host of auth related issues#2018-08-0321:25emergenceI saw there is an open bug around this#2018-08-0321:21Alex Miller (Clojure team)deps.edn does not have any way to include test scoped transitive deps#2018-08-0321:22emergenceSorry I wrote incorrectly, I meant test namespaces not transitive deps#2018-08-0321:22Alex Miller (Clojure team)I was confused by you saying jar file above#2018-08-0321:22Alex Miller (Clojure team)in tandem with github repo#2018-08-0321:23Alex Miller (Clojure team)or rather, I am still confused. perhaps you can back up. :)#2018-08-0321:23ghadiwe have a potential resolution for a host of auth related issues#2018-08-0321:24emergenceso in my deps.edn I have something like
mycompany/depname {:git/url "#2018-08-0321:24emergenceAnd forgive me if I'm overlooking something silly I'm trying tools.deps.alpha for first time s#2018-08-0321:25emergenceI want to access test namespaces from depname#2018-08-0321:26Alex Miller (Clojure team)in that project’s deps.edn, you could include :paths ["src" "test"]#2018-08-0321:26Alex Miller (Clojure team)but there is no way to do that externally#2018-08-0321:27Alex Miller (Clojure team)so I think my answer is still, you can’t do that#2018-08-0321:29emergenceThat's still helpful -- the project is still running off a project.clj file so I can see about merging in a secondary deps.edn for my purposes.#2018-08-0321:31Alex Miller (Clojure team)it is possible to use a :deps/root key to point to a subdir holding a deps.edn that included the test, but that’s likely to confuse people#2018-08-0321:34emergenceIf I did :deps/root "test" and stuffed the deps.edn in test/ do you think that would work?#2018-08-0321:35emergenceand then specified the appropriate relative paths#2018-08-0321:41emergenceEither way I'll play around with it and see what works for my team -- thanks for spending a few mins chatting on this 🙂#2018-08-0420:07lilactownhow would I go about adding a dependency (Java) that uses gradle (via Github)?#2018-08-0420:07lilactownno pom to be found#2018-08-0420:09dominicm@lilactown generally it's deployed to a maven repository#2018-08-0420:09dominicmEven if it's using gradle#2018-08-0420:10lilactownah. in this particular instance it’s not, since the code I want is actually a part of an application#2018-08-0420:11lilactownhttps://github.com/AsamK/signal-cli#2018-08-0420:11lilactownmy hope is that I could bring it in as a dependency and play with it in clojure#2018-08-0420:15dominicmI don't think so in this case. t.d.a doesn't have a reader for gradle files.#2018-08-0420:19lilactownI guess I could fork it and generate a pom?#2018-08-0420:21dominicmor just install it locally via gradle#2018-08-0510:53michalhi. is there any way to force clj to run in offline mode (to not fetch newest snapshots in particular) ?#2018-08-0520:28Alex Miller (Clojure team)Currently, no, but planning to add#2018-08-0515:27lilactownis add-lib going to make it’s way into master?#2018-08-0520:27Alex Miller (Clojure team)Yes#2018-08-0623:19kennyI have org.slf4j/slf4j-nop in my :exclusions for a dependency but it is still included in the tree printed out via clj -Stree. org.slf4j/slf4j-nop is a second level dependency (i.e. a dependency that my dependency depends on). Why is the org.slf4j/slf4j-nop not getting excluded?#2018-08-0623:36hiredmanwhat does -Spath say?#2018-08-0623:50kennyIt also includes org.slf4j/slf4j-nop.#2018-08-0700:11kennyDeleted .cpcache and it fixed it.#2018-08-0707:24dottedmagSounds like a bug. Why it was not invalidated on its own?#2018-08-0711:50Alex Miller (Clojure team)The only cache staleness bug I’m aware of is with local deps#2018-08-0711:51Alex Miller (Clojure team)Where downstream deps.edn that change don’t force a cache invalidation#2018-08-0717:32martinklepschI would really like to fix https://dev.clojure.org/jira/browse/CLJ-2037 but if I understand correctly it’s tricky with the current registry structure since keywords can’t carry additional metadata. Would be happy about any suggestions. Maybe @alexmiller can also comment to what degree this feature warrants a restructuring of the registry.#2018-08-0717:34martinklepsch(I assume the patch Felix Andrews describes in his comment won’t work with specs like (s/def ::thing ::other-thing) but I haven’t fully checked IIRC)#2018-08-0717:37Alex Miller (Clojure team)definitely on board with fixing it, but not going to happen until Rich has a chance to get back to spec work, and that will likely be a while#2018-08-0717:41martinklepschGot it. Guess I’ll run with a fork that has a fix for this issue then.#2018-08-0718:33dnolenis it not possible to pass JVM opts via :main-opts?#2018-08-0718:35dominicmThat's what jvm-opts is for? #2018-08-0718:39dnolenah thanks, I keep looking for stuff in the guide instead of the reference#2018-08-0718:41dnolenaliases don’t compose across deps.edn files right?#2018-08-0718:47seancorfieldNo, later aliases replace earlier ones with the same name.#2018-08-0718:47seancorfield(I think that's the safer behavior, TBH)#2018-08-0718:48dnolenyeah not saying it’s desirable#2018-08-0718:49seancorfieldHeh, I've wished for composable aliases from time to time... but always stop myself from creating a JIRA ticket 🙂#2018-08-0812:20rickmoynihanis it (or will it ever be) possible to put specific paths from inside a git dep on the class/resource path, without needing a deps.edn file at the project root?#2018-08-0812:20rickmoynihanthinking for resource deps#2018-08-0812:26Alex Miller (Clojure team)No?#2018-08-0812:37dominicm@rickmoynihan you could probably use gitlibs directly!#2018-08-0812:42rickmoynihan@dominicm: how do you mean?#2018-08-0812:43dominicmI suppose not if you wanted them on the classpath :thinking_face:#2018-08-0812:43dominicm@rickmoynihan https://github.com/clojure/tools.gitlibs if io/file is okay for you though. Obviously I don't have the context of your use case.#2018-08-0812:45rickmoynihanThanks hadn’t seen gitlibs before#2018-08-0812:46rickmoynihanit’s ok I think :local/root is better for this anyway#2018-08-0813:01Alex Miller (Clojure team)tools.deps uses gitlibs for its git deps{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-08-0818:58richiardiandreahas anybody run into problems with OpenJDK 9 and retrieving deps?#2018-08-0818:59richiardiandreaI have a weird failure in a cljs-tooling build: https://travis-ci.org/clojure-emacs/cljs-tooling/jobs/413721389#2018-08-0819:15lilactownis there a way to tell the CLI tools to “just download dependencies”?#2018-08-0819:15lilactowntrying to tune my docker build#2018-08-0819:17dominicmAs opposed to doing what#2018-08-0820:19richiardiandreareporting the full error here, is it a known issue or it is worth a Jira?#2018-08-0820:32richiardiandreait looks very similar to https://dev.clojure.org/jira/browse/TDEPS-20#2018-08-0820:56Alex Miller (Clojure team)that’s different#2018-08-0820:56Alex Miller (Clojure team)the problem here is buried - “Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact mount:mount:pom:0.1.13 from/to clojars (https://repo.clojars.org/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”#2018-08-0820:56Alex Miller (Clojure team)this is a certificate problem re the JDK being used and clojars#2018-08-0821:01Alex Miller (Clojure team)openjdk9 does not install with any root certs (I think this is fixed in 10)#2018-08-0821:02Alex Miller (Clojure team)https://github.com/travis-ci/travis-ci/issues/9368#2018-08-0821:49richiardiandreaoh cool ok, thanks a lot Alex#2018-08-0820:24ghadi@lilactown you can put RUN clojure -Spath in your dockerfile#2018-08-0820:25ghadishould have the intended effect, if an oblique approach#2018-08-0821:39lilactown@ghadi thanks! I think that almost worked… for some reason my uberjar command still downloads and installs all it’s deps 😭#2018-08-0821:39ghadilein uberjar?#2018-08-0821:40lilactownnot quite:
:uberjar
{:extra-deps
{pack/pack.alpha
;; {:local/root "/Users/will/Code/pack.alpha"}
{:git/url ""
:sha "c444ae3af083b0bfa68160daac576607232f556c"}}
:main-opts ["-m" "mach.pack.alpha.capsule" "dist/lilactown.jar"
"-a" ":server" "-m" "lilactown.core"]}
#2018-08-0821:40ghadiAll its deps, or just the pack deps?#2018-08-0821:41lilactownI think it’s just the pack deps#2018-08-0821:41ghadiRUN clojure -A:uberjar -Spath#2018-08-0821:41ghadishould do the the trick to handle that#2018-08-0821:42lilactownhere’s my relevant dockerfile stuff:
COPY deps.edn .
RUN clojure -A:server:uberjar -Spath
COPY . /usr/app
RUN clojure -A:uberjar
#2018-08-0821:43lilactownhrm. I wonder if changing it to -R:server:uberjar -Spath would change it#2018-08-0821:43lilactownseems more correct anyway#2018-08-0821:43ghadiare you setting WORKDIR /usr/app in the dockerfile?#2018-08-0821:43ghadi-A subsumes -R#2018-08-0821:44lilactownyes, I’m setting workdir too#2018-08-0821:47ghadi(I like multistage builds)
I'm surprised L27 redownloads things, to be honest. All should get downloaded on L21#2018-08-0821:48lilactownyeah me too 😕#2018-08-0821:49ghadiofftopic for this room, but I wouldn't trust openjdk-8-alpine until OpenJDK fully supports Alpine#2018-08-0821:49ghadiJDK 11 has official support, IIRC#2018-08-0821:49lilactown:thinking_face: I haven’t noticed any problems yet. anything in particular I should watch out for?#2018-08-0821:50ghadinot sure tbh. I've also used it in the past. Now I use jlink --add-modules to create slimmer packaged JVM#2018-08-0821:52ghadiFROM openjdk:10 as jdk
RUN jlink \
--strip-debug \
--compress=1 \
--vm=server \
--output /tmp/linkedjdk \
--add-modules ,jdk.management,jdk.unsupported,java.xml.bind,java.xml.ws.annotation
FROM debian:stretch-slim
COPY --from=jdk /tmp/linkedjdk /jdk/
# This is temporarily necessary as it appears the new OpenJDK TrustStore is incomplete
# Apparently, AWS's Step Function endpoint uses a different signing root than S3
COPY --from=jdk /docker-java-home/lib/security/cacerts /jdk/lib/security/
COPY my.jar /MYCOMPANY/
ENTRYPOINT ["/jdk/bin/java"]
#2018-08-0821:52ghadithat's my dockerfile @lilactown#2018-08-0821:53lilactownslick#2018-08-0821:53lilactownsaving this for later 😉 I remember you talking about this last week I think#2018-08-0821:55ghadidepending on what modules you need, you can get by with less#2018-08-0821:51ghadiI'd assume it would manifest as a hard VM crash. But if it works, it works!#2018-08-0917:44richiardiandreauhm, this baffles me, what am I doing wrong?
clojure -Srepro -Sdeps '{cljs-tooling {:mvn/version "0.3.0-SNAPSHOT" :exclusions [org.clojure/clojure org.clojure/clojurescript]}}' -Spath
src:/home/arichiardi/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/home/arichiardi/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/home/arichiardi/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar
#2018-08-0917:45richiardiandreain particular I am not seeing the dep I want to include#2018-08-0918:28dominicm@richiardiandrea is it the same for other dependencies, or just that one? I'm wondering about -Srepro and -Sdeps#2018-08-0918:30richiardiandreauhm dunno let me try smth else#2018-08-0918:31ghadiYeah I'm not sure how those interact together...#2018-08-0918:31richiardiandrea$ clojure -Srepro -Sdeps '{mount {:mvn/version "0.1.13"}}' -Stree
org.clojure/clojure 1.9.0
org.clojure/spec.alpha 0.1.143
org.clojure/core.specs.alpha 0.1.24
#2018-08-0918:31ghadi-Srepro might negate Sdeps#2018-08-0918:31richiardiandreaoh, maybe I need :deps{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-08-0918:32richiardiandreariiiiight#2018-08-0918:32richiardiandrea$ clojure -Srepro -Sdeps '{:deps {mount {:mvn/version "0.1.13"}}}' -Stree
org.clojure/clojure 1.9.0
org.clojure/spec.alpha 0.1.143
org.clojure/core.specs.alpha 0.1.24
mount/mount 0.1.13
#2018-08-0918:33ghadiOh, yeah that would help#2018-08-0918:34richiardiandrealol#2018-08-1004:38steveb8nis there a lib that provides a watcher (e.g. re-runs tests or a specific fn when it sees a file change) that works with deps.edn? I can’t find anything#2018-08-1005:02seancorfield@steveb8n Wouldn't that just be any command-line watcher than can (re-)run another command when files change?#2018-08-1005:02steveb8nprobably but I’ve only used the test watcher#2018-08-1005:02steveb8nwatchers#2018-08-1005:02seancorfieldThe clj script is a one-shot runner that loads deps.edn and runs whatever :main-opts are specified.#2018-08-1005:03steveb8nand the test watchers are all built for lein/boot#2018-08-1005:03seancorfieldThe lein/`boot` test watchers are built to re-run tests -- so it's very tightly coupled.#2018-08-1005:04steveb8nI could build my own, load it into classpath from a gist but I’d rather avoid the work if it’s already been done#2018-08-1005:05seancorfieldI actually think test watchers are a bit of an anti-pattern...#2018-08-1005:06seancorfieldThey sort of imply an external process -- external to your editor and your REPL -- that you have to keep watching while you are editing.#2018-08-1005:10seancorfieldWhen I'm working on code, I'm running code in the REPL, and I can run any tests in my editor with a hot key.... I wouldn't expect to re-run my entire test suite on every minor code change and file save.#2018-08-1005:11seancorfieldAnd my workflow usually involves evaluating code without even saving files anyway ¯\(ツ)/¯#2018-08-1005:12seancorfieldI can evaluate forms to the REPL without saving the file so why would I bother with that step? 🙂#2018-08-1005:13steveb8nI agree although I’m generating a diagram using dorothy and I have to switch from the layout ns to the driving ns every time I want to see the change#2018-08-1005:13steveb8nwould be much faster during dev to see updates without the switch#2018-08-1005:13dominicmKrei is already watching the classpath for changes. No reason it couldn't run tests in a classloader or something. #2018-08-1005:13steveb8ncool. I’ll take a look#2018-08-1005:14dominicmIt doesn't do that right now though. #2018-08-1005:14steveb8ndoh, bait-in-switch 🙂#2018-08-1005:16steveb8nin that case I’ll carry on switching ns for now. maybe one of the test runners will get this soon.#2018-08-1005:16steveb8nthanks @seancorfield @dominicm for the thoughts#2018-08-1005:20seancorfield@steveb8n One option to consider is to have a (comment ...) form in your layout ns that calls functions in your driving ns, then you can do everything in one namespace.#2018-08-1005:22seancorfieldI tend to have a (comment ...) block at the end of a lot of my ns's that let me run code to setup/test/teardown the code in that ns.#2018-08-1005:27steveb8nI have that but most of my iterative changes are in the driven ns, not the driving ns. I’ll try that idea in the driven ns though - might work#2018-08-1005:29steveb8nthat works! good idea, thanks. now re-loading the driven ns invokes the fn in the driver ns#2018-08-1005:30steveb8nthe simplest ideas are always best#2018-08-1006:11seancorfield@steveb8n Cool. I've gotten a lot more productive by dropping (comment ...) blocks all over the code base so I can stay in one file and run whatever code I need (and because it isn't compiled at load time, you can have circular references in such blocks). I use it for code that starts/stops Components, runs tests, sets up state etc.#2018-08-1006:12steveb8nyeah, it was the circular ref that was blocking my brain from trying that. great trick#2018-08-1006:12steveb8nmy fingers thank you#2018-08-1010:16thheller@alexmiller I'm looking into using the tools.deps.alpha CLJ API as the primary dependency resolver for shadow-cljs (was using pomegranate before). resolve-deps works nicely but I noticed that make-classpath just concats all paths together without sorting them properly. does the CLI itself also use this behavior?#2018-08-1010:17thhellerorder shouldn't matter but in case there are any conflicts on the classpath I'd rather have something deterministic rather than relying on the hash ordering of the lib-map#2018-08-1010:17dominicm@thheller This is intentional, my understanding is that the statement is "order doesn't matter".#2018-08-1010:18thhelleryeah just might be me running into issues with pomegranate where order definitely mattered#2018-08-1010:18thhellerbut it didn't resolve version conflicts properly, so shouldn't happen in tdeps#2018-08-1012:31Alex Miller (Clojure team)Order should only matter if you have two dependencies that contain the same thing#2018-08-1012:32Alex Miller (Clojure team)And if you have that something is wrong#2018-08-1012:32Alex Miller (Clojure team)Maven uses order for selection but we use version#2018-08-1013:48mhuebertis there any way to have multiple git dependencies in the same git repo, which depend on each other? eg. say I publish https://github.com/acme/MonsterLib, which contains two projects, A and B, in subdirectories, so we have /A/deps.edn and /B/deps.edn, and B depends on A. For local dev, I can put a :local/root entry in /B/deps.edn which points to "../A", but (as far as I know) this won’t be properly required by a consumer of B#2018-08-1013:49mhuebertmaybe a clearer way of asking is, “is there a way for tools.deps to resolve relative :local/root (or equivalent) dependencies in a git repo”#2018-08-1013:49dominicm@mhuebert I think you're looking for :deps/root#2018-08-1014:31rickmoynihanSo does :deps/root allow you to treat a sub project (i.e. a sub directory with a deps.edn file) within a git repo as a specific dep, instead of the top level project directory? Or am I misunderstanding?#2018-08-1014:37dominicmThat's exactly it#2018-08-1013:50mhuebertlooking for docs on that#2018-08-1013:51mhuebertnothing mentioned on https://clojure.org/reference/deps_and_cli#2018-08-1013:52Alex Miller (Clojure team)I don’t think there are any docs but it’s a thing#2018-08-1013:52mhuebertwould the entries be the same as with :local/root? ie. just put relative paths there#2018-08-1013:52Alex Miller (Clojure team)Yes#2018-08-1013:53mhuebertsounds like exactly what I was hoping for#2018-08-1013:53Alex Miller (Clojure team)Relative to local/root#2018-08-1013:55dominicm@alexmiller @mhuebert I think exactly as :local/root is wrong here, it would be just "A/" alone I think.#2018-08-1014:15mhueberthmm, I must still have something wrong. I’ve made a simple example, a tree like:
.
├── App
│ └── deps.edn
└── Monster
├── A
│ └── deps.edn
└── deps.edn
(repo: https://github.com/mhuebert/deps-root-test)
App/deps.edn depends on {:local/root "../Monster"}, which depends on {:deps/root "A"}, but if I run clj -Spath in App I get Coordinate type not loaded for library monster/A in coordinate #:deps{:root "A"}#2018-08-1014:16dominicm@mhuebert you need to do a :git/url too, alongside the :deps/root#2018-08-1014:18mhueberthmm. can Monster ever depend on the current version of A?#2018-08-1014:18mhuebertif they are in the same repo#2018-08-1014:20mhuebert(if you need the sha, which you don’t have until you have made the commit that would presumably contain that link)#2018-08-1014:20dominicmhttps://dev.clojure.org/jira/browse/TDEPS-74 I believe this would solve it.#2018-08-1014:21mhuebertyes, if :local/root was relative to its own location instead of the current working directory, then I could use that#2018-08-1014:22Alex Miller (Clojure team)it can’t be relative to itself#2018-08-1014:22dominicm@alexmiller relative to the manifest, no?#2018-08-1014:23Alex Miller (Clojure team)when you include a local dep by location, it can be relative to the project including the dep#2018-08-1014:23Alex Miller (Clojure team)instead of relative to cwd#2018-08-1014:24dominicmI think you're both saying the same thing, aren't you? Or am I misunderstanding?#2018-08-1014:24Alex Miller (Clojure team)maybe :)#2018-08-1014:24Alex Miller (Clojure team)I’m just trying to restate clearly for myself mainly#2018-08-1014:25mhuebertwhat I am wondering is, if that jira ticket was solved, could Monster/deps.edn contain a {:local/root "A"} pointer to the A subdirectory, and resolve for a consumer of Monster, eg. via :local/root (in the example above, App) or a git dep?#2018-08-1014:27Alex Miller (Clojure team)maybe?#2018-08-1014:32mhuebertI’m not sure how else one could consume multiple projects (which may have dependencies among themselves) from the same git repo#2018-08-1014:34mhuebertmonorepo woes.#2018-08-1014:41mhuebertI’ve updated the example to be more simple/realistic,
├── App ;; user-land, wants to use project A
│ └── deps.edn
└── Monster ;; library-land, a repo or local dir
├── A
│ └── deps.edn
└── B
└── deps.edn
the motivating case being that you want to publish project A, which depends on project B, in the same Monster repo.
The test case is, can App depend on A via git or local/root, and have tools.deps successfully follow the dependency to B.#2018-08-1014:51Alex Miller (Clojure team)what’s not being tracked at all in the code is the directory basis and you want that to shift as you examine different local or git dirs#2018-08-1014:52Alex Miller (Clojure team)so that it’s relative to the currently focused lib#2018-08-1014:52Alex Miller (Clojure team)and then use it to resolve relative dirs#2018-08-1015:00Alex Miller (Clojure team)which was actually kind of the intent of :deps/root#2018-08-1015:00Alex Miller (Clojure team)local deps don’t actually use deps/root right now to detect the manifest (git deps do)#2018-08-1019:59dominicmDoes it make sense for it to do so? Wouldn't you just provide a local path which included the deps/root?#2018-08-1020:17Alex Miller (Clojure team)yes, it makes sense, and no, I don’t think you would do that. You want to be able to specify a deps/root relative to the local root I think? but this is actually the less interesting part of the problem.#2018-08-1105:36dominicmI don't understand what additional functionality, beyond specifying a subdir, is provided by deps/root. What behaviour would be different?#2018-08-1015:00Alex Miller (Clojure team)so there are actually several issues here#2018-08-1218:42thheller@alexmiller does tools.deps support providing maven mirrors? in lein/pomegranate I could set :mirrors. Can't find anything related in tools.deps sources?#2018-08-1218:42Alex Miller (Clojure team)Not yet#2018-08-1218:50thhellerm2/settings.xml is read either right?#2018-08-1220:51seancorfield@thheller It reads the repos/credentials from that -- https://github.com/clojure/tools.deps.alpha/blob/884d7ae5b9c228ff795e4385291708102f1cd46d/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L71#2018-08-1308:36martinklepschI haven't looked around much, but just in case this is a known thing, some folks building docs for cljdoc locally get an error like this
Error building classpath. /var/folders/gm/p9fh4d3n5m11nnyjrsry1cm80000gq/T/cljdoc-expound-expound-0.7.2-SNAPSHOT6141675865145266918/impl-src is not a relative path
Does that look familiar to anyone? Are :paths required to be relative?#2018-08-1323:35richiardiandreasorry maybe an FAQ but can tools.deps read deps from a local folder?#2018-08-1323:36richiardiandreaoh :local/root awesome#2018-08-1405:45isaaccan clj handle java files that in paths?#2018-08-1405:47dominicmNo#2018-08-1405:47dominicmOr rather, not in source files. If they're from Maven it's okay. #2018-08-1405:48isaaccan i call javac from clj?#2018-08-1406:21dominicm@isaac absolutely.#2018-08-1406:21dominicmI do this in pack.#2018-08-1703:12alex-dixonGoing off isaac’s question from earlier, is there a good workflow for mixed java and clojure projects using deps.edn?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-08-1710:33jetzajaccan I add a lib that consists of multiple jars on disk?#2018-08-1712:07Alex Miller (Clojure team)Generally, no. Typically a lib consists of a single artifact#2018-08-1712:08Alex Miller (Clojure team)There are ways to hack it by listing the jars explicitly in :paths though#2018-08-1718:10lilactowndo multiple -Sdeps merge?#2018-08-1718:30lilactownanswer: nope#2018-08-1806:52cfleming@alexmiller Ugh, is it expected that users will be able to add jars via :paths? I’ll have to add support for that if so.#2018-08-1808:47Alex Miller (Clojure team)I would not consider that to be normal practice#2018-08-1915:30sveri@alexmiller Hi, how about native support for windows, did you come to work on it?#2018-08-1916:04Alex Miller (Clojure team)I’ve done a port to power shell but have not had time to finish all the build and release stuff, which interacts with some of the script#2018-08-1916:05Alex Miller (Clojure team)Hoping to be able to refocus on that soon and finish it off#2018-08-1918:44sveriOk, thanks for your work. I was trying to use cli on WSL, which works, but fighweel main for instance needs an x11 instance which I don't have installed right now.
#2018-08-2008:01levitanonghi all, is there a way to run clojure such that it only downloads the dependencies and exits? Similar to lein deps. Use-case is in docker, where caching is done by diffing USEd files.#2018-08-2009:02delaguardoclojure -e "(clojure-version)"#2018-08-2009:02delaguardothis will download everything, print out your clojure version and exit with exit code 0#2018-08-2009:07levitanong@delaguardo haha wouldn’t clojure -e "(System/exit 0)" work equally well in that case?#2018-08-2009:08delaguardosure)#2018-08-2009:08delaguardoI’m just copying my configuration where I’m using printed value as a part of deploy report)#2018-08-2010:37levitanongOkay, thanks @delaguardo!#2018-08-2010:54dottedmagclojure -e '' works too#2018-08-2012:52Alex Miller (Clojure team)clojure -Spath will work#2018-08-2016:26levitanong@dottedmag oh that’s nice and concise!
@alexmiller would clojure -Sdescribe also work?#2018-08-2016:26levitanongI ask because that would end up printing a lot less than -Spath#2018-08-2016:26ghadiclojure -Spath >/dev/null#2018-08-2016:26ghadisufficient for a dockerfile ^#2018-08-2016:28levitanongHaha @ghadi that’s an interesting solution, but I think I’d prefer clojure -e '' to that.#2018-08-2016:28ghadithat will launch the JVM twice, which is why alex suggested -Spath{:tag :div, :attrs {:class "message-reaction", :title "ok_hand"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👌")} " 4")}
#2018-08-2016:36levitanongoooh, so like, once for getting the deps, and then another for launching the repl?#2018-08-2016:36ghadisi#2018-08-2016:36ghadinot a big deal#2018-08-2016:42levitanongstill good to know how things work! Thanks for explaining 🙂#2018-08-2117:35ghadihttps://clojurians.slack.com/archives/C03S1KBA2/p1534872342000100#2018-08-2117:35ghadihttps://clojurians.slack.com/archives/C03S1KBA2/p1534872369000100{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-08-2118:39ghadi(Of course the absence of a convenience doesn't hold me back at all when I have a choice)#2018-08-2118:55Alex Miller (Clojure team)if you’re not using aliases that have M or O, then why do you care?#2018-08-2119:02ghadia test alias is a good example of when you want to launch a repl with deps but without (let's say) main options for some test-runner#2018-08-2119:09ghaditldr: because I am using an alias with M / O#2018-08-2123:19Alex Miller (Clojure team)I’m going to file this in the tiny violin category#2018-08-2123:52ghadiI'll take a tiny little 🎹 @alexmiller #2018-08-2123:57Alex Miller (Clojure team)Granted#2018-08-2200:54cflemingNow I have to create a Tiny Violin tag in my Github tracker.{:tag :div, :attrs {:class "message-reaction", :title "violin"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🎻")} " 12")}
#2018-08-2214:04dominicm@alexmiller is this something you would want to support?
{:git/url "../"
:deps/root "sub-dir"
:sha "abcabc"}
Essentially mono-repos with version pinning.#2018-08-2214:04dominicmI'm getting:
Error building classpath. Destination path ".." already exists and is not an empty directory
org.eclipse.jgit.api.errors.JGitInternalException: Destination path ".." already exists and is not an empty directory
at org.eclipse.jgit.api.CloneCommand.verifyDirectories(CloneCommand.java:257)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:189)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:49)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68)
at clojure.tools.gitlibs.impl$ensure_git_dir$fn__850.invoke(impl.clj:109)
at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:105)
at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
at clojure.tools.deps.alpha.extensions.git$eval894$fn__896.invoke(git.clj:41)
at clojure.lang.MultiFn.invoke(MultiFn.java:238)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:168)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:152)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:215)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:197)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invokeStatic(make_classpath.clj:59)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invoke(make_classpath.clj:52)
at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:70)
at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:64)
at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:109)
at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:84)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
which I suspect is due to how the folder name is being chosen.#2018-08-2214:07dominicmoh, weird, I can't supply an absolute path either, it is forced to be a relative one.#2018-08-2214:08dominicmLooks like paths aren't supported at all: Error building classpath. org.eclipse.jgit.transport.TransportLocal cannot be cast to org.eclipse.jgit.transport.SshTransport#2018-08-2214:08dominicmI'll open JIRA tickets#2018-08-2214:21Alex Miller (Clojure team)local git repos were intentionally out of scope#2018-08-2214:22Alex Miller (Clojure team)could possibly be added, but I would rank that below a lot of other things to work on#2018-08-2214:22Alex Miller (Clojure team)so I’ve classified these as minor enhancements in jira#2018-08-2214:29dominicmsounds very reasonable to me. I just don't touch the priority fields, based on what I think I'm supposed to do.#2018-08-2216:07onetom@dominicm the whole point of a monorepo is that you don't have to worry about the versions, because you are supposed to ensure that on every commit all modules/libraries/apps still build and work in your monorepo 😕#2018-08-2216:07dominicm@onetom I know 😞 sometimes we put things into life support though, quite useful under those conditions.#2018-08-2216:10onetomI was wondering how can I control the refresh period of -SNAPSHOT dependencies.
Or is it possible to pin them to a specific version?
For example, I have all these versions:
$ ls -lah ~/.m2/repository/com/bhauman/figwheel-main/0.1.8-SNAPSHOT/*.jar
-rw-r--r-- 1 onetom staff 135K Aug 14 20:07 /Users/onetom/.m2/repository/com/bhauman/figwheel-main/0.1.8-SNAPSHOT/figwheel-main-0.1.8-20180813.135516-2.jar
-rw-r--r-- 1 onetom staff 137K Aug 15 19:16 /Users/onetom/.m2/repository/com/bhauman/figwheel-main/0.1.8-SNAPSHOT/figwheel-main-0.1.8-20180814.194035-3.jar
-rw-r--r-- 1 onetom staff 140K Aug 21 08:29 /Users/onetom/.m2/repository/com/bhauman/figwheel-main/0.1.8-SNAPSHOT/figwheel-main-0.1.8-20180818.154912-4.jar
-rw-r--r-- 1 onetom staff 140K Aug 21 08:29 /Users/onetom/.m2/repository/com/bhauman/figwheel-main/0.1.8-SNAPSHOT/figwheel-main-0.1.8-SNAPSHOT.jar
I would be happy to just stick to figwheel-main-0.1.8-20180818.154912-4.jar until im on a slow internet connection and extra network traffic can cause several seconds of extra delay#2018-08-2216:12dominicmI think that "0.1.8-20180818.154912-4" works as a version#2018-08-2216:13onetomI was thinking that maybe one solution would be to just use a git dependency in this case, but then it also depends on SNAPSHOT versions of figwheel-core and figwheel-repl
https://github.com/bhauman/figwheel-main/blob/master/deps.edn#L6-L7#2018-08-2216:14onetom@dominicm thanks, indeed it does.
so i can just specify explicit versions of the transitive dependencies too and it should be fine.#2018-08-2216:15onetomstill, it would be good to be able to disable automatic SNAPSHOT updating somehow. iirc i could do that in boot#2018-08-2216:19dominicmin boot I pass --offline#2018-08-2216:21onetomyeah, that's the option i was thinking about!#2018-08-2618:21bhaumanIs there a central list of tools.deps tools like packr?#2018-08-2618:34dominicmJ/w Do you mean pack.alpha? #2018-08-2618:22bhaumanI remember there was one called mayven or something like that and I can’t seem to find it out there on webs#2018-08-2618:22seancorfieldThere's a partial list on repo's wiki#2018-08-2618:24bhaumanDo you remember the mayven project? I though metosin did it but nope.#2018-08-2618:24bhaumanIt’s not on the wiki#2018-08-2618:24bhaumanthanks for pointing out that page though#2018-08-2618:25seancorfieldhttps://github.com/clojure/tools.deps.alpha/wiki/Tools (for other folks wondering about the URL)#2018-08-2618:26seancorfieldI don't recall mayven (but I'm terrible with names)#2018-08-2618:26bhaumangood pin#2018-08-2618:26bhaumanand its a horrible name to search for#2018-08-2618:27seancorfieldYeah, Bing isn't turning up anything for me with mayven in it.#2018-08-2618:45Alex Miller (Clojure team)I recall it but don’t have a link. I don’t remember it being a tools.deps thing?#2018-08-2619:11bhaumanI thought it was a tools deps thing but that may be why I’m not finding it#2018-08-2619:51dominicmhttps://github.com/danielsz/meyvn#2018-08-2619:53dominicm@bhauman ^#2018-08-2621:37bhauman@dominicm Thanks!#2018-08-2701:17steveb8nhas anyone else used cognitect.test-runner in CI? I’m finding it returns status zero when there are test failures#2018-08-2701:26seancorfield@steveb8n Are you sure? I use it with clojure.java.jdbc and I'm pretty sure I get a non-zero status when there are failures.#2018-08-2701:26seancorfieldAre you using clojure or clj?#2018-08-2701:26steveb8nI’m using clojure#2018-08-2701:26steveb8nis that a possible explanation?#2018-08-2701:27steveb8nI can reproduce on my laptop also. have looked at source and it doesn’t jive#2018-08-2701:27seancorfieldclojure is the underlying script so it should be safe(r) anyway.#2018-08-2701:27seancorfieldAre you on an older version of clj/`clojure`?#2018-08-2701:28steveb8nnope. 1.9.0 in deps.edn#2018-08-2701:29steveb8nI might try debug in Cursive and see if that explains it#2018-08-2701:29steveb8nweird#2018-08-2701:31cfleming@steveb8n I think @seancorfield is talking about the clj/`clojure` CLI scripts, not Clojure itself.#2018-08-2701:31steveb8noh good point. I’ll check#2018-08-2701:32steveb8n1.9.0.375#2018-08-2701:35steveb8nwhen I run from repl I see correct results so I think Sean is right in that this is maybe the scripts masking the system/exit value#2018-08-2701:41steveb8njust updated clojure using brew. same result sadly. will keep digging#2018-08-2701:43steveb8njust spotted this commit. I’m on an old version of the test runner. testing new sha now#2018-08-2701:44steveb8nthat was it. thanks for the sounding board 🙂#2018-08-2701:47steveb8ninteresting challenge to keep the SHA’s in readmes current when a jar/snapshot not available in clojars. not sure how that can be improved.#2018-08-2702:25Alex Miller (Clojure team)rub some script on it#2018-08-2702:30seancorfield@steveb8n Sorry, stepped away. Glad you got it figured out. Not sure why a SHA is any harder to keep up to date than a Maven/Clojars dependency? You still have to go look at the project and see what's changed...#2018-08-2702:33steveb8nfair point. I agree. I guess the exception is snapshots but I generally don’t use them anyway. also the clojars build/badge provides the latest without a readme update I believe. that’s one way that clojars beats shas for readme maintenance. anyway I’m good and I’ll check latest shas from now on. thanks all#2018-08-2704:34levitanongWhat’s the difference between putting some code in "-e '(foo)'" and putting the same code in some namespace with -main, and putting it in an alias main-opt?#2018-08-2705:57steveb8n@alexmiller interesting turn of phrase. I might “borrow” that#2018-08-2818:35richiardiandrea@seancorfield question about clj-new's generators - how do I handle the case where I want to modify a file in the project root, like a deps.edn?#2018-08-2818:35richiardiandrea(and is it common/good practice?)#2018-08-2818:45seancorfield@richiardiandrea That sounds like a nice enhancement -- to expose the prefix argument somehow. Feel free to open an issue.#2018-08-2818:45richiardiandreaok#2018-08-2818:45richiardiandrea@seancorfield oh I think this is similar to https://github.com/seancorfield/clj-new/issues/8#2018-08-2819:29johnLooks like it's been implemented here: https://github.com/juxt/pack.alpha/blob/master/src/mach/pack/alpha/inject.clj#2018-08-2818:47seancorfieldThe main obstacle right now is that you can't specify the prefix (or any other parameters to generate-code under the hood), mostly because the command line processing isn't very sophisticated. Hence the open issue about looking at tools.cli for some of this.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-08-2819:34richiardiandrea@seancorfield sorry other question, can I build a template that accepts a single path in clj-new ?#2018-08-2819:40seancorfieldI'm not quite sure what you're asking there?#2018-08-2819:48seancorfieldIf your template function accepts [name & args] instead of the default [name], then any additional arguments you provides on the command-line are passed in...#2018-08-2819:49richiardiandreait seems like I always receive an error directly from clj-new if I pass the project name as my-template-test for instance#2018-08-2819:50seancorfield$ clj -Sdeps '{:deps {clj-new {:git/url "" :sha "108f27159501f8ce12564b398b0ea50def3892b1"}}}' -m clj-new.create template myt/thing
...
$ vi thing/src/clj/new/thing.clj # add & args and print them out
$ clj -Sdeps '{:deps {clj-new {:git/url "" :sha "108f27159501f8ce12564b398b0ea50def3892b1"} my/thing {:local/root "./thing"}}}' -m clj-new.create thing another.generated.thing with args
Generating fresh 'clj new' thing project.
args= (with args)
$ #2018-08-2819:50seancorfieldclj-new does not accept a single-segment name#2018-08-2819:50richiardiandreaoh ok that's what I feared#2018-08-2819:50seancorfieldIt must either be a multi.segment.name or a qualified/name#2018-08-2819:51seancorfieldPhil always regretted allowing that bad practice for lein new 🙂#2018-08-2819:51richiardiandreaoh ok got it#2018-08-2819:51seancorfieldYou can always say my-template-test.core to get the Leiningen behavior.#2018-08-2819:51richiardiandreatoo bad because it feels natural for JS devs (which are my audience now)#2018-08-2819:52richiardiandreayeah#2018-08-2819:52richiardiandreahave to explain that 😄#2018-08-2819:53seancorfieldThe docs say:
> The project name should be a qualified Clojure symbol, where the first part is typically your GitHub account name or your organization's domain reversed, e.g., com.acme, and the second part is the "local" name for your project (and is used as the name of the folder in which the project is created). An alternative is to use a multi-segment project name, such as foo.bar (the folder created will be called foo.bar and will contain src/foo/bar.clj).#2018-08-2819:53seancorfieldBTW, note above how I generated a template and then used it via :local/root 🙂 Gotta love the whole clj / deps.edn thing!#2018-08-2819:55richiardiandreayeah I am using the :local/root thing, it is a nice thing#2018-08-2819:56richiardiandreathe thing is, JS users do not care/are used to the Java wait of packaging - therefore they consider it an additional burden..#2018-08-2819:56richiardiandreasome people here at least 😄#2018-08-2819:56richiardiandreaso the naming now for them feels Java-y#2018-08-2819:56seancorfieldJS people should learn the benefits of globally unique names 🙂#2018-08-2819:57seancorfieldI have zero sympathy! :rolling_on_the_floor_laughing:#2018-08-2819:58richiardiandreawell, they have a group as well but very few people use it#2018-08-2820:01seancorfieldI will admit that when generating a new template, the qualified/multi-segment thing doesn't work as expected. If you use a qualified/name then the qualified part is ignored. If you use a multi.segment.name you get an illegal project (src/clj/new/multi.segment.name.clj). That's a bug that I haven't opened an issue for yet because there are restrictions on how template functions are named, including their namespace, so I'm not sure how best to resolve that conflict.#2018-08-2820:02seancorfieldThe simplest solution would be to allow simple names for templates but... ¯\(ツ)/¯{:tag :div, :attrs {:class "message-reaction", :title "smiley_cat"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😺")} " 4")}
#2018-08-2820:10richiardiandreaalso I don't see any util to get only the myproj from something like my.long.group/myproj#2018-08-2820:14seancorfieldIf you generate an actual project for my.long.group/myproj the qualifier part goes into the namespaces: (! 978)-> clj -Sdeps '{:deps {clj-new {:git/url "" :sha "108f27159501f8ce12564b398b0ea50def3892b1"}}}' -m clj-new.create app my.long.group/myproj
Generating a project called myproj based on the 'app' template.
Tue Aug 28 13:13:34
(sean)-(jobs:0)-(~/clojure)
(! 979)-> tree myproj
myproj
|____.gitignore
|____.hgignore
|____CHANGELOG.md
|____deps.edn
|____doc
| |____intro.md
|____LICENSE
|____README.md
|____resources
| |____.keep
|____src
| |____my
| | |____long
| | | |____group
| | | | |____myproj.clj
|____test
| |____my
| | |____long
| | | |____group
| | | | |____myproj_test.clj
#2018-08-2821:52richiardiandreauhm this does not seem to work in my template...I always get: Could not create directory /home/arichiardi/tmp/my.long.group/myproj. Maybe it already exists? Use -f / --force to overwrite it..
Super weird I am doing the same thing that your app is doing#2018-08-2822:50richiardiandreaoh found the problem, :name is actually used by ->files! didn't expect that 😄#2018-08-2820:14seancorfieldThat's how Leiningen and Boot work.#2018-08-2820:14richiardiandreayes that works but I would like to have {:project-name ...} in my data#2018-08-2820:14richiardiandreaworking on a patch#2018-08-2820:15seancorfield{:project-name ...} in what data?#2018-08-2820:16richiardiandreathe template data#2018-08-2820:16richiardiandreatogether with {:year ... :name ...}#2018-08-2820:16seancorfieldAh, yes, the template generator could do with a lot of enhancements.#2018-08-2820:16richiardiandreathis is something I really need, so patch will be coming#2018-08-2820:17seancorfieldThe app generator has :name for (project-name name)#2018-08-2820:17seancorfieldYou might consider sending a similar patch to boot-new since it's pretty much the same code, as I recall?#2018-08-2820:18seancorfieldThe template already has :raw-name by the way -- which is the whole name.#2018-08-2820:18seancorfieldhttps://github.com/seancorfield/clj-new/blob/master/src/clj/new/template.clj#L9-L14#2018-08-2820:19seancorfield(but the template template does not have all that -- and it probably should https://github.com/seancorfield/clj-new/blob/master/src/clj/new/template/temp.clj )#2018-08-2820:20richiardiandreakk will see what I can do#2018-08-2820:20richiardiandreamaybe I can just roll my own template#2018-08-2823:32cflemingAm I correct in thinking that there’s no way right now to get the CLI to output the actual Java command that would be executed?#2018-08-2823:32cflemingIs that cached, or is it only the classpath that’s cached?#2018-08-2823:58cflemingNever mind, poking through the code, I see the .libs, .cp, .jvm and .main files.#2018-08-2902:56Alex Miller (Clojure team)You’re correct in that there is no method to dump the actual full Java command#2018-08-2902:56Alex Miller (Clojure team)But it’s assembled from those pieces#2018-08-3001:34cflemingI don’t really understand the rationale for having separate alias types, e.g. R, C, O, M#2018-08-3001:35cflemingWhy not just have a single alias type and merge in whatever it contains? What’s the use case for separating them?#2018-08-3002:06kenny@cfleming I find it useful to have a :test alias that includes test dependencies and :main-opts, which I can optionally enable to actually run the tests.#2018-08-3002:16Alex Miller (Clojure team)you may have heard that Rich likes taking things apart#2018-08-3002:51cfleming@kenny But if you had a single alias for that that would work anyway.#2018-08-3015:40kennyAh yes, you're right. That simplifies things. I have no use case for the others now.#2018-08-3002:51cflemingThe dependencies would get used when calculation the classpath, and :main-opts would get picked up when running.#2018-08-3002:53cflemingYou don’t need the concept of separation for that.#2018-08-3002:53cfleming> you may have heard that Rich likes taking things apart
Sure, but I hope he also has reasons for that. I haven’t seen any for this particular case, which is why I asked.#2018-08-3002:54Alex Miller (Clojure team)the idea is that they can be used for different parts of the lifecycle and you can choose which part you’re affecting. Or you could say R,C,O,M are the simple part and A is the easy part#2018-08-3002:55cflemingBut surely the different parts of the lifecycle will just get picked up as needed anyway?#2018-08-3002:56cflemingIn Kenny’s test example, the :main-opts will just get ignored when building a classpath.#2018-08-3002:56cflemingIt seems like you’d only use that when you want to remove specific fields when applying an alias, and I can’t imagine a case where that would be useful.#2018-08-3002:56Alex Miller (Clojure team)building a classpath involves both the R and C parts#2018-08-3002:57Alex Miller (Clojure team)this is not a part of the design that I feel strongly about and I’m not going to give you an impassioned defense for it#2018-08-3002:58Alex Miller (Clojure team)as you know, sometimes decisions that seem important early seem less so later#2018-08-3002:58cflemingOk, thanks.#2018-08-3002:59Alex Miller (Clojure team)fwiw, I only use A these days#2018-08-3003:00cflemingYeah, that’s what I would do too. I’m mostly wondering how much support I should be adding for this to Cursive, or if I should just assume everything is implicitly A#2018-08-3003:00cflemingI think I’ll do that until someone complains about it.#2018-08-3003:00Alex Miller (Clojure team)I think that would get you far#2018-09-0408:42pesterhazyIs there a better way to get a new snapshot build than rm -rf .cpcache ~/.m2/repository/com/bhauman?#2018-09-0409:19delaguardoclj -Sforce should ignore .cpcache But why you want to delete your local maven repo?#2018-09-0409:20delaguardojust curious#2018-09-0409:29dominicm@pesterhazy @delaguardo I'm not certain, but I believe there is delegation to maven for this. So the usual 24h period is in place, regardless of ignoring the cpcache.#2018-09-0409:49pesterhazy@delaguardo when a new version gets uploaded to maven, I need to download it. By deleting the local cache I make sure the files get re-downloaded#2018-09-0409:50pesterhazyJust deleting .cpcache won't be sufficient AFAIK#2018-09-0409:54dominicm@pesterhazy j/w, do boot/lein/mvn provide options for this?#2018-09-0409:55delaguardohm… you can not upload to maven new release version (without -SNAPSHOT suffix) so when maven coordinate is changed in deps.edn (e.g. 1.2.0 -> 1.2.1) it will download it. For snapshoted versions it works differently: -SNAPSHOT suffix is replaced with the timestamp and after uploading new snapshot in maven you should have two with different timestamps.#2018-09-0409:55delaguardoall those cases already handled by tools.deps and there is no need to drop your local cache#2018-09-0409:56pesterhazyI was clearly talking about snapshot builds in my original question, no?#2018-09-0409:57pesterhazylet me check the timestamp theory#2018-09-0409:58delaguardojust try with -Sforce without deleting .m2#2018-09-0409:59pesterhazyI will, thanks!#2018-09-0409:59pesterhazyPretty sure that boot supports boot -u to re-download new snapshots#2018-09-0410:00pesterhazyWanted to check, but the web page doesn't list the command line arguments - it refers you to boot -h - and installing boot takes 5 min#2018-09-0410:00delaguardoboot -u is for upgrading boot tool itself#2018-09-0410:00pesterhazyOk, it's boot -U#2018-09-0410:00pesterhazyoh no#2018-09-0410:00delaguardosame, but for the latest snapshot#2018-09-0410:01pesterhazylooks like I misremembered#2018-09-0410:49Alex Miller (Clojure team)tools.deps uses daily update strategy for SNAPSHOTs currently so won’t update new ones even with -Sforce#2018-09-0410:50Alex Miller (Clojure team)I’m planning to change that but haven’t gotten to it yet#2018-09-0410:52Alex Miller (Clojure team)So for the moment you would need to both delete this portion of your .m2 and use -Sforce (no need to drop ALL of your .m2 though)#2018-09-0410:54Alex Miller (Clojure team)Or you could gen a pom with -Spom and use mvn -U dependency:tree to have Maven do it (followed by -Sforce to tell deps to update)#2018-09-0414:36pesterhazythanks for the explanation!#2018-09-0414:37pesterhazyI guess rm -rf is easy enough, it's just hard to explain to others in the team#2018-09-0415:45Alex Miller (Clojure team)Yeah#2018-09-0508:53maleghastHey there, everyone... I am a little lost, but probably just being stupid; I want to create an uberjar from my tools-deps based application. Any clues, links, etc..? (My Google-Fu has failed me completely)#2018-09-0508:59dominicmhttps://github.com/juxt/pack.alpha and https://github.com/healthfinch/depstar/#2018-09-0509:07maleghastThanks @dominicm#2018-09-0509:11maleghast@dominicm - Which Jar approach would you recommend via Pack for Docker-based deployment..?#2018-09-0509:12dominicm@maleghast I recommend capsule as a good default 🙂#2018-09-0509:12maleghast🙂#2018-09-0509:12maleghastThx#2018-09-0509:49maleghast@dominicm - I may be doing something stupid (very likely), but on running the following:
clj -A:pack mach.pack.alpha.capsule foundation.jar -e build-dir --application-id foundation --application-version "$(git describe)" -m foundation.main
I can find the file -> mach.pack.alpha.capsule <- but not my jarfile... 😞#2018-09-0510:53dominicmWhat output are you getting? None?#2018-09-0510:56maleghaster, hold on...#2018-09-0510:58maleghast@dominicm I get / got this ->
https://gist.github.com/maleghast/b671e5299d709c79813ca80b4a81f7bd#2018-09-0511:29dominicmThere's a file named after the namespace, odd! #2018-09-0511:30dominicmWhat's in it? #2018-09-0511:30dominicmAlso, how does your pack alias look? #2018-09-0511:31maleghasthuh?#2018-09-0511:32dominicmThere's a file named mach.pack.alpha.capsule in that folder #2018-09-0511:32maleghastYeah, I know#2018-09-0511:33dominicmWhat is it?#2018-09-0511:33maleghastI copied your command from the Pack Github page and just changed the name of the output jarfile and the app main namespace (it's up there ^^)#2018-09-0511:34maleghastI have no idea what it is, I thought it was supposed to be there.#2018-09-0511:34dominicmNope#2018-09-0511:34dominicmWhat does your pack section in deps.Edn look like? #2018-09-0511:36maleghastEr, one sec#2018-09-0511:36maleghast:pack {:extra-deps
{pack/pack.alpha
{:git/url ""
:sha "1d455f6c591f440560c57a29e6d930004d735dcf"}}
:main-opts ["-m" "mach.pack.alpha.capsule"
"-m" "foundation.main"]}
#2018-09-0512:35mike_ananev@maleghast I recommend you to stay with lein for some time. pack.alpha, depstar and other have some lacks... one of them copy src files into uberjar, another doesn't copy resource folder into uberjar. I've investigated all of them and IMO lein is still preferable way to build uberjar.#2018-09-0513:29dominicmpack does put src files in, because that's what you generally want. If you want to AOT, that is supported with pack, but you do it as an earlier step.
I'm happy to point you at how you can do this, and we're working on improving the documentation to make it much clearer on how to do this kind of thing.#2018-09-0609:17Andreas Liljeqvistplease point 🙂#2018-09-0609:20dominicmhttps://github.com/juxt/edge/blob/master/bin/uberjar#2018-09-0609:20dominicmhttps://github.com/juxt/edge/blob/master/bin/uberjar#L17-L26 in particular#2018-09-0512:37mike_ananev@maleghast most promising tool is https://github.com/luchiniatwork/cambada#2018-09-0513:00maleghastThanks @mike1452 I will take a look... The app I am trying to UberJar is built on top of a library that @dominicm works on, that uses tools.clojure and deps.edn and as such uses neither leiningen or boot, and I am loath to add either into the mix if I don't have to. That being said, if I have to...#2018-09-0513:21maleghast@dominicm - I think that I know what's wrong with the command I pasted above...#2018-09-0513:21maleghastthis works:
clj -A:pack foundation.jar --application-id foundation --application-version "$(git describe)" -m foundation.main
#2018-09-0513:27maleghast@dominicm - yeah, that does work... However I need to figure out how to package up the CSS etc. as the app is functioning, but looks AWFUL 😉#2018-09-0513:27dominicm@maleghast yeah, I figured that the alias you had already had the mach.pack.alpha.capsule in the alias 🙂#2018-09-0513:27maleghastYeah, that's what I worked out as well...#2018-09-0513:28dominicm@maleghast the -e build_dir option is a flag to provide an extra directory to be combined into the jar. So you can build your css/js into a directory, and have it run.#2018-09-0513:28dominicmIf you're using edge, there's an uberjar script in master which has these steps in 🙂#2018-09-0513:28maleghastAh... Thanks! 🙂#2018-09-0513:28maleghast(I am using edge)#2018-09-0514:56pesterhazyMade a script for grepping in your classpath: https://gist.github.com/pesterhazy/74f6bc235dfe5f64690250d2ee84969d#2018-09-0514:56pesterhazy#2018-09-0514:58pesterhazye.g. clojure -Spath | unpack-cp, then cd to that directory and run grep from there#2018-09-0518:17mike_ananev@dominicm Did you use pack/pack.alpha? can't understand how to avoid copying source files into uberjar using this tool#2018-09-0518:39dominicmI wrote it. Is having source files in there bad? #2018-09-0710:08mike_ananevWell it depends. I think it should as parameter, like lein does ( :omit-source true) . When I put my uberjar in external untrusted environment I don't want that my source files be availabale for other people.#2018-09-0710:09dominicmI haven't looked into that kind of use case particularly, but I understand now. Skipping the :paths seems like a possible flag that can be added though, it shouldn't be too difficult.#2018-09-0719:03mike_ananevYeah. It would be great.#2018-09-0621:37gashow do I clear the cache trying to do clj -A:new figwheel-main hello-world.app --reagent but getting yesterdays cached version of fighweel.main template?#2018-09-0622:08bhauman@gas 0.1.9 is out so you can just update to that#2018-09-0622:27gasyeah I see that however also saw you updated template so tryed to run clj -Anew but keep getting old template settings?#2018-09-0623:29seancorfield@gas does clj -Sforce work to clear the cache (that clj keeps)?#2018-09-0623:31seancorfield(or you can just delete the .cpcache folder where you are running clj and that should force it to recreate any classpaths)#2018-09-0623:31seancorfieldI think Alex mentioned a bug around snapshot handling...#2018-09-0623:31gastrying -Sforce @mo where is cpcache udner home?#2018-09-0623:31seancorfieldAh, here we go https://clojurians.slack.com/archives/C6QH853H8/p1536058198000100#2018-09-0623:32seancorfieldThe .cpcache folder exists wherever you run clj#2018-09-0623:32seancorfield(it creates that in every folder that you run it in)#2018-09-0623:33gasok -Sforce didnt work.#2018-09-0623:33seancorfieldPer Alex's comment?#2018-09-0623:33seancorfield(snapshots only get checked for once a day)#2018-09-0623:34gasyeah but not sure how bhaumann figwheel-main template works as it doesnt seem to have a version. presume its goes off git?#2018-09-0623:36gasdont seem to have .cpcache where I am running clj? on a mac not in home wither. hmm found it in .clojure/.cpcache#2018-09-0623:37gasdelete ~/.clojure/.cpcache ??#2018-09-0623:37bhaumantemplates are a bit different#2018-09-0623:38bhaumanI’m releasing a new template very soon#2018-09-0623:39seancorfieldclj-new requests RELEASE or SNAPSHOT versions as part of the dependency lookup as I recall... Let me go look at the code. It's been a while since I wrote it.#2018-09-0623:39bhaumanoh that would be cool#2018-09-0623:40gas@bhauman ok - I have copied your template and works great - FYI only issue in template at moment you need react as deps for reagent config for it to work with regent 0.8.1#2018-09-0623:42seancorfieldOK, so here's what happens in clj-new...#2018-09-0623:42seancorfield...it looks for <template>/clj-template then <template>/boot-template then <template>/lein-template, in that order.#2018-09-0623:43bhauman@gas weird, I tested every config, I’m surprised#2018-09-0623:44bhauman@seancorfield so no snapshot versions?#2018-09-0623:44seancorfieldFor the clj-template, if the template name looks like a git URL or a filesystem local path, it will only use that.#2018-09-0623:44bhaumanoh well thats good#2018-09-0623:46seancorfieldOtherwise it will either request a RELEASE version or a SNAPSHOT version -- but I'm checking whether the flag to expose snapshot is in place...#2018-09-0623:46bhauman@gas it shouldn’t need react at at all#2018-09-0623:46seancorfield...right now, that flag isn't exposed in clj-new (only in boot-new ... and lein new)#2018-09-0623:47bhaumanvery good to know#2018-09-0623:48seancorfieldBTW, a "git template" looks like <url to repo>@<SHA> and the last part of the URL is considered to be the template name.#2018-09-0623:48seancorfieldSo you can pull specific versions of templates down from GitHub using that.#2018-09-0623:48gas@bhauman ok I commented out react and it works this time! Only diff is i had the #framework part in my core.cljs but dont see any react stuff in that code#2018-09-0623:49seancorfieldAh, it won't work with https://github.com/bhauman/figwheel-main-template because it ends it -template rather than just the template name (`figwheel-main`).#2018-09-0623:50bhauman@seancorfield the snapshot part won’t work?#2018-09-0623:51seancorfieldThe git dependency won't work.#2018-09-0623:52seancorfieldclj -A:new hello-world.app#2018-09-0623:52seancorfieldThat would work if that was indeed the name of the repo -- but you have -template on it.#2018-09-0623:53bhaumangotcha#2018-09-0623:53bhaumanthere’s already a figwheel-main repo 🙂#2018-09-0623:53seancorfieldI could provide an option to override the actual template name (like I do with local paths: /path/to/some/code::my-template#2018-09-0623:53bhaumanI think the git url covers this case pretty well#2018-09-0623:54seancorfieldWell, the problem is that the template name doesn't match the repo path.#2018-09-0623:54bhaumanoh#2018-09-0623:54bhaumanok#2018-09-0623:54bhaumanhmmmmm#2018-09-0623:55seancorfieldIf I added that option, you could do clj -A:new hello-world.app
#2018-09-0623:55seancorfieldI should make it consistent on both git and local deps really...#2018-09-0623:56seancorfieldAnd figure out a way to enable snapshot lookup...#2018-09-0623:57seancorfieldhttps://github.com/seancorfield/clj-new/issues/11#2018-09-0623:59seancorfieldhttps://github.com/seancorfield/clj-new/issues/12#2018-09-0623:59seancorfieldI'll work on those tomorrow afternoon or over the weekend.#2018-09-0707:43levitanongHi guys, does anyone have a cljs tools-deps library I can look at? Trying my hand at it, but somehow the library can’t see itself. Lol#2018-09-0711:36dominicmI'm interested in solving TDEPS-12, @alexmiller have you thought about how you'd like it to be done? #2018-09-0715:48isaachow to use clj cli compile java code#2018-09-0715:48isaacI have some java code in classpath#2018-09-0716:13Alex Miller (Clojure team)@dominicm I don’t think anything on that ticket is very good thinking about this problem#2018-09-0716:14Alex Miller (Clojure team)the problem is that the assumption going into the design is that there is a 1-1 correspondence between library name and artifact#2018-09-0716:14Alex Miller (Clojure team)this is a case where that is not true#2018-09-0716:15Alex Miller (Clojure team)everything else flows from that and without addressing that fundamental issue, there is no way to fix the issue#2018-09-0716:16Alex Miller (Clojure team)I’ve been thinking about this for a long time and I think you have to solve the “name” part or you’re not fixing anything#2018-09-0716:19Alex Miller (Clojure team)the library name here is used for dual purposes - identification in tools.deps, and lookup in Maven (the groupId:artifactId). In many places we key maps based on #1 so that really needs a new dimension of context that is more than just #2.#2018-09-0716:19Alex Miller (Clojure team)that means encoding the classifier into the name (so you get uniqueness of identity) and have the Maven extension know how to decode that for lookup#2018-09-0716:20Alex Miller (Clojure team)that could be done with a name encoding, or by using something other than a symbol as the library name#2018-09-0716:22dominicmInteresting. When discussing with @malcolmsparks I dismissed the notion of using a vector or some encoding for the key.
Something not mentioned is that there's more than just classifier as a dimension in Maven, there's also type. I suspect encoding is a bad solution because it's not information dense enough. #2018-09-0716:24Alex Miller (Clojure team){:groupId "group.id", :artifactId "artifactId", :classifier "classifier"}#2018-09-0716:37Alex Miller (Clojure team)doing one does not preclude doing the other later#2018-09-0716:39dominicmI don't have one to hand, that's certain. In that case I will defer to your data. My objection was weak and I was erring on the side of exposing everything in the case of not having access to any data.
#2018-09-0716:40Alex Miller (Clojure team)tools.deps has been out for about a year and I haven’t seen a request or question about anything other than classifiers#2018-09-0716:40dominicmIf we did that, would it be a consistent return value? I'm guessing it would involve changing the current lib map spec? #2018-09-0716:40Alex Miller (Clojure team)what is “it” in above?#2018-09-0716:40dominicmThe map version of the lib name.#2018-09-0716:41Alex Miller (Clojure team)yes, would change (grow) the spec and a couple things in the Maven extension canonicalization#2018-09-0716:41Alex Miller (Clojure team)I really need to talk to Rich to make a decision on that#2018-09-0716:43dominicmAt the moment I rely on it being a symbol in pack, in order to perform naming. That seems like a breaking change for me. Am I relying on a property I shouldn't? #2018-09-0716:43Alex Miller (Clojure team)the code changes are pretty trivial. thinking clearly about the problem here is the important part.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-09-0716:52dominicm> choose whatever name you want with lowercase letters and no strange symbols.
I was just looking into which character was appropriate. Maven is surprisingly coy about what is allowed... #2018-09-0716:54dominicmI guess I will just have to test. #2018-09-0717:28lilactownlol so I blew out ~/.m2 and now when I try and run clj:Error: Could not find or load main class clojure.main
#2018-09-0717:44dominicm@lilactown I'm guessing because the classpath is cached#2018-09-0717:48Alex Miller (Clojure team)yes, if you manually muck with .m2, you’ll need to use clj -Sforce#2018-09-0717:48Alex Miller (Clojure team)currently clj does not do any checks to verify that all the dirs/files on the classpath exist#2018-09-0717:49Alex Miller (Clojure team)I’m a bit on the fence about it - that could be added, but has the downside of a) adding startup time and b) not being tolerant of (for example), a missing src path on the cp. although I guess you could maybe only check the files.#2018-09-0718:30zentropeMaybe just a warning to “if in doubt, clj -Sforce” on “missing” errors?{:tag :div, :attrs {:class "message-reaction", :title "point_up"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☝️")} " 4")}
#2018-09-0718:32lilactownyeah, I didn’t know about -Sforce. that would be a helpful enhancement#2018-09-0718:38zentropeOr, could do the path checking AFTER the error occurs, in which chase “start up time” isn’t a factor.#2018-09-0718:40zentropeVerify that there are missing paths for a good error message, not re-do the whole op.#2018-09-0719:56Alex Miller (Clojure team)the benefit of doing it before is that you can potentially fix it, not error#2018-09-0720:00Alex Miller (Clojure team)if stuff is stale, then force#2018-09-0720:01Alex Miller (Clojure team)the question I’d like to answer is - how long does it take to do the check for a reasonable size classpath#2018-09-0723:35zentropeRight. The “after” idea is to keep the speed benefits, but provide a clearer option for the user to try: same as it is now, just slightly clearer.#2018-09-0807:24malcolmsparks@alexmiller @dominicm One example where type is important is where a project might depend on a .war file, containing some functionality. See https://codesjava.com/eclipse-maven-java-web-project for example. This is more likely to happen inside a large enterprise using J2EE, I doubt you'd find an example in 'public' repos but there are vast private ones out there. This still is about the classpath (Maven itself is primarily about classpaths, not file distribution). There are other types too: https://docs.oracle.com/cd/E19830-01/819-4712/ablgz/index.html. I don't think it's worth adding support for them in tools.alpha, but at the same time it would be good if any design around supporting classifiers can be extended (via growing) to support types as well.#2018-09-0807:24malcolmsparkshttps://clojurians.slack.com/archives/C6QH853H8/p1536336844000100#2018-09-0807:30malcolmsparksI think the 1-1 correspondence is actually quite a nice design. If you think of the artifact as the logical entity defined in the pom.xml file (with its version and dependencies), rather than an actual file in the Maven repo. 99% of cases there is a single file for the artifact, but sometimes there are also ancillary files (one per classifier), sometimes called secondary artifacts. While Maven does support depending on a primary artifact file of, say, version 1.0, and a secondary artifact of a different version, say, version 1.1, this is such a questionable practice it is unlikely to be worth supporting.#2018-09-0807:33malcolmsparksSo in conclusion, I don't think there needs to be an extended lib name of {:groupId "group.id", :artifactId "artifactId", :classifier "classifier"}. The libname, with the :mvn/version, already uniquely addresses the logical artefact (defined in its pom). There should be a way of specifying the required classifiers, perhaps with a :mvn/classifiers entry.#2018-09-0807:37malcolmsparksOne minor point is that Maven makes you a declare a dependency on each artifact file you want to download (and you just normally repeat the GAV coord with a different classifier). Therefore there might need a way of saying "this classifier please, but include the primary artefact too", and also 'this classifier please, but not the primary artefact". I think the latter case could be supported later if required via growing the design.#2018-09-0808:08malcolmsparksHaving read https://github.com/technomancy/leiningen/issues/898 now I do understand the point about tools.deps being only concerned about the classpath and perhaps extraction of native libs needs to be out-of-scope.#2018-09-0809:04dominicmSome libraries do the work themselves, http://usb4java.org/nativelibs.html#2018-09-0812:39Alex Miller (Clojure team)I can give you at least one more example that has nothing to do with native stuff#2018-09-0812:41Alex Miller (Clojure team)ClojureScript relies on a few contrib libs (like tools.reader). All of hose libs get built in both a source form (and deployed with no classifier) and in aot form (deployed with :aot classifier). #2018-09-0814:53Alex Miller (Clojure team)the idea of pushing this into the coord is interesting. The current extension SPI already supports extensions returning multiple artifacts per lib. I do wonder about whether this has impacts for walking the transitive deps? Something to look at.#2018-09-0814:56Alex Miller (Clojure team)I guess there’s still only one pom defining deps for a lib, even in the case where there are multiple classifiers#2018-09-0816:06Alex Miller (Clojure team)as I looked at it more, I like this solution and I have implemented it and committed the change for the next release - see TDEPS-12 for more#2018-09-0817:54gfredericksI was thinking again about self-contained clojure files
I realized you could have a pretty clean impl at the cost of a mildly longer shebang line, if you publish a library (ideally with a very short name) that can read deps out of lines 2->N of a file, and execute everything after that using -Sdeps
so the shebang would be roughly #! clojure -Sdeps '{:deps {lib7 {:mvn/version,"1"}}}' -m lib7.main
I think the tricky part is that lib7 couldn't do a unix exec to replace the original jvm; so maybe the cleanest thing for it to do would be to have tools.deps as a dep, and call it internally, and make its own classloader with the resulting classpath...
so
A) are there any libs yet that accomplish anything comparable to this?
B) is this crazy or impossible?#2018-09-0817:59gfredericksStartup time would be interesting#2018-09-0818:29dominicmThere's a library to make isolated class loaders from a deps edn. #2018-09-0818:29dominicmStartup time would be boot-like#2018-09-0818:31gfredericksdo you recall the library's name?#2018-09-0818:38dottedmag@gfredericks Shebangs do not get parsed into arguments.#2018-09-0818:39dottedmagAnd yes, someone has done something like that.#2018-09-0818:40dottedmagIIRC via file that can be read either as a shell script (that runs clj) or as Clojure code (where shell header gets ignored via e.g. ;). With #!/bin/sh shebang.#2018-09-0818:41gfredericksoh if you can't in any syntax pass two args to clojure then I guess that would kill that idea#2018-09-0818:41dottedmag#!/bin/sh
; exec clj .... $0
<clojure code follows>
#2018-09-0818:42dottedmagSomething like this.#2018-09-0818:42gfrederickshaving lots of deps crammed into one line is unfortunate#2018-09-0818:43gfredericksI bet some fancy bash could support multiple lines#2018-09-0818:43dottedmagMultiple lines of what?#2018-09-0818:43gfredericksdeps#2018-09-0818:44dottedmagWhy? I don't remember a requirement that shell script has to fit on a punched card 🙂#2018-09-0818:44gfredericksjust annoying when viewing and editing#2018-09-0818:44gfredericksnot a deal breaker#2018-09-0818:44dottedmagI'd generate this script instead of editing it manually.#2018-09-0818:44gfredericksgenerate it from what?#2018-09-0818:45gfredericksI guess I mean the thing I'm going for is casual self-contained shell scripts#2018-09-0818:45gfredericksnot tied to anything else#2018-09-0818:45gfredericksso generating it kind of defeats the purpose#2018-09-0818:45dottedmagand yes, should be trivial: #!/bin/sh
; CLJ="clj ..."
; CLJ="$CLJ ..."
; CLJ="$CLJ ..."
; exec $CLJ
<clojure code follows>
#2018-09-0818:46gfredericksI wonder if you could take advantage of the deps map being a noop to have it outside the comments#2018-09-0818:46gfredericksso you get syntax highlighting and editor help#2018-09-0818:48gfredericksI get Syntax error: ";" unexpected for that sort of thing#2018-09-0818:55gfredericksthis is the holy grail I think
#!/bin/sh
... magic bash
{:deps {...
...
...
...}}
... more magic bash, with exec
(ns wahoo ...)
#2018-09-0819:46borkdudeThis also uses a similar trick:
https://github.com/borkdude/balcony/blob/master/balcony.clj#2018-09-0819:46borkdudeDoes anyone have an example of how to use clj/tools-deps with AOT + uberjar?#2018-09-0819:47gfredericks":"; seems designed to avoid the error I mentioned above#2018-09-0819:48borkdudeyes, it was the shortest bash command I could find that was also valid clojure#2018-09-0819:48borkdudeI think @mfikes came up with it#2018-09-0819:51gfrederickswould "" not work?#2018-09-0819:51gfredericksI guess not#2018-09-0819:51gfrederickswhat does : mean in bash?#2018-09-1212:42souenzzoit's not plain true
it's a (def : (constantly true)){:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-09-0819:51borkdudeit’s an alias for true#2018-09-0819:52gfrederickso_O#2018-09-0819:52gfrederickssure why not#2018-09-0819:53gfredericksso your AOT/uberjar question is about asking how to take a src dir and AOT compile it and put that in an uberjar?#2018-09-0819:54borkdudeyes, with tools.deps. With lein I can find docs on that, but with tools.deps not so much I guess#2018-09-0820:10Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools#2018-09-0820:10Alex Miller (Clojure team)There are some things there to do it#2018-09-0820:11borkdudeThanks!#2018-09-0819:54gfredericksI thought I saw an uberjaring lib at some point#2018-09-0819:55gfredericksw.r.t. AOT, it might be true that by calling compile enough you will get class files written to the classes dir#2018-09-0819:55gfredericksbut there might be lots of edge cases involved that I don't know about#2018-09-0819:56gfrederickse.g., if you have a top-level namespace that requires everything else, try clojure -e '(compile (quote that.ns))'#2018-09-0820:01borkdudeif [ ! : ]; then
echo "True!"
else
echo "False!"
fi
# prints "False!"
#2018-09-0820:07gfrederickswho wouldn't want to be able to write code like that!#2018-09-0820:10borkdudehowever, this doesn’t seem to work:
if [ : && : ]; then
echo "True!"
else
echo "False!"
fi
#2018-09-0905:05dominicm@borkdude https://github.com/juxt/edge/blob/master/bin/uberjar this does aot w/ pack#2018-09-0911:52gfredericksI figured it out!
#!/usr/bin/env bash
":"; DEPS=\
'{:deps
{org.apache.commons/commons-compress {:mvn/version "1.17"}}}'
":"; exec clojure -Sdeps "$DEPS" "$0"
#2018-09-1013:02rickmoynihanNeat. I’ve had a need for something like this this morning.
Have been meaning to ask why clojure doesn’t barf on the #!/usr/bin/env bash?#2018-09-1013:04rickmoynihan(read-string "#!/usr/bin/env bash") blows up… but clojure seems to somehow let it through…#2018-09-1013:38gfredericks#! is indistinguishable from ; AFAIK
your read-string call fails with ; as well, because there's nothing for it to return -- add a second line with a valid form and both will work#2018-09-1013:39rickmoynihanYeah scroll down #tools-deps 🙂#2018-09-1013:39gfredericksoh sorry#2018-09-1013:39rickmoynihannot at all; thanks for answering 🙂#2018-09-0911:52gfredericks☝️ that seems sufficient to let you keep your deps in the clojure-syntax part of the file#2018-09-0912:12gfredericksalso an advantage of not putting clojure in the shebang is that you don't have to give the fully qualified path to it#2018-09-0912:15gfrederickspro-tip: if you ever have to use a lib that contains a ' in its version string (which would threaten the bash quotation), use \u0027 instead#2018-09-0912:18dominicmYou can use a quote for a version? #2018-09-0912:20gfredericksprobably
{:mvn/version "12.43.01'-sure"}
I mean I haven't tried it#2018-09-0912:20dottedmag{haskell/haskell {:mvn/version "'"}#2018-09-0912:21gfredericksI put a fuller example here https://gist.github.com/gfredericks/900183dae706aae6361110b2383c8639#2018-09-0914:30borkdude@dominicm cool!#2018-09-0915:28pesterhazyWeird and wonderful #2018-09-1012:11aitemHi to all.
How i can omit source and obfuscate my target .jar file using deps.edn and clojure cli?
Previously i used leiningen and proguard and all works fine.
But in current project we are use clojure cli and juxt/pack.alpha for building .jar.#2018-09-1013:23rickmoynihan(read-string "#!/usr/bin/env bash") blows up… but clojure seems to be ok with it… This is the only bit I don’t understand, presumably the clj/`clojure` tools strip this out?#2018-09-1013:29dominicm@rickmoynihan they do not, this is in clojure.core#2018-09-1013:29rickmoynihanreally#2018-09-1013:29rickmoynihanwow#2018-09-1013:29dominicm! is technically a tagged literal which comments out the whole line#2018-09-1013:30rickmoynihanthat was my first thought… but why does read-string blow up?#2018-09-1013:30rickmoynihanahhh hold on maybe it’s because there’s no more input…#2018-09-1013:30rickmoynihanyeah that’s it#2018-09-1013:30rickmoynihanthanks#2018-09-1013:33rickmoynihan(read-string "#!/usr/bin/env bash\n 10") works (read-string "#!/usr/bin/env bash") fails#2018-09-1013:33dominicmboring answer 😉 Why is it always a boring answer#2018-09-1013:33dominicm(Because physics isn't broken){:tag :div, :attrs {:class "message-reaction", :title "grin"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😁")} " 4")}
#2018-09-1013:40rickmoynihanWell Rich had remarkable foresight when he added this feature ten years ago 🙂
https://github.com/clojure/clojure/blame/master/src/jvm/clojure/lang/LispReader.java#L116#2018-09-1013:40rickmoynihanThough I guess even then this was useful, even then… more so now we have the clj tools#2018-09-1014:09dominicmThis was clearly always the plan 😛#2018-09-1015:21Alex Miller (Clojure team)Rich plays the longest game#2018-09-1015:36dominicmI'm waiting for an IDE for clojure to appear in the clojure codebase.#2018-09-1016:20Alex Miller (Clojure team)No comment#2018-09-1023:24Joe LaneOh man this comment is going to keep me up tonight.#2018-09-1103:05Alex Miller (Clojure team)Mission accomplished#2018-09-1015:42sparkofreasonI need to use gen-class it implement an abstract Java class, but am having no luck getting it to load in the REPL for cursive. I assume it needs to be AOT'ed to be present on the classpath for clj. How is this accomplished with deps.edn? Or do I need a separate tool?#2018-09-1016:02sparkofreasonIf I use compile and add classes to :extra-paths I get this exception on import.#2018-09-1016:31niquola@aitem i’m curious too how distribute clojure closed source jars. How datomic is distributed? May be @alexmiller help us.#2018-09-1016:56Alex Miller (Clojure team)Don’t know - I’m not on the Datomic team :)#2018-09-1017:57niquolaWho can we ask about this?#2018-09-1018:26Alex Miller (Clojure team)ask in #datomic#2018-09-1016:33seancorfield@rickmoynihan Looking at that code makes me wonder what #< is for since, right now, both in EDN and Clojure, the reader for that simply throws an exception.#2018-09-1016:43rickmoynihan@seancorfield: yes that’s a curious one#2018-09-1016:43seancorfieldI'd forgotten about ## as well -- and #! was new to me too.#2018-09-1016:43dominicm #<foo> might be something?#2018-09-1016:44rickmoynihanyeah ## was new in 1.9#2018-09-1016:45seancorfieldCan you override how #< is read? Or do you mean "reserved for some future, unknown purpose"?#2018-09-1016:46rickmoynihanI didn’t think the reader was extensible/overridable; at least not beyond tagged literals.#2018-09-1017:08dominicm@seancorfield I'm guessing a bit. #2018-09-1017:21mfikes#< is for values that have no way to be read#2018-09-1019:24andy.fingerhutWow, I had never noticed that #! special case in the Clojure reader before. I won't say I never saw it, because I am sure I looked at the linked group of source lines at least a dozen times over the years.#2018-09-1020:42borkdudewhat’s an example with #<?#2018-09-1020:48andy.fingerhutAs in, is there some object that will print as a sequence of character beginning with #< ? I don't know if there is one today.#2018-09-1020:52andy.fingerhutScanning through the Clojure implementation for print and pprint I didn't notice any such case.#2018-09-1021:10Alex Miller (Clojure team)Not to my knowledge #2018-09-1021:10Alex Miller (Clojure team)I think that’s just reserving space in the reader#2018-09-1022:49gfredericksit was used for everything prior to #object wasn't it?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-09-1022:58Alex Miller (Clojure team)You think it’s that old object syntax?#2018-09-1022:58Alex Miller (Clojure team)Maybe so#2018-09-1023:11gfredericksmy interpretation was always that #<> was the way unreadable things were printed, and the #< entry in the reader was to produce the "unreadable form" exception whenever you accidentally tried to read those things#2018-09-1023:14gfrederickshere's the reader error messages from 1.2.1:
user=> (-> (Object.) pr-str read-string)
java.lang.RuntimeException: java.lang.Exception: Unreadable form (NO_SOURCE_FILE:0)
user=> (read-string "#,comma")
java.lang.RuntimeException: java.lang.Exception: No dispatch macro for: , (NO_SOURCE_FILE:0)
so I thought of the #< codepath as the way to avoid the less helpful No dispatch macro for: < message you'd get otherwise#2018-09-1023:15gfredericksand it's still useful today for reading data produced by older clojures#2018-09-1100:04johnI could have sworn I've seen the #<some-object> notation before#2018-09-1100:29gfredericks$ clojure -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.2.1"}}}' -e '(Object.)'
#<Object #2018-09-1100:39seancorfieldChanged in 1.7 Clojure 1.2.1
#<Object
#2018-09-1100:39seancorfield(the new CLI stuff makes this so easy to test!)#2018-09-1100:42gfredericksfor clojure in 1.2.1 1.3.0 1.4.0 ...#2018-09-1103:46richiardiandreaWe have a monorepo with this exact setup: https://clojure.org/guides/deps_and_cli#_using_local_libraries and sometimes we get the dreaded Manifest type not detected error...it is a local dep with a deps.edn and I wonder how I can debug what is going wrong...#2018-09-1103:50Alex Miller (Clojure team)probably related to relative paths if I had to guess#2018-09-1103:50Alex Miller (Clojure team)is it a transitive local dep?#2018-09-1103:52Alex Miller (Clojure team)there is a known issue that transitive local deps resolve paths in terms of your current directory, not in terms of the dep, which can lead to issues with finding the correct “dep root” (where the deps.edn would be expected)#2018-09-1103:54richiardiandrea@alexmiller yes it is a transitive dep that of one of the sub projects and one colleague has noticed that if he specifies it top level the problem goes away.#2018-09-1103:55richiardiandreaThanks then good to know the problem is known I will check Jira#2018-09-1103:57richiardiandrea@cfleming thanks this is the reason of the Cursive message ^#2018-09-1103:58Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-74#2018-09-1103:59Alex Miller (Clojure team)that’s the placeholder for it, there might be another one that’s same thing{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-09-1111:54cfleming@richiardiandrea Thanks for letting me know.#2018-09-1117:08dominicm@alexmiller someone in #leiningen just discovered that openjfx uses <type>pom</type> which wouldn't be supported by deps.edn#2018-09-1117:08dominicmjust as a data point#2018-09-1117:08Alex Miller (Clojure team)true#2018-09-1117:10Alex Miller (Clojure team)tools.deps is hardcoded to look only for jar extension#2018-09-1117:10Alex Miller (Clojure team)the original example in TDEPS-12 is all jars#2018-09-1117:33dominicmYep. This was something @malcolmsparks pointed out a little later.
Pom is something that is logically consistent with classpath building. I don't know how type interacts with classifier, I suspect it's multiplicative.#2018-09-1117:35malcolmsparksNot quite. Each classifier/type combination is specified in a separate dependency element in the POM. #2018-09-1117:38malcolmsparks@alexmiller thanks for comment in TDEPS-12, I've been able to build a local version of tools.deps and patch it into my /usr/bin/clojure script, no problem. Still testing.#2018-09-1117:43dominicmIt might be that a classifier key is wrong, and it should be named otherwise, and be maps with classifier and type in. #2018-09-1117:44Alex Miller (Clojure team)usually I just add the source directory of tools.deps.alpha to the front of the lib path in my clojure script :)#2018-09-1117:45Alex Miller (Clojure team)poms are used for determining transitive deps but are not themselves artifacts that get put on the classpath (these are independent parts of the tools.deps extension spi)#2018-09-1117:46Alex Miller (Clojure team)all classifiers in a group/artifact/version share the same pom#2018-09-1117:46Alex Miller (Clojure team)which is in some ways deeply weird#2018-09-1117:46Alex Miller (Clojure team)and really reinforces that GAV should primarily be about “one” artifact#2018-09-1117:47Alex Miller (Clojure team)I think you could argue pretty readily that the case in TDEPS-12 for example should be different artifactIds, not classifiers, but I don’t run the world :)#2018-09-1118:26dominicmI was thinking the same. But I did realize that having structure provides the opportunity for tooling to inspect it. Eg automatically fetching src & javadoc classifiers.
In the TDEPS-12 case, tooling has less of a place, but could feasibly be done by a maven plugin#2018-09-1118:29Alex Miller (Clojure team)source and javadoc are secondary info (just like Clojure meta) and make total sense to me#2018-09-1118:30Alex Miller (Clojure team)even things like aot as a secondary classifier make sense to me as a variant of the same artifact#2018-09-1118:31Alex Miller (Clojure team)any case where you’re putting more than one artifact from the same GAV on your classpath at the same time seems … weird#2018-09-1118:37dominicmI don't know GAV. Google isn't too revealing either, what is it? #2018-09-1118:38Alex Miller (Clojure team)sorry#2018-09-1118:38Alex Miller (Clojure team)I’ve merged with the borg apparently#2018-09-1118:38cflemingGroup Artifact Version (standard Maven ID)#2018-09-1118:38Alex Miller (Clojure team)groupId/artifactId/version#2018-09-1118:38Alex Miller (Clojure team)nice slack, out of order message delivery#2018-09-1118:39dominicmMy guess was surprisingly accurate. The version part didn't quite fit for me. #2018-09-1118:40dominicmJust so I'm clear, do you think javadoc should be a separate GAV with a pointer from the root? Or that context does make sense for a classifier?#2018-09-1118:57Alex Miller (Clojure team)no, I think the existing usage of classifiers for javadoc and source make sense#2018-09-1118:57Alex Miller (Clojure team)I think cases like the one in TDEPS-12 are weird#2018-09-1118:57Alex Miller (Clojure team)all those classifiers should be separate artifacts#2018-09-1120:33dominicmI wonder how hard it would be to have a tool which synchronized your Maven deps to add a profile with javadoc and source. I'd love that for stack traces.
A bit of a shame that it would be duplicative to isolate to an alias with the current classifier data shape. Not terrible I suppose though. #2018-09-1120:37dominicmI suppose you could use such a tool through -Sdeps too, but then cider would have to become aware of it... Unless it was a dependency of cider, then everyone would have it and be happy forever! #2018-09-1120:48seancorfieldNot everyone uses CIDER tho'...#2018-09-1120:49seancorfield(heresy, I know!)#2018-09-1120:53dominicmI think intellij is the only editor that doesn't. And atom. #2018-09-1121:01dominicmIntellij probably already does this, so is less interesting. Atom would miss out, but could integrate directly with the library. #2018-09-1122:06Alex Miller (Clojure team)intellij does this already#2018-09-1122:07Alex Miller (Clojure team)that is, download and use source and javadoc for libs#2018-09-1204:12cflemingRight, Cursive does that for lein libs, it doesn’t work for deps yet#2018-09-1204:43Alex Miller (Clojure team)ah#2018-09-1204:45cflemingClojure actually makes this pretty tricky, so in Cursive it works on demand - if you try to go to the source of a Java class IntelliJ will decompile it and prompt you to download the source at that point, it doesn’t do it up front since it’s basically impossible to know which artifacts have sources (most don’t in Clojure)#2018-09-1216:09isaacA javac tool & AOT tool https://github.com/gfZeng/tools.deps.task.compile#2018-09-1316:28abpWhen automating tests and builds with clojure we need to download maven dependencies to the project folder ideally, so we can use gitlab CI caching on the maven dependencies in the project folder. Is there any way to do this with the clojure shellscript and tools.deps? With leiningen we used local-repo https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L145#2018-09-1316:38Alex Miller (Clojure team)Yep, use a root attribute in deps.edn of :mvn/local-repo#2018-09-1316:44abpWe just found it too, but thanks anyway 🙂#2018-09-1318:25richiardiandreaHello folks! I am a bit confused by this:
clj -C:gen:new:tmpl -Stree <- no deps is shown
clj -R:gen:new:tmpl -Stree <- deps are shown
but the doc says that -C is for building the classpath...#2018-09-1318:26richiardiandreaif I am building the classpath, why is -C not showing my deps in -Stree?#2018-09-1318:27richiardiandrea(and of course I don't have them in the classpath as well)#2018-09-1318:32hiredmanis there a reason you aren't using -A#2018-09-1318:34hiredmanmy admittedly not very deep understanding is, -A does a lot of stuff, an the other flags like -C and -R do subparts of what -A, I suspect -C only merges the paths entries not the dependency entries. using -A seems to provide most of the kind of default what you would expect from "merging"#2018-09-1318:35hiredmanthere used to be a comment (maybe there still is) the default deps.edn file that said '''#2018-09-1318:35hiredman;; make-classpath aliases (-C) affect the classpath generation, options:
;; :extra-paths - vector of additional paths to add to the classpath
;; :classpath-overrides - map of lib to path that overrides the result of resolving deps
#2018-09-1318:36richiardiandrea@hiredman my alias contains :main-opts and I do not want to use it for new and gen for now#2018-09-1318:36richiardiandreaI am launching a REPL against them#2018-09-1318:37seancorfieldBeat me to it... Yeah, @richiardiandrea -C only pulls in paths and the library overrides. -R is for dependencies (`:deps`, :extra-deps, and :override-deps I think).#2018-09-1318:37richiardiandreaok I think I missed that part of the doc, thanks a lot, I thought classpath meant both path and deps#2018-09-1318:37richiardiandreaas customary in all the previous tools 😉#2018-09-1318:38seancorfieldYou mostly want -R or -M but if you are bringing in test code you probably want -C for that 🙂#2018-09-1318:38seancorfieldWhat I recommend is not having :main-opts in the same alias as anything else -- use a separate alias that is just for running that main.#2018-09-1318:39seancorfieldFor example, in several projects I have :test to set everything up for testing and :runner to actually run the tests.#2018-09-1318:40richiardiandreaso I am using new and telling my team to just do clj -A:new:tmpl cljs-template org-scope/my-project#2018-09-1318:40seancorfieldI tend to keep all my aliases small and orthogonal -- so they can be composed 🙂#2018-09-1318:40richiardiandreabut now I am just in need of a REPL with the deps#2018-09-1318:40richiardiandreaso I do not want -M I guess 😉#2018-09-1318:40seancorfieldWhich is exactly why you want the main opts separate from the deps 🙂#2018-09-1318:41richiardiandreaAnyways, this works clojure -R:gen:new:tmpl -A:nrepl#2018-09-1415:22niclasnilssonUsing deps.edn, when developing an app with some libs locally, what’s the intended way when App -> LibA and LibA -> LibB. Is there a way to do this without tainting App’s deps.edn with App’s dependencies dependencies (where all are local)?#2018-09-1415:23dominicm@niclasnilsson :local/root I think? #2018-09-1415:23mpenetyou can have a profile with local dependencies#2018-09-1415:23mpenetI mean an alias#2018-09-1415:23niclasnilsson@dominicm Only seems to work in one “layer”, not picking things up recursively#2018-09-1415:24niclasnilsson@mpenet, yep, that’s what I’d like to avoid. It’s just boring.#2018-09-1415:24mpenetwell you can add exclusions and explicit deps to other local deps#2018-09-1415:24mpenetbut yeah, it requires some work#2018-09-1415:24dominicm@niclasnilsson I see, you mean when all 3 are local. Transitive aliases don't work, I don't think there's anyway to configure libraries to load in transitive local mode. #2018-09-1415:25niclasnilssonAnd (I think) even when LibA is local and LibB is on github for instance, App doesn’t seem to pick it up. (Unless I did something wrong)#2018-09-1416:02richiardiandreaThe local dep issues are quickly becoming a problem for our Cursive users - we spend a lot of time clicking on the error dialogs popping up - we tried adding :manifest :deps but the the use from the command line broke for some reason.
I might invest some time digging into it...#2018-09-1416:03cfleming@richiardiandrea I think you want :deps/manifest :deps#2018-09-1416:04richiardiandreaAh yep I think with put that sorry 😃 will come up with a repro first then we'll see#2018-09-1417:15Alex Miller (Clojure team)@niclasnilsson don’t forget that you can put aliases in ~/.clojure/deps.edn too and then combine them with stuff in your project. so if you had a bunch of local dep overrides, you could declare them there (without touching your project deps.edn){:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-09-1423:13jcfSomething that just occurred to me. Languages like Ruby and Perl that have spent a lot more time optimising their CLIs have relatively succinct (but also somewhat cryptic) APIs for interactic with text. Consider the following examples, first in Ruby and the an approximation using clj:
$ echo 4 | ruby -pne '$_.to_i + 1'
4
$ echo 4 | clj -e '(inc (Long/parseLong (first (line-seq (java.io.BufferedReader. *in*)))))'
5
I'd argue that $_ is hard to remember and Google, but if you want to apply an operation to each line in some input the Ruby syntax is quite convenient (assuming you can remember it).
Are there plans on adding similar switches to the new clj executable to support process *in* with a bit less ceremony?#2018-09-1423:13Alex Miller (Clojure team)there is one#2018-09-1423:13Alex Miller (Clojure team)just append - at the end#2018-09-1423:15jcfHi @alexmiller! How would I change echo 4 | clj -e '(inc (Long/parseLong (first (line-seq (java.io.BufferedReader. *in*)))))' to make use of -? Sorry for being dumb but it's not immediately obvious to me.#2018-09-1423:15jcfI have used - with other programs before, but not with clj.#2018-09-1423:17jcfmain-opts
-m, --main ns-name
Call the -main function from namespace w/args
-r, --repl
Run a repl
path Run a script from a file or resource located at path
- Run a script from standard input
#2018-09-1423:18jcfI've found - in the man page for clj.#2018-09-1423:22jcfFor context I'm thinking I want to make it easier doing something like curl | jq '.users' | clj -e '...' so I can combine Clojure with other tools I have on the command-line.#2018-09-1423:22jcfI can of course do everything from Clojure, but just wondering if there's thought going into the API for passing in data to a new Clojure process.#2018-09-1500:09seancorfield@jcf Is this still too much code? (! 550)-> echo 4 | clj -e '(-> (read-line) (Long/parseLong) inc)'
5
#2018-09-1500:13seancorfieldIf you want to read every line: (! 552)-> (echo 4; echo 13) | clj -e '(while (some-> (read-line) (Long/parseLong) (inc) (doto (println))))'
5
14
{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-09-1500:25jcfDefinitely more terse than my snippet. I like the use of read-line and a threading macro for sure!
I think we're missing a trick still by not having the switches provided by Ruby.
The whole loop-and-read boilerplate becomes redundant when you pass -n, and with -p you don't have to print…
#2018-09-1500:31seancorfieldClojure is a fundamentally different execution model tho': you're building a classpath, firing up a JVM, compiling a set of forms, and then running the compiled code.#2018-09-1500:31seancorfieldWith Ruby, it can just take the expression and interpret it repeatedly...#2018-09-1500:33seancorfield...for Clojure, you'd need some in-JVM process that takes your expression and does the looping/reading for you -- you can't do it at the script level.#2018-09-1501:10seancorfieldOK, @jcf how about this for Ruby-like conciseness? (! 584)-> (echo 4; echo 13) | clj -Apne -e '($ (inc (Long/parseLong $_)))'
#'user/$
5
14
{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 8")}
#2018-09-1501:10seancorfieldOK, @jcf how about this for Ruby-like conciseness? (! 584)-> (echo 4; echo 13) | clj -Apne -e '($ (inc (Long/parseLong $_)))'
#'user/$
5
14
{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 8")}
#2018-09-1501:12seancorfield$ is a macro defined by the pne alias that takes the following expression and evaluates and prints it repeatedly in the context of $_ bound to subsequent (read-line) calls...#2018-09-1501:13seancorfieldAs for the pne alias... be afraid... be very afraid... (! 585)-> cat deps.edn
{:aliases,{:pne,{:main-opts,["-e","(defmacro,$,[&,body],(list,'while,(list,'when-let,'[$_,(read-line)],(let,[v,(gensym)],(list,'let,(vector,v,(list*,'do,body)),(list,'println,v))),'$_)))"]}}}
#2018-09-1501:14seancorfield(you can't use backtick easily with clj due to bash issues so you have to laboriously build the entire body with list and vector -- and you have to use , for whitespace in :main-opts because of reasons... 🙂 )#2018-09-1507:26jcfThat is gnarly, but I love the novelty of your solution.
I'm guessing a solution with similar behaviour to this won't land in tools.deps, but you've inspirited me to think about writing a small library that adds this functionality.
Thanks, @seancorfield!#2018-09-1506:10dottedmag@seancorfield Maybe instead of that one can add a dependency that populates user and make :main-opts require it?#2018-09-1506:13seancorfield@dottedmag I'm sure there are several ways to solve the problem... this was just the first thing that popped into my head 🙂#2018-09-1506:19seancorfieldFeel free to post your solution too...#2018-09-1623:27seancorfield@bhauman Thanks for adding that deps.edn -- see my comments in the now-closed issue... also my comment on the most recent commit (which introduces a syntax error).#2018-09-1623:33seancorfieldIs it a known bug that a local dep inside a git dep doesn't work properly? I expected tools.deps to find the deps.edn inside the cloned repo via the local/root... but it doesn't seem to 😞#2018-09-1623:34Alex Miller (Clojure team)Yes#2018-09-1623:40seancorfieldI couldn't find the JIRA ticket for that @alexmiller -- I assume there is one?#2018-09-1623:51Alex Miller (Clojure team)There are several I think - it’s all the local dep path stuff#2018-09-1700:12seancorfieldTDEPS-74 seems to get to the heart of it. OK, I guess I'll go vote for it!#2018-09-1723:46seancorfield@alexmiller @arrdem was offering a patch for the local/root issue I believe...#2018-09-1723:46Alex Miller (Clojure team)well I’ve thought about it a lot and haven’t come up with a solution I’m happy with#2018-09-1723:47seancorfieldIf the tools.deps reader converted paths (from :paths, :extra-paths, and :local/root -- and maybe others?) from relative to absolute, based on the location of the deps.edn file in which they occur, would that solve the problem?#2018-09-1723:47Alex Miller (Clojure team)I don’t think so#2018-09-1723:47seancorfield(or, rephrased, at what point does tools.deps convert paths to absolute ones -- they are absolute in the classpath at the end)#2018-09-1723:48Alex Miller (Clojure team)the extension does it when it resolves the paths for a dep#2018-09-1723:49Alex Miller (Clojure team)but it’s really about what the “current directory” means when you are resolving a transitive dep by relative path - currently nothing is shifting that context so you always have the root dep context#2018-09-1723:52seancorfieldAh, and paths can be in both the system deps and the per-user deps and of course you don't want those resolved against the file they're defined in... yeah...#2018-09-1723:53seancorfieldAnd by the time you've collapsed all the deps down, you've lost that context, right?#2018-09-1723:56Alex Miller (Clojure team)yeah. we actually conceived local deps primarily as working primarily with absolute paths - all the stuff people are doing with relative paths and monorepos is somewhat emergent#2018-09-1723:59arrdemSo for https://github.com/fundingcircle/lein-modules, we added a “//” notation somewhat stolen from pantsbuild - the idea was that paths which were not “//” prefixed were treated as relative and just passed around, “//” prefixed paths were treated as being absolute with respect to the project.clj the path occurred in - for us implicitly the “root” project.clj of the repo which emulates the meaning of “//” in Google Blaze derived systems where “//” is repo root when referring to build target names.#2018-09-1802:56Alex Miller (Clojure team)Yeah, I’m not doing that :)#2018-09-1805:21arrdem😉 #2018-09-1805:35arrdemPerhaps a more serious option here would be #file EDN extensions... maybe not suitable for general purpose but perhaps useful for us.#2018-09-1805:42Alex Miller (Clojure team)None of that seems necessary to me#2018-09-1805:43Alex Miller (Clojure team)I think the intent of this kind of setup is clear as is, the questions are in the implementation#2018-09-1723:59arrdemThat’s worked out OK at best for us.#2018-09-1800:05arrdemfile me under monorepo user probably pushing the boundaries of expected behavior.#2018-09-1800:05arrdemI’m currently exploring replacing lein-modules for our mid-size monorepo for a couple reasons.#2018-09-1801:14richiardiandreacouldn't we have a placeholder deps.edn in the root of a monorepo and subdeps consider that first if present for :local/root?#2018-09-1804:55dominicmI did wonder if relative paths would be a different key. #2018-09-1817:54arrdemChatted about this with a coworker some and I think we’re gonna take a swing at adding a resolver type which patches up paths, everything else seeming pretty janky in comparison. We’ll see how that goes.#2018-09-1818:07Alex Miller (Clojure team)what do you mean by a resolver type?#2018-09-1818:14arrdemExtension / coordinate type.#2018-09-1819:06Alex Miller (Clojure team)I feel like you’re going down a path that is a waste of time#2018-09-1819:07Alex Miller (Clojure team)Which you’re welcome to do, but there is no support for installing your own extensions in clj, so you’ll only be able to use this with some frankensetup#2018-09-1819:11dominicmWill there ever be support for custom extensions? #2018-09-1819:30Alex Miller (Clojure team)undecided#2018-09-1819:31Alex Miller (Clojure team)I mean, you can use them now if you use tools.deps.alpha as a library#2018-09-1819:31Alex Miller (Clojure team)it’s just yet another layer of installation complexity when it comes to clj#2018-09-1820:20dominicmI suppose it would require t.d.a to orchestrate itself in order to run again with additional dependencies. Probably not completely horrible if the context allows add-lib. #2018-09-2214:40raymcdermott@alexmiller just checking that the add-lib method from earlier SHAs should still work with 1.10-alpha7#2018-09-2214:41raymcdermottcos I am having fails#2018-09-2214:44Alex Miller (Clojure team)Haven’t tried it, but not sure why it wouldn’t off the top of my head#2018-09-2214:44Alex Miller (Clojure team)Fails how?#2018-09-2214:45raymcdermottgives back a false#2018-09-2214:45raymcdermottfor the example memoize (and others)#2018-09-2214:46raymcdermottcould be something on my end - just wanted to make sure that it wasn’t meant to be deprecated#2018-09-2214:49Alex Miller (Clojure team)$ clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.0-alpha7"} org.clojure/tools.deps.alpha {:git/url "" :sha "d492e97259c013ba401c5238842cd3445839d020"}}}'
Downloading: org/clojure/clojure/1.10.0-alpha7/clojure-1.10.0-alpha7.pom from
Downloading: org/clojure/clojure/1.10.0-alpha7/clojure-1.10.0-alpha7.jar from
Clojure 1.10.0-alpha7
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'org.clojure/core.memoize {:mvn/version "0.7.1"})
true
user=> (require 'clojure.core.memoize)
nil#2018-09-2214:50raymcdermottmaybe false is OK if the lib is already available#2018-09-2214:50Alex Miller (Clojure team)yeah, you’ll get true only if added a new lib(s)#2018-09-2214:50Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/blob/add-lib/src/main/clojure/clojure/tools/deps/alpha/repl.clj#L60#2018-09-2214:52raymcdermottok, great - thanks for checking#2018-09-2214:55Alex Miller (Clojure team)it can’t be deprecated - I haven’t even released it yet :)#2018-09-2214:58raymcdermottLOL fair point#2018-09-2214:58raymcdermottare you close to release or is the debate still ‘raging’?#2018-09-2215:00Alex Miller (Clojure team)been focused on other things :)#2018-09-2215:02raymcdermottwhat could possibly be more important#2018-09-2215:58raymcdermottthe problem I’m having might be related to Cursive as the tests all pass via the CLI#2018-09-2215:58raymcdermottand fail via the deps.edn default REPL there, so I’ll mention it on that channel later#2018-09-2408:35martinklepschAre there any known issues around depending on an artifact like this: https://repo.maven.apache.org/maven2/ml/dmlc/xgboost-jvm/0.72/#2018-09-2408:35martinklepschi.e. the only jar for the version has a -javadoc classifier #2018-09-2412:06Alex Miller (Clojure team)What are you expecting to happen?#2018-09-2412:06Alex Miller (Clojure team)What is your deps.edn?#2018-09-3008:07alanHi, sorry for the late response, I raised the issue because while I can build without any issue with leiningen I couldn't make the cljdoc build (which uses tools.deps) work#2018-09-3008:11alanThis is the tools.deps {:deps {org.clojure/clojure #:mvn{:version "1.9.0"}, ml.dmlc/xgboost4j-example #:mvn{:version "0.72"}, ml.dmlc/xgboost-jvm #:mvn{:version "0.72"}, org.clojure/clojurescript #:mvn{:version "1.10.339"}, org.clojure/tools.namespace #:mvn{:version "0.2.11"}, clj-boost/clj-boost #:mvn{:version "0.0.1"}, org.clojure/java.classpath #:mvn{:version "0.2.2"}, org.clojure/data.csv #:mvn{:version "0.1.4"}, ml.dmlc/xgboost4j #:mvn{:version "0.72"}, codox/codox {:exclusions [enlive hiccup org.pegdown/pegdown], :git/url "", :sha "4b0720941083fda9643d905f0854fabea55b175f", :deps/root "codox/"}}}#2018-09-3008:13alanIt seems impossible to get the XGBoost stuff in this way. While in leiningen I have [ml.dmlc/xgboost4j "0.72" :extension "pom"] and everything's fine#2018-09-3012:51Alex Miller (Clojure team)pom extension is not supported - what are you expecting that to do?#2018-09-3013:18alanI'm not using it, is in cljdoc to build the package, but now I guess there's not much it can be done about it#2018-09-3014:03Alex Miller (Clojure team)Sorry, I’m not understanding the problem. It would be very helpful to have a description of the problem, with the things needed to reproduce it, what you expect to happen, and what is actually happening. I feel like the above has half of that but it’s very confusing to me what you’re doing.#2018-09-3014:07alanI have a dependency, the only way to get it is to use the :extension "pom" flag, I develop, build and release my library with lein, no issues. I wanted to build docs with cljdoc which uses tools.deps and the build fails because of that dependency#2018-09-3014:21Alex Miller (Clojure team)“the only way to get it is to use the :extension "pom" flag” - why?#2018-09-3014:23alanhttps://mvnrepository.com/artifact/ml.dmlc/xgboost-jvm/0.80#2018-09-3014:25martinklepsch@alexmiller because the dependency doesn’t come with a regular .jar file — I don’t know too much about why you package libraries like this but it seems to be a thing in this case
And as far as I understand nobody (using tools.deps) will be able to use @U7P0DPQ1H s library which depends on this artifact #2018-09-3014:37Alex Miller (Clojure team)If it doesn’t have a jar, then i don’t get how it affects the classpath. #2018-09-3014:39Alex Miller (Clojure team)I guess maybe by having deps#2018-09-3014:40Alex Miller (Clojure team)Which I would actually expect to work#2018-09-3014:41Alex Miller (Clojure team)If someone could file a jira in TDEPS about this, I will look at it more closely later this week#2018-09-3014:56martinklepschYeah would assume that the point of this kind of artifact is usually its deps, I.e. it’s a kind of Uber-module #2018-09-3014:57martinklepschwont have time to file a ticket in the next few days...#2018-10-0415:39alan@alexmiller I just opened an issue https://dev.clojure.org/jira/browse/TDEPS-98 hope it's ok{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-09-2410:28mkvlrMy issue https://dev.clojure.org/jira/browse/TDEPS-50 still reproduces after it has been marked as a duplicate of https://dev.clojure.org/jira/browse/TDEPS-12 which has been marked as fixed. Can someone with access please reopen this for me?#2018-09-2412:04Alex Miller (Clojure team)Are you using the latest release?#2018-09-2412:05Alex Miller (Clojure team)And what is your deps.edn? There is some new syntax from TDEPS-12 with an example in the comments there.#2018-09-2412:09Alex Miller (Clojure team)Specifically you’ll want the lib just once with a coordinate that has :classifier [“” “native”]#2018-09-2412:13Alex Miller (Clojure team){:deps {com.github.jnr/ffi {:mvn/version “1.2.16” :classifier [“” “native”]}}}#2018-09-2412:27bherrmannAm I nutty or does the example "hello.clj" example from https://clojure.org/guides/deps_and_cli not work until you add :paths ["."] to deps.edn ??#2018-09-2412:28dominicm@bherrmann that surprises me for two reasons, one is that it should be "src", the other is that it's a default. #2018-09-2412:29dominicmWhere did you put hello.clj?#2018-09-2412:35bherrmannok, yea. hello.clj just needs to be under "src"#2018-09-2412:35bherrmannmy bad... I'm automating a manual process (converting and excel file) and was thinking more script minded, less software development minded (where a src sub directory is more assumed)#2018-09-2414:06Alex Miller (Clojure team)@mkvlr I looked at this and have reopened - you’re right that this is related to TDEPS-12 but needs a bit more work#2018-09-2414:06Alex Miller (Clojure team)but I’m not going to get to that this week#2018-09-2414:09mkvlr@alexmiller thanks a lot! Yes, we can work around by manually specifying it for now so no rush from us. It’s just missing to pick up the classifier when reading the pom if I understand it correctly.#2018-09-2414:13bhaumanI’m writing some docs on resolving dependency conflicts, is there a tools deps equivalent to lein pedantic?#2018-09-2414:15Alex Miller (Clojure team)yes#2018-09-2414:15Alex Miller (Clojure team)https://github.com/Olical/depot#2018-09-2414:33bhaumanthanks @alexmiller, while that works and is good to know about it implements lein-ancient it doesn’t find dependency conflicts like pedantic 🙂#2018-09-2414:33Alex Miller (Clojure team)oh, sorry. then no :)#2018-09-2414:34bhauman-Stree helps of course#2018-09-2414:34Alex Miller (Clojure team)there is a totally unsupported subject to removal and alteration verbose mode for tools.deps that is primarily designed to be helpful to me#2018-09-2414:35bhaumanoh I’ll put that in the docs then 😉#2018-09-2414:35Alex Miller (Clojure team)definitely an area where more could be done#2018-09-2414:36bhaumanI moved figwheel to Jetty thinking it was a smart move to get https support and to play nicely with larger environment#2018-09-2414:36bhaumanbut dang its prone to tricky conflicts#2018-09-2414:37bhaumanthe way the package is divided up#2018-09-2414:37bhaumanwhen I say tricky I mean tricky for newcomers#2018-09-2414:38bhaumanthe conflict between ring and datomic cloud is the main problem{:tag :div, :attrs {:class "message-reaction", :title "point_up"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☝️")} " 4")}
#2018-09-2416:22lilactownyeah I spent about 3 hrs early on trying to get Jetty (for local dev) and datomic cloud working. I switched to http-kit and it is working well so far#2018-09-2521:55bhauman@U4YGF4NGM I’ve got docs on this now{:tag :div, :attrs {:class "message-reaction", :title "man-bowing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-09-2521:56bhaumanhttps://figwheel.org/docs/jetty_conflicts.html#2018-09-2521:56bhaumanpublished it this weekend#2018-09-2521:56bhaumansorry about that#2018-09-2521:57lilactownawesome! ty!#2018-09-2421:05seancorfieldIf I have some.group/artifact {:mvn/version nil} and an :override-deps entry of some.group/artifact {:mvn/version "1.2.3" :exclusions [foo/bar com.quux/blah]} should I expect the :exclusions from the override to be applied to the main dep?#2018-09-2421:05seancorfield(and what happens if there are :exclusions on both the main dep and the override?)#2018-09-2421:05Alex Miller (Clojure team)the override overrides#2018-09-2421:06Alex Miller (Clojure team)as in replace the original (including its exclusions)#2018-09-2421:06seancorfieldCool. So I can rely on specifying :exclusions just once in the overrides -- that's excellent!#2018-09-2707:07tianshuwhen try to uberjar, meet this error
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.quartz-scheduler:quartz-parent:pom:2.3.0 in terracotta-snapshots ()
This dependency, will be handled correct when using leiningen. I don't know how to investigate the issue, where should I start?#2018-09-2712:21dominicmThis looks like TDEPS-50 also?#2018-09-2713:44orestisIs there a tools.deps friendly way to run Eastwood from the command line?#2018-09-2714:35orestisAlso, what is a no-fuss way to build an uberjar for a deps.edn project? I tried pack.alpha, depstar so far and I get weird errors.#2018-09-2714:36orestisdepstar: I get this issue: https://github.com/healthfinch/depstar/issues/5, too much fuss to do manifests etc.#2018-09-2714:37orestispack.alpha: NOTE: /alpha/bootstrap/onejar/src/com/simontuffs/onejar/JarClassLoader.java, line -1: Some input files use unchecked or unsafe operations.
NOTE: /alpha/bootstrap/onejar/src/com/simontuffs/onejar/JarClassLoader.java, line -1: Recompile with -Xlint:unchecked for details.
~/d/n/c/october (master|✚1…) $ java -jar output.jar
Exception in thread "main" java.lang.ClassNotFoundException: clojure.main
at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:713)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at com.simontuffs.onejar.JarClassLoader.loadClass(JarClassLoader.java:630)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.simontuffs.onejar.Boot.run(Boot.java:336)
at com.simontuffs.onejar.Boot.main(Boot.java:168)
#2018-09-2716:37seancorfield@orestis For pack.alpha, that seems to indicate it didn't include the Clojure runtime -- no idea about that since I haven't tried it but @U09LZR36F (the author) is very helpful. For depstar, adding the manifest manually is an easy step and I like the raw simplicity of it. Haven't tried cambada so thanks for finding that and posting the link!#2018-09-2716:38orestisIt’s up on the tools.deps wiki and has a very good README :)#2018-09-2716:39seancorfieldYeah, just looking it over. Very nice!#2018-09-2716:59dominicmI think the pack bug would be fixed by using the latest version. #2018-09-2716:59dominicmI haven't updated the installer, and I should! #2018-09-2714:40orestisHooray - https://github.com/luchiniatwork/cambada seems to work out of the box, with the only addition being an extra command-line parameter to specify the main namespace#2018-09-2716:32andy.fingerhutI do not have a tools.deps friendly way handy and worked out for running Eastwood from the command line, but if you look at this section of the Eastwood docs on how you can do it from the REPL, without Leiningen involved, you or someone better at tools.deps knowledge than I might be able to figure it out. I'd be happy to add a note like that to the Eastwood README if it isn't too difficult to explain: https://github.com/jonase/eastwood#running-eastwood-in-a-repl#2018-09-2716:33andy.fingerhutWarning: The latest Eastwood 0.2.9 release has a bug in which when you try to use it, it will download (if you don't already have it) and use Eastwood 0.2.8 instead. You may want to simply specify Eastwood 0.2.8 for now, since that is what you will end up using anyway.#2018-09-2716:47seancorfield@orestis clj -Sdeps '{:deps {jonase/eastwood {:mvn/version "0.2.8"}}}' -e "(require '[eastwood.lint :as e])" -e '(e/eastwood {:source-paths ["src"] :test-paths ["test"]})'#2018-09-2716:48seancorfieldYou could squirrel most of that away in your ~/.clojure/deps.edn file as an alias...#2018-09-2716:51seancorfield:eastwood {:extra-deps {jonase/eastwood {:mvn/version "0.2.8"}}
:main-opts ["-e" "(require,'[eastwood.lint,:as,e])"
"-e" "(e/eastwood,{:source-paths,[\"src\"],:test-paths,[\"test\"]})"]}
(note that you need , instead of space!)#2018-09-2718:09orestisI ended up doing this: :eastwood {:extra-deps {jonase/eastwood {:mvn/version "0.2.8"}}
:main-opts ["-m" "eastwood.runner"]
:extra-paths ["test" "tools"]}
And I made a tools directory to contain simple namespaces like that. In this case: (ns eastwood.runner
(:require [eastwood.lint :as e]))
(defn -main [& args]
(e/eastwood {:source-paths ["src"] :test-paths ["test"]}))#2018-09-2718:55seancorfieldOh, that's nice! :thumbsup::skin-tone-2:#2018-09-2717:01orestisYeah, I saw that trick on your dot-Clojure and was about to ask! Is that a known bug?#2018-09-2717:02orestisI guess though I can just as easily make a tiny namespace to do this. #2018-09-2717:07seancorfield@orestis It's to do with how the strings are run through the clojure shell script and into the main opts cache file and then read back. So, yeah, it's a "known feature" 🙂#2018-09-2717:07orestisRight, so it’s not going away any time soon I guess?#2018-09-2717:08orestisI remember some mentions of caching compiled dependencies to speed up startup time - any ideas about that?#2018-09-2717:10orestisMore questions - in deps.edn, can you actually override the entire top-level deps entry? Or can you only append?#2018-09-2717:12orestisRationale: most of my aliases so far have to do with running lint tools, packaging, test runners etc. These namespaces have no need to access my actual app’s classpath. I guess there has bound to be some overhead when the class path is unnecessarily large?#2018-09-2717:13orestisI know that clj will cache the classpath and it’s actually very noticeable. #2018-09-2904:07miridiusHi all, what's the best practice for running a deps.edn based project in production? Is it ok to run it "dynamically" with clojure -Aprod -m my.app.main or is it better to AOT compile the app -- and if so, how?#2018-09-2906:06seancorfieldAOT really only saves you startup time.#2018-09-2906:06dominicmWe do that sometimes. No complaints. #2018-09-2906:06seancorfieldWe still use JAR files for production deployment but we never AOT anything.#2018-09-2906:07seancorfieldWe use uberjars for the convenience of knowing our production system will never be dependent on reaching out to Maven or Clojars at application startup. But we use clojure.main as our Main-Class.#2018-09-2906:10miridiuswhat do you use for creating the uberjar? juxt/pack.alpha -> Capsule?#2018-09-2906:10seancorfieldWe're still using Boot but we're moving toward clj/`tools.deps`.#2018-09-2906:11seancorfieldI'm leaning toward depstar for the simplicity of the tooling, but cambada seems to be very slick.#2018-09-2906:11miridiusah ok 🙂 thanks. I'm currently using lein for the uberjar task and literally nothing else#2018-09-2906:13seancorfieldWe outgrew Leiningen years ago and switched to Boot -- we've built almost our entire build/test/deploy pipeline on top of that.#2018-09-2906:13seancorfieldBut we're in the process of retooling to something simpler, based on deps.edn etc.#2018-09-2906:13miridiusdepstar and cambada both look really cool! thanks for those#2018-09-2906:14miridiusYeah I've tried both leiningen and boot and found them both to increase the amount of complexity I face rather than decrease it. Troubleshooting them when they don't behave is especially painful! So I've happily moved everything to tools.deps, and the only piece missing now is production deployment#2018-09-2906:15seancorfieldOur build.boot file is close to 2,000 lines. We have a ways to go 🙂#2018-09-2906:15miridiuswow...#2018-09-2906:17miridiusI use docker so theoretically don't have any need to uberjar anything, as long as I can make sure all the deps are there. I was thinking of adding a :quit command-line arg to my main class that quits immediately and just running that at build time so that it downloads all the deps. Kind of hacky though...#2018-09-2906:18seancorfieldCan't you do clj -Stree to force the downloads?#2018-09-2906:20miridiusoh! I didn't know it would download the deps, I thought it just prints them#2018-09-2906:20miridiushey btw, I use your dot-clojure repo, it's really helpful 🙂#2018-09-2906:21miridius-Stree works... nice#2018-09-2908:15dominicmWe (JUXT) use pack in many projects now, it's gone very well overall. #2018-09-2906:07seancorfield^ @miridius#2018-09-2906:09miridiusthanks for the tips 🙂#2018-09-2921:15bbrinckHi, I just made a small utility (using clj-new- thx @seancorfield) to validate deps EDN from stdin - https://github.com/bhb/check-deps. Feedback/questions welcome 🙂#2018-09-3002:12richiardiandreaNice!#2018-09-2921:17bbrinck(and suggestions)#2018-09-3012:47krukowRunning into an issue migrating from lein to tools deps. REPL works, but when I try to uberjar the project I get the following error:
Exception in thread "main" org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.httpcomponents:httpcore:jar:4.4.9
https://gist.github.com/krukow/b2b6ef6536cb8cf07cc8f613afaa8108
Would someone here be kind enough to help be figure out what's going on?#2018-09-3012:51krukowHmmm - resolved by cleaning .m2 and adding
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
#2018-09-3012:51krukowdid this mean I had a corrupt .m2#2018-09-3012:55Alex Miller (Clojure team)Those repos are already included by default so shouldn’t need those#2018-09-3012:55Alex Miller (Clojure team)But maybe a bad m2#2018-09-3012:58krukowthanks Alex - I'll remove and simplify, I think I'll clean the entire .m2 and re-build#2018-09-3012:58krukowmigration from lein to deps was very easy 🙂 thanks!#2018-09-3013:00krukowwait - this is weird ... I'll double check but what I'm seeing now is#2018-09-3013:02krukowremoving :mvn/repos from deps.edn => results in the above ArtifactDescriptorException: exception#2018-09-3013:03krukowadding
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
=> uberjar builds#2018-09-3013:07krukownot happy about using http though#2018-09-3013:09krukowException in thread "main" org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.httpcomponents:httpcore:jar:4.4.9
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:323)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:192)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:253)
at clojure.tools.deps.alpha.extensions.maven$eval2646$fn__2648.invoke(maven.clj:79)
at clojure.lang.MultiFn.invoke(MultiFn.java:243)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:170)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:152)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:215)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:197)
at cambada.uberjar$get_dep_jars.invokeStatic(uberjar.clj:109)
#2018-09-3013:11krukowthis is independent of the state of the .m2 dir#2018-09-3013:12krukowseems that this call cambada.uberjar$get_dep_jars.invokeStatic(uberjar.clj:109) => clojure.tools.deps.alpha$resolve_deps behaves differently depending on whether I have
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
or not in deps.edn#2018-09-3013:12krukowstrange#2018-09-3020:25seancorfield@krukow I just tried with your deps.edn and that build command and it worked just fine. I got this warning tho' Warning: The Main-Class specified does not exist within the jar. It may not be executable as expected. A gen-class directive may be missing in the namespace which contains the main method, or the namespace has not been AOT-compiled.
#2018-10-0114:14krukow@seancorfield thanks for helping out - I ended up with this uberjar command:
:uberjar
{:extra-deps
{luchiniatwork/cambada {:mvn/version "1.0.0"}}
:main-opts ["-m" "cambada.uberjar"
"-m" "vsts-flow-metrics.cli"]}}
which should help for the main warning.
Then without specifying a :mvn/repos in deps.edn I get:
Exception in thread "main" org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.httpcomponents:httpcore:jar:4.4.9
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:323)
...
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:197)
at cambada.uberjar$get_dep_jars.invokeStatic(uberjar.clj:109)
But If deps.edn contain
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
the command completes#2018-10-0114:14krukowI'll see if I can create a minimal repro#2018-10-0114:14krukowand I use clj -A:uberjar -Srepro -Sforce to create the uberjar#2018-10-0114:46timgilbertSay, using an s3p private repo I get a lot of messages like these when starting clojure:
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from
Apparently this file is just missing, but I definitely don't have it in my private repo. Is there any way to tell clojure and/or maven to only use the private repo for artifacts from a specific namespace?#2018-10-0116:05timgilbertAlternately, is there a way to suppress that message from appearing? It's not so good for shell-script type tasks#2018-10-0116:08timgilbertThe full panoply of messages is more like this:
Downloading: com/google/code/findbugs/jsr305/maven-metadata.xml from
Downloading: com/google/guava/guava/maven-metadata.xml from
Downloading: commons-codec/commons-codec/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from
Downloading: com/amazonaws/amazon-kinesis-client/maven-metadata.xml from
#2018-10-0116:20timgilbertI guess what I'd like is the ability to set the <updatePolicy> value on my S3 repository to never, which I think would make this behavior go away. https://maven.apache.org/pom.html#Repositories#2018-10-0119:08zane:jvm-opts isn't a valid top-level key, right? Only inside an alias in :aliases?#2018-10-0119:43Alex Miller (Clojure team)correct#2018-10-0119:43Alex Miller (Clojure team)there is a ticket wrt default jvm-opts#2018-10-0120:31zaneOkay. Thanks, Alex!#2018-10-0205:45kommenis there a way to figure out in a clojure process started with clj which aliases were used to start it?#2018-10-0206:10seancorfield@kommen No.#2018-10-0206:11seancorfieldThe aliases are only used to build the classpath (and main opts etc). By the time your program is running, none of that is available.#2018-10-0206:11seancorfieldWhat is the problem you're trying to solve here?#2018-10-0206:11kommenI’m looking into a solution for this cider issue: https://github.com/clojure-emacs/cider/issues/2308#issuecomment-426029865#2018-10-0206:13kommencider checks if dependencies for certain functionality are on the classpath. until now, simply by splitting the classpath entries and interpreting that. this falls apart when using tools.deps git and local coordinates#2018-10-0206:15seancorfieldYup. Well, I think CIDER is trying to be far too clever here and making all sorts of unwarranted assumptions -- but it sounds like Bozihdar is aware of the problem and will figure it out -- he's been doing amazing stuff with CIDER...#2018-10-0206:16seancorfieldclj and tools.deps are making a lot of people reconsider their assumptions 🙂#2018-10-0206:23kommenok, thanks#2018-10-0206:49dominicmThe safe way to do it is to check for the existence of namespaces/classes, not to search the classpath for them.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-0220:13krukow@seancorfield @alexmiller I have a simple repro (on my machine) - from a clean install. Unzip the .zip file into an empty directory and run clj -A:uberjar -Srepro -Sforce - would any of you be so kind as to confirm / invalidate the repro so I can make a formal issue?#2018-10-0220:15krukowFYI the repro sequence was: with an empty /.m2, /.clojure on OS X high sierra with brew install clojure, `clj -Sdeps '{:deps
{seancorfield/clj-new
{:git/url "https://github.com/seancorfield/clj-new"
:sha "21ca1b27f46dc324be084ba839beca555aeda387"}}}' \
-m clj-new.create \
app \
krukow/artifact-exception` and then clj -A:uberjar -Srepro -Sforce in the folder where simple-repro.zip is unzipped.#2018-10-0220:18krukowthen add
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
#2018-10-0220:18krukowto deps.edn and things work#2018-10-0220:20krukowFYI: posting this to https://github.com/luchiniatwork/cambada/issues#2018-10-0220:34dominicm@krukow this makes sense to me. Cambada doesn't consider the system deps.edn file. #2018-10-0220:36seancorfieldOuch! That's quite the gotcha!#2018-10-0220:43krukow@dominicm I think that's correct! I'll add to: https://github.com/luchiniatwork/cambada/issues/15#2018-10-0220:43krukowthank you!#2018-10-0220:44krukowbut the workaround is easy then - I should have thought of this 😮#2018-10-0220:44krukowOne more general question: where can I find the source code for the clojure & clj CLI tools#2018-10-0220:46seancorfieldThe brew-install and tools.deps.alpha repos under the clojure org.#2018-10-0220:46seancorfield(the former is the shell scripts etc, the latter is the library they use for all the dependency stuff)#2018-10-0220:47krukow:thumbsup: thank you Sean#2018-10-0222:41amar@krukow this PR should fix the issue you were having https://github.com/luchiniatwork/cambada/pull/14#2018-10-0305:09dominicmAlso means it doesn't need to merge in paths#2018-10-0305:29zaneIs it just me or is the -v / --var option for com.cognitect/test-runner broken? Setting it appears to trigger a crash because set is being called on a single-argument call to map, which returns a transducer: https://github.com/cognitect-labs/test-runner/blob/master/src/cognitect/test_runner.clj#L20...L22#2018-10-0305:29zaneHoping to get one more set of eyes on this before I file an issue.#2018-10-0305:46seancorfieldIt's missing var as the second argument to the map call. Def. a bug.#2018-10-0313:51zaneThat was my guess too. Thanks!#2018-10-0307:06krukow:thumbsup: amar!#2018-10-0314:49hlolliI added this into my deps.edn :jvm-opts ["-Xms2G" "-Xmx5G"] started repl with cider, but I can't in any way see if these options are present. This is not present in the command line args if I look at the java process in my system monitor.#2018-10-0315:06ghadiI don't think these take effect except within an alias{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-0315:13hlollioh ok, good to know 🙂 @ghadi#2018-10-0315:26orestis@hlolli see in #cider for a way to start cider with a specific alias. #2018-10-0315:26orestishttps://clojurians.slack.com/archives/C0617A8PQ/p1538572945000100{:tag :div, :attrs {:class "message-reaction", :title "point_up"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☝️")} " 4")}
#2018-10-0316:03hlollibingo works, I've been bit turned off by dir-locals due to "unsafe" popup. But the popup didn't appear this time.#2018-10-0406:36dominicmI think I understand why, but should it be the case that, if an alias specifies main-opts (mine has -m in), then -e and -i are ignored? The output of clj --help doesn't exactly suggest that -e and -m are mutually exclusive, especially as they are "different".
Currently cider tries to use -e in order to start nrepl without interrupting the users choice of main-opts.#2018-10-0406:39dominicmOne solution might be to add a java class to cider releases which triggers in response to a jvm option, but that's not going to be my first approach if I can help it. -e is just lovely as a general tool.#2018-10-0412:36genekimNot sure where I should ask this, but wanted to put this out there —
After listening to a @apropos_cast, I took the plunge and tried clj and tools.deps.
I was blown away by how easy it was! I used lein-tools-deps by @rickmoynihan — it took only 10m to transition my first project over, and was able to quickly do three other lein projects. It was fantastic to be able to share library code w/o publishing to local maven repo. It made using the monorepo pattern fantastically simple.
Marvelous work, Rick — and tools.deps is fantastic!
However, on my laptop, I found that “clj -m foo.main” takes 15s to startup, 2x longer than “lein run”. (Which is not what I expected. I expected startup times to be the same, or lein to be worse, given my understanding that there’s one more JVM being started?)
It’s not in a public repo, so it’d take some time to come up with something that you could reproduce — but if anyone is interested, I can capture some traces, or with some help, I can try to come up with a reproducible test case?#2018-10-0413:06rickmoynihanAlex Miller deserves almost all of the credit. ltd does almost nothing. But thanks for the thanks.#2018-10-0413:52Alex Miller (Clojure team)On the timing, was that first time or second time (after classpath is cached)?#2018-10-0414:39genekim@U064X3EF3 brilliant work! It’s a marvel! Thank you!
On timing: I ran all the timings 5-10 times, so I’m presuming this is with classpath caching.#2018-10-0412:42mpenetI think lein has default jvm flags optimised for startup time, clj I don't think does anything like that#2018-10-0412:42mpenetI could be wrong tho#2018-10-0413:52Alex Miller (Clojure team)I doubt you’ll see that much of a difference from those flags#2018-10-0413:53Alex Miller (Clojure team)If you’re using :aot with lein then that could be a big difference#2018-10-0413:53mpenetYeah in general when people complain about startup time I reply that their workflow is prolly not really optimum#2018-10-0413:53Alex Miller (Clojure team)Lein would use the compiled classes #2018-10-0413:54mpenetdoing cli tools with clojure is another matter, solved with other tools probably#2018-10-0415:07bhaumananyone agree? https://github.com/clojure-emacs/cider/issues/2473{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 8")}
#2018-10-0416:22cflemingI’m having a problem using tools.deps as a library. If I don’t set the CWD of the process using it to the correct directory, then the manifest type detection is broken.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-0416:25cflemingIs that expected, a known bug, or something I should file an issue for? I can’t see anything immediately obvious in the tracker.#2018-10-0416:36Alex Miller (Clojure team)If you are using absolute paths, then I presume things are fine. If you are using relative paths, then it seems reasonable to expect that they are relative to the current working directory#2018-10-0416:37Alex Miller (Clojure team)Transitive local deps with relative paths are a problem and that’s a known issue#2018-10-0416:37cflemingI would have assumed that they would be relative to the file in which the ref was found.#2018-10-0416:39cflemingIn this case, I have a root project which has a :local/root of “modules/something”. That deps.edn has another local ref with a relative path, which sounds like it’s a known issue?#2018-10-0416:39Alex Miller (Clojure team)That’s the known problem - TDEPS-74 #2018-10-0416:40cflemingI don’t think that’s the problem that I’m seeing, I’m seeing issues with manifest detection there too.#2018-10-0416:41cflemingBut since the manifest detection is relative to the CWD, that will never work.#2018-10-0416:43cflemingI assume it’s all related to the same problem.#2018-10-0416:48cflemingI can’t see a good fix for this. detect-manifest would require the full path to the original file, and the only way I can see to pass that would be to add a new key to local coords when calling manifest-type.#2018-10-0416:49cflemingEven expand-deps doesn’t have that info right now, although I’m assuming that could be passed as a param.#2018-10-0416:50cflemingUgh, in fact, not even resolve-deps has that info, since it’s passed a deps map, not the original files.#2018-10-0416:53Alex Miller (Clojure team)Yes, it’s difficult to fix, which is why it’s still open#2018-10-0416:53Alex Miller (Clojure team)I have a couple ideas on it to look at though#2018-10-0416:53cflemingIt seems like the best place to fix it would be to absolutise those paths in read-deps.#2018-10-0416:57cflemingIn fact, if that happened as part of canonicalisation in slurp-deps, then that should also work for transitive deps.#2018-10-0416:59Alex Miller (Clojure team)The related problem is that for local deps, it can’t build the proper set of deps.edn to merge - they’ve already been merged upstream#2018-10-0417:00Alex Miller (Clojure team)This relates to the notion of naming and separating the deps files#2018-10-0417:00cflemingThe idea would be to merge local deps with the system and user deps files too?#2018-10-0417:00Alex Miller (Clojure team)Well the key is to NOT get the original project deps.edn#2018-10-0417:00cflemingRight.#2018-10-0417:01Alex Miller (Clojure team)And all of this stuff also is related to -Sdescribe and how things are installed on the various platforms#2018-10-0417:02cflemingThe only way to reliably not get that is to pass either the paths or the read maps around, I guess, since merging is destructive.#2018-10-0417:02Alex Miller (Clojure team)I need some focus time to sit down and wrk through all these things.#2018-10-0417:02cflemingI guess you could merge system and user, and then pass that around and merge it with the other deps files, including the main project one, when you read them.#2018-10-0417:03cflemingThat changes the whole API though, unless you stored that in the map you pass around I guess.#2018-10-0417:03cflemingUnder a ::base-config key or something.#2018-10-0510:00mpenetis there a workaround to https://dev.clojure.org/jira/browse/TDEPS-51 ?#2018-10-0510:01mpenetI guess classpath-overrides is a possibility but that's a bit gross#2018-10-0515:22genekimGot it — thanks for the insights on lein compiles vs. clj, @alexmiller @mpenet !#2018-10-0517:38adamfreyIf I'm making a library with tools.deps and it includes a Java file in it, how can I have that library be consumable in a way that allows people to import the java class? I know it the .java file needs to get compiled into a .class file at some point, which makes sense in an uberjar, but what about pulling in my library via git or :local/root? Would it work / is it advisiable to put the .class file into version control?#2018-10-0518:15Alex Miller (Clojure team)You can’t - you should use a build tool to create and deploy an artifact #2018-10-0520:29nwjsmithI’ve been playing around with tools.deps using Maven as a build tool, and it’s been a good experience. clj -Spom -Srepro does mess with my pom.xml in undesirable ways, but testing is straightforward with the clojure-maven-plugin, and building uberjars with maven-shade-plugin works great. The most pleasant thing for me has been the speed relative to Leiningen, it’s much faster.#2018-10-0909:30martinklepschreminds me of this: https://github.com/danielsz/meyvn#2018-10-0520:30nwjsmithI’m not sure why I haven’t seen this set up more often in the wild#2018-10-0520:30nwjsmithFear of XML maybe?{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 4")}
#2018-10-0810:34isaacclojure cli don’t care -Dmaven.repo.local?#2018-10-0811:57eval2020@isaac you can add :mvn/local-repo "some/path" to deps.edn.#2018-10-0812:15isaacIs the :mvn/local-repo will be instead of “~/.m2”?#2018-10-0812:25eval2020the provided path will be used instead of ~/.m2/repository#2018-10-0905:00Michael FianoHello all. I just noticed that -A must be supplied before the -r argument for them to take effect. Is that intentional or a bug?#2018-10-0905:01Michael FianoCancel that. I see in the usage notes that is intended#2018-10-0923:35kennyGetting UnknownHostKey: . RSA key fingerprint is when running clj -Spath. Any ideas?#2018-10-0923:37kennyAh needed to add github to known_hosts. That should produce a better error message.#2018-10-1005:43dominicm@kenny the issue is because of ecdsa I believe. #2018-10-1008:39henryw374It's a bit frustrating to have to wait so long for Windows support. https://dev.clojure.org/jira/browse/TDEPS-67 has been around for a while... Please add your vote! (not sure if it'll make a difference but it can't hurt). Thanks#2018-10-1015:39dominicmWe're developing a situation where a few aliases have been split up in the name of DRY. This means that to start the project in "dev mode" requires a difficult number of aliases. Has anyone tried tackling this yet?
I'm currently thinking of writing something that merges together aliases so you can create a "super-alias" that's easier to maintain.#2018-10-1109:11Andreas LiljeqvistWe use makefiles to handle starting tricky combos#2018-10-1109:13dominicmThat doesn't go very well with cider-jack-in though, or any other editor integration?#2018-10-1109:17Andreas LiljeqvistYou are right, we use it more for building releases and such#2018-10-1109:17Andreas LiljeqvistDon't have a good solution for launching dev#2018-10-1015:40dominicmThis is really just a workaround for the lack of ability for aliases to depend on one another.#2018-10-1015:41seancorfield@dominicm Yeah, we're running into that a bit as we convert our Boot-powered monorepo to clj/`deps.edn`...#2018-10-1016:38zaneI'd love to hear more about this if you'd like to share! Why you're doing it, etc. #2018-10-1017:35seancorfield@U050CT4HR We've had our dependencies in external EDN files for ages (years, maybe) and if fact called them deps.edn until clj/`tools.deps` appeared 🙂 I'm running into problems with Boot due to the fileset and pod stuff (I think because our project is large -- 83K lines across 400+ files -- but we haven't managed to tie down a repro case yet). Several of our Boot tasks take a long time to fire up and run, whereas running similar tasks via clj is much faster (since there's a lot less machinery involved).#2018-10-1017:36seancorfieldI'm also a big fan of keeping close to the bleeding edge with Clojure releases and "core" tooling so having fewer dependencies in our system would make me happy.#2018-10-1020:00zaneThat's really helpful Sean. Thanks so much!#2018-10-1015:41dominicm@seancorfield have you had any particular ideas on how to tackle it? I'm quite happy to open source my solution. It will likely be a key piece in edge.#2018-10-1015:42Alex Miller (Clojure team)have you considered just repeating yourself? :)#2018-10-1015:45dominicm@alexmiller In some cases, yep! For a single dependency it's usually not so bad. When it's 2 or 3 aliases with 2 or 3 changes, the maintenance burden of bumping in multiple places is failing quite often. I'll admit that I'm trying to add in more "easy" onto this design than usual, as I can't impose pull request standards onto all projects.#2018-10-1015:48seancorfieldWe've had to repeat a chunk of our testing context in two separate aliases, which is a bit annoying. The alternative is a longer string of aliases. Right now we have a dummy project containing a deps.edn with mostly aliases. That has a :default aliases which has all our :override-deps where we "pin" versions of libraries. Then we have a deps.edn in each project that specifies the deps for that project (mostly as some/lib {} since the actual version comes from the :default alias) and references to other subprojects as worldsingles/something {:local/root "../something"}. This works pretty well for working within any given subproject and then we'll likely have a couple of shell scripts for running tests across the whole suite of projects.#2018-10-1015:49seancorfieldDoing this has actually highlighted some problematic cross-project dependencies that Boot sort of hid for us, so that's been nice. And of course being able to use a much more lightweight process is always nice 🙂#2018-10-1015:49dominicmheh, I actually just caught myself this second failing to update one of two places. I only caught it because I realized that a :main-opts looked out of place in the alias I had updated.#2018-10-1015:53seancorfieldOne thing I do find is that I'd desperately love to be able to parameterize deps.edn sometimes. For example, we can have an alias that starts an nREPL server to work with a project in an editor, but we can't affect which port it starts on -- that has to be hardcoded in the alias. So I'm thinking we may end up with one or more aliases that bring in (and run) certain small Clojure scripts that exist purely to support dev/test work. That way we could do clj -A:some:stuff:here <port-number> and run a -main in a dev namespace that starts nREPL on that port.#2018-10-1015:53seancorfieldI'd be interested to hear more about how Cognitect folks deal with this in large projects. hint, hint 🙂#2018-10-1015:53Alex Miller (Clojure team)don’t make large projects?#2018-10-1015:54seancorfieldscowl#2018-10-1015:54seancorfieldIs 63K lines of production Clojure and 20K lines of tests "large" tho'?#2018-10-1015:56Alex Miller (Clojure team)deps.edn being “just data” (and not having parameters, references, etc) has vast benefits for “simple” and is not something I will give up easily#2018-10-1015:56Alex Miller (Clojure team)at some point of complexity, you are really doing programming here#2018-10-1015:56seancorfieldOh, true. I totally get the benefits. I just haven't (yet) found the ideal balance yet, with the new tooling.#2018-10-1015:56Alex Miller (Clojure team)you can use clj to run programs - so pushing it into a script (which can take args) is one path#2018-10-1015:56dominicm@seancorfield I'm in favor of a main for the nREPL situation.#2018-10-1015:57Alex Miller (Clojure team)you can also wrap clj in another program (probably a shell script)#2018-10-1015:57Alex Miller (Clojure team)but could also be, say, make#2018-10-1015:58Alex Miller (Clojure team)make has the nice capability to define dependencies on smaller pieces which I think often is useful in build situations where you are making dependent subtasks#2018-10-1015:58Alex Miller (Clojure team)ditto ant#2018-10-1015:58seancorfieldBeing able to do something like this to run all subprojects tests is convenient enough: for d in * ; do (cd $d; CLJ_CONFIG=../versions clj -A:defaults:test:runner); done
#2018-10-1015:59seancorfieldWe're trying hard to get away from Ant. All that XML shudder#2018-10-1015:59Alex Miller (Clojure team)I don’t think running a standard “program” per project is any different though#2018-10-1016:00seancorfieldYeah, hence why I'm coming around to having a "dev" project with short Clojure scripts with -main entry points. And, to be honest, that's not hugely different from having tasks in a build.boot file which are all just functions.#2018-10-1016:09dominicm@seancorfield would the dev project be a dependency of all projects?#2018-10-1016:13seancorfieldOnly under an alias for dev/test tooling -- it wouldn't be a regular dependency.#2018-10-1016:19dominicmsorry, yeah.#2018-10-1018:40dominicm@seancorfield once add-lib is stable, it might be a good idea to dynamically fetch dependencies based on the task(s) being run. e.g. nrepl will grab cider/tools.nrepl on demand. That would probably reduce the number of aliases way down. Basically making most of the project static (deps.edn) but providing a dynamism for the non-simple stuff.#2018-10-1615:01sparkofreasonWhat's add-lib? Sounds intriguing.#2018-10-1619:36dominicmIt's a function in a branch of tools.deps#2018-10-1018:42seancorfieldI dream of a world where tooling bootstraps itself on top of prepl instead of needing server-side dependencies 🙂#2018-10-1018:43seancorfield(I love unravel for that reason)#2018-10-1018:48dominicmsocket repl already does that, but I agree 🙂. But I think the same applies (for me) for loading in cljs to do a one-time build.#2018-10-1018:55seancorfieldWell, specifically the tooling. I'd like to see a version of CIDER that could bootstrap itself over a socket REPL or prepl. So you didn't need all the CIDER deps on the project side of the equation. Same with ProtoREPL for Atom.{:tag :div, :attrs {:class "message-reaction", :title "pray"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙏")} " 8")}
#2018-10-1018:57seancorfieldWhat I like about unravel is that I can spin up a bare Socket REPL (e.g., on a production JAR, via JVM options) and then connect via unravel and when I (require 'compliment.core) it sends it over the wire from the REPL so I don't need it in my project. Then I have a nice interactive REPL with history, documentation hinting, completion...#2018-10-1019:00seancorfieldYour project then becomes tooling neutral. You don't need any dependencies in it to support tooling. And then you can use any tooling with it. I know CIDER can inject itself into a newly-started REPL process, but that's a heavyweight server containing all of those dependencies. I basically want CIDER-like functionality over a bare Socket REPL started from a JAR that doesn't have those dependencies 🙂#2018-10-1019:24dominicmI just discovered I can use find-deps as a filter! Nice. This means I can do !aE in vim, followed by my find-deps command to auto-update the buffer I'm working on. Cool!#2018-10-1019:29dominicm@alexmiller have you had any thoughts about comments being obliterated by -Sresolve-tags?#2018-10-1019:30dominicmI should open a jira if there isn't one#2018-10-1019:30Alex Miller (Clojure team)reader doesn’t preserve comments#2018-10-1019:31Alex Miller (Clojure team)I’m not planning on doing any more work on resolve-tags. most likely will actually extract it all out into a separate project and remove it entirely#2018-10-1019:33Alex Miller (Clojure team)as an external program you could invoke it from clj if needed#2018-10-1019:40dominicmIf it's an external program that shifts expectations I think, cool. I would be more inclined to reimplement it with rewrite-clj so I could preserve comments/formatting.#2018-10-1019:47dominicmI've been thinking about whether a wrapper over rewrite-clj that implements IAssoc is feasible.#2018-10-1020:26Alex Miller (Clojure team)Well feel free to rewrite it and make it do whatever :)#2018-10-1020:41arrdem@seancorfield have you seen https://github.com/Unrepl/unrepl? It does pretty much that self-bootstrapping dance.#2018-10-1022:11seancorfield@arrdem yes I use that with socket server REPLs a lot. #2018-10-1022:12seancorfieldIt can bootstrap compliment into a running REPL to give you auto-completion, which I like. #2018-10-1022:29andy.fingerhutI guess bootstrap here just means "send the necessary text across the socket to define functions/etc. to create the necessary code on the server?"#2018-10-1022:32seancorfieldYeah. In this case, "send the Compliment library across the socket..."#2018-10-1101:36seancorfieldI needed very simple uberjar functionality with clojure.main as the Main-Class in the manifest. I forked and updated depstar to provide that https://github.com/seancorfield/depstar/commit/b49f15b0fc341f3004cb1bbf037334e3acbcb900 (a PR is in to @ghadi but my last PR to update the readme to explain the lack of manifest hasn't garnered a response yet so...).#2018-10-1101:38seancorfield;; uberjar building:
;; - see for a minimalist approach:
;; - uses my fork that adds manifest support
;; - clj -A:depstar result.jar -m
;; - sets clojure.main-main as the entry point
;; - clj -A:depstar result.jar my.project
;; - sets my.project/-main as the entry point
:depstar {:extra-deps
{com.healthfinch/depstar
{:git/url ""
:sha "a328136da59ea9b83483189451fb71aafd13bdb1"}}
:main-opts ["-m" "hf.depstar.uberjar"]}
#2018-10-1106:35Michael FianoAs a beginner to tools.deps and Clojure in general, I just banged my head against the wall for a couple hours figuring this implementation detail out, because the documentation leaves off this edge case. I discovered something with alias loading I thought would be worthwhile to share here in any case:
Assuming you have 2 aliases with :main-opts keys, regardless if other keys from other alias types are present in them or not:
If -M is specified on the command line for both aliases, only the last alias is used as documented.
If -A is specified on the command line for both aliases, only the last alias is used as per the documentation for -M.
If -A is specified for one, and then -M is specified for the other, only the last one is used.
If -M is specified for one, and then -A is specified for the other, still only the -M one is used, even though -A is later.
It's that last rule that got me good. In short, you have to be very careful here.{:tag :div, :attrs {:class "message-reaction", :title "grimacing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😬")} " 4")}
#2018-10-1108:12martinklepschAnyone recommendations for a thing that will package my deps.edn lib into a jar and install it into the local repo?#2018-10-1108:16martinklepschPack doesn't do regular jars, cambada doesn't install them into local maven, meyvn seems a little more involved#2018-10-1108:33orestisI think the other one is depstar? No idea if it does what you want though.#2018-10-1108:46mpenetyeah they all are a bit tricky to use, each with their own set of tradeoffs#2018-10-1108:48mpenetpersonnaly I am half tempted to just rely on having clj installed and run apps with it, more or less what you do with cljdocs I think @martinklepsch right (bash script that downloads the archive, upacks it and runs)?#2018-10-1108:51mpenetcapsule based build seemed quite a good idea (one of the options on juxt.pack), but I am not sure where that project is heading#2018-10-1109:02dominicm@mpenet happy to answer questions you might have!#2018-10-1109:05martinklepschI'm actually referring to regular jars (not uberjars)... e.g. when publishing a library#2018-10-1109:13dominicm@martinklepsch fwiw, you could use pack with skinny jar mode and write a pom.xml. But you'd have to install that jar yourself (with maven I guess?)#2018-10-1111:25mpenet@dominicm nothing special in mind. pack is imho the most complete solution so far. One minor missing piece would be the ability to specify a main on the one-jar option, but i can live without it#2018-10-1116:24dominicmThere should be an issue open for this. It is something I want to fix. I don't want to require AOT from users, so I want to patch onejar a little to support a default param to clojure.main.#2018-10-1112:23orestisWould be nice if there was only one well supported solution for packaging. I stared with cambada but had to fork to apply some PRs#2018-10-1114:50seancorfield@orestis sounds like me forking depstar to apply a PR to add a minimal manifest 😀#2018-10-1114:51seancorfield@martinklepsch is there a reason you need a jar in your local repo? Can't you refer to that local library source via :local/root?#2018-10-1115:07martinklepsch@seancorfield I can but I have some wierd interdependencies that aren't supported fully by Cursive/tools.deps#2018-10-1115:08cflemingRight, transient :local/root deps are not well supported (by deps itself, and also Cursive - I’m fixing that right now)#2018-10-1115:12orestisEven though I don’t use Clojure professionally yet, I’d chip in for an Arachne-like Kickstarter, but for a new build tool based on tools.deps and simple libraries. #2018-10-1115:13orestisThere’s so much duplication of effort these days - jar building, linters, formatters and so on. #2018-10-1115:15orestisI have a dream for a build tool that would just provide an (overridable/extendable) set of library decisions and some glue code to invoke them from the command line. #2018-10-1115:28Andreas LiljeqvistWhat you are describing is probably a program 😕#2018-10-1115:31Andreas LiljeqvistBut yeah, there is a lot of churn in the dev space right now#2018-10-1115:31Andreas LiljeqvistWill probably be a while before it simmers down with some kind of best practices#2018-10-1115:33orestisThe whole point is to rally everyone around a single tool. I think this has multiplier benefits for the community and newcomers. See for example Elixir and their mix tool. #2018-10-1115:40Andreas LiljeqvistThere is certainly value for newcomers.#2018-10-1115:40Andreas LiljeqvistAtm Lein is probably good for them#2018-10-1115:41Andreas LiljeqvistBut tools-deps is the first system that let me understand wtf was going on when starting up#2018-10-1115:41Andreas Liljeqvistsooo..#2018-10-1115:43orestisYeah, I’m all for tools.deps for dependencies and classpath stuff. #2018-10-1116:28seancorfieldI've been recommending clj to newcomers -- with the caveat that most books/tutorials use lein. At least with clj, you can point newcomers at "official" documentation in a central place and you know the tooling is both simple and well-supported.#2018-10-1116:34orestisAs a relative newcomer, clj is just so much less cognitive overhead than Lein. #2018-10-1115:34seancorfieldFor a long time, at work we've done no AOT and we build over uberjars to have clojure.main as an entry point. Our needs are very simple. We used to run a lot of stuff from source directly via lein and later boot but I think packaging as an uberjar for java -jar is simpler and safer (no need to fetch dependencies onto the target systems).#2018-10-1115:38Andreas Liljeqvist+1 for building uberjars#2018-10-1115:35seancorfieldClojureScript has had years of churn but seems to be settling down now. Maybe Clojure needs a bit of churn to shake things out and give us better, simpler tools too?#2018-10-1115:36Andreas Liljeqvisttools-deps is a great step in that direction. But causes a bit of chaos in the transition#2018-10-1115:46orestisTo be clear, IMO tools-deps enables the creation of such a mostly-glue tool. Otherwise making a new build tool from scratch is a huge undertaking. #2018-10-1116:05arrdem@martinklepsch the official story there seems to be write a pom.xml yourself, use the -Spom task to keep the dep lists in sync? (I don’t really understand what it does) and mvn install yourself.#2018-10-1116:26dominicmHaving attempted to write that build tool, I'm not happy with anything I've come up with. Build tooling is hard to get right, especially around tools like figwheel. #2018-10-1116:35orestisShare notes perhaps? Any attempt to explore the space is very valuable.#2018-10-1116:36dominicmKrei.alpha and kick.alpha are the two attempts by JUXT. We also did mach in the past which we used for alternative purposes. #2018-10-1116:38dominicm@U064X3EF3 keeps mentioning that builds are programs, so I suspect that both of those attempts to be data driven are fatally flawed.
I've been thinking of implement a paper by peyton jones for creating build systems a la carte, maybe that will unlock the pieces needed, but I'm not sure yet. #2018-10-1116:55orestisMy naive attempt so far is to just invoke some namespaces - either external or my own via aliases. My need is for more well rounded and robust libraries to do things like jars and so on. #2018-10-1116:57dominicmCaching. Minimal builds. Early cut off. These are the general problems to solve. Cloud building is a nice to have in some contexts, but none I'm in. #2018-10-1117:01orestisI’ll look at krei and kick, thanks. #2018-10-1117:05orestisI don’t think that there is any way around the current slow startup of clojure, is it? Even with precompiled stuff, you will get at least a handful of seconds which is already too much. Hard stuff. #2018-10-1117:09rickmoynihanyou could always fire up a build-tooling repl once, that puts you in a namespace with functions like (build) (test) (do-everything)#2018-10-1117:16zane@orestis Have you seen the results people are getting with GraalVM?#2018-10-1308:54dominicmSomething quite hard to build at a REPL is AOT-compiling namespaces. You have to start a new JVM to do that. You might be able to get away with something like a pod, but that's the best I have. Just as a result of how it decides to cache/skip namespaces.#2018-10-1116:31lilactownespecially since the UX is so dependent on perf. everyone wants things to be fast#2018-10-1116:31lilactownor at least, fail fast#2018-10-1116:39dominicmTo be honest, I think you have to just make it a library that you can run from the repl. It has to happen around the Clojure command line tools, else there is too much friction.#2018-10-1116:42dominicmEven shell scripts get in the way of tooling. Cursive won't work if you require funk.#2018-10-1116:42seancorfieldI really do like depstar's minimalism but it's frustrating that it omitted a minimal manifest -- and I'm not sure why it did that. Everything else in the build space is much more complex 😐#2018-10-1116:44dominicm@seancorfield the easy answer is that generating manifests is out of scope. War is another format that needs a manifest to be written. Simpler to let users decide how to handle extra files in their classpath. #2018-10-1116:44dominicmI don't mean to speak for the author, just a thought I've had working on pack.#2018-10-1117:04rickmoynihanyes, not so minimal when you start having to do that…#2018-10-1117:09seancorfieldAdding a manifest with just a main class is only a few lines of Clojure when you're building a JAR tho'... 🙂#2018-10-1117:28dominicmBut it conflicts with any additional things a user may want to insert into a manifest themselves. Users can also generate a manifest in a few lines of code. In most cases of can just be hardcoded by the user. #2018-10-1118:47seancorfieldRight, which is why it's important that adding the minimal manifest is optional. depstar specifically excludes META-INF/*.MF files from the classpath so the resulting JAR is guaranteed to have no manifest. My fork allows users to conditionally add a minimal manifest that specifies Main-Class (only). That seems to be such a common need -- for java -jar ... -- that baking it in as an option seems worthwhile.#2018-10-1118:48seancorfieldThe code is based on Boot's create-manifest so it could also take a hash map of key/value pairs to add as extra manifest attributes, but that's not exposed to the command-line in my fork (yet).#2018-10-1118:48dominicmI see. That seems quite wrong. It also removes license files which is not allowed under some licenses like Apache#2018-10-1118:49dominicmFun fact: run manifest doesn't capture the full capabilities of manifests! #2018-10-1118:49seancorfieldYeah, I noticed that and thought it was weird. I don't know why it does that (and Ghadi hasn't responded to any of my questions about it yet).#2018-10-1118:51seancorfieldWe're not redistributing our JARs so the license issue isn't a problem -- and depstar filters out a lot of other stuff too so it's pretty brutal in that respect.#2018-10-1118:35arrdemIs there an algorithmic reason beyond wanting to track the path for every expanded dep that expand-deps couldn’t group deps into resolution batches by manifest type? The Maven implementation used to read manifests seems designed for reading many manifests in one go, and less so for reading a single manifest at a time.#2018-10-1121:03arrdemThe hard thing is proving that letting Maven potentially solve range constraints instead of using deps’ behavior for everything can’t produce a different final dependency solution 😕#2018-10-1122:19Alex Miller (Clojure team)Maven will produce a different solution, and we don’t want it#2018-10-1123:28arrdemAh. Yeah if you have a non-Maven pinning (or even just winning coordinate) for a dep, and Maven resolves something else, you can’t really back the transitive consequences of that out of Maven’s solution so you loose minimality for sure which can have other consequences.#2018-10-1119:11Michael FianoRegarding my message above about aliases, does anyone think this deserves an issue opened to at the very least document this edge case?#2018-10-1119:15arrdemThe order independence of -M winning when -A and -M is present is probably worth documenting.#2018-10-1119:15arrdemSo yes IMO.#2018-10-1119:20dominicmI would prefer to see it fixed, but documentation would be an improvement over the current situation. #2018-10-1119:23Michael FianoPersonally I'd like to see the behavior change. The reason why is this:#2018-10-1119:23Michael Fiano:
I have a :dev alias that is only ever project-local. I use it for example to require my namespace when starting up an nrepl in cider. This requires a :main-opts key, making it of the -M type, but as per the documentation, only the last -M type is ever used. I also want to compose this with a debugging alias to say depend on and require spyscope with another alias. Currently this is impossible without putting that in the :dev alias and thus copying it to be project-local instead of a global alias.#2018-10-1119:24Michael FianoUnless there is some other way to compose a global main-opts with a project local one that I do not know of#2018-10-1119:25Michael FianoMaybe I'm just asking too much from this too new tool?#2018-10-1119:28seancorfieldFWIW, I only ever use -A and I split aliases out so I run into this issue (downside: you need to specify multiple aliases for some things).#2018-10-1119:29Michael FianoPerhaps @U064X3EF3 can weigh in here as I see he is the project lead. When i get home later I'll open an issue if only to document this edge case so other newcomers don't bang their heads like I did.#2018-10-1119:33andy.fingerhutIt is better to ask about whether it can change when things are still alpha, than later. It is really up to folks like Alex, Rich, and Stu Halloway on how they choose to make the tool work, so don't be surprised with the answer "doc enhancement, please"#2018-10-1119:35seancorfieldMy sense is that it is "by design" that aliases are combined in order, left-to-right and that -A:foo is just shorthand for -R:foo -C:foo -M:foo ... -- so the behavior you observe is intentional and likely considered to be already well-documented.#2018-10-1119:36seancorfieldIf you specify your project-local alias with -R and/or -C instead of -A then you'll get the behavior you want.#2018-10-1119:37Michael FianoThe confusing bit is that -M is documented as only the last one being used, but if -A is indeed shorthand for that, an -A after an -M needs to have a note mentioning that only the last explicit -M is applied, and -A is ignored.#2018-10-1119:37seancorfieldThe -A is not ignored -- the aliases are all combined left-to-right, but with :main-opts it's always last one wins.#2018-10-1119:37Michael FianoBut that is not the case.#2018-10-1119:38Michael Fiano-M:foo -A:bar, when both foo and bar have :main-opts keys, then foo wins.#2018-10-1119:42seancorfieldAh, gotcha. Here's why: val="$(join '' ${resolve_aliases[@]})|$(join '' ${classpath_aliases[@]})|$(join '' ${all_aliases[@]})|$(join '' ${jvm_aliases[@]})|$(join '' ${main_aliases[@]})|$deps_data"
#2018-10-1119:43seancorfieldSo -M is always applied last. OK, yes, that should be documented. Do you want to create a JIRA ticket for that (against the CLJ JIRA project I believe)?#2018-10-1119:43seancorfieldI'm happy to create the ticket if you want me to.#2018-10-1119:44Michael FianoI very much want to so others don't run into the headaches I did. I won't be able to do that until tonight or tomorrow though when I'm at my dev machine, unless you don't mind.#2018-10-1119:46seancorfieldI'll go ahead and create it... against the website since it's really an error in the Reference page for the CLI#2018-10-1119:46Michael FianoThanks @seancorfield!#2018-10-1119:54seancorfieldhttps://github.com/clojure/clojure-site/issues/306#2018-10-1122:00Alex Miller (Clojure team)This is a bug, not a doc issue. Order matters, should be respected.#2018-10-1123:33seancorfieldOK, good to know. So the shell script would need to expand each -A as it encountered it, adding the aliases as if all four were really specified explicitly?#2018-10-1123:33seancorfieldDo you want a JIRA issue (and maybe a patch) against CLJ for that instead?#2018-10-1123:46Alex Miller (Clojure team)Jira yes, patch if you like. I suspect it affects the scripts too so may require some coordination#2018-10-1123:52seancorfieldUgh! Yes, fixing the clojure script would solve one side of things -- that's easy -- but tools.deps also needs to be fixed too. OK. JIRA for now. I'll look at a patch if I get enough free time but it's a bigger change than I had initially thought.#2018-10-1201:38seancorfieldhttps://dev.clojure.org/jira/browse/TDEPS-100#2018-10-1201:42Michael FianoThank you everyone.#2018-10-1119:54seancorfieldhttps://github.com/clojure/clojure-site/issues/306#2018-10-1201:38seancorfieldhttps://dev.clojure.org/jira/browse/TDEPS-100#2018-10-1201:27arrdemRe: yesterday’s discussion of depstar, this is apparently floating around https://github.com/luchiniatwork/cambada#2018-10-1201:37seancorfieldYeah, that is a lot more comprehensive but has a lot of knobs and dials -- and has its own set of surprises, I gather, from people who've used it.#2018-10-1206:03tianshuis that in tools.deps, there's no exclusions?#2018-10-1206:08seancorfield@doglooksgood :exclusions work fine in deps.edn#2018-10-1206:11seancorfieldDo you have a specific case where you think it doesn't work?#2018-10-1206:13dominicm@doglooksgood there isn't a top level exclusions though, which is something people try though, only a per-dependency exclusions.#2018-10-1206:15tianshu@seancorfield When using leiningen, usually I found there're some conflicts in my dependencies, but with clojure -Stree, I don't get any information like this.#2018-10-1206:15tianshuSo I don't know if there's conflicts or not#2018-10-1206:18dominicm@doglooksgood I think tools.deps operates on a different principle to maven/lein, so conflicts are less of an issue. tools.deps always picks the newest version, as opposed to maven which picks the first one it finds.#2018-10-1206:24tianshuSLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tianshu/.m2/repository/org/slf4j/slf4j-nop/1.6.2/slf4j-nop-1.6.2.jar!/org/slf4j
/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tianshu/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2
.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
I want to remove one of the binding, but don't know where introduce the multiple bindings.#2018-10-1206:27tianshuwith lein I got some information like this:
Possibly confusing dependencies found:
[org.quartz-scheduler/quartz "2.3.0"] -> [org.slf4j/slf4j-api "1.7.7"]
overrides
[conman "0.8.1"] -> [hikari-cp "2.4.0"] -> [com.zaxxer/HikariCP "3.1.0"] -> [org.slf4j/slf4j-api "1.7.25"]
and
[org.quartz-scheduler/quartz "2.3.0"] -> [com.zaxxer/HikariCP-java6 "2.3.13"] -> [org.slf4j/slf4j-api "1.7.10"]
and
[ch.qos.logback/logback-classic "1.2.3"] -> [org.slf4j/slf4j-api "1.7.25"]
Consider using these exclusions:
[conman "0.8.1" :exclusions [org.slf4j/slf4j-api]]
[org.quartz-scheduler/quartz "2.3.0" :exclusions [org.slf4j/slf4j-api]]
[ch.qos.logback/logback-classic "1.2.3" :exclusions [org.slf4j/slf4j-api]]
#2018-10-1206:28tianshuthen I know I can remove the binding by add exclusions on the org.quartz-scheduler/quartz dependency.#2018-10-1206:30dominicm@doglooksgood this is different to conflicting dependencies. clj -Stree gives you the information you need to fix this, you need to find which dependency is bringing in the logback you don't want, either nop of logback-classic.#2018-10-1212:25Alex Miller (Clojure team)I don’t think -Stree actually does provide info to do that - it just lists what is used, not what isn’t #2018-10-1212:25Alex Miller (Clojure team)This is an area that needs some more work #2018-10-1212:26Alex Miller (Clojure team)If you don’t like the one you’re getting, you can try excluding that one and see which one you get instead#2018-10-1214:57dominicm@alexmiller that's what I meant. It will show where logback-nop comes from.#2018-10-1216:02seancorfield@alexmiller Any thoughts on adding some sort of "pedantic" check to tools.deps?#2018-10-1218:04Alex Miller (Clojure team)in terms of reporting or also failing on pedanticness?#2018-10-1218:18Michael FianoAre there any plans to allow whitespace in strings?#2018-10-1218:19Michael FianoI was reading the edn spec and it says commas are not treated as whitespace within strings, but I must use them for arbitrary code using -e for example#2018-10-1218:36Alex Miller (Clojure team)would like to allow, in practice it’s painful due to the interaction with the shell#2018-10-1218:37Michael FianoI see. Editor word-wise commands etc get confused so it's painful to edit#2018-10-1410:05isaacdoes deps.edn supports set env or properties?#2018-10-1410:41dominicm@isaac Yes. Although I'm not certain that I fully understand what you're asking.#2018-10-1410:43dominicme.g.
❯ clj -J-Dx=y -e '(System/getProperty "x")'
"y"
works.#2018-10-1410:48isaac:local/root "${LIB_ROOT}/xxx.jar"#2018-10-1410:48dominicm@isaac no, this doesn't work.#2018-10-1410:51isaacthere have difference normally between local dev env and production env#2018-10-1410:53isaacIMO, deps.edn supports env variable (or property) is necessary.#2018-10-1410:54dominicmYou can emulate this by using -Sdeps. e.g. clj -Sdeps "{:deps {foo/bar {:local/root \"${LIB_ROOT}/xxx.jar\"}}}"{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-1410:54dominicmNormally we solve this by using different aliases for dev and production.#2018-10-1411:17isaacwhy :main-opts ["-e" "(println 'hello)"] throws EOF while reading?#2018-10-1411:24isaacit seems the :main-opts splits arguments by space#2018-10-1412:18mfikes^ Known issue https://dev.clojure.org/jira/browse/TDEPS-56#2018-10-1412:23mfikesWorkaround with Corfield comma:
:main-opts ["-e" "(println,'hello)"]
#2018-10-1412:50aisamu>Corfield comma
🙂{:tag :div, :attrs {:class "message-reaction", :title "tophat"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🎩")} " 8")}
#2018-10-1418:32richiardiandreaCan clojure conjoin vectors in deps.edn ? I have a feature request in nrepl/nrepl that could really use that: https://github.com/nrepl/nREPL/issues/55#2018-10-1419:25dominicmYour vector is just a string#2018-10-1419:27dominicmYou may be able to compose them, but I don't think so. #2018-10-1419:27dominicmI think the correct fix is multiple -m flags#2018-10-1420:25richiardiandreaYep i agree! See suggestion in the nRepl issue 😄 #2018-10-1510:10dominicmUnexpected side effect of gitlibs with "optional" namespaces, e.g. for myapp.ns.spec. tools.namespace will try and load the namespace regardless. This is also frustrating for :local/root dependencies, where I actually want to be able to refresh and work on both in parallel, but not necessarily on the specs.#2018-10-1510:27dominicmThis is similar to what clojure.core goes through I suppose.#2018-10-1515:31danielnealI'm doing something derp-y here - can anyone halp - what have I missed out?
clj -Sdeps '{cheshire {:mvn/version "5.8.1"}}'
Clojure 1.9.0
user=> (require '[cheshire.core :as json])
FileNotFoundException Could not locate cheshire/core__init.class or cheshire/core.clj on classpath. clojure.lang.RT.load (RT.java:463)#2018-10-1515:32Alex Miller (Clojure team)clj -Sdeps '{:deps {cheshire {:mvn/version "5.8.1"}}}'#2018-10-1515:32Alex Miller (Clojure team)^^ missed :deps layer#2018-10-1515:32danielnealah awesome! Thanks#2018-10-1515:32danielnealso excited using the cli for the first time#2018-10-1515:33Alex Miller (Clojure team)I’ve been thinking about adding a warning as this is the nth time someone has asked a question like this{:tag :div, :attrs {:class "message-reaction", :title "point_up"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("☝️")} " 4")}
#2018-10-1515:33Alex Miller (Clojure team)(inc n)#2018-10-1515:33danielnealhaha so yup I can't read but at least it's not just me#2018-10-1515:33danielnealwarning++#2018-10-1515:34danielnealso nice just to get a quick clojure repl to prod this json object - I never can remember how to get jq to do what I want.#2018-10-1516:01seancorfieldYeah, I've been using clj for ages and I still do this, omitting the :deps layer! Probably at least once a day.#2018-10-1516:15mfikesAnother warning that could prove useful is if you omit the :mvn/version layer as in
{:deps {cheshire "5.8.1"}}
I do this, say, once per week.#2018-10-1516:15seancorfieldYou do get an error for that -- but it's a bit obscure.#2018-10-1516:16mfikesAhh, cool#2018-10-1516:16seancorfieldError building classpath. Coordinate type not loaded for library cheshire/cheshire in coordinate "5.8.1"
#2018-10-1516:17seancorfield(and, yeah, I do that all the time too!)#2018-10-1516:20orestisSounds like a need for some kind of tool that allows you to write specifications for data?{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 4")}
#2018-10-1516:20mfikesYeah, one that is in alpha 🙂#2018-10-1516:22orestisI’ve never typoed the maven thing, but it could be I’m just editing an existing file with plenty of examples. #2018-10-1516:36andy.fingerhutor perhaps a lint-like tool that has custom non-spec code that looks for and warns about data structures that look like they may be mistakes...#2018-10-1516:40andy.fingerhutFYI, a transcript of Alex Miller's "Dependency Heaven" talk is now available here: https://github.com/matthiasn/talk-transcripts/blob/master/Miller_Alex/DependencyHeaven.md#2018-10-1516:51seancorfieldWe've made a lot of progress in this last week on moving from Boot to clj/`deps.edn` for our monorepo.#2018-10-1516:52seancorfieldThe issues with relative paths in :local/root has meant we have to have everything "side by side" in one part of the repo.#2018-10-1516:53seancorfieldAnd we found we really needed a baseline deps.edn (in the project) and a per-subproject deps.edn (for each subproject in the repo).#2018-10-1516:54seancorfieldSo we ended up having a dummy subproject and pointing CLJ_CONFIG at that folder, so we can have all our "pinned" versions in :override-deps in an alias that every command has to use CLJ_CONFIG=../versions clj -A:defaults:other:aliases
#2018-10-1516:55seancorfieldWe have a small "build" tools subproject with a bunch of -main entry points for various tasks and we bring it in with aliases (in ../versions/deps.edn).#2018-10-1516:58seancorfieldWe'll probably have a couple of shell scripts to provide some sugar over those invocations but we've got nREPL capabilities, test, build a JAR, run migrations (SQL/data) all up and running with "just" clj and a few aliases. I need to convert two more pieces this week and we'll be close to just removing Boot from our pipeline.#2018-10-1517:17lwhortonhey with the new tools.deps i’m tooling around with a local jar file via the :local/root "/path/to/jar", but changes to the jar don’t seem to be reflected in the repl session. what do i have to do to (reload) or re-require after changing the jar so that the latest code is around in the repl?#2018-10-1519:13Alex Miller (Clojure team)I’m not sure that you can get the default Java classloaders to do this - they don’t pick up modified jar files after startup. So even reloading your Clojure namespace is probably not going to help afaik.#2018-10-1519:30lwhortonthat seems to be what’s happening. for now we’re just hard-restarting the whole repl#2018-10-1520:15Alex Miller (Clojure team)Something like a boot pod would give you a fresh classloader and the ability to do this#2018-10-1517:29seancorfieldThat sounds like a plain ol' REPL workflow issue and you'd need to tell Clojure to reload namespaces from that JAR, regardless of which tool you were using.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-1517:29seancorfieldMind you, if you're rebuilding the JAR locally, why not point :local/root at the source code instead and remove the JAR step?#2018-10-1517:30seancorfield(you'll still need to tell Clojure to reload the code -- after all, the classpath is not changing and Clojure doesn't know to re-read files on the classpath unless you tell it to)#2018-10-1517:33lwhorton> reload namespaces from that JAR
is that a good old (require '[foo.bar]) again?#2018-10-1517:56mfikes@lwhorton (require '[foo.bar] :reload){:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-1518:02dominicm@seancorfield what layout did you have before? #2018-10-1518:13seancorfield@dominicm We had a top-level deps.edn (equivalent) and our misc. build tooling was in a completely separate folder build/
<tooling here>
clojure/
deps.edn equivalent
subproject-a/
deps.edn
subproject-b/
deps.edn
...#2018-10-1518:17seancorfieldNow we have build/
<wrapper shell script>
clojure/
build/
deps.edn
build/dev-related Clojure code
versions/
deps.edn -- pinned versions and aliases
subproject-a/
deps.edn
subproject-b/
deps.edn#2018-10-1518:19dominicm@seancorfield not super different then. Interesting solution using CLJ_CONFIG. Hard to get people to set that up, but not a problem in a smaller team though. #2018-10-1518:20seancorfieldThat's why we have a wrapper shell script 🙂#2018-10-1518:23seancorfieldThe shell script is basically build task subproject and it maps mostly onto (cd path/to/<subproject>; CLJ_CONFIG=../versions clojure -A:defaults:<task>)
#2018-10-1518:24seancorfieldbuild nrepl datamapper or build test environment or build migrate sql etc#2018-10-1518:43dominicm@seancorfield that doesn't seem very amenable to cursive or ciders jack in? How are you editing? #2018-10-1519:14seancorfieldWell, I'm using Atom/ProtoREPL which doesn't support clj in any useful way and my colleague who uses Emacs/CIDER always starts his REPLs by hand and then connects to it via CIDER...#2018-10-1519:15seancorfieldBecause of the whole CLJ_CONFIG=../versions clojure -A:defaults thing, it's a pain to configure tools to start clj-repls that way anyway.#2018-10-1519:33dominicmYeah. It is as a result of that 🙂. You could restore that functionality by automating some kind of alias merging across your deps.edn files.#2018-10-1519:33dominicm(This is something I expect I will end up writing, but I'm still fiddling with ways to avoid it)#2018-10-1519:47seancorfieldYou'd either need to have support for multiple deps.edn files in a clj command or your user deps.edn file would need everything we have in versions/deps.edn tho'... (if you mean to remove CLJ_CONFIG=../versions from the command itself)#2018-10-1519:47seancorfieldEither way, not a problem for the sort of usage we have/need.#2018-10-1519:47dominicmI understand. I was looking to you for a silver bullet, as I'm currently getting Edge more & more tools.depsy.#2018-10-1519:48seancorfieldThere are no silver bullets (tm)!#2018-10-1519:48dominicmI can feel Fred Brooks channeling through you#2018-10-1520:40andy.fingerhutWhat are your views on bronze bullets? 🙂#2018-10-1521:02arrdem@seancorfield I’d be curious what you think of https://github.com/arrdem/katamari - I just got it to the point of being able to build classpaths, uberjar and self-host last night. It’s designed to solve the sorts of many targets and shared configuration in a single monorepo friction you’re encountering as I hit them too in my first pass at deps for our monorepo.{:tag :div, :attrs {:class "message-reaction", :title "champagne"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🍾")} " 4")}
#2018-10-1521:34zane🎶 Na na~ 🎶#2018-10-1521:08seancorfieldWill take a look. Thanks @arrdem#2018-10-1521:19arrdemIt’s extremely pre-alpha and currently extremely under-documented, but I’m happy to answer questions and will be trying to clean it up as I go.#2018-10-1602:03kennyI have this code:
(ns where-print
(:require
[clojure.spec.alpha :as s]))
(s/def ::foo string?)
(defn -main
[& args]
(when-not (s/valid? ::foo 1)
(s/explain ::foo 1)
(System/exit 1)))
If I run clj -m where-print nothing is printed to stdout. Any idea how to prevent this from happening?#2018-10-1602:45seancorfield(System/exit 1) is a pretty brutal way to exit the program and it can quit the JVM before output is flushed. If you add (flush) before that exit, you should get your output.#2018-10-1602:49seancorfieldIt would also work if you did this: (defn -main
[& args]
(when-not (s/valid? ::foo 1)
(println (s/explain-str ::foo 1))
(System/exit 1))) since println will flush stdout to the newline (although your output will now have two newlines) @kenny#2018-10-1603:07kenny@seancorfield thanks. By brutal, are you hinting at a better way to exit the process with a unsuccessful exit code?#2018-10-1603:14seancorfieldNo. Just observing that it it immediately kicks off a shutdown of the JVM and some things may not get done that you are expecting 🙂#2018-10-1606:01dottedmagJVM — Unleash your inner Hulk!#2018-10-1707:37dominicmSimilar data point to knowing about conflicts, I'd really like to know if a dependency wants a newer version of a dependency than I am giving it.
project depends on b 0.1
project depends on a
a depends on b 0.2#2018-10-1718:55seancorfieldAs of this morning, our entire CI pipeline (including SQL migrations, JAR building and deployment to servers) runs on clj/`deps.edn` for most of our projects -- previously that was all on Boot.#2018-10-1809:03martinklepschdo you also build jars for libraries which you push to a maven repo? I find the existing libs mostly focus on uberjars or don't help with the creation of the pom file/actual deployment.#2018-10-1814:49seancorfieldWe only create uberjars for deployment. #2018-10-1720:10ghadidang!#2018-10-1720:20Alex Miller (Clojure team)!#2018-10-1720:20Alex Miller (Clojure team)New tools.deps.alpha and clj out: https://groups.google.com/d/msg/clojure/xCa1mU2sOWs/XcEAGJ4RBQAJ#2018-10-1720:21Alex Miller (Clojure team)Mostly a fix/recommendation around connecting to private s3 maven repos{:tag :div, :attrs {:class "message-reaction", :title "tada"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🎉")} " 12")}
#2018-10-1811:55pauldHow can I debug the clj tool chain? clj hangs when I put org.openjfx/javafx-graphics {:mvn/version "11"} in the deps.edn file.#2018-10-1811:59gfredericksdid it successfully download the jar?#2018-10-1812:02bronsaI see a massive cpu spike after clj finishes downloading the artifact#2018-10-1812:04pauldI don't know. I can get it working it boot, so boot was able to download the jar.#2018-10-1812:04pauldmaybe I need to wait longer? Still it works just fine in boot, so there is some issue here with the clj tool.#2018-10-1812:09bronsait looks to me like there's some infinite loop during dependency resolution#2018-10-1812:10mpenetThe latest clj should fix that#2018-10-1812:10mpenetAt least there was an issue related to this#2018-10-1812:10bronsayeah I remember that, but I'm on latest tools.deps#2018-10-1812:10mpenetTDEPS-79{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 4")}
#2018-10-1812:10mpenetoh #2018-10-1812:10bronsaat clojure.lang.APersistentVector.hasheq(APersistentVector.java:162)
at clojure.lang.Util.dohasheq(Util.java:177)
at clojure.lang.Util.hasheq(Util.java:168)
at clojure.lang.PersistentHashMap.hash(PersistentHashMap.java:120)
at clojure.lang.PersistentHashMap.valAt(PersistentHashMap.java:152)
at clojure.lang.PersistentHashMap.valAt(PersistentHashMap.java:156)
at clojure.lang.RT.get(RT.java:753)
at clojure.core$get.invokeStatic(core.clj:1492)
at clojure.core$get.invoke(core.clj:1486)
at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
at clojure.core$reduce1.invokeStatic(core.clj:934)
at clojure.core$get_in.invokeStatic(core.clj:6062)
at clojure.core$get_in.invoke(core.clj:6062)
at clojure.tools.deps.alpha$excluded_QMARK_.invokeStatic(alpha.clj:87)
at clojure.tools.deps.alpha$excluded_QMARK_.invoke(alpha.clj:83)
at clojure.tools.deps.alpha$include_coord_QMARK_.invokeStatic(alpha.clj:113)
at clojure.tools.deps.alpha$include_coord_QMARK_.invoke(alpha.clj:106)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:180)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:165)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:228)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:210)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invokeStatic(make_classpath.clj:59)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invoke(make_classpath.clj:52)
at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:70)
at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:64)
at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:109)
at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:84)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
#2018-10-1812:11bronsahere's a thread dump as clj hangs#2018-10-1812:12bronsaoh that ticket hasn't been resolved yet @mpenet#2018-10-1812:12bronsaso maybe it's exactly that#2018-10-1812:19mpenetPatched was applied tho i think#2018-10-1812:20pauldI'm using .397#2018-10-1812:20bronsait's been reverted @mpenet#2018-10-1812:20bronsahttps://github.com/clojure/tools.deps.alpha/commit/b03942bbc75780537d8bfd91535b3a21d79a93ee#2018-10-1812:21mpenetAh#2018-10-1812:25Alex Miller (Clojure team)It was accidentally applied - I am working on alternate patch{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-1812:30pauldMore background: https://dev.clojure.org/jira/browse/TDEPS-79#2018-10-1813:07Alex Miller (Clojure team)@pauld can you drop a comment on TDEPS-79 with your deps.edn so I can double-check it is the same thing?#2018-10-1813:09pauldsure#2018-10-1813:19pauld@alexmiller done#2018-10-1813:20Alex Miller (Clojure team)thx#2018-10-1816:12hlollipart java question, part tools.deps question. I'm trying to leverage the git/url dep functionality on this java repo
com.kunstmusik/CsoundJNA {:git/url ""
:sha "a717dc3af6bf268b2962cac854d3fa9c0e2901a9"}
it's downloaded to ~/.gitlibs and is added onto the classpath, all good there.
But the importing is the tricky part, knowing little about java, could it be that the pom.xml file is ignored or is missing some information, https://github.com/kunstmusik/CsoundJNA/blob/master/pom.xml or is my import on the java class wrong?
(:import [com.kunstmusik.csoundjna CsoundJNA])
in my deps.edn file I have this as well
org.clojure/tools.deps.alpha {:mvn/version "0.5.460"}
#2018-10-1816:18Alex Miller (Clojure team)you can’t use a Java dep like this as something has to compile the java file to a class file. For that, you either need a build tool or an artifact with compiled classes in it.#2018-10-1816:18Alex Miller (Clojure team)git deps are only going to be useful with Clojure source only (which does not require a build)#2018-10-1816:19hlolliah, I understand now. Thanks @alexmiller! I'll ask my friend to just publish it to maven, probably easier 🙂#2018-10-1818:01bastiIs there a way to check for the latest version of dependencies defined in deps.edn?#2018-10-1818:01dominicmTool named depot by olical @basti {:tag :div, :attrs {:class "message-reaction", :title "ok_hand"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👌")} " 4")}
#2018-10-1818:08bastiThanks I’ll check it out!#2018-10-1819:32seancorfieldWhy does (reader/clojure-env) care about actually building a classpath?#2018-10-1819:35seancorfieldBackground: we use an env var for CLJ_CONFIG so we have a known "baseline" for aliases and override-deps etc; but our actual deps require an alias -- -Adefaults -- to resolve since they all have {} for the coords and the actual versions are provided by the alias and :override-deps. So we can run CLJ_CONFIG=../versions clojure -Adefaults -Sdescribe
but if we omit the alias, (reader/clojure-env) blows up with Error building classpath. Coordinate type not loaded for library selmer/selmer in coordinate {}
Because of course we can't pass the alias into the clojure-env shell call to clojure -Sdescribe#2018-10-1819:36seancorfield@alexmiller Is there a reason that describe needs to build the classpath? If so, would you be open to a patch to allow additional arguments to be passed into clojure-env to support required aliases etc?#2018-10-1819:39dominicmI think I was supposed to open a ticket for that, sorry#2018-10-1819:43seancorfieldLooks like it fails because it thinks it is "stale" and therefore runs the classpath generation code.#2018-10-1819:55Alex Miller (Clojure team)@seancorfield there is a ticket for this already (and no, I don’t think so)#2018-10-1819:56Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-82#2018-10-1819:58seancorfieldYup, confirmed. I modified my local clojure script to set stale=true when describe is true, and also set cp= to avoid loading the (possibly non-existent) .cpcache file, and then clojure-env runs as expected in my environment.#2018-10-1821:21seancorfieldI attached a patch to that ticket matching what I changed locally to make that work @alexmiller /cc @U09LZR36F#2018-10-1821:21Alex Miller (Clojure team)thx!#2018-10-1916:46seancorfield@alexmiller Do you think TDEPS-82 will get fixed and released "soon"? We have an "outdated dependencies" step in our build chain that we'd like to run other places that just my dev machine so we'd need an updated clojure release for that.#2018-10-1918:13Alex Miller (Clojure team)I would like to fix TDEPS-79 and would then be happy to pull 82 along with that into a release{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-10-1819:55ro6@alexmiller I think the documentation formatting is broken at https://clojure.org/reference/deps_and_cli#_maven_s3_repos#2018-10-1819:56Alex Miller (Clojure team)thx, I just modified that yesterday, will fix#2018-10-1819:58ro6Looks like maybe the code block was supposed to end after <servers>
...
<server>
<id>my-private-repo</id>
<username>AWS_ACCESS_KEY_HERE</username>
<password>AWS_SECRET_ACCESS_KEY_HERE</password>
</server>
...
</servers>
but didn't.#2018-10-1819:59Alex Miller (Clojure team)yeah, just fixed#2018-10-1914:31jeff.terrellIs there a clj/clojure CLI equivalent to lein deps :tree?#2018-10-1914:55seancorfieldThe closest is clj -Stree but it doesn't show conflicts, just the resolved dependencies. Folks have asked for this a few times. You may be able to get more information with -Sverbose -Sforce. #2018-10-1914:56jeff.terrellThanks. Nice to know for sure that this feature doesn't exist (yet). And -Sverbose -Sforce doesn't illuminate the conflict.#2018-10-1914:58seancorfieldI'm not at my computer right now but there is a way to get tools.deps to tell you what versions it finds and what it selects -- which is close. #2018-10-1914:34jeff.terrellOr more generally: how can I diagnose a dependency conflict in a deps.edn project? I searched in various places (web, FAQ, deps & CLI guide and reference) but didn't see an answer. Seems like the kind of relatively undocumented thing that might go in a FAQ entry, which I'm willing to write if there's a good answer.#2018-10-1914:38jeff.terrellAnd by the way, is there a canonical adjective to use for projects that manage dependencies with deps.edn? I'm looking for something parallel to 'A Leiningen project' and 'A Boot project'. A 'tools.dep' project? A 'deps.edn' project? A 'clojure deps' project?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-1914:57seancorfieldI've been calling them 'A clj project' to parallel the use of the tool to run them but there's nothing official as far as I know. #2018-10-1914:58jeff.terrellThat's better than circumlocution, and maybe not too ambiguous. simple_smile Thanks.#2018-10-1915:07Alex Miller (Clojure team)I would say not tools.deps, but either deps.edn or Clojure tools project{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-10-1914:55seancorfieldThe closest is clj -Stree but it doesn't show conflicts, just the resolved dependencies. Folks have asked for this a few times. You may be able to get more information with -Sverbose -Sforce. #2018-10-1914:57seancorfieldI've been calling them 'A clj project' to parallel the use of the tool to run them but there's nothing official as far as I know. #2018-10-1923:55jimmyIs there a way to run clj and just have it pull down dependencies listed in deps.edn? Not start a repl or run anything, just download the dependencies.#2018-10-2000:00lilactownI run clj -Stree to do that#2018-10-2000:29Alex Miller (Clojure team)-Spath{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-2005:36richiardiandreaSilly question: when clojure emits a Downloading: I guess it did not find the dep already in .m2 right?
I am trying to figure out why I have some many "Downloading" in my CI build#2018-10-2005:42richiardiandreaYeah checking the code it seems so. It seems to me very odd that I am downloading the same deps over and over. There must be some maven setting wrong#2018-10-2005:43richiardiandreaRunning on Azure Pipelines I wonder if somebody has had any problem#2018-10-2012:21Alex Miller (Clojure team)In CI, often you don’t get the .m2 dir preserved. Some providers let you mark that in different ways.#2018-10-2015:23richiardiandreaIt is odd because I am running it all the tests/commands in the same shell#2018-10-2218:39jeff.terrellIf you have a deps.edn project, how do you push it to Clojars?
The Clojars "pushing" wiki page, for example, gives instructions for Leiningen, Boot, and Maven, but not for deps.edn:
https://github.com/clojars/clojars-web/wiki/Pushing#2018-10-2218:40seancorfieldYou don't 🙂#2018-10-2218:40seancorfieldA deps.edn project can be depended on directly on GitHub.#2018-10-2218:41seancorfieldA more helpful answer to your question is that you will need to use something like cambada to create a "thin" JAR and then do something appropriate to generate pom.xml and deploy it up to Clojars (Maven, I guess?).#2018-10-2218:59jeff.terrellThanks, Sean. Yeah, I knew that about depending directly on GitHub commits. simple_smile But since the CLI tools don't work on Windows yet, I'm using Leiningen on my project, and Leiningen can't depend on GitHub coordinates (even with the lein-tools-deps plugin).#2018-10-2219:00jeff.terrell(I don't use Windows, but some members of my team do, and installing a Linux environment is a bit of a tall ask at this point.)#2018-10-2219:09seancorfieldI'm surprised lein-tools-deps can't depend on git coords... boot-tools-deps supports those...#2018-10-2219:13jeff.terrellIt can depend on git coords, but that functionality doesn't work on Windows. It complains about not being able to find the clojure command.#2018-10-2219:24Alex Miller (Clojure team)you can use clj -Spom to create a pom, which could be modified into a deployment path too#2018-10-2219:25Alex Miller (Clojure team)(in any answer to prior questions)#2018-10-2219:25Alex Miller (Clojure team)-Spom syncs, so if you change it the pom, those changes (outside the dependencies) will be respected{:tag :div, :attrs {:class "message-reaction", :title "eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👀")} " 4")}
#2018-10-2219:26jeff.terrellNice, good to know, thanks Alex.#2018-10-2317:37dominicmI might have a try tonight at getting pack to work with clojars (ish). The parts are there. {:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-10-2318:15dominicmHow desperate are people to avoid mvn the cli tool? I think pack already works with mvn#2018-10-2318:37seancorfieldmvn just has such a nasty CLI compared to, say, lein deploy or boot deploy...#2018-10-2318:41dominicmhttps://clojars.org/io.dominic/super-duper-octo-barnacle but it also works 😄 (almost first time)#2018-10-2318:42dominicmI find it a little weird that deploy-file doesn't look at pom.xml to figure out the url automatically, but meh#2018-10-2318:44dominicmI'll write up the process with mvn and put it in pack's readme, and point at the working example.{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-10-2318:57dominicmhttps://github.com/juxt/pack.alpha#uploading-to-clojars-or-maven really simple 🙂 If someone knows a mvn-less way to upload to maven/clojars, I'm all ears. Seems like a good thing for a small targeted library to do really well (like with really secure credentials settings from day #1).{:tag :div, :attrs {:class "message-reaction", :title "clap"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👏")} " 4")}
#2018-10-2319:20seancorfieldTo save me the trouble of actually trying to build a skinny JAR with pack and opening it up to look inside, does it add a manifest and, if so, what attributes are in it?#2018-10-2319:27dominicmThe mvn deploy:deploy-file can take a pom.xml externally to use. I guess it uploads separately. You could put it inside the jar by adding a mkdir extra && cp pom.xml extra/ then passing -e extra to pack when building.#2018-10-2319:27dominicmI don't think things generally depend on the pom.xml being inside the jar though.#2018-10-2319:37seancorfieldSo... no manifest added nor needed?#2018-10-2319:38seancorfieldAnd what is the difference between a "regular" skinny jar from pack and one that also has --no-libs?#2018-10-2319:38dominicm@U04V70XH6 not inside the jar, nope. You do need a pom.xml, but it's generated out of band. I didn't actually check if it's in my jar, one sec, I will#2018-10-2319:38dominicmNo pom.xml in my jar. It runs fine.#2018-10-2319:38seancorfieldI'm thinking back to the discussion/complaints about depstar not adding even a basic manifest into the uberjar (so you couldn't just java -jar it).#2018-10-2319:39dominicm@U04V70XH6 That's a MANIFEST.MF, you can use the -e flag in pack to add it to your classpath at build time 🙂#2018-10-2319:39seancorfieldIt wasn't a big deal to add a minimal manifest with Main-Class but of course you don't actually need to -- you can just do java -cp ... clojure.main ... and run the JAR anyway.#2018-10-2319:40dominicm@U04V70XH6 regarding --no-libs, the general idea of skinny jar mode is to produce something that's really cheap to rsync or something onto s3. So you generally output:
app.jar
lib/clojure-1.8.0.jar
lib/github-0.1.0.jar
And then upload app.jar + any files from libs that are missing.
This was requested by a company doing java, this is their upload process.#2018-10-2319:40seancorfieldYeah, my point/question was "Does pack add a manifest by default?" and the answer seems to be "no" and, implied, does a library published to Clojars need a manifest and I think the answer is also "no".#2018-10-2319:41dominicmCapsule/Onejar do add a manifest, because it's required for them to function. I'm not entirely happy with the process though. I think more could be done here, but there's nothing to drive change currently.#2018-10-2319:41seancorfieldAh, so "skinny" means no libs in the JAR but instead copy them to a lib/ folder?#2018-10-2319:42dominicm@U04V70XH6 Yep!#2018-10-2319:43dominicmIn the future I might add a dedicated maven namespace to pack. I want to end up where you go via use-case rather than "underlying details". UberJAR complicates this a little, as there's choice.#2018-10-2319:44seancorfieldPart of me is just contemplating what I might do with my fork of depstar -- to keep it as minimal as possible but open up the possibility of producing a thin JAR and being part of a workflow that could produce a pom.xml and mvn it up to Clojars... or another repo...#2018-10-2320:23dottedmagPardon my ignorance, does Lein use mvn to upload?#2018-10-2320:24andy.fingerhutProbably more likely to find someone who knows the answer to that in #leiningen channel#2018-10-2320:26dottedmagFigured it out already 🙂 It uses cemerick/pomegranate - a wrapper around Eclipse Aether.#2018-10-2320:43arrdemtools.release(.alpha) 😉{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-2403:18seancorfieldI gave in and put seancorfield/clj-new and seancorfield/depstar on Clojars. Releases 0.5.4 and 0.1.2 respectively. JARs built with (my fork of) depstar. Initial pom.xml built by clj -Spom. Pushed to Clojars with mvn deploy:deploy-file (ugh!).#2018-10-2417:18dominicmTopic of today. I want multiple aliases which compose and start multiple things. The goal would be using able to start a program like normal, but perhaps side load nrepl in.
Best idea I have so far is a custom main and jvm options. The custom main would evaluate code based on the provided options, and then finally run the -m it was provided.
Any other suggestions for ways to interface with such a thing?
I did consider taking a list of mains, but then you have to detect when they are done in an arbitrary way, but that might be fine by looking at the thread state? Also main-opts don't concatenate. #2018-10-2509:06codonovanYes. I think we had to use lein-figwheel "0.5.16" and figwheel-sidecar "0.5.14" to get it working. https://github.com/drapanjanas/re-natal/issues/190 might be helpful#2018-10-2515:20xfyreInteresting, let me check this in Xcode. I was experiencing reload issues before I even tried to upgrade Figwheel#2018-10-2515:34xfyreI’m seeing
2018-10-25 18:34:41.958455+0300 DcnetApp[90232:23128804] [] nw_socket_handle_socket_event [C224.1:1] Socket SO_ERROR [61: Connection refused]
#2018-10-2418:55seancorfield@dominicm I thought about this long and hard as we switched our dev/test/build pipeline over to clj from boot -- and ended up with the "custom main and jvm options" approach, with several small "microprojects" containing the various -main we want that are added via aliases... :main-opts don't concatenate for a (good) reason but that does sometimes make it hard to "compose" multiple tasks in a "build pipeline". Our compromise there was to create a build shell script that runs clojure for each set of command line arguments (loosely... it implements a sort of mini-DSL around aliases and subprojects to make it easy for us to kick off a sequence of tasks from a single command line).#2018-10-2418:56seancorfield(as of yesterday, we no longer use boot for any of our dev/test/qa/prod stuff -- with two legacy exceptions, one of which I'm working on right now 🙂 )#2018-10-2419:28dominicm@seancorfield Just so I am clear, you have a custom main which calls other -main? Or do you look at JVM opts in order to determine what to start in your one custom main?#2018-10-2419:29seancorfield"It Depends". We have some -main functions that use the supplied command line arguments to decide what to run (namespace-qualified function names, usually). But also, now we have our little build wrapper shell script, we often just run clojure multiple times with different (main) options...#2018-10-2419:30seancorfieldBut the former approach could easily use JVM options instead of command line arguments. Or as well as.#2018-10-2419:31seancorfield(the main issue is determining what arguments -- command line or otherwise -- to call each of the functions with, based on just a single -main entry point and its own command line arguments)#2018-10-2419:32arrdemit’d be interesting to have a “multi clojure.main” that took say an EDN array of mains and ran each one in its own future / send-off.#2018-10-2419:32arrdemor let you have “mixins” that get sent off and one “main”.#2018-10-2419:33dominicm@arrdem I'm toying with that idea. The hard limitation being that if you actually want the main to run once and then block, it's more difficult. Also, if you want ordering (e.g. producing some state picked up for the next main), then you have to "detect" when the main is done.#2018-10-2419:33arrdemConcatenating programs into a single JVM seems like … a job for a real program not some crazy main. We have a thing which does that, it’s a REPL 😉#2018-10-2419:34arrdemFor Katamari I’ve been toying with the idea of how to let kat reuse a JVM between taks if your dependencies haven’t changed. Not sure you can actually blow away enough Clojure state (in an unmodified Clojure) to do that safely or if it buys you anything.#2018-10-2419:35dominicmI'm trying to leverage existing standards (e.g. -main). But I agree. I'm feeling like I might have to bite the bullet and "force" something new onto people for daemons like nrepl.#2018-10-2419:36arrdemYeah. Remote daemonized nREPL servers with a client hooked to your terminal is the best I’ve got for stuff like this.#2018-10-2419:36arrdemI need to take a poke at pack.alpha, looks like it may provide more of the machinery I want for Katamari’s jar, uberjar and maybe pom targets than depstar does.#2018-10-2419:38dominicm@arrdem I'm trying to move towards having some kind of API for consumers. But I've prioritized -main support for now. I'm very welcome to improvements to the "api" 🙂#2018-10-2419:38arrdem@dominicm cool - I’ll see what I find 😉#2018-10-2419:38arrdemkat is already a pile of alpha APIs what’s one more.#2018-10-2419:39dominicmHehe.#2018-10-2419:39arrdem@seancorfield curious what wound up in your build script.#2018-10-2419:44seancorfield@arrdem Well, our basic command to run anything is essentially cd <repo>/clojure/<subproject>
CLJ_CONFIG=../versions clojure -A:defaults:<alias> <some> <args>
so mostly the build script maps pairs of <task> <subproject> on the command line into runs of that command pair.#2018-10-2419:45seancorfieldSince arguments are inherently variadic, we treat <task> <subproject> as a command with no args and [ <task> <subproject> <some> <args> ] as a command with some args. So build parses the [ and ] delimiters around commands.#2018-10-2419:46arrdemInteresting.#2018-10-2419:47seancorfieldPlus a couple of convenient conventions: <task>+test means "map <task> to its alias(es) and also add :test:test-deps; and <task>s means "run <task> for each of the (subproject) arguments provided" -- so you can easily build multiple uberjars or run multiple subprojects' tests.#2018-10-2419:48seancorfieldOh, and <task>-all which expands a list of subprojects to include all the others (recursively) that are pulled in as :local/root deps (so we can easily "test a subproject and everything it depends on")#2018-10-2419:49seancorfieldVery minimal really. And of course we could do it all from the command line but... we're lazy programmers...#2018-10-2419:50seancorfieldEverything beyond that "DSL" is handled with aliases (and command line arguments passed to clojure).#2018-10-2419:52arrdemHum. Do you just have standardized eg /test paths for the -all alias or how did you get dependees tests recursively included.#2018-10-2419:54seancorfieldA monorepo with subprojects all side-by-side in the clojure directory, each with src and test folders beneath them. Is that what you meant?#2018-10-2419:54arrdemYep.#2018-10-2419:54arrdemCool, thanks for the info!#2018-10-2419:54arrdemInteresting to see what other folks are coming up with.#2018-10-2419:55seancorfieldWe deleted most of our 2,000 line build.boot file yesterday 🙂#2018-10-2419:55seancorfieldI expect to delete almost all of the rest of it by week's end.#2018-10-2420:14dottedmag@seancorfield How big is the replacement?#2018-10-2420:16seancorfieldA very well commented and spaced out bash script with about 70 lines of help output -- total just under 300 lines 🙂#2018-10-2420:17dottedmagSo, 2000 lines of build.boot was just a dispatcher?#2018-10-2420:17seancorfieldNo... it's kinda hard to explain 🙂#2018-10-2420:18seancorfieldWe used to dynamically determine cross-subproject dependencies using tools.namespace -- seeing which namespaces a subproject required and which subprojects those were actually defined in. All that is gone and replaced with explicit :local/root dependencies now.#2018-10-2420:20seancorfieldPlus we had accrued a lot of now-unused build tooling as we evolved the project over several years (we switched to Boot at the end of 2015). We had a lot of per-subproject-specific stuff in the Boot file which was never made generic (which the shell script does).#2018-10-2420:20seancorfieldAnd, lately, this Boot bug was killing us https://github.com/boot-clj/boot/issues/715#2018-10-2420:21dottedmagI see.#2018-10-2420:23seancorfieldBoot had encouraged us to build complex tooling because it was easy to do so. The reason we switched from Leiningen to Boot three years ago was because we wanted to do more programmatic stuff -- which was much, much easier in Boot -- but we sort of got carried away 🙂 And moving to clj and deps.edn and aliases kind of focuses the mind on what's actually required.#2018-10-2420:31dottedmagIt reminds me of one of build steps (implemented in Go, to boot) I'm guilty of writing: it extracts from a Go binary a list of versions of packages it was built from, parses list of packages current module is dependent upon, and then decides whether that binary needs to be rebuilt. And to add to the insult, the list of binaries to check is created by parsing a Go source file.#2018-10-2420:33dottedmagVery nifty, cuts down maybe 10s from the full build, but it's more than 300 lines of code just for that.#2018-10-2420:34seancorfieldAh, yes, an easy trap to fall into 🙂#2018-10-2420:36arrdemCurious what makes you see that as a trap.#2018-10-2420:37seancorfieldWhen your build is "just code" it's easy to just start writing code -- without thinking about whether it is a) needed or b) an over-complicated way to achieve your goal.#2018-10-2420:51dottedmagAlso this code is usually easy and fun to write, so one can waste enormous amount of time on it.#2018-10-2505:09steveb8nI’m trying to use a private bitbucket repo as a git dep but getting “auth fail” from the jgit layer. I’ve tested ssh access from the terminal and that’s working. anyone else had any luck with this?#2018-10-2505:11seancorfieldI think that is only partially supported at he moment...?#2018-10-2505:11seancorfield(I'm trying to find the public discussions about it)#2018-10-2505:12seancorfieldHmm, OK, looks like ssh keypair should work for git deps...#2018-10-2505:13seancorfieldSee Procurers > Git Configuration here https://www.clojure.org/reference/deps_and_cli#_procurers#2018-10-2505:13seancorfieldIn particular "Note: user/password authentication is not supported for any protocol."#2018-10-2505:14seancorfieldHow do you have your repo configured in deps.edn @steveb8n?#2018-10-2505:15steveb8nthx for quick response. I’m using an SSH key with ssh-agent on OSX i.e. not using un/pw#2018-10-2505:15steveb8nin deps.edn I follow the pattern I’m using for some public repos i.e. same keys#2018-10-2505:15steveb8nonly difference is :git/url is ssh: instead of https:#2018-10-2505:16steveb8nwhen I run ssh -v in CLI, that shows successful connection#2018-10-2505:19steveb8nhmm, it looks like I am missing a “registered identity”#2018-10-2505:19steveb8nI’ll dig deeper into that#2018-10-2505:19steveb8nand will report back#2018-10-2505:30steveb8nI found the problem. I am using the OSX ssh-agent with the OSX keychain and it’s not persistent between reboots#2018-10-2505:30steveb8nssh-add -K fixed it#2018-10-2505:30steveb8nthanks for pointing me in the right direction#2018-10-2505:33seancorfieldCool! The whole ssh keys/identity thing can be a bear...#2018-10-2505:44steveb8nyep. in this case it’s a different behaviour in zsh CLI vs jgit (part of tools.deps). I’ve written it down in my readme for next time 🙂#2018-10-2510:47steveb8n@seancorfield a heads up for future. when using git private deps in CI, you must invoke the ssh-agent before invoking clojure i.e. eval clojure -A:dev only gives me this:#2018-10-2508:14Christian Johansenclj -A:dev
2018-10-25 10:06:47.433:INFO::main: Logging initialized @10380ms
Exception in thread "main" java.lang.ExceptionInInitializerError
at clojure.main.<clinit>(main.java:20)
#2018-10-2508:14Christian Johansencan this be improved somehow?#2018-10-2508:14Christian Johansena stack trace, anything to point me towards the problem?#2018-10-2508:19dominicm@christian767 my immediate thought is to try Clojure 1.10 alphas. As there's been some big changes around this stuff. #2018-10-2508:20Christian Johansenthis is with 1.10.0-alpha8#2018-10-2508:20Christian JohansenI’ll try an even newer one#2018-10-2508:30Christian Johansenwith clojure-tools-1.9.0.397 and Clojure 1.10.0-beta4:#2018-10-2508:31Christian Johansenclj -A:dev
2018-10-25 10:30:49.453:INFO::main: Logging initialized @11217ms
Exception in thread "main" java.lang.ExceptionInInitializerError
at clojure.main.<clinit>(main.java:20)
#2018-10-2508:31Christian Johansen😢#2018-10-2508:32seancorfieldSo that's failing before it even gets Clojure up and running...#2018-10-2508:32seancorfieldYou'll need to share some of your deps.edn (possibly both your ~/.clojure version and your project one).#2018-10-2508:33seancorfieldDoes it start a REPL without -A:dev?#2018-10-2508:33seancorfieldIf so, your :dev alias is the problem.#2018-10-2508:33Christian Johansengood tip#2018-10-2508:33Christian Johansensure does#2018-10-2508:33seancorfieldOK, so what's in your :dev alias?#2018-10-2508:33Christian Johansenfound the culprit#2018-10-2508:34Christian JohansenI have dev/user.clj, and in it I’m still using a dependency I ditched yesterday#2018-10-2508:34Christian Johansen[yonatane.timbre-json :as timbre-json]#2018-10-2508:35Christian Johansenis there something I could’ve done to have this reported in an understandable way?#2018-10-2508:35Christian Johansen(there was an :extra-paths in :dev)#2018-10-2508:35seancorfieldMaybe wrap the contents of your user.clj file in (try ... (catch Throwable t (println "dev/user.clj broke" t)))#2018-10-2508:36Christian Johansencan you try around the ns form?#2018-10-2508:36Christian Johansenanyway, what if the error was in my source? can’t try-catch all my code#2018-10-2508:36seancorfieldSure. ns is just code.{:tag :div, :attrs {:class "message-reaction", :title "eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👀")} " 4")}
#2018-10-2508:37Christian Johansenseems strange to have to try-catch all my code to see compilation failures#2018-10-2508:37seancorfieldWell, true, if your user.clj won't actually compile that's just going to break. user.clj is pretty special so you really should be very sparing about what you do in it.#2018-10-2508:37seancorfieldYou could always debug it with clj dev/user.clj I think?#2018-10-2508:38Christian Johanseninteresting#2018-10-2508:38seancorfield(without the -A:dev alias, it wouldn't be on your classpath so it wouldn't automatically run at Clojure startup)#2018-10-2508:38Christian Johansenmy impression was that I always got this behavior with any compilation errors, but you’re saying this is happening because it’s in user?#2018-10-2508:39seancorfieldRight, because it's executed as part of starting Clojure.#2018-10-2508:39Christian JohansenI see#2018-10-2508:39Christian Johansenby extension, this would happen with anything required from user too then?#2018-10-2508:40seancorfieldWell, wrapping the ns form in try/`catch` should deal with that.#2018-10-2508:40Christian Johansenright#2018-10-2508:40Christian Johansenleiningen manages to report this more clearly, I wonder why that is#2018-10-2508:40seancorfieldBecause Leiningen starts two JVMs perhaps?#2018-10-2508:41Christian Johansenaha#2018-10-2508:41seancorfieldOK, it's crazy late hear. I'm off to bed (nearly 2am).#2018-10-2508:41Christian Johansenwhoa, It’s 10 in the morning over here 🙂#2018-10-2508:41Christian Johansenthanks for your help#2018-10-2508:41seancorfieldHopefully other folks will be coming online soon if you have other Qs.#2018-10-2508:42Christian Johansen👍#2018-10-2509:16Christian JohansenI reported this as an enhancement: https://dev.clojure.org/jira/browse/TDEPS-102#2018-10-2509:16Christian Johansenit’s quite the buzzkill for new tools.deps users#2018-10-2510:40Christian Johansennot putting code in user.clj seems like the better approach for now. is there a tools.deps analog to leiningens :repl-options {:init-ns my-app.dev}?#2018-10-2510:43dominicmIn edge we solve this by having a dev.clj#2018-10-2510:50Christian Johansenhow do you load it?#2018-10-2512:24Alex Miller (Clojure team)I suspect this is really a Clojure problem, not a tools.deps problem#2018-10-2512:26Christian Johansenyes, after discussing with more people I realize it probably is#2018-10-2512:26Christian Johansensorry for misplacing the Jira issue…#2018-10-2512:46Alex Miller (Clojure team)No worries, it can be moved#2018-10-2519:29arrdemthanks for the depstar fix @seancorfield - it works like a charm!#2018-10-2519:32seancorfieldI plan to actively maintain my fork, going forward, because we're relying heavily on it at work.#2018-10-2519:33seancorfieldI've also started using it for packaging library JARs for deployment to Clojars now (there's an hf.depstar.jar entry point for thin JARs now, then clj -Spom -- and one-off updates to add/update license, SCM, etc -- and then mvn deploy:deploy-file).#2018-10-2520:33arrdemCool. I’ll probably be looking long and hard at depstar vs pack.alpha shortly just because I need to start generating manifests from Katamari, but if I stick with depstar I’d be happy to support your copy if its intent is to be active.#2018-10-2520:46ghadiMy experiment worked!#2018-10-2520:47ghadiThe experiment was: Git deps allow you to be a shitty maintainer because someone can just fork and change the url, without having to deploy an artifact#2018-10-2520:48ghadiI learned this by making a simple PR to cognitect/test-runner that was ignored for months:
https://github.com/cognitect-labs/test-runner/pull/16
but I had no problem because I forked it internally and pointed to our un-busted fork#2018-10-2522:52zaneI've been waiting for that PR to be merged. 😞#2018-10-2520:49ghadi@seancorfield didn't have to do anything to unblock himself on depstar, which I neglected to maintain#2018-10-2520:50arrdem:thinking_face: maybe {:mvn/version } could let you pass it an override lib to get the same place without refactoring your deps lists …#2018-10-2605:30dominicmThere's a ticket for this#2018-10-2521:03seancorfield:rolling_on_the_floor_laughing: @ghadi Although I have now deployed artifacts of seancorfield/depstar -- using the modified version of depstar! 🙂#2018-10-2521:04ghadiif you break the API in a significant way I'll kindly ask you to rename#2018-10-2521:04ghadibut I'm happy that you're happy sean!#2018-10-2521:05ghadiIt's a really good name if I do say so -- it's tar for deps, and a deathstar{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-10-2521:07seancorfieldI like the simplicity of it. So far, I've only fixed bugs in the existing code and added a new namespace/entry point (for thin JARs). No API changes 🙂#2018-10-2522:52steveb8nwhen I use a :local/root dep, I’m finding that all my aliases are unavailable. workaround is to add copies of all aliases (with empty maps) into the dependencies deps.edn. Is this known?#2018-10-2523:22arrdem@steveb8n AFAIK this is intended behavior - clj/`clojure` do not yet support merging “shared” extra deps files eg. for aliases besides the install global file and the user config.#2018-10-2523:23arrdemI think seancorfield has done some hacking to be able to share a defaults file within a monorepo as have I.#2018-10-2523:25steveb8nhmm, in that case 1/ it would be good to doc this in the :local/root docs and 2/ does the workaround I discovered make sense?#2018-10-2523:26steveb8nI’m good to go with the workaround but just asking for others who will inevitably hit this#2018-10-2523:30steveb8nthat’s my version of “asking for a friend”#2018-10-2523:42seancorfieldIt doesn't make sense for aliases in a deps project you depend on to bleed into your project's deps.#2018-10-2523:43seancorfield:local/root is nothing special -- just a want to specify a dependency. You wouldn't want all the aliases from a random :git/url deps file to bleed into your project, would you?#2018-10-2523:44seancorfieldThe aliases available to you should always be: system deps + user deps (which can be anywhere based on CLJ_CONFIG env var) + project deps.#2018-10-2523:58arrdemI’ve been kicking around whether it makes sense to want to build deps with aliases - eg what if I wanted to let every one of my dependencies apply its :test alias so that I get a single classpath with all the machinery required to run all my deps’ tests.#2018-10-2600:00arrdemKinda gives you a way to have a parameterized debug build or such.#2018-10-2600:04steveb8ninteresting idea. I guess we’ll see some best practices emerge around primary and dependent aliases#2018-10-2607:09Oliver GeorgeThis is probably a silly question. Should it be possible to clone clojure locally and use it via deps.edn via :local/root?#2018-10-2607:24dominicmNope! It has Java in#2018-10-2607:26Oliver GeorgeThanks. Not really surprised. I did get it working by building first (e.g. ant install) and pointing at the jar with {:deps {org.clojure/clojure {:local/root "/Users/olivergeorge/repos/fork-clj/clojure/target/clojure-1.10.0-master-SNAPSHOT.jar"}}}{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-2607:26Oliver GeorgeWas wondering more broadly how easy it was getting to tweak clojure source and see it in a REPL.#2018-10-2612:40Alex Miller (Clojure team)I do something similar when working on Clojure#2018-10-2621:49Oliver GeorgeCool. Pretty productive given the clojure build process is all of 10s on a 4yo macbook#2018-10-2706:24dominicmhttps://maven.apache.org/shared/maven-artifact-transfer/apidocs/index.html if someone wanted to write a tools.release, the maven library exists to do that 🙂#2018-10-2706:44dominicmI've got a complete first pass of "ednup" a tool for rewriting edn without losing whitespace. It's not great, but it is definitely sufficient to handle resolve tags! Don't expect this to work beyond this commit, I'm going to clean up with new namespaces, but:
{:paths ["src"]
;; cool comment
:deps
{potemkin {:mvn/version "0.1"}
tools.deps.alpha {:git/url ""
;; totally rad
:tag "tools.deps.alpha-0.5.460"}
;; woah dude
tools.deps.alphax {:git/url ""
:tag "tools.deps.alpha-0.5.460"
;; zoom
:sha nil}}}
Running
~/src/github.com/SevereOverfl0w/clj-resolve-tags master* 13s
❯ clj -Sdeps '{:deps {resolve-tags {:git/url "" :sha "44315f501e3b7fba40961563ba7fcfd0441225fa"}}}' -m resolve-tags not-deps.edn
Results in
{:paths ["src"]
;; cool comment
:deps
{potemkin {:mvn/version "0.1"}
tools.deps.alpha {:git/url ""
;; totally rad
:tag "tools.deps.alpha-0.5.460"
:sha "ca2aa1eb320687ce940a4f142c8d1af8f9a018bc"}
;; woah dude
tools.deps.alphax {:git/url ""
:tag "tools.deps.alpha-0.5.460"
;; zoom
:sha "ca2aa1eb320687ce940a4f142c8d1af8f9a018bc"}}}
#2018-10-2719:12andy.fingerhutIs there already an issue suggesting the addition of a "show version of clj/clojure and exit" option to clj and/or clojure? If not, would one be welcome? Seems useful in some debug interactions between people.#2018-10-2719:13dominicmIt does I think? -Sdescribe#2018-10-2719:16andy.fingerhutGot it. thx. Didn't realize that, and didn't see the more traditional -v or --version command line options.#2018-10-2719:17dominicmHave those been added? Neat. I haven't tracked really closely. #2018-10-2719:17andy.fingerhutI don't think so. I didn't see them.#2018-10-2719:22dominicmAh. I will stick to describe them 😊 #2018-10-2723:43jeff.terrellI've also been surprised by the lack of -v / --version options, FWIW.#2018-10-2805:10richiardiandreaI can trigger a (swap! verbose-surprise-count inc) as well :smiling_face_with_smiling_eyes_and_hand_covering_mouth: {:tag :div, :attrs {:class "message-reaction", :title "white_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✅")} " 4")}
#2018-10-2904:00steveb8nWhen using :git/url deps, is there any way to emulate latest or snapshot style behaviour? I'm having to manually updates shas a lot.#2018-10-2906:36dominicmI don't think so - as an intentional decision.#2018-10-2906:37dominicmYou could write a script to update them automatically though.#2018-10-2908:13steveb8nInteresting idea. It seems like I won't be the only one wanting this. I wonder if there's another best/good practice to emulate maven in this way#2018-10-2910:13dominicmThe code is about 100 lines. You could modify the resolve-tags library I write quite easily. {:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-10-2910:16dominicmmodify the resolve-git-dep function with a hardcoded branch, basically https://github.com/SevereOverfl0w/clj-resolve-tags/blob/master/src/io/dominic/deps/resolve_tags.clj#2018-10-2910:33steveb8nHmm, maybe I should be writing a new “procurer”. I’ll look into that as a solution. thanks for the nudge in a new direction#2018-10-2906:38dominicmhttps://github.com/clojure/tools.gitlibs#revs supports branch names (e.g. master)#2018-10-2912:08Alex Miller (Clojure team)It does, but tools.deps does not (gitlibs is intentionally broader and more generic)#2018-10-2909:36arrdemIf you're coordinating changes between many repos all the time I think it's worth considering whether the apparent decoupling you have at repo boundaries maps at all to stable designed API boundaries or whether you'd be better off admitting that you don't yet have decoupling and would be better served putting everything in one place for now.#2018-10-2910:31steveb8nGood point. In my case it’s not de-coupling I want, but shared specs. The specs are changing a lot currently, hence the need for regular updates#2018-10-2909:38arrdemIn slightly related news I got a Katamari v0.0.5 out the door which successfully does content addressing of build products (jars, uberjars, javac'd classfiles). At this point it's fully self-bootstrapped and stable as such. Incremental rebuilds were the big thing I wanted to get working before this week's meeup - if I can get repl and test tasks sorted out I'll be happy enough to really try and push this thing.
https://github.com/arrdem/katamari/releases/tag/v0.0.5 for your amusement.#2018-10-2910:58Christian JohansenI cleared out ~/.m2/repos/org/apache, and now I’m seeing this error with tools.deps:#2018-10-2910:58Christian JohansenException in thread "main" org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.httpcomponents:httpcore:jar:4.4.9
#2018-10-2910:59Christian Johansenafter googling a little, it seems to go away with this in deps.edn:#2018-10-2910:59Christian Johansen:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
#2018-10-2910:59Christian Johansenwhat’s going on here?#2018-10-2911:33Christian Johansenbtw, I can observe that the jar in question is both being downloaded by the Clojure process, and is available in my ~/.m2 after its run (and failed)#2018-10-2912:10Alex Miller (Clojure team)Those maven repos are already set in the install deps.edn so that shouldn’t make much difference#2018-10-2912:10Alex Miller (Clojure team)Can you post the entire deps.edn for repro?#2018-10-2913:18jerger_at_ddaI had just the time to watch "Dependency Heaven" an like the arguments.
I'm working quite a lot with git dependencies on systems like gitbook (make books out of git-repos containing md) or smeagol (a git-repo / MD based wiki)
But I find it quite hard to find a good abstraction for addressing git resources.
Are there any plans to support other git servers like gitlab (the SaaS and all the on prem instances) or gitblit, gittea et al?
What about private repos?
In context of privisioning git repos I use this schema: https://github.com/DomainDrivenArchitecture/dda-git-crate/blob/master/doc/reference_domain.md
The relevant parts are Repository and GitCredential. These informations will lead to completely different urls like
>){:tag :code, :attrs nil, :content nil}
deps.edn, asking them to add one.#2018-10-3016:41seancorfieldI've been trying to make sure I do that with every project I maintain.#2018-10-3016:43neural@seancorfield nice#2018-10-3016:43seancorfield(in fact, in the Contrib libraries I maintain, I've removed project.clj because it's misleading -- many people assume the version in that file, or the specific dependencies in that file, are definitive, whereas for Contrib projects, it's driven by pom.xml)#2018-10-3016:51neuralmaybe this is an other channel related question... but you guys know if clj-refactor/refactor-nrepl works with deps.edn???#2018-10-3017:07seancorfield@neural.works.com As far as I know, the most recent version of CIDER works with clj/`deps.edn` -- but maybe better to ask in #cider for more detail...?#2018-10-3017:08neural@seancorfield tks#2018-10-3019:43dominicmIt does. I set that up. #2018-10-3121:08timgilbertSay, anybody got something that works similarly to lein-try, where I can lein try clojure-lanterna and fire up a REPL with the latest version loaded?#2018-10-3121:11jeff.terrellRelative newbie here, but I think the -Sdeps option covers that use case, no?#2018-10-3121:12timgilbertI'm thinking in a bare directory with no existing deps.edn present#2018-10-3121:12ghadiyup, that's what it's for#2018-10-3121:12dominicmI would use find deps#2018-10-3121:13dominicmhttps://github.com/hagmonk/find-deps#2018-10-3121:15timgilbertThink that'll work, thanks @dominicm#2018-10-3121:46lilactownI use these bash aliases to try out new deps in a nice REPL: https://gist.github.com/Lokeh/a67bb2dc9295d7990d9f8525f478457c#2018-10-3121:50seancorfieldclj -Sdeps '{:deps {clojure-lanterna {:mvn/version "RELEASE"}}}' -- or you can use "LATEST" to look for snapshots instead.#2018-10-3121:50seancorfieldI use a command like that all the time when I just want to try out a library without any setup.#2018-10-3121:54seancorfieldThat looks like a fun library @timgilbert -- are you planning to build something public or is it just for fun?#2018-10-3123:02timgilbertJust for fun, was looking at some roguelike things from a HN post earlier today#2018-11-0101:42zaneHere's the post in case anyone else is interested: http://www.gamasutra.com/blogs/JoshGe/20181029/329512/How_to_Make_a_Roguelike.php#2018-11-0109:28borkdudeIs it possible to combine multiple R aliases into one alias?#2018-11-0109:42dominicmHow do you mean?#2018-11-0109:55borkdudeSay alias A says extra deps d1 d2, alias B says extra deps d3 d4, now I want to make an alias AB which combines A and B so it has extra deps d1 .. d4#2018-11-0109:57borkdudeBasically wondering if I could do something like https://github.com/borkdude/boot-bundle using deps.edn#2018-11-0110:07dominicm@borkdude there's no inheritance. So the only way to do it is by modifying the edn file. I've been writing https://github.com/SevereOverfl0w/ednup with the goal of it eventually supporting this.#2018-11-0116:50timgilbertSay, I'm trying to figure out how to effectively deal with the .cpcache directory in circleci. The way it works is that circleci can store a directory in a cache which is then available to subsequent steps (although the subsequent steps might themselves run on different VM instances).#2018-11-0116:50timgilbertSo what I currently do is run clojure -A:foo:bar -Spath to download a bunch of stuff and then cache the .m2 and .cpcache directories#2018-11-0116:51timgilbertMy question is, will tools.deps create a single .cpcache entry for the combined :foo:bar aliases, or will it create a :foo cache and a :bar cache? (Or all three?)#2018-11-0116:52timgilbertI'm wondering if it would be more efficient to run clojure -A:foo -Spath; clojure -A:bar -Spath#2018-11-0116:52Alex Miller (Clojure team)combined#2018-11-0116:52Alex Miller (Clojure team)the cache key is based on the set of aliases used#2018-11-0116:52timgilbertAha, makes sense. Thanks @alexmiller#2018-11-0116:53Alex Miller (Clojure team)https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/clojure#L230-L237#2018-11-0117:35fmjreyHey, I'm migrating some lein project to deps.edn and somehow it looks like my dev/user.clj isn't loaded in the repl, even though I use clj -R:dev and the :dev alias does have :extra-paths ["dev" "fixtures"]. My user.clj is very simple:
(:require
[com.stuartsierra.component.user-helpers :refer [dev go reset]]))
clj -Stree -R:dev shows the component lib is loaded.
Still it's as if user.clj isn't loaded:
$ clj -R:dev dev/user.clj
$
No output or error.
$ clj -R:dev
Clojure 1.9.0
user=> (dev)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: dev in this context, compiling:(NO_SOURCE_PATH:1:1)
user=>
What could I be missing?#2018-11-0117:44hiredmanuse -A not -R{:tag :div, :attrs {:class "message-reaction", :title "ok_hand"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👌")} " 4")}
#2018-11-0117:47fmjreyThanks, feel stupid now, wondering how I ended up using -R#2018-11-0117:49hiredmansimilar questions have come up a few times, it makes me wonder if there is a tutorial or documentation somewhere that is using -R in examples where it should be using -A#2018-11-0117:51fmjreyI think it must have been: https://clojure.org/guides/deps_and_cli#extra_deps#2018-11-0117:51fmjrey> You can add this dependency to your classpath by adding the :bench alias to modify the dependency resolution: clj -R:bench.#2018-11-0117:57fmjreyNot even sure I understand the difference between -A and -R from the cli doc#2018-11-0117:58fmjreyAhh this should explain it: https://clojure.org/reference/deps_and_cli#_aliases#2018-11-0117:51richiardiandreaquestion about CI and .m2 - is there anything I have to do to set Clojure CLI on a machine for being able to cache artifacts in .m2?#2018-11-0117:52richiardiandreathis machine does not seem to have Maven, do I need it? or .m2/settings.xml#2018-11-0117:52hiredmannope#2018-11-0117:53hiredmanif you are ci'ing inside some kind of container you might want to set things up so ~/.m2 is stored in a way that persists across different runs#2018-11-0118:42richiardiandreais .m2 created for me?#2018-11-0118:43hiredmanyes#2018-11-0118:43hiredmantools.deps will store git deps in ~/.gitlibs too#2018-11-0118:46richiardiandreaok thanks, my CI executes things in a weird way I think...every call to clojure resets .m2 so it downloads everything again, investigating...#2018-11-0118:48hiredmanyeah, you are likely building in some kind of container environment with nothing persisted between builds#2018-11-0118:48richiardiandreaI am on Azure Pipeline..#2018-11-0118:48richiardiandreayeah..#2018-11-0118:50hiredmanin theory that should be fine, m2 is a cache, so if the cache is missing it will just rebuild it, but it is much better to persist m2#2018-11-0118:50hiredmanyou actually might be running in to another caching issue too#2018-11-0118:50richiardiandreayeah I see it is redownloading the whole thing#2018-11-0118:51hiredmanthe clj tool fetches deps and builds the classpath once and sort of caches that, if the deps it fetches go missing, but the cache file it generates is still there, it will blow up#2018-11-0118:51richiardiandreamakes sense yeah#2018-11-0118:51richiardiandreathank you so much#2018-11-0119:45richiardiandreawhich command creates .m2? I know first execution does, but I would like to see it there before running my actual other tasks#2018-11-0119:45richiardiandreacan I just do clojure -Sdescribe?#2018-11-0119:45richiardiandreaactually let me try locally#2018-11-0119:46richiardiandreayep that works#2018-11-0119:46richiardiandrea$ mv .m2 .m2-backup
$ clojure -Sdescribe
Downloading: org/clojure/clojure/1.10.0-alpha6/clojure-1.10.0-alpha6.pom from
Downloading: org/clojure/spec.alpha/0.2.168/spec.alpha-0.2.168.pom from
Downloading: org/clojure/pom.contrib/0.2.2/pom.contrib-0.2.2.pom from
Downloading: org/clojure/core.specs.alpha/0.2.36/core.specs.alpha-0.2.36.pom from
Downloading: org/clojure/clojure/1.10.0-alpha6/clojure-1.10.0-alpha6.jar from
Downloading: org/clojure/spec.alpha/0.2.168/spec.alpha-0.2.168.jar from
Downloading: org/clojure/core.specs.alpha/0.2.36/core.specs.alpha-0.2.36.jar from
{:version "1.9.0.397"
:config-files ["/home/arichiardi/.local/lib/clojure/deps.edn" "/home/arichiardi/.clojure/deps.edn" ]
:install-dir "/home/arichiardi/.local/lib/clojure"
:config-dir "/home/arichiardi/.clojure"
:cache-dir "/home/arichiardi/.clojure/.cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
$ ll ~/.m2
total 24
drwxrwxr-x 3 arichiardi arichiardi 4096 Nov 1 12:45 ./
drwxr-xr-x 120 arichiardi arichiardi 16384 Nov 1 12:45 ../
drwxrwxr-x 3 arichiardi arichiardi 4096 Nov 1 12:45 repository/
#2018-11-0119:49richiardiandreait is downloading alpha6 by default#2018-11-0120:24Alex Miller (Clojure team)No, it’s not - that’s an artifact of your deps.edn files#2018-11-0120:27Alex Miller (Clojure team)By default, clj will use 1.9.0 based on the ~/.clojure/deps.edn#2018-11-0120:46richiardiandrearight#2018-11-0120:47richiardiandreaI should have use -Srepo maybe?#2018-11-0120:47richiardiandreaanother question I have is if I can have aliases in parent dir and consume in children dir, in a monorepo setup for instance#2018-11-0121:05richiardiandreaI tried with CLJ_CONFIG to the root but I don't see it working...#2018-11-0121:06richiardiandreaoh now I see it working cool#2018-11-0201:03richiardiandreaHow long does builds take for you folks in average? I am being told node builds take 5-10 minutes and I want to make sure I am not doing anything wrong before answering. At the moment a monorepo build takes 22 minutes and it downloads ~50 deps#2018-11-0204:06richiardiandreaThis is more of a #clojurescript question actually#2018-11-0207:42slipsetFirst of all, I know tools.deps/clj is not meant as a build tool. But still:
We now have a bunch of utils like pack, Eastwood, depot, testrunner, etc which are useful tools to use in a build pipeline.
I know I can add aliases to all of these in my home-directory’s deps.edn, but that doesn’t really work across multiple machines.
Having all the aliases duplicated across multiple projects doesn’t seem like a great solution either.
One solution I could think of is that deps.edn could refer to other deps.edns and use their aliases. Would that be possible?#2018-11-0209:19Christian Johansenwhat exactly is the problem in duplicating between projects? lack of standardization?#2018-11-0209:20Christian JohansenI have stuff like this:#2018-11-0209:20Christian Johansen:ancient {:extra-deps {olical/depot {:mvn/version "1.2.0"}}
:main-opts ["-m" "depot.outdated.main"]}
#2018-11-0209:21Christian Johansenyou want to make that implicit in your deps.edn?#2018-11-0209:39slipsetI was imagining that when I wanted to bump the version on olical/depot, I’d have to check out all my projects and bumpt it.#2018-11-0209:39slipsetAlso, some of the aliases are somewhat more involved than this.#2018-11-0209:41Christian Johansenyup. it’s a difficult trade-off. it’s tempting to standardize or “dedupe” some of this boilerplate, but that creates problems when things aren’t as equal anymore#2018-11-0209:42Christian Johansene.g. you loose the flexibility#2018-11-0209:43slipsetI’ve come to realise this. But that’s another discussion we could have over a cup of coffee IRL 🙂#2018-11-0209:43Christian Johansenif you have a lot of very similar projects I guess you could make your own dev package, and encode the aliases in it, and then use it like so:#2018-11-0209:43Christian Johansen{:aliases
:dev {:extra-deps {my.company/my-bundle {:mvn/version "1234"}}}
:test {:main-opts ["my.bundle.test"]}
:ancient {:main-opts ["my.bundle.ancient"]}
:build {:main-opts ["my.bundle.build"]}}
#2018-11-0209:44Christian Johansenyes, for sure 🙂#2018-11-0209:45Christian Johansenbut that approach does quickly leads to entangled framework world, use with caution#2018-11-0209:48slipsetI guess so.#2018-11-0209:50Christian JohansenI don’t have enough similar projects to warrant the effort, but this is an approach I would consider to contain the duplication and lightly enforce some standardization. you’d still have to bump the shared dependency in all projects, but at least you reduced n deps/aliases to 1#2018-11-0209:52Christian JohansenI’m sure there are other ways to approach this as well#2018-11-0210:02orestisOne thing I noticed really is that when most of those things are just standalone utilities, they shouldn’t even be in my main deps.edn file, because they don’t need all my “production” stuff to run.#2018-11-0212:34Alex Miller (Clojure team)Exactly - put them in your ~/.clojure/deps.edn#2018-11-0213:29orestisIs there then a way for the clj to ignore any local deps.edn?#2018-11-0213:30orestisMeaning, when the Classpath is created, it only has the stuff from ~/.clojure/deps.edn and nothing from a local deps.edn file.#2018-11-0213:48Alex Miller (Clojure team)no, why would you want that?#2018-11-0214:13orestisIn the context of a standalone library, e.g. clj-new, cljfmt etc etc that does “build tool” stuff — it has no need to bring in the application-level deps. I’m not sure what the ramifications of having more things in the classpath are though — apart from the potential version conflicts which I think are addressable right now.#2018-11-0210:03orestisI’m always thinking that perhaps it’s time for a new “dev-time” tool that provides a wrapper and some porcelain around those single-purpose libraries. Sure you lose the flexibility, but you also don’t have to make all those decisions when you are a newcomer to the ecosystem, plus the single-purpose libraries can still be used standalone and can evolve at their own pace.#2018-11-0210:09Christian Johansenwherever it is, all the deps and scripts required to build and deploy the app should reside in the repo at least#2018-11-0210:09dominicm@orestis would such a thing be put into each project? Or be part of the setup guide for ~/.clojure/deps.edn when you're new to a project?#2018-11-0210:11Christian JohansenI strongly believe that developing any app should require as few pre-installed tools as possible, and that the code/repo should be as self-contained as possible#2018-11-0212:27rickmoynihanI agree with this sentiment too. tools.deps is quite good here, as you can effectively ship tooling with a repo; and version lock the tools - without having to include binaries etc.#2018-11-0210:11orestisI think it could be its own standalone CLI, like lein or boot.#2018-11-0210:12orestisBut in the end all it would do is generate the correct clj/clojure invocation for you.#2018-11-0210:13Christian JohansenI’m currently using deps.edn + Makefile to get the job done. I see some things that will probably seem like tedious repetition at some point, but so far I’m loving the simplicity of it. no magic, you can see everything{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-11-0210:13orestisI see your point though @christian767 — though with tools.deps you need to have clj installed — otherwise, well, you have to fallback to java -cp ..., no?#2018-11-0210:14Christian Johansenyes, I think clj and java is a good baseline requirement for clojure projects at this point#2018-11-0210:14orestisRight, a more opinionated Makefile is what I’m trying to describe.#2018-11-0210:14Christian Johansensure#2018-11-0210:14orestisMy experience in trying to setup a new deps.edn based project was a few days trying out the various uberjar libraries, then a linter, then a formatter, then an outdated-deps-finder etc etc etc.#2018-11-0210:15orestisI think the community would benefit if this is done for you, while still maintaining the much-needed transparency and zero magic.{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-11-0210:15Christian JohansenI agree#2018-11-0210:16orestisI wonder how could this move forward? Ask people to upload their deps.edn and makefiles and shell scripts somewhere and compare approaches? I’m too much of a newb to do this based only on my assumptions, and different projects will have wildly different needs.#2018-11-0210:18dominicm@orestis JUXT are solving this with https://github.com/juxt/edge#2018-11-0210:19dominicmYou start a project by cloning edge, and then modifying it. Everything is set up for you. clj is all you need.#2018-11-0210:19dominicmTemplates also solve this problem.#2018-11-0210:20Christian Johansenbasically, scaffolding#2018-11-0210:21dominicmI wouldn't say scaffolding is a good comparison. It's more of a cookie cutter or starting point than anything.#2018-11-0210:21orestisThe issue with templates/scaffolding is that after the initial project creation, you are on your own.#2018-11-0210:21Christian Johansenmy issue with this approach is that it sets you off for an overwhelming start, and it is too easy to end up with a bunch of stuff laying around that you don’t really need#2018-11-0210:21dominicm@orestis that's why we suggest cloning. You can upgrade by doing git merge edge/master#2018-11-0210:22dominicmI've been making some adjustments to reduce the chances of conflicts to very low. It should only happen if you modify things (in which case, what do you expect?)#2018-11-0210:22dominicmThe documentation around this isn't great, but it's how we expect edge to be used, and how we use it internally.#2018-11-0210:24dominicmblugh, s/modify things/modify shared areas in a separate folder/#2018-11-0210:26orestisI see your point — and it might be hugely valuable for a consulting company, since you tend to start projects very often, right? I’m more after something that decouples some very common concerns for most people, while imposing no opinions on how your actual app is structured. Meaning, I just want to run a command to give me outdated dependencies, a command to run all my tests, a command to build my uberjar. I’d like newcomers to the community to have a single entry point, and if/when they want more power, be able to drop down to the layer below.#2018-11-0210:27orestisCrucially, they shouldn’t have to know the underlying libraries if they don’t care. Similarly to how you don’t really know how lein builds uberjars.#2018-11-0210:27dominicmI know 🙈 You don't want to#2018-11-0210:27orestisAnd also crucially though, the uberjar building library is standalone and people can use it as they see fit.#2018-11-0210:27Christian Johansenkoacha does this for tests#2018-11-0210:28Christian Johansenyou could create a binstub on your box, run koacha in your app, and it’ll “just work”#2018-11-0210:28dominicmWe start, and maintain, a lot of projects, yep 🙂.
edge is slightly more opinionated than that, it assumes integrant for example. But I think the implementation would work for this? Use a git repo, and merge periodically to have the uberjar builder updated by maintainers upstream.#2018-11-0210:29orestisExactly @christian767, kaocha is very good at that, it has its own CLI - like thing, but you can also use its API if you want to.#2018-11-0210:30Christian Johansenyup#2018-11-0210:31dominicmBinstubs are only part of the puzzle though 🙂
What would provide:
> I think the community would benefit if this is done for you, while still maintaining the much-needed transparency and zero magic.
?#2018-11-0210:31orestisSo what I’m after is — a cli tool that you could install, then run XXX test and it would do the right thing, using kaocha under the hood. Kaocha then can keep evolving and people can use it standalone without the XXX tool.#2018-11-0210:32orestisYou wouldn’t know that kaocha is used unless you really want to — kaocha is a very good example since it plays well with clojure.test. Uberjars can be more fiddly to configure, I guess?#2018-11-0210:33slipsetThis is sort of the discussion I was looking for#2018-11-0210:33slipsetBut in one way, I guess we’re discussing yet another lein or boot#2018-11-0210:33dominicmjars need a lot of configuration in my experience 🙂 Clients come up with all sorts of novel environments in which we have to deploy.#2018-11-0210:33dominicm@slipset I wouldn't like to see another lein or boot fwiw.#2018-11-0210:34slipsetI know, but the ideas I’m sort of looking for very quickly lead in that direction, unfortunatly#2018-11-0210:35slipsetEven if it’s only @christian767 Makefile or Juxt’s edge.#2018-11-0210:35slipsetI could imagine ending up in:#2018-11-0210:36slipset$ make -f /usr/local/etc/default-makefile-for-app setup#2018-11-0210:36slipsetwhich is basically lein new app or clj -A:clj-new app ...#2018-11-0210:36slipsetor some sort.#2018-11-0210:38orestisI’m not 100% sure if this only an “ergonomics” thing — or if some plumbing is still missing from tools.deps.#2018-11-0210:39dominicmGiven Stu's advocacy for CLJ-2373 (I've memorized it he's said it so much!), let's take a leaf out of the Clojure/core approach to solving problems.
What is the problem statement here?#2018-11-0210:40dominicmIf we were only trying to solve the assembly problem, then templates are fine. So there's another element too I'm guessing?#2018-11-0210:46dominicmI kinda get the impression we're each looking for different additional properties. I've seen mention of "project-local isolation", "baseline as clj", and "not caring about underlying library".#2018-11-0211:02dominicmfwiw, I care more about "baseline as clj" because it's the minimum I can expect anyone to have done, ever. If we're doing training as a company, getting binstubs installed, covering all the distros is more hassle than it's worth for me.
I would probably find it acceptable if a project had some kind of alias like clj -A:tool-name which would then do the rest for itself. add-lib is pretty compelling here, as the tool can dynamically fetch it's dependencies as needed.#2018-11-0306:02seancorfield@alexmiller merged master back to the add-lib branch and we just added a :deps alias at work to that SHA so we can use add-lib in our REPLs. It's nice. I need update my dot-clojure file/repo with that too.#2018-11-0212:13orestisNice breakdown. Fully agree.#2018-11-0212:24orestisPossibly relevant: https://www.reddit.com/r/Clojure/comments/9tj6d1/katamari_a_clojure_build_tool_a_talk_by_reid/#2018-11-0216:00seancorfieldI uploaded my https://github.com/seancorfield/dot-clojure as a guide for others some time ago -- I think sharing these is useful while the tool is still new. #2018-11-0216:04seancorfieldAs for a self-contained project: yes, very important -- it's why we have a shell wrapper that sets CLJ_CONFIG (to point to a repo-specific "dot Clojure" folder) and we have a deps.edn file for each subpar object in the repo. #2018-11-0216:26richiardiandreaFunny I was doing the same yesterday but it seemed like a hack ^#2018-11-0216:27andy.fingerhutBut today, it no longer seems like you are hacking. Today, you are corfielding 🙂#2018-11-0216:28richiardiandreaIt seems like everybody is going to corfield their deps.edn sooner or later 😄#2018-11-0216:29andy.fingerhutSean, if this goes seriously weird, I sincerely apologize{:tag :div, :attrs {:class "message-reaction", :title "laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😆")} " 4")}
#2018-11-0216:44andy.fingerhutIf it only goes slightly weird, then .... you are welcome? 🙂#2018-11-0218:07seancorfieldWeird is good. I like weird. :rolling_on_the_floor_laughing:{:tag :div, :attrs {:class "message-reaction", :title "zany_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🤪")} " 8")}
#2018-11-0218:07seancorfieldWeird is good. I like weird. :rolling_on_the_floor_laughing:{:tag :div, :attrs {:class "message-reaction", :title "zany_face"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🤪")} " 8")}
#2018-11-0218:10seancorfieldBack to the CLJ_CONFIG stuff (I was on my phone earlier)... we've found the ability to specify a "home" deps.edn via CLJ_CONFIG is great! It means a project can have a "home" configuration for aliases that provide project-specific tooling as well as :override-deps (which we rely on heavily to "pin" versions across subprojects within the overall project.#2018-11-0218:13seancorfieldI talked a bit our shell wrapper a week or two back I think. Mostly it takes commands of the form build alias(es) subproject and it goes into that subproject and runs CLJ_CONFIG=../versions clojure -A:defaults:aliases (edited to add our #2018-11-0218:19Alex Miller (Clojure team)Is that only for aliases?#2018-11-0218:19seancorfieldIt does a few mini-DSL-y sort of things around plural aliases or aliases with certain prefix/suffix strings as well as expanding some internal "aliases" but it's only a few tens of lines of shell script.#2018-11-0218:20seancorfield@alexmiller We have the "pinned" version of Clojure specified as a regular :deps in that versions/deps.edn file but everything else is just aliases I think.#2018-11-0218:23seancorfieldOh, and it adds our Archiva Maven repo as well.#2018-11-0218:27seancorfieldIt's definitely shown the need for all deps-based tooling to support correct deps.edn merging with support for :override-deps and aliases, if it doesn't just execute in the context of your project's classpath (Depot, for example, already supports aliases but needed a PR to support :override-deps merging).#2018-11-0218:27seancorfieldIt would definitely simplify usage if :override-deps could be top-level and not need an alias.#2018-11-0219:06dominicm@seancorfield fwiw, I don't like the CLJ_CONFIG solution because it takes away the ability to have aliases (e.g. A custom repl) from a user.
I understand your choice and think it is pragmatic though. #2018-11-0219:40seancorfieldYeah, it's a compromise. We have REPL-related aliases in that deps file, as well as other tooling.#2018-11-0219:41seancorfieldWhat I'd really like would be the ability to specify an extra deps.edn file easily between user-deps and project-deps.#2018-11-0223:36seancorfield@alexmiller Is this something you'd entertain as an enhancement? Essentially to support something like <system>/deps.edn, <user>/deps.edn, <tooling>/deps.edn, (project) deps.edn.#2018-11-0223:37seancorfieldI wasn't sure if you just missed this in the flow of questions in the channel since you didn't comment -- and I sort of expected you to 🙂#2018-11-0300:04Alex Miller (Clojure team)Didn’t see it#2018-11-0300:05Alex Miller (Clojure team)Don’t know - have been hoping to move towards fewer deps.edn files, not more :)#2018-11-0300:08seancorfieldFair enough. The underlying t.d.a. lib supports an arbitrary number of deps.edn files tho'... and it would be nice and generic to surface that in clj/`clojure` 🙂#2018-11-0300:09seancorfieldYou can easily merge extra stuff at the end with -Sdeps but you can't insert anything before the project deps.edn which is where I find the need -- and I think I'm not alone.#2018-11-0300:10seancorfieldWe can get away with "replacing" the user-deps via CLJ_CONFIG for our needs but it does mean some duplication and inconvenience.#2018-11-0300:28Alex Miller (Clojure team)Other things are pointing away from having an arbitrary number of deps so that’s much more likely to die#2018-11-0300:28Alex Miller (Clojure team)But I asked because I’m interested in the use cases#2018-11-0300:28seancorfieldInteresting. I wonder how we'll make things work if that goes away. It's kind of fundamental to our workflow right now.#2018-11-0300:32seancorfieldI guess we'd need to maintain a variant of the clojure script that allowed us to specify an extra deps file around these lines # Chain deps.edn in config paths. repro=skip config dir
if "$repro"; then
config_paths=("$install_dir/deps.edn" "deps.edn")
else
config_paths=("$install_dir/deps.edn" "$config_dir/deps.edn" "deps.edn")
fi
config_str=$(printf ",%s" "${config_paths[@]}")
config_str=${config_str:1}
#2018-11-0300:34seancorfield(which part are you saying might "die"? support for CLJ_CONFIG or support for the --config-files command line argument in t.d.a.?)#2018-11-0219:41seancorfieldSo we'd have <system>/deps.edn, <user>/deps.edn, <tooling>/deps.edn, (project) deps.edn.#2018-11-0219:42timgilbertSay, I'm getting this error trying to mvn install a tools.deps.alpha repo I've got checked out locally, anyone know if there is some special stuff I need to do in order to build it?
[INFO] --- clojure-maven-plugin:1.7.1:test (clojure-test) @ tools.deps.alpha ---
Error: Could not find or load main class clojure.main
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
#2018-11-0219:46Alex Miller (Clojure team)Can you provide more info? Is there a pom? Did you gen it or make it? What do you mean by a “tools.deps.alpha repo”?#2018-11-0219:47timgilbertSure. I just cloned https://github.com/clojure/tools.deps.alpha and ran mvn install. There is a pom, which I've left unchanged from the repo. I couldn't find any docs about a build process, but I'm sure I could have missed something.#2018-11-0219:49timgilbert(I get the same error running mvn test, which makes sense)#2018-11-0219:50timgilbertHere is the full output:
% mvn install
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building tools.deps.alpha 0.5.461-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- build-helper-maven-plugin:1.12:add-source (add-clojure-source-dirs) @ tools.deps.alpha ---
[INFO] Source directory: /mnt/c/Users/tim/Dropbox/src/cljs/tools.deps.alpha/src/main/clojure added.
[INFO]
[INFO] --- build-helper-maven-plugin:1.12:add-resource (add-clojure-source-dirs) @ tools.deps.alpha ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.12:add-test-source (add-clojure-test-source-dirs) @ tools.deps.alpha ---
[INFO] Test Source directory: /mnt/c/Users/tim/Dropbox/src/cljs/tools.deps.alpha/src/test/clojure added.
[INFO]
[INFO] --- build-helper-maven-plugin:1.12:add-test-resource (add-clojure-test-source-dirs) @ tools.deps.alpha ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ tools.deps.alpha ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /mnt/c/Users/tim/Dropbox/src/cljs/tools.deps.alpha/src/main/resources
[INFO] Copying 18 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ tools.deps.alpha ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ tools.deps.alpha ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /mnt/c/Users/tim/Dropbox/src/cljs/tools.deps.alpha/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.0:testCompile (default-testCompile) @ tools.deps.alpha ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ tools.deps.alpha ---
[INFO]
[INFO] --- clojure-maven-plugin:1.7.1:test (clojure-test) @ tools.deps.alpha ---
Error: Could not find or load main class clojure.main
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.073 s
[INFO] Finished at: 2018-11-02T19:48:28+00:00
[INFO] Final Memory: 19M/295M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.theoryinpractise:clojure-maven-plugin:1.7.1:test (clojure-test) on project tools.deps.alpha: Clojure failed. -> [Help 1]
#2018-11-0219:51timgilbertSorry, I could put this all in a ticket too. Just thought I might be missing something obvious.#2018-11-0219:52Alex Miller (Clojure team)Ok, so you actually mean tda#2018-11-0219:52Alex Miller (Clojure team)No magic here, it is just a maven build and I run mvn test on it all the time#2018-11-0219:52Alex Miller (Clojure team)What java version?#2018-11-0219:53Alex Miller (Clojure team)Prob needs 8+#2018-11-0219:57timgilbert% java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
% mvn --version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-17134-microsoft", arch: "amd64", family: "unix"
#2018-11-0219:57timgilbertThis is running under ubuntu in the "windows subsystem for linux" which is a bit of a weird environment but all the usual clojure stuff works fine#2018-11-0219:57Alex Miller (Clojure team)I’m using Java 11 and Maven 3.5.4 at the moment, but I don’t have any reason to believe your versions are not sufficient#2018-11-0219:58Alex Miller (Clojure team)the build box is certainly using much older stuff to build and deploy that lib#2018-11-0219:58Alex Miller (Clojure team)works on my machine :)#2018-11-0219:58Alex Miller (Clojure team)and the build box :)#2018-11-0219:58Alex Miller (Clojure team)try throwing -X -e on your mvn command#2018-11-0220:00timgilbertI was wondering if I might need to build clojure locally first or something#2018-11-0220:00Alex Miller (Clojure team)nope#2018-11-0220:01Alex Miller (Clojure team)it’s just using the released clojure 1.9.0#2018-11-0220:01timgilbertHere's the exception with -X -e:
[ERROR] Failed to execute goal com.theoryinpractise:clojure-maven-plugin:1.7.1:test (clojure-test) on project tools.deps.alpha: Clojure failed. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.theoryinpractise:clojure-maven-plugin:1.7.1:test (clojure-test) on project tools.deps.alpha: Clojure failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Clojure failed.
at com.theoryinpractise.clojure.AbstractClojureCompilerMojo.callClojureWith(AbstractClojureCompilerMojo.java:464)
at com.theoryinpractise.clojure.AbstractClojureCompilerMojo.callClojureWith(AbstractClojureCompilerMojo.java:366)
at com.theoryinpractise.clojure.ClojureRunTestWithJUnitMojo.execute(ClojureRunTestWithJUnitMojo.java:138)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
#2018-11-0220:02Alex Miller (Clojure team)It’s failing in the test step, which is kinda weird. You could use -Dmaven.test.skip=true if you wanted to just move past that#2018-11-0220:03timgilbertAha! That did it.#2018-11-0220:03Alex Miller (Clojure team)it’s failing to run Clojure to run the tests. not sure why that would be#2018-11-0220:04Alex Miller (Clojure team)I’ve seen a lot of maven build failures, but I haven’t seen this one :)#2018-11-0220:04timgilbertMy next question was going to be how to run the tests 😉#2018-11-0220:04timgilbertIt's possible that some local modifications I made messed something up, let me try on a clean master#2018-11-0220:05Alex Miller (Clojure team)here’s the build box running it: https://build.clojure.org/job/tools.deps.alpha/153/console (Java 8 / Maven 3.2.5)#2018-11-0220:09timgilbertHmm, still getting the same thing on a clean master. I'll see if I can repo at home on my mac and file a ticket if so. Thanks for looking.#2018-11-0220:17seancorfieldIt works on my Mac. I haven't tried to build/install any Clojure Contrib projects on WSL on my Windows 10 laptop. I'll try to remember to test that later today.#2018-11-0220:55timgilbertThe root of the problem seems to be that I'm getting the clojure dependency with scope provided.
% mvn dependency:tree | grep org.clojure
[INFO] org.clojure:tools.deps.alpha:jar:0.5.461-SNAPSHOT
[INFO] +- org.clojure:clojure:jar:1.9.0:provided
[INFO] | +- org.clojure:spec.alpha:jar:0.1.143:provided
[INFO] | \- org.clojure:core.specs.alpha:jar:0.1.24:provided
[INFO] +- org.clojure:data.xml:jar:0.2.0-alpha5:compile
[INFO] | \- org.clojure:data.codec:jar:0.1.0:compile
[INFO] +- org.clojure:tools.gitlibs:jar:0.2.64:compile
[INFO] \- org.clojure:tools.cli:jar:0.3.5:compile
#2018-11-0220:56timgilbertAnyways, I'll mess around a bit more and put a ticket up. I'm guessing this is something local to my machine.#2018-11-0220:56Alex Miller (Clojure team)interesting. I wonder why that’s different here vs there#2018-11-0221:12seancorfield@U08QZ7Y5S FYI, if you're relying on WSL, avoid Insider build 18272 -- they broke WSL in that build (they know about it, not sure when a fix will flight tho').#2018-11-0221:13timgilbertAfter the October fiasco, I'm avoiding Windows updates altogether for some time#2018-11-0221:13Alex Miller (Clojure team)October Fiasco is my new hardcore band btw{:tag :div, :attrs {:class "message-reaction", :title "the_horns"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 8")}
#2018-11-0221:20seancorfieldI'm on Fast Ring Insider builds so my WSL is bricked on both my Windows machines. Fortunately, I still have my Mac for dev work! 🙂#2018-11-0717:02timgilbertFWIW, filed https://dev.clojure.org/jira/browse/TDEPS-105 to track this mishegas. My home mac is sad so I haven't had time to test it there#2018-11-0717:20Alex Miller (Clojure team)@U08QZ7Y5S I believe you’re experiencing this, and I feel bad about that, but priority-wise has to fall into my “won’t fix” bucket, so I’m going to decline it. If you manage to find a fix, happy to see it re-opened and consider.#2018-11-0717:25timgilbertThat's totally fine with me, I mostly just wanted to get it documented in case somebody else was seeing it. I have to confess that I have no idea how mvn test ever works without a concrete clojure.jar in the dependency tree, though.#2018-11-0717:33Alex Miller (Clojure team)prob via the clojure maven plugin (which is in the classpath)#2018-11-0717:44timgilbertI thought of that, but it's not in there: https://github.com/talios/clojure-maven-plugin/blob/clojure-maven-plugin-1.7.1/pom.xml#2018-11-0219:44johnWhat are people using as an environ replacement? Does it only work with a project.clj?#2018-11-0219:47zanejuxt/edge is deps.edn-based and uses Aero: https://github.com/juxt/aero{:tag :div, :attrs {:class "message-reaction", :title "ok_hand"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👌")} " 8")}
#2018-11-0219:47seancorfieldWe have our own, internal, configuration library that can use env vars but mostly relies on EDN files.#2018-11-0223:05dominicmI'd enjoy evangelizing aero to you 🙂 Any big differences between aero and your system?#2018-11-0223:34seancorfieldI'd have to go look at Aero again. We did a review of pretty much everything available out there when we decided to "roll our own" a while back...#2018-11-0311:09yogidevbear(whispers "aero" into Sean's ear while he's sleeping){:tag :div, :attrs {:class "message-reaction", :title "rolling_on_the_floor_laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-11-0219:48seancorfieldWe've considered open sourcing it, but it does contain some proprietary stuff that we haven't had time (or inclination) to refactor out yet.#2018-11-0220:04kennyIf I have this in my :deps:
compute/integrations {:git/url "
And I run clj -Stree -Sforce, I get this:
compute/integrations
Why is commons-logging/commons-logging included?#2018-11-0220:07Alex Miller (Clojure team)dunno, that’s weird#2018-11-0220:07Alex Miller (Clojure team)feel free to file a ticket with a repro if you can slim it down#2018-11-0220:08Alex Miller (Clojure team)it being under a git dep might be part of the answer, not sure#2018-11-0220:08kennyWill see if I can get a repro.#2018-11-0220:51kennyFirst cut at the obvious repro case and I cannot reproduce. I made a git repo with a deps.edn that has clj-http as the dep. I made directory with a deps.edn on my computer that added that lib as a dep with the commons-logging exclusion and it behaved correctly. Is there something else I should be looking for in my current problem case that may shed some light here?#2018-11-0220:53kennyIf I add that exact dependency to my new project:
{:deps {compute/integrations {:git/url "
commons-logging is not in the tree, as expected.#2018-11-0220:55kenny#2018-11-0221:00kennyThis dep excludes httpclient, as expected.
compute/integrations {:git/url "#2018-11-0221:01kennyIs there a way to see all the deps that have commons-logging as a dep?#2018-11-0221:03kennyRemoving all other deps in my project besides Clojure and compute/integrations causes the exclusion to work correctly.#2018-11-0221:39kennyGot a repro. There is something seriously funky going on. Repro is here: https://github.com/kennyjwilli/clj-exclusions. cd into the app directory and run clj -Stree -Sforce. I get this:
org.clojure/clojure 1.9.0
org.clojure/core.specs.alpha 0.1.24
org.clojure/spec.alpha 0.1.143
integrations/integrations ..
clj-http/clj-http 3.9.1
commons-codec/commons-codec 1.11
org.apache.httpcomponents/httpasyncclient 4.1.3
org.apache.httpcomponents/httpcore-nio 4.4.6
slingshot/slingshot 0.12.2
commons-io/commons-io 2.6
org.apache.httpcomponents/httpcore 4.4.9
org.apache.httpcomponents/httpclient-cache 4.5.5
org.apache.httpcomponents/httpclient 4.5.5
commons-logging/commons-logging 1.2
potemkin/potemkin 0.4.5
clj-tuple/clj-tuple 0.2.2
riddley/riddley 0.1.12
org.apache.httpcomponents/httpmime 4.5.5
com.amazonaws/aws-java-sdk-cloudwatch 1.11.399
com.amazonaws/jmespath-java 1.11.399
com.fasterxml.jackson.core/jackson-databind 2.6.7.1
com.fasterxml.jackson.core/jackson-core 2.6.7
com.fasterxml.jackson.core/jackson-annotations 2.6.0
com.amazonaws/aws-java-sdk-core 1.11.399
joda-time/joda-time 2.8.1
com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.6.7
software.amazon.ion/ion-java 1.0.2
Now if I change the name of the dependency from integrations to lib, I get this output from clj -Stree -Sforce:
org.clojure/clojure 1.9.0
org.clojure/core.specs.alpha 0.1.24
org.clojure/spec.alpha 0.1.143
com.amazonaws/aws-java-sdk-cloudwatch 1.11.399
com.amazonaws/jmespath-java 1.11.399
com.fasterxml.jackson.core/jackson-databind 2.6.7.1
com.fasterxml.jackson.core/jackson-core 2.6.7
com.fasterxml.jackson.core/jackson-annotations 2.6.0
com.amazonaws/aws-java-sdk-core 1.11.399
joda-time/joda-time 2.8.1
com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.6.7
software.amazon.ion/ion-java 1.0.2
org.apache.httpcomponents/httpclient 4.5.5
commons-logging/commons-logging 1.2
lib/lib ..
clj-http/clj-http 3.9.1
commons-codec/commons-codec 1.11
org.apache.httpcomponents/httpasyncclient 4.1.3
org.apache.httpcomponents/httpcore-nio 4.4.6
slingshot/slingshot 0.12.2
commons-io/commons-io 2.6
org.apache.httpcomponents/httpcore 4.4.9
org.apache.httpcomponents/httpclient-cache 4.5.5
potemkin/potemkin 0.4.5
clj-tuple/clj-tuple 0.2.2
riddley/riddley 0.1.12
org.apache.httpcomponents/httpmime 4.5.5
It is different and all I did was change the name. What is going on here?#2018-11-0221:52Alex Miller (Clojure team)the ordering is somewhat dependent on traversal order as to which one is encountered first, but you shouldn’t be getting a dep that is excluded#2018-11-0221:53Alex Miller (Clojure team)there are a couple tickets out there that I’ve looked at and there are a couple issues to resolve - it’s likely this is the same as one of them#2018-11-0221:54kennyOk. Which tickets?#2018-11-0221:55seancorfieldThey're only different because org.apache.httpcomponents/httpclient is coming in via two dependencies, not just one.#2018-11-0221:56seancorfieldIn the integrations case, httpclient and logging come in via clj-http. In the lib case, httpclient and logging are coming in via aws cloudwatch (via aws core)#2018-11-0221:57kennyThe inconsistency is quite misleading though.#2018-11-0221:58Alex Miller (Clojure team)TDEPS-58 (and TDEPS-79)#2018-11-0221:59kennyYep that sounds like it.#2018-11-0221:58seancorfieldIf you have the exclusion on clj-http, it won't stop it being pulled in if the cloudwatch dep is walked first -- since it won't have seen the exclusion at that point.#2018-11-0221:59seancorfieldWe've had to be very aggressive with exclusions to stop up all the "holes" where certain things leak in...#2018-11-0221:59Alex Miller (Clojure team)there have been some requests for “global” exclusions#2018-11-0221:59kennyAggressive meaning exclusions on a lot of your deps?#2018-11-0222:00Alex Miller (Clojure team)I’m not necessarily opposed to that, but haven’t thought through all the implications#2018-11-0222:00kennyI think Boot has that.#2018-11-0222:00seancorfieldYes. We've had to add the same/similar exclusions on several deps across our subprojects in order to completely "hide" them.#2018-11-0222:01kennyWould global exclusions solve your problem?#2018-11-0222:01seancorfieldWe've used global exclusions with Boot. It's a bit of a sledgehammer. We used it to exclude Clojure and ClojureScript across everything.#2018-11-0222:03seancorfieldYes, global exclusions would "solve" the problem but it's really a huge hammer to crack a little nut. And the fall out is that you then need to ensure you bring in all those things explicitly in the right subprojects -- which is a pain.#2018-11-0222:03Alex Miller (Clojure team)seems like that would make it hard to use Clojure or ClojureScript :)#2018-11-0222:03seancorfieldYou can bring them in at the top-level. It just ensures nothing else brings them in.#2018-11-0222:05Alex Miller (Clojure team)so I’d say that’s global-except-at-top-level :)#2018-11-0222:06seancorfieldYeah, well, Boot is nothing if not "pragmatic" 🙂#2018-11-0222:07Alex Miller (Clojure team)I’m just saying, that’s a subtlety#2018-11-0222:03kennyDependency resolution wouldn't solve that for you?#2018-11-0222:06Alex Miller (Clojure team)with Maven dependency, no#2018-11-0222:06Alex Miller (Clojure team)Maven takes the first version it encounters#2018-11-0222:06seancorfieldTwo things: 1. we wanted ClojureScript excluded completely for all our backend projects, 2. we run "pedantic" style checks (or, at least, we did in lein and boot) so if potentially conflicting versions came in, we would get an exception -- and we didn't want to add those exclusions to nearly every Clojure dependency, since everything depends on different versions of Clojure out there 🙂#2018-11-0222:11kennyAnyway, thanks for the help. Excluding commons-logging from my other top level AWS deps fixed the issue. I voted and followed TDEPS-58.#2018-11-0300:24lwhortoni want to report a big-ol THANK YOU to the hard work behind tools-deps. i just started a greenfield clojure project and was able to create an nrepl “reloaded” workflow using only 1. deps.edn file 2. makefile for convenience 3. cat. no more boot, no more lein (for now, still have to look at generating jars).#2018-11-0300:24lwhortonis this what cleanliness feels like? have i found it?! troll#2018-11-0300:25Alex Miller (Clojure team)Living the dream#2018-11-0300:28seancorfieldI'll second that THANK YOU since we have essentially eliminated Boot from our primary Clojure workflow across all tiers (dev through QA through production) and we like the simplicity of it all.#2018-11-0310:28borkdude@lwhorton mind sharing your deps.edn? I’m interested in enabling nREPL for a project so I don’t have to use lein#2018-11-0316:52lwhortonsure:
:aliases {:dev {:extra-paths ["dev"]
:extra-deps {org.clojure/tools.namespace {:mvn/version "0.2.11"}
org.clojure/tools.deps.alpha {:mvn/version "0.5.460"}
nrepl/nrepl {:mvn/version "0.4.5"}
reply {:mvn/version "0.4.2"}}
:main-opts ["-m" "nrepl.cmdline"]}}}
#2018-11-0316:52lwhortonmakefile:
NREPL_PORT=$(shell cat .nrepl-port)
dev:
clj -A:dev
.PHONY: dev
clj-repl:
clj -R:dev -m reply.main --attach $(NREPL_PORT)
#2018-11-0316:55lwhortonand my user.clj file (pulled in from the extra-paths in deps.edn):
(ns user
(:require
[ :as io]
[clojure.tools.namespace.repl :refer [refresh refresh-all]]
[clojure.pprint :refer [pprint]]
[clojure.inspector :as i]
[gerbil.core :as c]
)
)
(def ^:dynamic system nil)
(defn init []
(alter-var-root #'system (constantly (c/system))))
(defn start []
(alter-var-root #'system c/start))
(defn stop []
(alter-var-root #'system
(fn [s] (when s (c/stop s)))))
(defn go []
(init)
(start))
(defn reset []
(stop)
(refresh :after 'user/go))
#2018-11-0316:56lwhortonyou can drop the reply stuff if you want and just use nrepl’s “basic” client#2018-11-0311:26Christian JohansenI'll third that THANK YOU! 👍#2018-11-0312:10slipset@lwhorton juxt pack generates jars for you #2018-11-0315:56orestis@borkdude :cider-nrepl {:extra-deps {nrepl/nrepl {:mvn/version "0.4.5"}
cider/cider-nrepl {:mvn/version "0.18.0"}}
:main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]}
:nrepl {:extra-deps {nrepl/nrepl {:mvn/version "0.4.5"}
org.clojure/tools.nrepl {:mvn/version "0.2.13"}}
:main-opts ["-m" "nrepl.cmdline"]}#2018-11-0315:56borkdudethanks#2018-11-0315:57orestisHere’s what I use to launch nREPL — the first variant is what I actually use to connect CIDER.#2018-11-0316:11cap10morganAsking this here too since it’s not really Datomic-specific: I have a datomic-pro download in a dir (so it has the datomic-pro-version.jar at the root, a lib subdir with deps jars, etc. I am trying to setup a tools-deps project to run some clojure code using datomic and all of its deps w/o triggering a download of them. Is there a more efficient way to do it than having to specify each individual jar in the :paths vector in deps.edn?#2018-11-0316:29cap10morganAnswering my own question:
:paths ["src" "/path/to/datomic/datomic-pro-version.jar" "/path/to/datomic/lib/*"] was the answer. (Not "/path/to/datomic/lib/" nor "/path/to/datomic/lib/*.jar")#2018-11-0506:55slipsetI see that https://github.com/healthfinch/depstar/ and https://github.com/juxt/pack.alpha both are in the business of packaging your project where depstar seems a bit more focussed on uberjars. Any thoughts on this?#2018-11-0506:57slipsetReason I’m asking is that I’m pondering creating a deps-shipit which would generate/refresh the pom, update the versions in it, create your choice of (skinny/uber) jar and then upload it to Clojars.#2018-11-0507:58dominicmUberjars aren't really suitable for uploading to clojars#2018-11-0508:07slipsettrue enough#2018-11-0508:07dominicm@slipset fwiw, I'd be interesting in a tool which only concerns itself with uploading jars to maven repos. Maybe with support for s3 wagons too.#2018-11-0508:09dominicmHaving a tool which starts a JVM to run this shell script isn't immediately on my list of problems I need solving:
#!/bin/sh
clojure -A:pack -m …
clojure -Spom
clojure -A:shipit output.jar pom.xml
#2018-11-0508:09slipsetoh, like https://github.com/slipset/deps-deploy ?#2018-11-0508:10dominicmThis is fantastic ❤️ 2 comments:
1. Can we look at getting gpg support for credentials, I think this is really important. Environment variables are far from secure.
2. I'd love to have this replace my current documentation in pack for using mvn:deploy-file 🙂#2018-11-0508:11slipset1) Yes. If I could ask you to file an issue/submit PR, that’d be fantastic#2018-11-0508:11dominicmI asked for this to exist about a week ago, so I'm glad it now does. I'm going to have to try imagining more things. Pile of Gold. Pile of Gold.#2018-11-0508:12slipset2) I’m more than happy to improve it in any way 🙂#2018-11-0508:12slipsetIn fact, I was about to submit a PR against pack suggesting the use of deps-deploy this morning 🙂#2018-11-0508:16slipset@dominicm the gpg support for credentials, that would basically be supporting passing private-key-file to aether, right? https://github.com/cemerick/pomegranate/blob/master/src/main/clojure/cemerick/pomegranate/aether.clj#L325#2018-11-0508:16dominicm@slipset I don't think so. I am not aware of that feature at all though 🙂#2018-11-0508:17slipsetOk, then if you could outline a solution in a ticket, I’d be very happy. I haven’t worked much with gpg.#2018-11-0508:18dominicmI've put quickly jotted down some thoughts, I've tried to capture my overarching goal of "moar secure" and provide some options.#2018-11-0508:19slipsetThank you! I’ll have a closer look this evening.#2018-11-0508:20slipsetBtw, concerning a potential deps-shipit, you’d actually end up just firing up one jvm, as the plan for deps-shipit would be to compose the three steps you are outlining into one.#2018-11-0508:23dominicmI know 🙂 But it's similar to this also:
#!/usr/bin/env clojure
(require '[pack.onejar])
(require '[deploy])
(require '[t.d.a])
(pack.onejar/jar)
(t.d.a/generate-pom)
(deploy/deploy jar pom)
Having said that, I'm usually happier to automate myself than others are 🙂#2018-11-0508:24slipset🙂#2018-11-0508:28dominicmSupporting jar signing seems important too, trying to find some good references on it#2018-11-0508:29slipsetShouldn’t be too hard, lein already provides this. Just need to RTFS a bit.#2018-11-0508:36dominicmyeah. Reading the source looks like the best option, given that maven + gpg signing isn't giving me anything useful.#2018-11-0512:21rickmoynihanIn my experience gpg and lein has been a bit awkward to setup… and I’ve seen lots of people struggle to do it. Is there an option to use something like bouncy castle here ( https://www.bouncycastle.org/java.html )? Or perhaps something that ships with the JVM?
I’m assuming the main thing gpg brings to the table (aside from the crypto) is being able to interact with the OS keychain.#2018-11-0512:21rickmoynihanNot sure if bouncy castle has any support for that#2018-11-0513:04dominicm@rickmoynihan You've highlighted my concern with BouncyCastle immediately. It would be good to enumerate the exact issues people have. I would expect it's around gpg vs gpg2. But I'm potentially projecting the issue I personally have.#2018-11-0513:04dominicmI would expect using Bouncy Castle with PGP brings in a new class of issues, like support smart cards.#2018-11-0513:05dominicmOne option is to search & present options. e.g. if only gpg/gpg2 are found, use those. If both, then ask the user.#2018-11-0513:35rickmoynihanYeah; I suspect gpg/gpg2 are the best options… as lein and others have been through this before and everyone always seem to settle on gpg. It’s just frustrating that that choice often leaves to brittleness and installation/configuration issues… but I guess it’s always going to be a bit of a cross platform nightmare 😞#2018-11-0513:46dominicmIt doesn't have to be I think 🙂#2018-11-0513:46dominicmLein also doesn't prompt the user ever. There are other mistakes there.#2018-11-0514:15rickmoynihanI can’t really recall my issues with gpg… it could well have been gpg vs gpg2. IIRC mine might also have been due to config cruft from .bashrc which I ported from linux to mac years back. In my experience it’s just one of those areas which can be fiddly, where you really don’t want something fiddly.#2018-11-0514:59dominicmIf gpg is hard to use, I'd be inclined to not tackle the problem 🙂#2018-11-0514:59dominicmBut that's just me.#2018-11-0514:59dominicmWhat I mean is, if gpg is easy to shoot yourself in the foot with, that's gpg's problem. I don't think it's a good use of time to handle the thousands of config combinations.#2018-11-0515:30slipsetlein actually prompts me for my gpg thing (if I don’t turn off the signed artifacts thingy)#2018-11-0515:43dominicm@slipset your gpg passphrase you mean?#2018-11-0516:00slipsetyes, I guess that’s the correct term.#2018-11-0516:04seancorfieldAfter some updates to software on my Mac I could never get gpg working again so I just told leiningen not to sign things. I mostly use boot to upload to clojars to avoid gpg anyway. #2018-11-0516:06seancorfieldAnd I have my clojars credentials in my dot file unencyrpted - again so I can avoid trying to deal with gpg. #2018-11-0516:06seancorfield(And I was never able to get anything to work on windows in this area)#2018-11-0516:08seancorfieldPhil was always the one pushing gpg. No one used to bother until he bugged everyone 🙃 and the boot crowd never pushed it. #2018-11-0516:15dominicm@seancorfield do you mean exclusively with lein it broke, or it's broken altogether?#2018-11-0516:15dominicm@slipset I was thinking more about prompting if both commands gpg and gpg2 are available. Things like that.#2018-11-0516:16slipset@dominicm I’ll look into https://github.com/slipset/deps-deploy/issues/2 later tonight.#2018-11-0516:16dominicmNo rush. I think it's good to think on & compare.#2018-11-0516:16dominicmHammock time 🙂#2018-11-0516:16slipsetI’d be really happy if you could comment on my comment there so I have something to hammock on 🙂#2018-11-0516:17dominicmSorry, I've got my emails turned off 😄#2018-11-0516:17slipsetNothing to be sorry about, it’s actually very smart if you want to get things done 🙂#2018-11-0516:17dominicmWe have a large number of (non-)errors in a production environment at the moment, only way I can stay sane#2018-11-0516:38rickmoynihanFWIW on mac with GPG this is what I use and it seems to work: https://gpgtools.org/#2018-11-0517:39seancorfieldYeah, that's what I was using but it stopped working and uninstalling/reinstalling didn't fix it so I gave up. The whole thing seemed more hassle than it was worth -- given the only thing that seemed to care about it was Leiningen...#2018-11-0518:47dominicmI consider gpg an important part of my security system, I think its a shame more people don't use it with jars.
But I can't speak to how well the gnu toolchain works elsewhere. #2018-11-0519:09slipset@dominicm https://github.com/slipset/deps-deploy/pull/4 if you could have a look?#2018-11-0519:10slipsetI used gpg --encrypt .clojars_creds.edn to encrypt my creds.#2018-11-0519:10slipsetwhich may or may not be what you were asking for, as this is a symmetric encryption.#2018-11-0519:13dominicmI'll see if I can get it working with my smart card. Looking at the code, I don't expect it will, as I don't have a gpg passphrase.#2018-11-0520:34slipsetMay I ask what kind of smart card you have?#2018-11-0521:16dominicmYubikey#2018-11-0616:42timgilbertSay, are there options available for customizing the groupId, version, etc in pom.xml during the course of clojure -Spom?#2018-11-0616:42dominicm@timgilbert update the generated pom.xml, then -Spom won't override them.#2018-11-0616:43timgilbertHrm. I don't love having half-generated code in my codebase that I'll need to create another git commit for whenever I update deps.edn#2018-11-0616:46timgilbertI suppose I could write a tool that would run -Spom and then update the relevant values. What I'm actually looking for is something that will create the version number based on git tags on my repo in the same way lein-v does#2018-11-0617:01dominicmI imagine you're into "write another tool" territory 🙂#2018-11-0617:04arrdemIf someone writes a manifest generator, let me know 😉 I’m on the hunt for one.#2018-11-0617:21seancorfield@arrdem I forked depstar and added basic manifest generation -- then decided I really didn't need it and removed it.#2018-11-0617:21ghadiheh#2018-11-0617:23arrdemyagni#2018-11-0617:24arrdemI’m somewhat surprised how far bare jars have gotten me so far - we’ll see.#2018-11-0617:25ghadiI'm surprised by all the git deps users wanting to have artifacts back#2018-11-0617:25seancorfieldWe build/deploy bare uber JARs and run them that way in production. No AOT. java -cp path/to/the.jar clojure.main -m entry.point#2018-11-0617:34arrdemAt this point I’m convinced that between Katamari / git deps I’ll probably only use external libraries from Maven again. Which is great. But I have internal users who aren’t on that workflow yet and who I want to be able to deploy artifacts with manifests and poms for.#2018-11-0617:35ghadiI get it from a transition strategy or reach perspective#2018-11-0617:35arrdemmaking nicer / faster psuedo-uberjars in docker is easy enough, doesn’t require any of this and that’s my only deployment need.#2018-11-0619:02orestisCan someone share some knowledge about the difference between bare jars and Uberjars and what manifests are for?#2018-11-0619:02orestis(Links welcome, no need to type it out here)#2018-11-0619:02orestisI finally got that Pom.xml is a maven thing :)#2018-11-0619:07orestisI’d love to be able to just rsync individual jars instead of an Uber jar, then somehow launch my app with the correct classpath. I guess if you have clj command line on the server installed, you can do that but you pay for the startup compilation time. #2018-11-0619:08dominicm@orestis Check out pack's skinny jar mode 🙂#2018-11-0619:15seancorfield@orestis FYI, we deploy source uberjars to production -- no AOT -- and with all their dependencies included, they're still only 20-30MB so it's a pretty small payload to upload across servers (but, yes, would take a while to upload from a dev machine I guess).#2018-11-0619:21orestisI tried AOT jars and they still are roughly 30-40MB so still pretty small. It’s the compiling that takes time :)#2018-11-0619:23orestisAnd it’s annoying that you compile the same dependencies over and over. I’m fairly sure I’ve seen Alex & co discussing this as a potential improvement in the future. #2018-11-0710:38rickmoynihanYes, I saw this the other day on the clojure mailinglist and was excited by it too.
It’s a mild annoyance that uberjaring and AOTing a moderately sized app takes 2 minutes. I understand why @U04V70XH6 likes to avoid AOTing apps; I’m assuming to have quicker builds… and I’ve been contemplating only AOTing nightlies, to bring CI times down… but still have uberjars start quickly.
I think this problem could be solved if we could easily AOT libraries. I can’t recall what the issues are here; I suspect it’s to do with how clojure.core/compilation/read/eval get linked in to libraries.#2018-11-0715:33seancorfieldOne big problem with AOT for libraries is that they can locked into specific versions of Clojure (this was a problem back in the day when some did compile their libraries for distribution). #2018-11-0715:33seancorfieldWith source-only libraries they should always work with all future versions of Clojure. #2018-11-0715:37seancorfieldAOTing an application is safe because it's an entire system. Self-contained. For us, the startup time hit just isn't worth adding an AOT step. New Relic's instrumentation at startup is a much bigger overhead for us than Clojure's compilation. #2018-11-0716:35orestisOh, can you share that new relic thing please?#2018-11-0716:45seancorfield@orestis What do you mean? We just use the regular Java agent for New Relic. I think we had to blacklist a particular classloader in Clojure to avoid prohibitively slow NR startup, but otherwise you just add the JVM startup options to point at their library and config file.#2018-11-0716:46seancorfieldhttp://corfield.org/blog/2016/07/29/clojure-new-relic-slow-startup/ -- talks about the slow startup issue.#2018-11-0716:46seancorfieldhttp://corfield.org/blog/2013/05/01/instrumenting-clojure-for-new-relic-monitoring/ -- this covers how to add Trace annotations to Clojure functions for use with New Relic.#2018-11-0716:47seancorfieldHappy to answer any other specific Qs you might have.#2018-11-0716:50orestisRight, that! Thanks so much!#2018-11-0703:25seancorfieldDeleted the last vestiges of Boot from our system today (on dev). I decided to even convert two legacy system pieces since they are going to be around for a few months and it seemed silly to keep Boot around just for those. Very pleased with nice, simple, lean processes we have now. And we're on Clojure 1.10 Beta 5 in dev as well. Several blog posts to come 😉{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 4")}
#2018-11-1004:18jwhitlarkThose blog posts are eagerly awaited. Do you think tools.deps has removed the need for boot, in general?#2018-11-1005:16seancorfieldI think it depends what you want from a build tool. #2018-11-0707:57cflemingI’ve talked about some variant of this in the past, but I can’t remember the exact details now. It seems to be impossible to use tools.deps as a library from a process which doesn’t have the CWD set to the directory containing the main deps.edn file. This seems like a significant limitation - it means you can never have a process which resolves deps from two separate deps.edn files, for example.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 16")}
#2018-11-0707:58cflemingThe problem is that if your deps.edn file references another using :local/root, the paths are incorrect - the paths from the dependency are relative to the CWD, not to the deps.edn file location.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 16")}
#2018-11-0708:01cflemingThe problem is here: https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/deps.clj#L29#2018-11-0708:02cflemingHere, root is the local :deps/root, which is generally relative.#2018-11-0708:02cflemingIs this intentional? If not, I’ll file a bug.#2018-11-0708:06cflemingI was somewhat convinced that this wasn’t previously the case, but it seems to have been this way for some time according to git.#2018-11-0708:07dominicmAlways been this way, yeah#2018-11-0708:13cfleming@dominicm Is there an issue for it? There’s TDEPS-74 but that’s the transitive one. I can’t see anything else likely-looking.#2018-11-0708:22mpenetI think that also might be one of the reason katamari has vendored/forked deps for its use#2018-11-0708:24cflemingInteresting, I’ll have to check that out.#2018-11-0708:28cflemingRight, looks like it uses raynes/fs#2018-11-0708:30cflemingThis is kind of a show-stopper for Cursive, unfortunately.#2018-11-0708:31dominicmUncertain. I think there should be. #2018-11-0717:08orestisWhat about CLJ_CONFIG? I remember @seancorfield uses it for something similar?#2018-11-0717:18dominicm@orestis that's for having multiple deps.edn files. It's bad for tooling as it stops users from adding personal aliases to use.#2018-11-0717:24seancorfieldWell, it's good for projects that want a specific deps.edn "base" file instead of using whatever the developer happens to have in their .clojure/deps.edn file -- so it really depends on your perspective.#2018-11-0717:27seancorfieldFor our monorepo, we specify all our defaults (some aliases, but mostly :override-deps) in a versions/deps.edn file and we use CLJ_CONFIG to use that as our basis for all the subprojects' deps.edn files.#2018-11-0717:28seancorfieldThat doesn't help Cursive tho', since it affects the "wrong" deps.edn file -- the "user" one, not the "project" one.#2018-11-0717:29seancorfieldIn one place, we shell out to the clojure command and when we do that (via clojure.java.shell/sh) we specify the :dir that it should be executed in so CWD is controlled. I'm not sure how Cursive invokes this stuff -- I suspect it is trying to use t.d.a. directly and that's where the problem occurs...?#2018-11-0717:33dominicmcursive is java, so it's using it as a library rather than shelling out. I think @cfleming mentioned that $PATH isn't set in cursive.#2018-11-0717:38seancorfieldIn our Clojure code, we still shell out to run clojure 🙂#2018-11-0718:53cflemingWhat I want is to get the full Java command line that I need to execute, so that I can execute it with IntelliJ’s command running infrastructure. There’s no way to get that from the command line with clj.#2018-11-0718:58cfleming@dominicm The issue is that if you use t.d.a as a lib, there’s no way to set the CWD of a running Java process. It’s not a $PATH thing.#2018-11-0719:36mpenetApparently there's a way with jruby https://github.com/arohner/lein-daemon/blob/master/daemon-runtime/src/leiningen/daemon/runtime.clj#2018-11-0719:36mpenetI guess it's jni under the hood#2018-11-0720:29cflemingYeah, but that would probably break all sorts of things with IntelliJ (where the logs go, etc).#2018-11-0719:19dominicmDuh, of course #2018-11-0719:32rickmoynihan@cfleming: is an entirely insufficient workaround to launch idea from the command line in your projects working directory?#2018-11-0719:34rickmoynihanIIRC I’ve done that in the past with intellij; when I’ve run into issues with macos/quartz-UI not being a child of a shell environment#2018-11-0719:35rickmoynihanthough I don’t normally use intellij#2018-11-0720:10arrdem@rickmoynihan yeah - most IDEA launches are through a GUI, and that defeats IDEA’s multi-project/workspace capabilities.#2018-11-0720:13rickmoynihan… but it does work around the issue?#2018-11-0720:27cfleming@rickmoynihan Only if you ever use one project - I usually have 3 open.#2018-11-0720:30rickmoynihanas I said an entirely insufficient workaround#2018-11-0720:28cflemingI actually don’t even know how to run a Mac app from a specified directory, or if it’s even possible.#2018-11-0720:30rickmoynihan/Applications/IntelliJ\ does it for me#2018-11-0720:30cflemingFor now, I’ll probably just have to do the same horrible hacks I do for leiningen using hooke or the moral equivalent.#2018-11-0720:32cfleming@arrdem Has raynes/fs worked well for your vendored fork?#2018-11-0720:32cflemingLooking at the code, I assume this sort of problem is what you’re trying to deal with?#2018-11-0720:32arrdem@cfleming Yeah. It was a bit fiddly to figure out where I needed it, but I haven’t had problems with it since.#2018-11-0720:33arrdemYou’re right, that is why I patched it in.#2018-11-0720:33cflemingAre there places where you don’t want it? If I use hooke, then that’s pretty global.#2018-11-0720:34arrdemCan you hooke or alter-var-root on io/file? That seems like a really big hammer and may play badly with static linking.#2018-11-0720:35arrdemIn general I couldn’t prove that things wouldn’t behave precisely as expected and erred on the side of caution, so I did the refactor by hand and it took a couple passes.#2018-11-0721:47cfleming(sorry, got called away there)#2018-11-0721:47cflemingYeah, that’s what I do. It’s scoped to a particular sequence of calls, but during that sequence it’s globally modified, yes.#2018-11-0721:48cflemingIt seems to work, and I think it works because the code calling io/file is not statically linked, so it uses the var.#2018-11-0721:48cfleming@arrdem ^^#2018-11-0721:49arrdemhummm#2018-11-0721:49arrdemexciting#2018-11-0721:49cflemingI’m not sure whether that’s better or worse than vendoring a specific version of deps, which is then tied to that version of Cursive.#2018-11-0721:50arrdemFor katamari it’s as much a proof that katamari’s heavily vendored monorepo workflow actually does work.#2018-11-0721:50cflemingAll options suck, basically.#2018-11-0721:50arrdemfork && vendor && move on#2018-11-0721:51cflemingI can’t wait for the bug reports saying that Cursive xxx doesn’t support the functionality that was introduced in deps yyy#2018-11-0721:52arrdemyeah I need to write some bash + sed automation to let me packport merged diffs ~automatically.#2018-11-0722:12cflemingThe problem I have with that is that I can’t rely on everyone upgrading Cursive, so older versions will still be out there with older versions of tools.deps baked in.#2018-11-0813:21onetombtw, we are relying more and more on project specific nix-shells (activated via [`direnv`](https://direnv.net) by simply cding into a project's directory)
and intellij doesn't expects such approach at all, instead it's providing tons of custom dialogs with very similar path and option settings for the various tools in different plugins 😞
i really envy emacs in this respect, since they have https://github.com/wbolster/emacs-direnv{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-0815:56richiardiandreaDefinitely nice will try it out#2018-11-0813:59dominicmNeat, it supports vim too 🙂#2018-11-1005:57jwhitlarkPSA, there's work going on with docker-clojure: https://github.com/Quantisan/docker-clojure/pull/56{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-11-1107:43jwhitlarkI was playing around with https://clojure.org/reference/compilation#_gen_class_examples and I've noticed that with 1.10.0-beta5, I couldn't compile the example code until I'd created a classes directory. Is that expected behavior?#2018-11-1107:50jwhitlark#2018-11-1108:18dominicmIt is normal behaviour from my past testing on this. The output directory is supposed to already be in the classpath. #2018-11-1108:31jwhitlarkOk.#2018-11-1108:53orestisDoes tools-deps have a story for compiling Java code that lives in the same repo?#2018-11-1112:43danierouxJoined to ask this exact question. I have one measly .java file that I want to (:import), and i really don't want to make a jar just for that one file.#2018-11-1108:53orestis(No immediate use case, just curious)#2018-11-1109:02jwhitlarkI'm curious about that too.#2018-11-1109:24dominicmNot at all @orestis. #2018-11-1109:25dominicmI've done some work on this, but it would be the same level as clojure.core/compile I suspect. Riddler is a good option to look at I think#2018-11-1112:46danierouxwhat is Riddler, where can I find it?#2018-11-1113:13dominicmApparently I got mixed up, I meant this: https://github.com/ztellman/virgil#2018-11-1112:43danierouxJoined to ask this exact question. I have one measly .java file that I want to (:import), and i really don't want to make a jar just for that one file.#2018-11-1113:02dottedmagtools.deps is about building classpath, clj is about running Clojure. You can run javac from one of clj aliases if that's a single .java file.#2018-11-1113:13dominicmApparently I got mixed up, I meant this: https://github.com/ztellman/virgil#2018-11-1123:17jwhitlarkWow, digging through virgil's code answers a lot of questions.#2018-11-1200:54gfredericksthere's no way to add a jvm-opt without having to enable an alias via CLI opts, is that right?#2018-11-1201:34mfikes@gfredericks -J#2018-11-1201:35gfredericks@mfikes sorry, I mean I'd like to put it in the deps.edn#2018-11-1201:35mfikesExample: clj -J-Xmx512m#2018-11-1201:35gfredericksand have, e.g., clojure pick it up#2018-11-1201:35gfrederickswithout any extra args#2018-11-1201:36mfikesAhh, I think you are right in that case. I can't think of a way to just "make it so"#2018-11-1201:38mfikesIt would seem, a top-level thing, sibling of :deps or :paths is where a :jvm-opts could live#2018-11-1201:39gfredericksyeah; I can't tell if that is just a missing feature, or if there's some philosophical conflict#2018-11-1201:39Alex Miller (Clojure team)It’s missing, no philosophical objection{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-11-1219:11adamfreyhere's a command that I can run from the terminal to start a cider-ready nrepl, but I'm having a problem when putting it into an aliase in a deps.edn file.
clj -Sdeps '{:deps {nrepl {:mvn/version "0.4.5"} cider/cider-nrepl {:mvn/version "0.18.0"} refactor-nrepl {:mvn/version "2.4.0"}}}' -m nrepl.cmdline --middleware '[cider.nrepl/cider-middleware refactor-nrepl.middleware/wrap-refactor]'
the --middleware option is a vector with a space in the middle, but I can't find any way to make it work from deps.edn.
:main-opts ["-m" "nrepl.cmdline"
"--middleware" "[cider.nrepl/cider-middleware refactor-nrepl.middleware/wrap-refactor]"
"--interactive"]
by the time it gets to the -main function the args binding is set to
("--middleware" "[cider.nrepl/cider-middleware" "refactor-nrepl.middleware/wrap-refactor]" "--interactive")
#2018-11-1219:36dottedmag@adamfrey try , instead of a space#2018-11-1219:39adamfreyoh good call! it works because it gets passed as one thing to the nrepl main and then nrepl parses it from a string using edn/read-string Thanks!#2018-11-1221:53martinklepschAlso known as the Corfield Comma 😂#2018-11-1219:39adamfreyoh good call! it works because it gets passed as one thing to the nrepl main and then nrepl parses it from a string using edn/read-string Thanks!#2018-11-1219:56aviI seem to be bumping up against a dep conflict but I’m not quite understanding the behavior I’m seeing from tools.deps + CLI-tools; since I had a lot of text to share I started a topic over at ClojureVerse. If anyone has a minute to take a quick look and possibly enlighten me as to what I’m missing, I’d much appreciate it!
https://clojureverse.org/t/tools-deps-not-downloading-all-deps-when-run-with-r-aliases/3146?u=avi#2018-11-1220:22seancorfieldAnswered on ClojureVerse.#2018-11-1220:22aviThanks Sean! 🙏#2018-11-1220:23seancorfieldTL:DR; dependency version resolution has a different set of versions to consider with one alias vs multiple aliases.#2018-11-1220:24aviI think I get it; makes sense 😅#2018-11-1516:01aviJust wondering if anyone’s figured out a way to get tools.deps to use a http proxy when downloading Maven deps, other than applying the patch in https://dev.clojure.org/jira/browse/TDEPS-20 ? I’m just being lazy I guess, putting off learning how to patch tools.deps and install and use the patched version…#2018-11-1516:07Alex Miller (Clojure team)I don’t think there is any way to do it but to fix that (and would be happy to see a patch for it)#2018-11-1516:07Alex Miller (Clojure team)oh, there is a patch for it, sorry#2018-11-1516:07Alex Miller (Clojure team)well, then you just have to wait :)#2018-11-1516:08Alex Miller (Clojure team)I’m probably not going to have much time to look at tools.deps stuff until post-conj#2018-11-1516:54avitotally understandable, no worries! Thanks for the clear and helpful response!#2018-11-1517:34martinklepschIs there any documentation on how tools.deps resolves deps? Specifically wondering what happens when two dependencies depend on 'A but in different versions#2018-11-1517:35Alex Miller (Clojure team)the general idea is to always take the newest#2018-11-1517:36martinklepschOk, that's great. Does that "general idea" imply any exceptions to that rule where the older artifact would be chosen?#2018-11-1517:36Alex Miller (Clojure team)the complexities come when you consider the full graph#2018-11-1517:37Alex Miller (Clojure team)A -> B1 -> C2
-> B2 -> C1#2018-11-1517:37Alex Miller (Clojure team)do you get {A, B1, C2} or {A, B2, C1} or {A, B2, C2}#2018-11-1517:38timgilbertI'm glad I'm not the one who needs to figure that out 😉#2018-11-1517:39Alex Miller (Clojure team)it’s actually NP-complete I believe#2018-11-1518:10hiredmanhttps://research.swtch.com/version-sat#2018-11-1518:14hiredmanhttps://cwiki.apache.org/confluence/display/MAVENOLD/SAT+Based+Dependency+Resolution#2018-11-1517:39Alex Miller (Clojure team)in the general case#2018-11-1517:42Alex Miller (Clojure team)fortunately most actual cases are pretty “easy”#2018-11-1517:52timgilbert...unless they involve jackson, I'm guessing#2018-11-1518:00Alex Miller (Clojure team)yes, that is the exception :)#2018-11-1518:00Alex Miller (Clojure team)and unfortunately, every dependency set includes at least 9 versions of jackson{:tag :div, :attrs {:class "message-reaction", :title "picard-facepalm"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "picard-facepalm", :src "https://emoji.slack-edge.com/T03RZGPFR/picard-facepalm/37b1e28762160297.gif"}, :content nil})} " 8")}
#2018-11-1518:56andy.fingerhutA new complexity class: jackson-complete{:tag :div, :attrs {:class "message-reaction", :title "laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😆")} " 4")}
#2018-11-1518:56andy.fingerhutAlthough that might be equivalent to: undecidable 🙂#2018-11-1518:58mfikesEntjaksonsproblem{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 4")}
#2018-11-1519:06andy.fingerhutOh, that is good. You even Deutsch-ified the spelling of Jackson. Love the attention to detail there 🙂#2018-11-1519:13Alex Miller (Clojure team)now that’s a t-shirt with sufficiently small audience#2018-11-1519:16andy.fingerhutA sufficiently small audience to .... not make it?#2018-11-1519:25Alex Miller (Clojure team)to be fun :)#2018-11-1519:27Alex Miller (Clojure team)a joke t-shirt with an audience of 1 is not fun, because you can’t share it. And a joke t-shirt with an audience of large N, is cliche. So you have to find that sweet spot in the middle. :)#2018-11-1519:27Alex Miller (Clojure team)this is bigger than 1, but not by much :)#2018-11-1519:29andy.fingerhutGot it. It is pretty easily explained to anyone at the Conj, if they don't get it immediately (I wouldn't without this conversation)#2018-11-1519:32andy.fingerhutAs a reminder, if you think Rich would see a "channeling Rich sayings" T-shirt as a tribute (as it is intended), not a weirdness or offensive, I would be happy to bring some to the Conj 🙂#2018-11-1519:39Alex Miller (Clojure team)prob not the best idea#2018-11-1519:39andy.fingerhutBut also happy to never bring it up again if he doesn't want to see such things.#2018-11-1519:39Alex Miller (Clojure team)for someone with such great care for words, I hate to ascribe any words to him#2018-11-1519:41andy.fingerhutUnderstood. He isn't the only person like that, and such people should get to be upset with people that misquote them, especially if the meaning is changed. FYI, I do take pretty significant care in transcribing his talks. I think I've listened to the ones I have transcribed at least 4 times each now.#2018-11-1519:41seancorfieldMaybe we should make some (rich (comment forms))
T shirts? off-topic#2018-11-1519:42Alex Miller (Clojure team)I don’t get that#2018-11-1519:42Alex Miller (Clojure team)I appreciate the care in the transcripts - he’s never complained about those#2018-11-1519:42seancorfieldStu talked about Rich Comment Forms in his "Running With Scissors" talk.#2018-11-1519:42Alex Miller (Clojure team)and I know he appreciates having slides+text together for context#2018-11-1519:42Alex Miller (Clojure team)oh, I haven’t watched it yet :)#2018-11-1519:43seancorfieldAnd observing it was a pun on the (comment ...) forms being "rich" with exploratory code and it also being a habit of Rich to use comment that way.#2018-11-1519:44Alex Miller (Clojure team)clearly the circle is too small for a t-shirt :)#2018-11-1519:44Alex Miller (Clojure team)fwiw, I made a custom “O(log32 n) is good enough” t-shirt for Conj 1 :){:tag :div, :attrs {:class "message-reaction", :title "smile"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😄")} " 4")}
#2018-11-1519:54dpsuttonif you donated that design to clojurist together i bet they could raise some money for projects by selling the t shirt. i would for sure get one#2018-11-1520:06Alex Miller (Clojure team)well feel free to make one yourself, mine was certainly nothing special and I don’t have it#2018-11-1520:09andy.fingerhuthttp://customink.com makes it really easy to create T-shirt designs, I found out recently#2018-11-1520:10andy.fingerhutI have never bought anything from them, so don't have knowledge of the full experience front to back.#2018-11-1520:13dpsuttoni made some tshirts for my team at my last company through them. I still get emails to make mouse pads and other things with that design 2.5 years later#2018-11-1520:13dpsuttonbut the shirts were/are great#2018-11-1601:28gfredericks...mouse...pads?#2018-11-1601:28gfredericksI don't think I've even seen a mause pad for at least fifteen years#2018-11-1519:45andy.fingerhutI guess we'll have to live with these continuing to be the only Rich-approved fan club T-shirts: https://clojure.org/community/swag#2018-11-1519:46Alex Miller (Clojure team)well, all the conference t-shirts too :)#2018-11-1520:18cflemingI still love my conj 2014 t-shirt.#2018-11-1520:18dpsuttonmy clojure west 2016 and strangeloop 2016 are my two favorite shirts#2018-11-1520:18cflemingNot many other conference t-shirts stick around for long.#2018-11-1520:19andy.fingerhutAnd here I thought the biggest benefit of the conferences was listening to the talks, and the hallway track 🙂#2018-11-1520:20cflemingNo, listening to the talks is totally overrated, it’s only the hallway track 🙂#2018-11-1520:20cflemingAt least for me, since I can watch talks online, but seldom get to chat with others who understand what I’m talking about.#2018-11-1520:21andy.fingerhutBut clearly I have missed out on some great T-shirt buying opportunities that are not available elsewhere. This should be in the conference advertisements!#2018-11-1609:05martinklepschhas there been any discussion around preventing things like this:
(tdeps/resolve-deps
{:deps '{clj-time {:mvn/version "0.8.0"}
clj-time/clj-time {:mvn/version "0.7.0"}}
:mvn/repos default-repos}
nil)
#2018-11-1613:14Alex Miller (Clojure team)Should be fine - there is a canonicalization step#2018-11-1617:51tcrawleyI'm working on migrating a project from lein to tools.deps, and need to access an authenticated repo. https://clojure.org/reference/deps_and_cli#_procurers says that the credentials are pulled from ~/.m2/settings.xml, which doesn't work well for us. Is there another method for specifying credentials, or plans to expose one?#2018-11-1617:51Alex Miller (Clojure team)in short, no#2018-11-1617:52tcrawleyI'd be fine with embedding the credentials in deps.edn - ours are generated by make, and are never checked in#2018-11-1617:52Alex Miller (Clojure team)we are not fine with putting any kind of credentials in deps.edn so will not be doing that#2018-11-1617:53Alex Miller (Clojure team)perhaps there is some other way to pull credentials from a trusted source#2018-11-1617:55tcrawleyWhat about a tools.deps-specific file for them? ~/.tools-deps.edn would possibly work, and is as secure as ~/.m2/settings.xml#2018-11-1617:56dominicm@tcrawley settings.xml is more secure. It supports encryption. Unless tools.deps adds encryption.#2018-11-1617:56Alex Miller (Clojure team)why is that different for you from making a ~/.m2/settings.xml?#2018-11-1617:56Alex Miller (Clojure team)have no desire to do the encryption stuff#2018-11-1617:56tcrawley@dominicm does tools.deps support encrypted ~/.m2/settings.xml?#2018-11-1617:57Alex Miller (Clojure team)it uses the Maven apis, so should{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-1617:57tcrawleyah, that makes sense#2018-11-1617:57Alex Miller (Clojure team)can’t say that I’ve actually done it{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-1618:00dominicm@tcrawley if you wanted to get really crazy. You could generate a server into ~/.m2/settings.xml and use a reference to that in deps.edn.#2018-11-1618:00dominicmxml is just data after all.#2018-11-1618:01tcrawleyright, I'd just like to avoid that if possible :) I can output an alternate settings.xml, then tell tools.deps to use that if the maven libs support setting that via an environment variable and it actually works#2018-11-1618:01tcrawleyI'll try that first#2018-11-1618:05dpsuttonthere's also a bug with the credentials in settings.xml. its on github issues for s3 wagon maybe? alex had very useful information in it#2018-11-1618:05dpsuttonit affects my ubuntu coworkers but not me on fedora right now#2018-11-1618:05dpsuttonhttps://github.com/s3-wagon-private/s3-wagon-private/issues/31#2018-11-1618:18Alex Miller (Clojure team)that is not an issue with tools.deps (as I am using a different version than s3-wagon-private)#2018-11-1618:19Alex Miller (Clojure team)the stuff documented at https://clojure.org/reference/deps_and_cli should work#2018-11-1618:29kennyWhy does this produce a REPL instead of running my main opts?
clj -Sdeps '{:aliases {:jar {:extra-deps #:com.healthfinch{depstar {:git/url "", :sha "2879672df7fd296855563c22dd9ba9de2022a0b4"}}}, :main-opts ["-m" "hf.depstar.uberjar" "foo.jar"]}}' -A:jar
Clojure 1.9.0
user=>
#2018-11-1618:29ghadibecause the ns'ed map syntax is qualifying :sha incorrectly#2018-11-1618:29kennyWow, :main-opts is at the wrong level. Oops...#2018-11-1618:30kennyGreat start to the day 🙃#2018-11-1618:30ghadimain opts isn't supported top-level, AFAIK#2018-11-1618:30ghadibut yeah it's at the wrong level#2018-11-1618:31kennyShouldn't there be some validation there?#2018-11-1618:31kennyI passed a vector instead of a map to an alias.#2018-11-1621:28tcrawleyI figured out how to tell the maven lib to use a different settings.xml file by setting the org.apache.maven.user-settings sysprop. However, clojure doesn't expose a way to pass options to the make-classpath java invocation. Is that something you would be interested in exposing @alexmiller?#2018-11-1621:29tcrawleyI can work around it with a custom copy of the clojure script for now#2018-11-1621:31Alex Miller (Clojure team)Would prob rather add a top level :mvn/ key but would be good to have a ticket regardless#2018-11-1621:32tcrawleyI was thinking about a general way to pass options to that invocation, but a top-level :mvn/settings-file (or whatnot) would work for this use-case. I'll file a ticket.#2018-11-1621:40dominicmI initially thought it would not work, but as long as -Sdeps is considered, it works quite well. #2018-11-1621:43tcrawley@dominicm what "it" are you referring to?#2018-11-1621:44dominicmA top level key rather than an environment variable @tcrawley #2018-11-1621:44tcrawleygotcha#2018-11-1718:25richiardiandreaLittle question, is there a way to override the information that tools.deps.pom/gen-pom uses below
https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/gen/pom.clj#2018-11-1718:26richiardiandreaI am trying to generate a pom with version and project coords coming from the outside#2018-11-1718:26Alex Miller (Clojure team)No, but it syncs - so you can change it and it won’t be removed#2018-11-1718:28richiardiandreaYep I saw the sync but in my case the info is coming from another file (either project.clj or package.json) and I think it would be awesome to have the pom.xml generated somehow - not persisted. Is it something worth discussing about?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-1718:32richiardiandreaThis is the only missing piece for an automated - deps.edn + project descriptor - no manual steps flow 😃
I know there exists probably a maven plugin for bumping versions in a pom but ...it is no very lean#2018-11-1718:33richiardiandreaAnd thanks for the quick answer as well #2018-11-1718:37richiardiandreaOr I can write some bash but when this happens it always feels there is something hacky I don't know why simple_smile #2018-11-1800:20ro6as tools.deps stands today, if I'm co-developing an app+library locally, do I need to cycle my clj repl to refresh changes from the library?#2018-11-1800:40Alex Miller (Clojure team)Depends but you can probably just reload the namespace#2018-11-1806:21dominicm@robert.mather.rmm tools.namespace refresh will reload namespaces in local libraries. {:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-1919:06martinklepschThis fails to find org.jboss.naming:jnpserver:jar:5.0.3.GA. org.immutant/messaging has the jboss repo specified in it's pom but only transitively depends on jnpserver. The the immediate dependencies of org.immutant/messaging do not specify the jboss repository.
clj -Sdeps '{:deps {org.immutant/messaging {:mvn/version "2.1.10"}}}' -Stree
#2018-11-1919:06martinklepschIs this something worth opening a JIRA ticket for?#2018-11-1919:10Alex Miller (Clojure team)I think there is one?#2018-11-1919:12martinklepschYou're right: https://dev.clojure.org/jira/browse/TDEPS-46#2018-11-1919:12Alex Miller (Clojure team)Well that’s for -Spom right? Related but different#2018-11-1919:13Alex Miller (Clojure team)Well no, guess not#2018-11-1919:13Alex Miller (Clojure team)I find the way that ticket is written confusing#2018-11-1919:13martinklepschyeah... but there's a ticket for -Spom as well#2018-11-1919:27martinklepsch@alexmiller btw, I checked up on canonicalize-deps and either it's not working or I'm misunderstanding what it's supposed to do:
(let [cfg {:deps '{clj-time {:mvn/version "0.8.0"}
clj-time/clj-time {:mvn/version "0.7.0"}}
:mvn/repos mvn/standard-repos}]
(canonicalize-deps (:deps cfg) cfg))
;; => [[clj-time #:mvn{:version "0.8.0"}]
;; [clj-time/clj-time #:mvn{:version "0.7.0"}]]
#2018-11-1919:35Alex Miller (Clojure team)there’s actually two different points where things get canonicalized#2018-11-1919:36Alex Miller (Clojure team)the other is one of the extension methods that gets called by resolve-deps, and I think that’s where this gets adjusted#2018-11-1919:37Alex Miller (Clojure team)I started doing this earlier in canonicalize-deps and then ended up incorporating it later in the process instead#2018-11-1919:37Alex Miller (Clojure team)sorry, don’t have time to look at it now#2018-11-1920:49borkdudeIn the case of git deps, which paths of the repo get added to the classpath? Does it look at a pom file? Can you control this? E.g. also add the test dir so you can run the tests of a git dep?#2018-11-1921:43Alex Miller (Clojure team)git deps will detect the manifest type (deps.edn or pom.xml) and use the paths indicated#2018-11-1921:46Alex Miller (Clojure team)you can override it with :deps/manifest :deps or :deps/manifest :pom#2018-11-1921:48borkdudeI guess that’s only useful if both are present, else it will choose whichever is present?#2018-11-1921:48Alex Miller (Clojure team)prefers deps.edn#2018-11-1921:48Alex Miller (Clojure team)then pom#2018-11-1921:48Alex Miller (Clojure team)if neither will throw, but if you explicitly set to :deps, it is tolerant of a missing deps.edn#2018-11-1921:48Alex Miller (Clojure team)and just treats it as a dep with no transitive deps#2018-11-1921:49borkdudein that case it chooses ["src"] as the default :paths?#2018-11-2208:30mkvlrI’m trying to get add-lib working with prepl. I’m using the latest add-lib branch and clojure:#2018-11-2208:30mkvlrclj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.0-beta7"} org.clojure/tools.deps.alpha {:git/url "" :sha "f6c080bd0049211021ea59e516d1785b08302515"}}}' -J-Dclojure.server.repl="{:port 5556,:accept clojure.core.server/io-prepl}" #2018-11-2208:32mkvlrto get it working I have to set the dynamic class loader first:#2018-11-2208:32mkvlr(let [cl (.getContextClassLoader (Thread/currentThread))
dcl (clojure.lang.DynamicClassLoader. cl)]
(.setContextClassLoader (Thread/currentThread) dcl))
#2018-11-2208:33mkvlris that how it’s supposed to work or wouldn’t it be nice if prepl did this for me like repl does? https://github.com/clojure/clojure/blob/b182982007df934394f0bc68b3a238ca9f200dd1/src/clj/clojure/main.clj#L389-L390#2018-11-2208:40mkvlror is adding functionality like this the job of the tool using prepl?#2018-11-2210:19Ben HammondHi guys I'm working on a project that extends
https://github.com/juxt/edge
which daisy-chains a bunch of small dep.edn files together using
{:local/root "../edge.kick"}
I'm trying to get my head round how this plays with alias declarations
⚓#2018-11-2210:20Ben Hammondso I've just discovered that it doesn't
aliases are only sampled at the root level#2018-11-2210:22Ben Hammondwhich implies to me that I have misunderstood the intention of :local/root#2018-11-2210:22Ben Hammondis there a description of what :local/root should, and should not be used for#2018-11-2210:22Ben Hammondthat you can point me at#2018-11-2210:23Ben HammondI guess I'm trying to strong-arm deps into working like an OSGI style module system#2018-11-2210:24Ben Hammondand I'm wondering if that would be considered abusive#2018-11-2210:39rickmoynihanI’ve had similar ideas about module systems.
In particular I have a nagging feeling that integrant/duct + deps.edn could be such a thing.#2018-11-2210:44Ben Hammondyes you are probably right#2018-11-2210:44Ben HammondI think my confusion is this#2018-11-2210:45Ben Hammond:local/root dependency allows you to see the internal development gubbbins of a dependency#2018-11-2210:46Ben Hammondwhich creates the temptation that you can manipulate those other aliases#2018-11-2210:47Ben Hammondbut at some point you should expect to move that dependence to a fixed binary#2018-11-2210:48Ben Hammondtherefore the thought process should always be How would I use this if it was a binary artefact#2018-11-2216:40mfikesThe fact that :local/root behaves just like :git/url or :mvn/version with respect to aliases is a good thing#2018-11-2312:39preporHello. I want to suggest another way how to build uberjars with tools.deps. boot-tools-deps amends boot's configurations. Our approach doesn't complement uber task but replaces it.
https://gist.github.com/prepor/ad40e061287793660c9faff21de23b82
This task simply adds classpaths (and paths) from tools.deps to fileset. And it works with git and local dependencies too. The final task could look like this:
(deftask build
[]
(comp (deps-uber)
(aot :namespace '[app.core])
(jar :file "standalone.jar"
:main 'app.core)
(sift :include #{#"standalone.jar"})
(target)))
#2018-11-2319:00dominicm@alexmiller https://dev.clojure.org/jira/browse/TDEPS-17 has become a priority for me. I can't use our git fork because it's coming in transitively.
I'd like to work on it this weekend, any thoughts you've formed so far? I'm thinking that the simplest version is that something like this will work:
org.clojure/tools.namespace {:mvn/version "1.0.0" :lib pro.juxt/tools.namespace}
Version is first priority. 2 with equal versions and different libs throws an error like comparing git to mvn does.#2018-11-2402:13eoliphanthi @alexmiller, running into an issue I posted over in #clojure
https://clojurians.slack.com/archives/C03S1KBA2/p1543000601943800
Just trying to understand if this is expected behavior vs a bug, etc#2018-11-2421:04jwhitlarkI haven't seen a bash/zsh completion script for clj, but it occurs to me that a) it wouldn't be that hard, b) the project specific part could use direnv to set that portion when you entered a clojure project directory. Any input?#2018-11-2510:26dominicmI have an interesting conundrum in pack. I'm trying to suppress this output:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/dominic/.m2/repository/org/slf4j/slf4j-nop/1.6.2/slf4j-nop-1.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/dominic/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
Suppressing it for my projects is easy enough, I add exclusions to pack for slf4j-nop.
However, on projects which don't already have their own logging stack, it then results in:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
Which isn't a better result imo.
The ideal is that the -main function's in pack would tell slf4j what to use if there's multiple bindings, suppressing the warning. But I don't see an obvious API for that. Anyone else got any good ideas?#2018-11-2512:55borkdudeCommon mistake of mine:
$ clj -Srepro -Sdeps '{:deps {org.clojure/test.check {:mvn/release "RELEASE"}}}'
Error building classpath. nil
java.lang.NullPointerException
Maybe an error message could help here?#2018-11-2617:41rickmoynihanDoes anyone know if this issue likely to be resolved?
https://dev.clojure.org/jira/browse/TDEPS-82
I think it causes lein-tools-deps (which first uses -Sdescribe) to resolve dependencies unnecessarily and too soon#2018-11-2618:50Alex Miller (Clojure team)Yes, would like to fix{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-2620:01seancorfieldI patched the clojure script locally and was hoping to provide a patch to that ticket but my weekends have not been conducive to that, unfortunately.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-11-2620:01seancorfield(I'm not sure my quick'n'dirty local patch is an ideal solution to submit yet)#2018-11-2620:27Alex Miller (Clojure team)yeah, sorry I haven’t had a chance to look at it yet#2018-11-2620:27Alex Miller (Clojure team)I’m hoping to have some cycles post-conj to catch up on tools.deps stuff#2018-11-2715:45adamfreycan :jvm-opts be put at the top level of a project's deps.edn file?
like:
:jvm-opts ["-Xms512M"
"-Xmx1g"
"-Djava.io.tmpdir=./tmp"
"-Dcatalina.base=./tmp"]
#2018-11-2715:46adamfreyIt doesn't seem to be working but I can't find an example in the docs so maybe that's not a supported feature#2018-11-2715:49adamfreyah nevermind, it doesn't exist yet. I found a discussion about the same thing from a few days ago: https://clojurians-log.clojureverse.org/tools-deps/2018-11-12#2018-11-2716:00dominicmI guess it doesn't make much sense for that to exist does it?#2018-11-2716:03adamfreywhy doesn't it make sense? I would like to start a a clojure process from my project and have a bunch of jvm opts applied without having to specify them in the cli each time.#2018-11-2716:03adamfreyThere's a JIRA ticket for it already: https://dev.clojure.org/jira/browse/TDEPS-63#2018-11-2716:10dominicmbecause it works from an alias#2018-11-2716:15dominicmI'm thinking about the expectations I have, which is that :aliases is ignored from deps.edn transitively, and everything else is transitive.#2018-11-2720:39aisamuI'd say it follows the same logic as a root-level :main-opts#2018-11-2721:53dominicmThat doesn't exist either! #2018-11-2722:04aisamuPrecisely simple_smile#2018-11-2716:03adamfreywhy doesn't it make sense? I would like to start a a clojure process from my project and have a bunch of jvm opts applied without having to specify them in the cli each time.#2018-11-2716:03adamfreyThere's a JIRA ticket for it already: https://dev.clojure.org/jira/browse/TDEPS-63#2018-11-2716:29eval2020Can one require deps in the init-script?#2018-11-2716:31eval2020This succeeds: clj -e '(require (quote [clj-time.core :as t])) (prn (t/now))' but I cannot make it work from init-script.#2018-11-2716:36eval2020I was doing weird stuff: yes this works as expected#2018-11-2718:39dominicmthought this was a need trick:
#!/bin/sh
deps="{}"
if [ "$#" -gt 0 ]; then
deps="$(clojure -A:user/find-deps "
I named it replme.{:tag :div, :attrs {:class "message-reaction", :title "ok_hand"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👌")} " 8")}
#2018-11-2722:25winkywooster@dominicm that’s cool, thanks for sharing#2018-11-2810:56eval2020@dominicm tools-deps equivalent of lein-try - very nice! 👌:skin-tone-2:#2018-11-2812:33borkdudewhat does user/find-deps do?#2018-11-2812:34dominicm@borkdude delegates to https://github.com/hagmonk/find-deps#2018-11-2814:32eval2020Published something similar (ie rebel-readline + deps) earlier this year.
The above inspired me to actually document it: https://gitlab.com/eval/deps-try/tree/master#deps-try#2018-11-2816:28dominicmWhat's the :repl/try about?#2018-11-2820:28borkdudeIs there a way to get -R test + -C test + also the possibility to override -m? With -A:test I cannot do this.#2018-11-2820:30eval2020@dominicm it’s a hook from rebel that kickstarts the loading of the libs (calling (unreleased) tools-deps’ add-lib). (Must be an easier way to start a repl where you want some commands to be executed before taking input)#2018-11-2821:54dominicmThere is, edge does it#2018-11-2821:55dominicmRebel takes the same flags as clojure main#2018-11-2821:56dominicmhttps://github.com/juxt/edge/blob/master/main/aliases/rebel/edge/rebel/main.clj{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 4")}
#2018-11-2822:26kennyIs there a way to get rid of the logback.xml file that aws-maven brings in via tools-deps?#2018-11-2822:29dominicm@kenny I solved it by making a fork and opening a jira.#2018-11-2822:31dominicmhttps://github.com/SevereOverfl0w/tools.deps.alpha/tree/add-lib?files=1 I based it on the add-lib branch. I can live without s3 wagon support. #2018-11-2822:31kennyUnfortunately I need S3 support because Datomic Cloud stores its artifacts there 😞#2018-11-2822:32dominicmhttps://dev.clojure.org/jira/browse/TDEPS-108#2018-11-2822:34Alex Miller (Clojure team)I am interested in fixing that for you, after conj (although there may be new ways to do so)#2018-11-2906:09dominicmThat's great, thanks!#2018-11-2822:34dpsuttonwould make a fun unsession to fix a bug with alex miller#2018-11-2822:37Alex Miller (Clojure team)Oh yeah so fun ;){:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 8")}
#2018-11-2822:37dpsutton"work during the conference. thanks" lol#2018-11-2822:51mattyulrichHi all. I'm having an issue that looks like transitive dependency problems with a corporate repo. I have the repo listed in :mvn/repos - but it looks like transitive deps from java libraries that I reference from that repo are not resolving. When trying to resolve the deps, it seems to be looking at clojars instead. Is this a known issue or something that should be working?#2018-11-2822:55hiredmanyou might checkout the poms of those transitive deps to see if they do anything kind of odd like relocations https://dev.clojure.org/jira/browse/TDEPS-8#2018-11-2822:57hiredmanthere are a lot of issues in the tools.deps jira, so in order to discern you are experiencing a known issue or not, you may have to narrow it down#2018-11-2823:20amarHi everyone. Has anyone been able to use a private git library as a dependency in a CI environment like Google Cloud Build? The key is there and if I use git from the command line, the repo can be successfully cloned.
I'm seeing
com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available:
at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.java:120)
at clojure.tools.gitlibs.impl$fn__843.invokeStatic(impl.clj:30)
at clojure.tools.gitlibs.impl$fn__843.invoke(impl.clj:28)
at clojure.lang.Delay.deref(Delay.java:42)
#2018-11-2906:10dominicmYou need to start an ssh agent, and add the key to it#2018-11-2921:06amarGreat. Thanks!#2018-11-2900:00mattyulrichThanks @hiredman - don't see a dependency relocation, but it does look like a good list of Jiras to review. I'll dig into there and see if I can see my issue.#2018-12-0117:59pesterhazyLooks like clj main-opts don't compose. For example, I tried to define two aliases that both set main-opts: https://github.com/pesterhazy/advent2018/blob/e0f8fba23339052eb112958fed712da97c7c270b/deps.edn#L3#2018-12-0118:01pesterhazyUnfortunately with clj -A:nrepl:in-ns, the -e (...) argument for the second alias overrides the -e (...) of the first#2018-12-0118:01pesterhazyIs this intended behavior? Is there another way for an alias to execute code?#2018-12-0118:02pesterhazyNote that clj -A:nrep -e (...) does work#2018-12-0118:21dominicmNon composition is expected. #2018-12-0118:26pesterhazyThose are two enhancements to clj that would make aliases more useful then: avoiding the need for Corfield Commas, and being able to stack side effects (like starting an nREPL server) on top of each other#2018-12-0121:46Alex Miller (Clojure team)You can stack effects with -m - main args are appended to the end. But, there are other things going on in your example above (no support for -e aliases, ordering of -A vs other aliases, etc)#2018-12-0211:18dominicmhttps://clojure.org/reference/deps_and_cli
> If multiple -M alias maps are activated, only the last one will be used#2018-12-0211:23pesterhazyso in this case it's better to use -M:alias1:alias2 than -A:alias1:alias2?#2018-12-0211:48dominicmThe former would have no meaning #2018-12-0211:48dominicmWell, the former is equivalent to -M:alias2#2018-12-0211:49pesterhazyOk, I'll give that a try#2018-12-0211:50dominicmI opened an nrepl issue to start a discussion about how this might work. I've not come up with anything good. #2018-12-0211:58pesterhazy@U09LZR36F can you link to the issue?#2018-12-0212:05dominicmhttps://github.com/nrepl/nrepl/issues/54#2018-12-0212:05dominicmI should update it to mention the actual problem statement#2018-12-0212:05dominicmI did think about a main Hopper #2018-12-0212:06dominicmA simple version could be put together in an afternoon#2018-12-0214:27Alex Miller (Clojure team)sorry, I meant you could stack effects from last alias + command line, not across aliases. I think making ordering do stuff beyond that is too fragile#2018-12-0214:28Alex Miller (Clojure team)the order of “last” should be unambiguous (but is currently not, and that’s a bug (https://dev.clojure.org/jira/browse/TDEPS-100 which may be irritating to fix)#2018-12-0214:29Alex Miller (Clojure team)at some point, you are putting too much on CLJ and should recognize that you are writing a program and push this complexity into a Clojure program I think#2018-12-0214:29Alex Miller (Clojure team)but I’m willing to consider that I’m wrong on that :)#2018-12-0215:06dominicmIt's a little bit complicated, because I think mains have been pushed as a common interface. But they don't compose exactly.
But anything you want to do, can be done with jvm opts and a custom main.#2018-12-0321:26pesterhazyclojure -Sdeps '{:aliases {:a1 {:main-opts ["-e" "(println,:a1)"]} :a2 {:main-opts ["-e" "(println,:a2)"]}}}' -Srepro -M:a1:a2 will print out only :a2#2018-12-0321:26dominicmCorrect.#2018-12-0321:27dominicmBest bet is a custom main, and use jvm opts to compose.#2018-12-0321:27pesterhazyAlso you can only ever specify one -m/--main option#2018-12-0321:28pesterhazyTo me, those two facts combined diminish the usefulness of aliases, because you often need to produce a side effect to make an alias effective#2018-12-0321:28pesterhazyAs an example, to start an nREPL server, you need to call nREPL's start-server function#2018-12-0321:29dominicmYep! This is a limitation from Clojure itself, inherited from Java. I've given this some thought, and I don't know what multiple mains look like.
In parallel presumably? #2018-12-0321:29pesterhazySo one person in the team uses nREPL so they, so they run,e e.g. clojure -M:nrepl#2018-12-0321:31pesterhazyThat to me is how composition of aliases would help - in different situations you can enable different sets of features#2018-12-0321:32pesterhazyUsing :main-opts with ["-e" "(do (require ...) (start-server))]" is a little crude but works-ish - except for the fact that main-opts don't compose#2018-12-0321:33dominicmYou can call functions, as long as they exit, but nrepl main doesn't, else it would be fairly useless as a main#2018-12-0321:33pesterhazyIf not through aliases, I'm not sure how I would accomplish the same thing#2018-12-0321:34dominicmEdge has a workaround for this already, it looks at the system properties, and loads things based on flags there.#2018-12-0321:35pesterhazydo jvm opts compose?#2018-12-0321:35dominicmYep! That's what I was trying to get across 😊 #2018-12-0321:36pesterhazyGotcha. It seems a little arbitrary though that jvm-opts get concatenated, whereas main-opts are replaced#2018-12-0321:39dominicmJvm opts naturally concatenate in the underlying system. I think they're unordered.
Main opts is talking about clojure's main-opts though.
You wouldn't be able to do what you want with nrepl and other mains even if concatenation happened. #2018-12-0321:39pesterhazywhy?!#2018-12-0321:40pesterhazythis does work https://github.com/pesterhazy/advent2018/blob/e0f8fba23339052eb112958fed712da97c7c270b/deps.edn#L6#2018-12-0321:41pesterhazyis the problem with concatenating main-opts that the ordering of different aliases is not well defined?#2018-12-0321:43pesterhazyhere's where the behavior is defined btw https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha.clj#L26#2018-12-0321:52pesterhazy@U09LZR36F I guess your idea is to use an alias like {:jvm-opts ["-Dclojure.run.foo=foo.core"]} and then have a main that scans through system properties and calls those in some order?#2018-12-0322:01dominicm@U06F82LES somrthing like that, yeah#2018-12-0322:02dominicm@U06F82LES nrepl.main blocks forever. So rebel would not run afterwards.#2018-12-0322:10pesterhazynrepl.server/start-server doesn't block#2018-12-0322:11dominicmYou're thinking of clojure mains opts, I'm thinking of plain mains. My mistake.#2018-12-0322:12dominicmI don't think you can pass -e and -m anyway.#2018-12-0400:08Alex Miller (Clojure team)Something that we have discussed is a “start” option that would invoke an arbitrary function#2018-12-0118:28pesterhazyCorfieldization is a fun exercise but a clearer way to achieve this effect would be great#2018-12-0118:57dominicmThe comma thing is a bug afaik#2018-12-0120:50seancorfieldI'm sure that if someone can come up with patches to the clojure shell script that eliminate Corfield, lots of people will be very happy. Me included.#2018-12-0121:12dominicmFixing it is difficult I would guess. You have to figure out how to generate properly escaped strings from edn.#2018-12-0121:27seancorfieldMainly you need to be able to round-trip arbitrary strings from bash to file to bash again.#2018-12-0121:27seancorfieldPortably 🙂#2018-12-0121:41dpsuttonWill the corfield comma work in a powershell implementation of clj?#2018-12-0121:44Alex Miller (Clojure team)Sean’s description is pretty good, happy to have someone figure that out#2018-12-0121:44Alex Miller (Clojure team)I suspect powershell is prob easier#2018-12-0121:49dottedmagAt least it's bash, not an arbitrary POSIX shell.#2018-12-0121:54Alex Miller (Clojure team)One issue is with nested quotes, another is needing to write the second half in bash (first half is in Clojure)#2018-12-0121:55dottedmag@alexmiller if you can give me an example of what needs to be done, I'll try to stretch my shell muscles.#2018-12-0122:25Alex Miller (Clojure team)Sorry, boarding a flight but there is a ticket in TDEPS#2018-12-0122:27seancorfieldhttps://dev.clojure.org/jira/browse/TDEPS-56#2018-12-0122:27seancorfield(there may be others but that's the first one I found)#2018-12-0122:29seancorfield@dottedmag ^ yup, that seems to be the whole issue.#2018-12-0123:23pesterhazyI’ll take a look as well#2018-12-0211:04pesterhazyI'm trying to tackle TDEPS-56, but am having trouble building the project. I've discovered two repos, brew-install and tools.deps.alpha. In the former I can run cd src/main/resources && bash -x install.sh ~/tmp
but that fails because *.jar is missing#2018-12-0211:06pesterhazyPresumably the jar is built from the tools.deps.alpha repo, but am I supposed to copy that over to brew-install/src/main/resources ?#2018-12-0211:21pesterhazyNext up, I'm running into this issue: bin/clojure: line 174: $install_dir/libexec/clojure-tools-${project.version}.jar: bad substitution#2018-12-0211:22pesterhazy${project.version} is not valid Bash syntax so that must be related to the maven "filtering" feature#2018-12-0211:43pesterhazylooks like I was confused - master in the brew-install repo is an outdated branch#2018-12-0214:08pesterhazyok figured it out - in a nutshell you need to go through the package.sh script and before that mvn install tools.deps.alpha (also point the pom to the snapshot version you're testing)#2018-12-0214:31Alex Miller (Clojure team)yes, exactly#2018-12-0214:09pesterhazyAdded a patch: https://dev.clojure.org/jira/browse/TDEPS-56?focusedCommentId=50847#comment-50847 cc: @seancorfield @dottedmag#2018-12-0317:38seancorfield@U06F82LES I share Alex's concern about both the eval aspect and the lack of compatibility with existing cached .main files -- I wonder if the script could verify the file contents, ensuring it was all \c\h\a\r\a\c\t\e\r\s and only calling eval in that case? I think that would retain compatibility with existing files and also address the security issue? /cc @U064X3EF3#2018-12-0317:43Alex Miller (Clojure team)I think we should not use eval. I think probably it would be best to read the file as an array and then array splat it into the command line. I think that largely fixes the spacing problem, but not sure what still needs to be solved for quoting.#2018-12-0317:44pesterhazyhappy to try a different tack#2018-12-0317:45pesterhazyto read the file into an array from dash, we'll need to use a separator#2018-12-0317:45pesterhazywere you thinking of separating arguments by newline?#2018-12-0317:46Alex Miller (Clojure team)do you need a separator? I thought space was the separator#2018-12-0317:47pesterhazyright, then we'll need to escape space#2018-12-0317:48pesterhazyso we'd write this -co {:aot-cache\ true}
#2018-12-0317:49Alex Miller (Clojure team)do you? the whole idea here is to write the string with spaces, then read it back into an array (breaking on spaces), then drop that into the args list#2018-12-0317:50Alex Miller (Clojure team)I think that’s actually fine and the issue is really what happens when you have Clojure quoted things inside the args#2018-12-0317:50pesterhazyhere's how I'm thinking it could it could work:
- in the clojure code, we'd replace " " with "\\ " and "\\" with "\\\\"
- in the bash code, we'd read the file char by char, replace "\\ " with a space, replace "\\\\" with "\\" and else when encounter an unescaped space, start a new argument#2018-12-0317:51pesterhazywe'd basically implement our own quote/unquote scheme#2018-12-0317:52pesterhazythe main point is that space-as-arg-separator and space-as-literal-space need to be distinguished in the bash code#2018-12-0317:56Alex Miller (Clojure team)newline separator might also be a better choice - I think you may be able to automatically get some of what we want by changing the IFS too#2018-12-0317:57Alex Miller (Clojure team)ideally, I think the main file should have as little markup as possible#2018-12-0317:57pesterhazysure, though the main-opts strings may contain newlines as well#2018-12-0317:58Alex Miller (Clojure team)the less magic that’s in the Clojure code the better. we’re just creating portability problems the more coordination is required between these two pieces of code#2018-12-0317:58pesterhazyagreed#2018-12-0317:58pesterhazymaybe keep a Windows version in mind as well#2018-12-0317:58Alex Miller (Clojure team)I think new lines in the args is much less common than multi-word / quoting#2018-12-0317:59Alex Miller (Clojure team)I am keeping Windows in mind :) (I suspect powershell has better options for this stuff though)#2018-12-0318:00Alex Miller (Clojure team)I need to do some other stuff, checking out#2018-12-0318:00pesterhazyeven newlines aren't common, it shouldn't break if they do appear#2018-12-0318:00pesterhazyi'll give it another try#2018-12-0319:17pesterhazyUsing newlines as arg separators, I wrote an explode Bash function that takes a string and generates an array: https://github.com/clojure/brew-install/pull/3/files#diff-b3212e45e19f61de4754a755466b793f - it allows you to escape newlines by prefixing them with a backslash. So the file format is simple.#2018-12-0319:17pesterhazyIf that looks like the right direction, I'll prepare a new patch#2018-12-0314:37jeroenvandijkI'm having an issue creating an uberjar using tools.deps with boot-clj via boot-tools-deps. The clj files of my local and git dependencies are not included if I don't AOT these files. Is this the right place for such an issue?#2018-12-0314:38jeroenvandijkI've created an example project with the issue here https://github.com/jeroenvandijk/tmp-boot-tools-deps-issue#2018-12-0314:38jeroenvandijkThe files https://github.com/jeroenvandijk/tmp-boot-tools-deps-issue/blob/master/uberjar.txt#L21-L23 are in the fileset, but not in the jar#2018-12-0316:19seancorfieldMy advice would be to use one of the uberjar utilities designed for deps.edn instead of Boot. #2018-12-0316:22seancorfieldboot-tools-deps was intended as a simple bridge for starting with a Boot project and moving the primary dependencies into deps.edn - there are several inherent limitations of the approach. #2018-12-0316:23seancorfieldAt work, we moved off Boot completely. #2018-12-0316:23seancorfield@jeroenvandijk ^#2018-12-0316:30jeroenvandijk@seancorfield Ah good to know, thanks 😬. Out of curiosity what are you using for uberjars?#2018-12-0316:31dominicm@jeroenvandijk I'll plug pack.alpha which I authored. Sean maintains a depstar fork.#2018-12-0316:33jeroenvandijk@dominicm cool thanks. What's the best choice for creating uberjar's? Capsule, onejar?#2018-12-0316:34jeroenvandijk(btw I tried pack.alpha, but it seems to force me to choose a main namespace as jar entry. Something i'm not used to with lein and boot-clj)#2018-12-0316:35dominicm@jeroenvandijk capsule works in more contexts. I'm surprised you say that, because pack works on the opposite, clojure.main is the default unless you set it.#2018-12-0316:36jeroenvandijkok i'll have to try again 🙂 Maybe i gave up too quickly#2018-12-0316:36jeroenvandijkThanks, i'll come back with the results tomorrow#2018-12-0316:37dominicm@jeroenvandijk feel free to bother me if something doesn't work.#2018-12-0316:38jeroenvandijkawesome, thanks!#2018-12-0316:52seancorfield@jeroenvandijk My fork of depstar -- https://clojars.org/seancorfield/depstar#2018-12-0316:58jeroenvandijkthanks!#2018-12-0316:53seancorfieldWe create all our production artifacts with that (and run them with java -cp path/to/the.jar clojure.main -m entry.point for them to start at entry.point/-main).#2018-12-0412:30borkdudedoes tools.deps support git deps that have an edn file in a subdirectory? do you have to reflect this in the git url?#2018-12-0412:30borkdudee.g. my-project/lib2, my-project/lib2#2018-12-0412:30dominicm@borkdude you can use :deps/root to specify a sub-directory{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 8")}
#2018-12-0412:30borkdudeperfect!#2018-12-0417:32shaun-mahoodSpecifically for use with REBL and openjdk, is there a way to tell REBL bring in the correct version of JavaFX as part of the extra-deps? I don't actually need it at the moment (I switched to the Java version to get it to work), but it seems like it would be useful to prevent problems (I ran into it on 2 of my machines that I use regularly for Clojure work).#2018-12-0417:33seancorfieldMaybe ask in the #rebl channel?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-12-0420:48dottedmagIt's a pity that existing .main-files are lossy: both arguments are separated by spaces and spaces within arguments are written as spaces.#2018-12-0420:49dominicm@dottedmag you mean via :main-opts?#2018-12-0420:49dottedmagAny idea how to invalidate those? I could use readarray Bash builtin to read \0-delimited file into a Bash array.#2018-12-0420:49dottedmag@dominicm yes.#2018-12-0420:52dottedmagHmm, I can change $ck, but that invalidate all existing cached files, no matter if :main-opts had had spaces in them or not.#2018-12-0420:53dottedmag@alexmiller ^ would that be acceptable?#2018-12-0420:55dominicmhttps://dev.clojure.org/jira/browse/TDEPS-56 🙂 some work is happening now. Ideas are being thrown around. \0 delimiters might be interesting, but can't parameters have \0 in them?#2018-12-0420:56dottedmag@dominicm That's the ticket I'm working on. No, Bash cannot keep \0 in variables or arguments.#2018-12-0420:56dottedmagThere is even a hack in Bash to use '' as a "delimiter" to denote "use \0 as a delimiter".#2018-12-0420:56dominicmDidn't realize you were down in the trenches with this 🙂#2018-12-0420:57dottedmagAlso, execve(2) API accepts NUL-terminated strings, so no way \0 can be passed through.#2018-12-0420:58dottedmagWell, the patch itself is not so complicated. The only remaining issue is backward compatibility with previously written .main files.#2018-12-0420:59dominicmOn a personal note, it seems to me that updating clj may/should invalidate all my caches#2018-12-0421:07Alex Miller (Clojure team)it is supposed to, as the install-level deps.edn is included in the cache calculation#2018-12-0421:08Alex Miller (Clojure team)but “invalidate” != “delete”#2018-12-0421:08Alex Miller (Clojure team)as the idea of writing a script that goes around rm’ing your dirs scares the crap out of me#2018-12-0421:09Alex Miller (Clojure team)using a .main2 seems perfectly acceptable too#2018-12-0421:15timgilbertI wonder if this would be usable for the string-splitting problem: http://mywiki.wooledge.org/BashPitfalls#hosts.3D.28_.24.28aws_....29_.29#2018-12-0421:16timgilbert...or if it would be acceptable to fall back to a tool written to handle this kind of thing, like awk or something#2018-12-0421:18timgilbert...or if the string could be written to .main separated by commas, and then read into the array on the bash side via readarray -d ,#2018-12-0421:43Alex Miller (Clojure team)would prefer to a) not add any additional deps if at all possible and b) have the clojure program write something as dumb and obvious as possible to the .main file#2018-12-0421:56dottedmag@alexmiller Only locations of install-level deps.edn files are included in the key of cache, not their contents, right?#2018-12-0421:57pesterhazyi'd like to point out that I've put some thought into this and proposed a solution for quoting: https://github.com/clojure/brew-install/pull/3/files#diff-b3212e45e19f61de4754a755466b793f#2018-12-0421:57pesterhazy(and submitted a first pass at a patch)#2018-12-0421:58dottedmag@alexmiller How about this one? Or I can do .main2, it's even simpler.#2018-12-0421:59pesterhazyif I'm not on the right track, happy if others take a swing at this, but I wouldn't want effort needlessly wasted if possible#2018-12-0421:59pesterhazydiscussion here: https://clojurians.slack.com/archives/C6QH853H8/p1543759744032700#2018-12-0422:00dottedmag@pesterhazy Seems much easier to do \0-based output and read back using readarray like this: - main_cache_opts=($(cat "$main_file"))
+ declare -a main_cache_opts
+ readarray -d '' -t main_cache_opts < "$main_file"
#2018-12-0422:00dottedmagThe only remaining change is to ignore .main and and read/write .main2 instead.#2018-12-0422:01dottedmag@pesterhazy t.d.a change is also trivial: --- a/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj
+++ b/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj
@@ -76,7 +76,7 @@
(when (.exists jf)
(.delete jf))))
(if-let [main-opts (seq (get (deps/combine-aliases deps-map (concat aliases main-aliases)) :main-opts))]
- (io/write-file main-file (str/join " " main-opts))
+ (io/write-file main-file (str/join "\u0000" main-opts))
(let [mf (jio/file main-file)]
(when (.exists mf)
(.delete mf))))))#2018-12-0422:02dottedmag@pesterhazy I have to go right now, so I can't finish it at the moment.#2018-12-0422:03Alex Miller (Clojure team)sorry, I don’t have time to work on this atm but will hopefully have a slice of deps time in a day or two#2018-12-0422:03dottedmag@pesterhazy I don't mind if you pick it up from here, if you think this approach is good enough.#2018-12-0422:04Alex Miller (Clojure team)if there are two (or N) options, that’s fine, just describe the tradeoffs in a table (and separate the file impl aspect from the “not breaking existing stuff” aspect - those I think are orthogonal)#2018-12-0422:07pesterhazythe obvious downside of readarray is that it's not available in bash3, which is what ships with macOS#2018-12-0422:07pesterhazyI think that rules it out#2018-12-0422:08dottedmagOh, too bad.#2018-12-0422:10pesterhazyalso not sure how reliable reading NUL character in bash is - that's the kind of thing where bash if flaky#2018-12-0422:11pesterhazyaccording to this https://stackoverflow.com/a/6571549/239678: "In Bash, you can't store the NULL-character in a variable."#2018-12-0422:12dottedmagThat's why they have readarray in first place.#2018-12-0422:12pesterhazymany languages with C strings have trouble with NUL characters in strings, for the obvious reasons#2018-12-0422:12dottedmagAnd special case for -d '' there#2018-12-0422:13dottedmagread -r -d '' should work with an explicit loop.#2018-12-0422:13dottedmagAs of now all NUL-characters are stripped from .main when read by Bash, so no backward compatibility problems here.#2018-12-0422:14dottedmagUnless someone relied on NULs being stripped 😄#2018-12-0422:14pesterhazywell try it - I didn't have luck reading the "\\ " character sequence using read#2018-12-0422:14pesterhazyno it's not backward compatible - it would turn potentially multiple arguments into a single argument, i.e. change the semantics#2018-12-0422:15pesterhazyI don't think there's a way around using a different file extension#2018-12-0422:16dottedmag@pesterhazy $ echo '00000000: 310032003300' | xxd -r | while read -r -d '' i; do printf ">$i<\n"; done
>1<
>2<
>3<
#2018-12-0422:18dottedmag@pesterhazy By backward compatibility I meant backward compatibility for code that relied on \u0000 passed from deps.edn unchanged to the app. But that didn't work, so we don't need to care about it.#2018-12-0422:27pesterhazy@alexmiller I've updated the ticket with a summary#2018-12-0423:41dominicmThe ticket doesn't mention back porting readarray, https://stackoverflow.com/a/49322365 has some hints. Would this work? #2018-12-0503:50baritonehandsI created a PR for more Deps/CLI docs: https://github.com/clojure/clojure-site/pull/315#2018-12-0503:50baritonehandsnot sure what the process is for evaluating these#2018-12-0503:52seancorfieldMy simple guide for AOT: do not do it.{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-12-0503:52seancorfieldAOT is both unnecessary and a giant pain in the ...#2018-12-0503:57seancorfield(for all my snarkiness, the tip about needing to create classes and having it on your classpath is a good one -- it bit me when I tried to manually compile stuff)#2018-12-0503:58baritonehandsas much as it is not preferred, it is still necessary in some cases#2018-12-0503:58seancorfield@gfredericks mentioned the classes thing in his talk at Conj -- I think it's a widely not-known thing 😞#2018-12-0503:59seancorfieldYeah, AOT is a necessary evil in some very rare cases. Knowing how to successfully compile stuff when building JAR files, for example.#2018-12-0504:00seancorfieldDon't worry, I'm just very grumpy about AOT 🙂#2018-12-0504:02baritonehandsAh @gfredericks fellow Chicagoan{:tag :div, :attrs {:class "message-reaction", :title "raised_hands"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙌")} " 8")}
#2018-12-0504:17seancorfieldHave you watched his talk from Conj? Definitely worth it.#2018-12-0504:18baritonehandsNot yet, still working my way through all of them#2018-12-0504:18baritonehandsgot distracted by #adventofcode lately#2018-12-0504:19seancorfieldAh, I've avoided that every year. https://www.youtube.com/watch?v=-Qm09YiUHTs&index=25&list=PLZdCLR02grLpMkEBXT22FTaJYxB92i3V3&t=0s for the direct link. Really good stuff!#2018-12-0609:28kirill.salykinhi,
I am trying to build skinny jar with pack and got this error:
Exception in thread "main" java.lang.IllegalArgumentException: No matching clause: [nil :jar]
at mach.pack.alpha.skinny$run_steps.invokeStatic(skinny.clj:107)
at mach.pack.alpha.skinny$run_steps.invoke(skinny.clj:99)
at mach.pack.alpha.skinny$_main.invokeStatic(skinny.clj:175)
at mach.pack.alpha.skinny$_main.doInvoke(skinny.clj:158)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
#2018-12-0609:28kirill.salykinbut jar is created#2018-12-0609:29kirill.salykinplease advice what I am doing wrong#2018-12-0609:29kirill.salykinpack version:
{pack/pack.alpha {:git/url ""
:sha "ddbbdbd4001c75647deddaad0f8bb7dc954fcb6f"}}
#2018-12-0609:32kirill.salykinUPD: I switched to latest version in master and everything seems to work#2018-12-0609:36kirill.salykinone thing tho - skinnyjar stucks if target directory contains prev version…#2018-12-0609:41dominicm@kirill.salykin master is a pretty big rewrite of pack. This bug isn't terribly surprising 😊
What do you mean by "contains previous version"?#2018-12-0609:41kirill.salykinif you run it twice in a row#2018-12-0609:41kirill.salykinrm -rf ./target/*
clojure -A:pack mach.pack.alpha.skinny
#2018-12-0609:42kirill.salykinthis fixes it#2018-12-0609:42dominicm@kirill.salykin what is the output in the case that it gets stuck? I expect a thread will error something? #2018-12-0609:42kirill.salykinno output, just takes a lot of time#2018-12-0609:42kirill.salykinmore then 1 minute on machine#2018-12-0609:42kirill.salykini was not patient to wait longer…#2018-12-0609:42dominicmHmm. I'll try and debug that today. I'm not sure what could cause that.#2018-12-0609:43kirill.salykinif you clean the target directory - everything works perfectly#2018-12-0609:43dominicmMaybe there's something accidentally recursive.#2018-12-0609:45kirill.salykinshall I create the issue?#2018-12-0610:22dominicmFeel free, I'll try and get round to it today 🙂#2018-12-0610:23kirill.salykinwill do, thanks for your help!#2018-12-0610:42dominicm@kirill.salykin I'm unable to reproduce what you're seeing 😟 I might need some more context, maybe it only happens under certain deps.edn files or something?#2018-12-0610:42kirill.salykinI’ll try to create some minimal repro case#2018-12-0610:43kirill.salykinthanks for looking#2018-12-0610:43dominicmI'll mirror this on github 🙂 Not sure which channel you prefer for tracking this#2018-12-0610:44dominicmfwiw, I tested against pack itself like this: clojure -m mach.pack.alpha.skinny --lib-dir /tmp/lib --project-path skinny.jar#2018-12-0610:47kirill.salykinmaybe it is something in our project#2018-12-0610:47dominicmWhat does your :pack alias look like?#2018-12-0611:01kirill.salykinhm
:pack {:extra-deps {pack/pack.alpha {:git/url ""
:sha "4a00565dce29225a1dd2a810c438b90b14f9e111"}}
:main-opts ["-m"]}}}
#2018-12-0611:01kirill.salykinI am bit confused with empty main-opts#2018-12-0611:02kirill.salykinlemme try w/o main-opts at all#2018-12-0611:02dominicm@kirill.salykin it really just saves you from typing -m 😄#2018-12-0611:02kirill.salykinso it doesnt matter?#2018-12-0611:03dominicmnot particularly 🙂#2018-12-0611:11kirill.salykinworks on empty project
I assume it is something within my project#2018-12-0611:36dominicmif you can track down what's within your project, I'd be interested#2018-12-0612:01kirill.salykinI’ll try, but later. It seems not urgent, also it is nice to clean target directory anyway (for outdated libs for instance)
so not with rm -rf target/* it just works#2018-12-0612:02kirill.salykinbtw, pack dies if target directory not created#2018-12-0614:16kirill.salykinQuestion regarding juxt/pack, do I understand correctly that using --lib-type keep with skinny-jar should keep clojure file in jar?
the issue I see now that git dependency packed as empty jar (there is no gen-class) and clojure complaints that can find file on classpath#2018-12-0614:17dominicm@kirill.salykin --lib-type keep means that any dependencies that are jars, will stay as jars, and git dependencies which are directories, will stay as directories.#2018-12-0614:19kirill.salykinis it by design that git dependency packed into empty jar? or do I missing something in config?#2018-12-0614:20dominicmthat sounds wrong. I haven't seen that in my testing#2018-12-0614:21kirill.salykinwill try to create something that can reproduce the issue#2018-12-0614:21kirill.salykinthanks!#2018-12-0614:25dominicm@kirill.salykin my mistake, I've reproduced#2018-12-0614:25dominicmworking on it#2018-12-0614:25kirill.salykinThanks!#2018-12-0614:27mccraigmccraigi've got a lein-modules based project in a monorepo which has common deps in the top-level project.clj and per-module deps in each module subdirs own project.clj. it builds and tests all modules in a sensible (poset) order and correctly outputs versioned jars for each module - is there a way of achieving similar behaviour for deps.edn based project ?#2018-12-0614:27dominicm@mccraigmccraig what's poset order? I would just loop over each directory with bash & use pack 😄#2018-12-0614:28mccraigmccraigit figures out the correct order itself, based on the dependency graph#2018-12-0614:28dominicm@kirill.salykin I've figured it out, dumb typo#2018-12-0614:33dominicm@kirill.salykin Try 24863d1ff5a54bb7cc783ff62272212b51c8b316#2018-12-0614:35kirill.salykinwow, it blazingly fast, thanks!
trying#2018-12-0614:36dominicm@kirill.salykin I go on holiday soon, and I've wrapped up work 🙂 I have a few spare cycles.#2018-12-0614:38kirill.salykinyay! works! thanks#2018-12-0614:39dominicmGreat 🙂#2018-12-0615:09kirill.salykinbtw, stuck issue seems fixed as well#2018-12-0615:14dominicmoh, hmm :thinking_face:#2018-12-0615:16dominicm¯\(ツ)/¯#2018-12-0619:13rpkarlssonI ran into some trouble using java9+ and Cider with deps.
Deps builds abbreviated paths ("src", as specified in :paths) instead of full paths ("/home/rpkarlsson/Projects/mc-kartan/src/") as other classpath builders seem to do.
Cider with java1.8 and lower determines the classpath in a different manner and is not effected by this.
Are abbreviated paths something that could potentially lead to other problems and because of that should be fixed in deps or should abbreviated paths be expected in tooling?
More info at:
https://github.com/clojure-emacs/cider/issues/2533#2018-12-0622:34dottedmagComments, sugestions? https://dev.clojure.org/jira/browse/TDEPS-56?focusedCommentId=50940&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-50940#2018-12-0622:35dottedmagI've got the spaces work in :main-opts#2018-12-0623:15timgilbertLooks reasonable to me, personally, though I'd prefer newlines to NUL characters.#2018-12-0623:40pesterhazy ¯\(ツ)/¯#2018-12-0623:41pesterhazyI’m not playing the game of uploading competing patches#2018-12-0623:45Alex Miller (Clojure team)if there are competing approaches, please summarize the tradeoffs#2018-12-0623:45Alex Miller (Clojure team)I need that to make decisions and I will have a slice of time to do so soon#2018-12-0701:05dottedmag@timgilbert Newlines -> escaping comes back. Not good.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-12-0717:17donaldballWhat tools do folk like to use to build and publish jar artifacts to maven repos, particularly private s3 maven repos?#2018-12-0717:31dominicmhttps://juxt.pro/blog/posts/pack-maven.html{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-12-0717:50seancorfieldAt work we use https://github.com/seancorfield/depstar to build our deployment uberjars and I also use it now to build "thin" JARs to deploy on Clojars.#2018-12-0717:51seancorfieldFor the actual "push", I use mvn deploy:deploy-file ...#2018-12-0717:52seancorfield(the same, nasty invocation that @U09LZR36F has in his blog post, wrapped up in a shell script -- that also updates pom.xml with the SHA of the latest commit so that Clojars can link back to the commit that created the release)#2018-12-0717:57donaldballThanks for the pointers!#2018-12-0720:23timgilbertFWIW, I'm working on a tool to post-process the results of clojure -Spom to add in the groupId/artifactId and pull the version from a git tag, since checking in partly-generated stuff into source control bugs me#2018-12-0720:25timgilbert(with the goal of automating everything up to the jar creation and then using pack for the rest)#2018-12-0719:14harriganIf I create a pom.xml with the following dependency:
<dependency>
<groupId>org.lmdbjava</groupId>
<artifactId>lmdbjava</artifactId>
<version>0.6.1</version>
</dependency>
and run mvn dependency:tree I see lmdbjava with 10 transitive dependencies. If I create a deps.edn with the same dependency:
{:deps {org.lmdbjava/lmdbjava {:mvn/version "0.6.1"}}}
and run clj -Stree I see 9 transitive dependencies. com.github.jnr:jffi:jar:native:1.2.16:runtime is missing. Could this be an issue with tools.deps?#2018-12-0719:23seancorfieldSounds like https://dev.clojure.org/jira/browse/TDEPS-50 maybe?#2018-12-0719:28harriganAh that’s it exactly — thanks for the link.#2018-12-1014:10Brendan van der EsHi all, question: Suppose I'm in a REPL in the middle of a datomic session with large cache in memory and I need to pull in a new library to continue processing. Is 'add-lib' the only way to pull in this dependency without losing the in-memory state of the jvm session? In other words, is there any way to have the CLI preserve this session state in memory in between reboots for new deps? Assuming the answer to this is a negative and suppose 'add-lib' fails, any other ways to preserve datomic memory cache between repl reboots?#2018-12-1015:55Ben Hammondis https://docs.datomic.com/on-prem/valcache.html any good for you?#2018-12-1015:56Ben HammondI know its probably too late for this repl session ...#2018-12-1014:19Alex Miller (Clojure team)probably a better question for #datomic but afaik, the answer is no#2018-12-1014:36Brendan van der EsYea, was afraid of that. I'm exploring ideas for a more datomic-centric repl experience but would need to make this kind of reloading seamless. Thanks for the quick response; will try #datomic.#2018-12-1015:03jeremysHello, did someone try to install clojure with nix on os x? I am not using homebrew atm and I was wondering if the clojure package from nix included the necessary stuff to run tools deps and cli#2018-12-1015:32Alex Miller (Clojure team)don’t know - it’s not built or maintained by the core team#2018-12-1015:42dominicm@jeremys I don't know package it is, but one of them does include tools deps.#2018-12-1015:43jeremys@alexmiller Ok thx!#2018-12-1015:44jeremys@dominicm looking at it there https://github.com/NixOS/nixpkgs/blob/b352d47e42b5babe82d937b2d5c77476b663dd88/pkgs/development/interpreters/clojure/default.nix it fetches https://download.clojure.org/install/clojure-tools-${version}.tar.gz#2018-12-1015:47jeremysI am thinking it fetches the necessary tools, I was about to take a look at the brew formulae to get a sense where things might differ but the tool part of the url on clojure’s website and the deps mention in the nix package leads me to think it’s there. Thx.#2018-12-1015:47Alex Miller (Clojure team)seems fine. that’s a couple versions back but what it does is intentionally pretty dumb.#2018-12-1015:48Alex Miller (Clojure team)the brew recipe is https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/clojure.rb but it mostly calls the install script https://github.com/clojure/brew-install/blob/1.9.0/src/main/resources/install.sh#2018-12-1015:49jeremysI’ll take a look! thank you.#2018-12-1016:26jeremysJust for information the nix package comes with the tools deps machinery.#2018-12-1022:41rschmuklerHey all 🙂 . Is it possible to use my deps.edn to point to a github repo that is using a leiningen project.clj? I naively tried pointing at it but got hit with an error regarding detecting the manifest. Makes sense, and I realize it might be out of the scope of what tools.deps is trying to achieve... but also feels quite useful#2018-12-1022:45seancorfieldtools.deps, which is what does the dependency resolution under the hood, doesn't know how to use a project.clj file. Perhaps open an issue against that repo and see if they'll add a deps.edn file?#2018-12-1022:46Alex Miller (Clojure team)or if it has no transitive deps, you can force it to be a deps.edn project by putting :deps/manifest :deps in the coordinate#2018-12-1022:47seancorfieldPresumably, you could do that anyway and just add that project's deps to your own deps.edn file anyway @alexmiller?#2018-12-1022:47Alex Miller (Clojure team)yes, that should work#2018-12-1022:47Alex Miller (Clojure team)sneaky! :)#2018-12-1022:47rschmuklerThanks for the quick replies! Sadly, it's of almost no value but for its transitive dependencies (which I needed to update). I ended up just excluding its (outdated) dependencies and then manually updating the dependencies in my deps.edn#2018-12-1022:48rschmuklerThanks again for the fast responses#2018-12-1110:23borkdudeIs there an overview of how people can get a REPL in their editor for a deps.edn based project (without adding nREPL-specific deps to the deps.edn, so by setting up an alias in their profile)?#2018-12-1110:55orestisAh, you’re looking for a proper guide, not just an alias?#2018-12-1110:55orestisHere’s my cider-nrepl alias, FWIW: :cider-nrepl {:extra-deps {nrepl/nrepl {:mvn/version "0.4.5"}
cider/cider-nrepl {:mvn/version "0.18.0"}}
:main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]}#2018-12-1110:56orestisI start my REPL by doing clojure -A:cider-nrepl and then connect CIDER to it via cider-connect. I think that any nREPL-aware editor could do the same?#2018-12-1111:13borkdudeyeah, maybe I could just add that to the README#2018-12-1117:10richiardiandreaI do the same the ClojureScript one is very similar#2018-12-1117:11richiardiandreaMore info here: https://nrepl.xyz/nrepl/usage/server.html#2018-12-1117:12borkdudecool, that’s the page I was looking for#2018-12-1117:12borkdudeI guess it has to be CIDER agnostic since some people use VSCode or whatnot#2018-12-1117:14borkdudehttps://github.com/borkdude/advent-of-cljc/blob/master/README.md#dev#2018-12-1117:14richiardiandreaYep well cider is just a middleware. Also keep in mind that Calva for VSCode does use the cider Middleware#2018-12-1117:15borkdudeHonestly I don’t know how that all works, I just use it#2018-12-1117:15richiardiandrea👍 #2018-12-1110:23borkdudeI’d like to link to something like that in https://github.com/borkdude/advent-of-cljc#2018-12-1120:04pesterhazy@borkdude The problem with using an alias that sets "-m" "nrepl.cmdline" is that it doesn't allow you to also specify -r or -m rebel-readline.main#2018-12-1120:05pesterhazyTo me an nREPL-connected editor is only one part - the other part is a terminal/rebel-readline based prompt#2018-12-1120:06pesterhazyThat could be solved if nrepl.cmdline supported a --next-main or --chain argument, like --chain rebl-readlin.main. What it would do is to spawn the server in the background and then call the next main function#2018-12-1120:07pesterhazyIt could even do clean-up when that chained main exits#2018-12-1120:08pesterhazy@alexmiller has mentioned a start option, which could be in the same area#2018-12-1120:08orestis+1 for that. I never got to try it but I think you can combine -e to evaluate something arbitrary with -m ?#2018-12-1120:08pesterhazyyes you can -e something -e something-else -m some.main, though -m must be last#2018-12-1120:09pesterhazybut those don't work if the settings are specified in different aliases - the latter alias will override the earlier one#2018-12-1120:33viestiA colleague of mine ran into this: https://github.com/gfredericks/user.clj, maybe one could start nrepl server at load of user.clj? #2018-12-1120:41pesterhazyyes, a library could be used for this purpose#2018-12-1121:06borkdude@pesterhazy cool, if you have anything better to link to, I would update the README with it#2018-12-1121:17dominicmIn edge I have this working #2018-12-1121:17dominicmhttps://github.com/juxt/edge/blob/master/bin/rebel#2018-12-1121:18dominicmIt depends on a process in user.clj which searches for jvm options like load_xyz, which are then translated to namespaces to load. This means you can start arbitrary background services. #2018-12-1201:57potetmI write a clj script and wrap it w/ a bash script: https://github.com/potetm/advent-of-code/tree/master/bin#2018-12-1201:59potetmYou can put those on your path, add namespaced aliases to your ~/.clojure/deps.edn, and you can run nrepl from any dir containing a deps.edn.#2018-12-1216:12timgilbertSay, what directories do I need to cache in between invocations to take advantage of tools.dep's caching? I'm running builds on circleci and I'm caching ~/.m2, ~/.gitlibs, and .cpcache after running clojure -A:foo -Spath, but even after the cached directories are restored, when it invokes clojure -A:foo I see it downloading a bunch of jars from clojars.#2018-12-1216:15timgilbertOh, also caching ~/.clojure which has its own .cpcache#2018-12-1216:33dominicmM2 & gitlibs should be enough#2018-12-1216:36Alex Miller (Clojure team)the ~/.clojure/.cpcache is only used if you are running clj without a deps.edn, so probably doesn’t apply#2018-12-1216:36Alex Miller (Clojure team)and you would need the local .cpcache if you want to avoid the second pre-jvm hit of building the classpath#2018-12-1216:37Alex Miller (Clojure team)but all of those file checks are based on timestamps, so when you say “restored” I wonder if that changes the timestamps#2018-12-1216:37Alex Miller (Clojure team)particularly in relation to the deps.edn file#2018-12-1216:54timgilbertHmm, yeah, not certain how it works. Maybe I can "negative touch" the deps.edn file to decrease its last-modified time#2018-12-1217:15Alex Miller (Clojure team)easier to touch the cpcache before you try to use clj?#2018-12-1217:15Alex Miller (Clojure team)I guess we could have a flag that was the opposite of -Sforce#2018-12-1217:16Alex Miller (Clojure team)that would use the cached cp regardless of the timestamps#2018-12-1217:17Alex Miller (Clojure team)although then you’d have to make sure you did something else to force any deps changes to be picked up#2018-12-1217:42timgilbertJust confirmed on circleci that my deps.edn winds up being newer than the .cpcache and .m2 directories. I presume that circleci saves the mtimes on cache and restores them on restore. I'll try recursively touching those directories as an initial workaround#2018-12-1218:50hiredmanI dunno, the touch thing will likely work, but I am suspicious of deps.edn always being newer and you always download jars#2018-12-1218:51hiredmaneven if those files are older, it shouldn't cause you to re-download jars into your m2#2018-12-1218:57hiredmanyeah, if you are re-downloading jars over and over in to m2 that seems like the up stream issue and fixing it will take care of down stream#2018-12-1218:59hiredmanI dunno what facilities circleci provides, but you might stick a ls -R ~/.m2 at the start of your build before running clojure#2018-12-1219:28timgilbertHmm, the touch doesn't seem to have any effect:
ls -lRt --time-style=full-iso deps.edn .cpcache ~/.m2 || true
-rwxr-xr-x 1 circleci circleci 1660 2018-12-12 19:25:36.984493358 +0000 deps.edn
.cpcache:
total 36
-rw-r--r-- 1 circleci circleci 7989 2018-12-12 19:25:38.420533085 +0000 2750907078.cp
-rw-r--r-- 1 circleci circleci 24551 2018-12-12 19:25:38.420533085 +0000 2750907078.libs
-rw-r--r-- 1 circleci circleci 64 2018-12-12 19:25:38.420533085 +0000 2750907078.main
/home/circleci/.m2:
total 4
drwxr-xr-x 15 circleci circleci 4096 2018-12-12 19:25:38.380531978 +0000 repository
/home/circleci/.m2/repository:
total 52
drwxr-xr-x 3 circleci circleci 4096 2018-12-12 19:25:38.388532199 +0000 aopalliance
drwxr-xr-x 3 circleci circleci 4096 2018-12-12 19:25:38.388532199 +0000 commons-io
#2018-12-1219:29timgilbertSame results with ls -lRtc to get the ctime#2018-12-1219:42timgilbert...hmm, might be a PEBKAC problem#2018-12-1219:50timgilbertArgh, turns out it was a problem in my config where I changed the cache key things were being saved into but not the one where they were being restored from. Sorry for the noise! Once I fixed that up everything worked as expected and the deps are not being redownloaded in the second step.#2018-12-1219:52Alex Miller (Clojure team)yay#2018-12-1300:05Nolanhey everyone! have been loving tools.deps and have had no issues until i abruptly started receiving Error building classpath. with no apparent configuration change. has anyone seen this? i’ve been using deps in this way for a few weeks. anyone have any ideas on how to start debugging?#2018-12-1300:15Nolanfor the next person: ssh-add -l may reveal that the identity previously being used is no longer there. still not sure exactly why that happened#2018-12-1300:24Alex Miller (Clojure team)maybe your ssh-agent died and restarted?{:tag :div, :attrs {:class "message-reaction", :title "wizard"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "wizard", :src "https://emoji.slack-edge.com/T03RZGPFR/wizard/22f9331056bd2361.png"}, :content nil})} " 4")}
#2018-12-1301:38Nolanmust’ve. but it’s happily back at it now. i really appreciate what tools.deps has done for us. can’t say it enough—thank you for all your work on it.#2018-12-1300:28richiardiandreawhen I create clojure -Spom can I tell clojure not to include org.clojure/clojure? I don't have it in my project deps.edn#2018-12-1300:32hiredmanclojure merges in .clojure/deps.edn#2018-12-1300:32hiredman-Srepro Ignore the ~/.clojure/deps.edn config file#2018-12-1300:33richiardiandreauhm I have in the back my head a voice that says that that does not work...#2018-12-1300:33richiardiandreawill try it#2018-12-1300:34hiredmanit will likely break other things if you don't include a dependency on clojure somewhere#2018-12-1300:34richiardiandreaI am trying to build a pom.xml for ClojureScript so it should be fine#2018-12-1300:35richiardiandreayeah it still adds
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure</artifactId>
<version>1.9.0</version>
</dependency>
#2018-12-1300:35hiredmanare you sure something else isn't puling it in?#2018-12-1300:36timgilbertI was just looking at this bit of code, I’m pretty sure -Spom will include your system-wide deps too, eg /usr/local/lib/clojure/deps.edn or whatever it’s called#2018-12-1300:37richiardiandreawell, I run clojure -Srepro -Spom after having deleted it...my deps.edn seems clean...dunno...#2018-12-1300:38timgilbertOn the subject of pom generation, I’ve just published garamond, which rewrites the pom.xml that -Spom generates with a version ID from git tags: https://github.com/workframers/garamond#2018-12-1300:39richiardiandreamaaaan that's awesome I was fiddling with scripts 😄#2018-12-1300:40timgilbertI still have a few features I’m working on, but it’s ready for use. It uses itself plus juxt/pack and deps-deploy to publish itself to clojars sans a project.clj#2018-12-1300:40richiardiandreausing those as well at the moment 👍#2018-12-1300:41richiardiandreathank you so much, this will definitely simplify my scripts!#2018-12-1300:41hiredmancheckout clojure -Srepro -Sdescribe#2018-12-1300:41timgilbertAwesome#2018-12-1300:42richiardiandreaoh /home/arichiardi/.local/lib/clojure/deps.edn is something I haven't created#2018-12-1300:42richiardiandreaand indeed it contains the clojure version above#2018-12-1300:42hiredmanyeah#2018-12-1300:42richiardiandreatricky trick! thank you! 😄#2018-12-1300:42hiredmanI think the install scripts create that with the idea that it is the "system" level deps.edn#2018-12-1300:42richiardiandreacan I just get rid of :deps in there?#2018-12-1300:42hiredmanI don't see why not#2018-12-1300:43richiardiandrealemme do that then 😄#2018-12-1300:43hiredmanor just delete the file#2018-12-1300:44richiardiandreayeah that's right it works!#2018-12-1300:46richiardiandreathank you very much @hiredman#2018-12-1301:20Alex Miller (Clojure team)note that installing / upgrading clj will rewrite that file{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-12-1301:33seancorfieldDeleting the "system" level deps.edn will remove the default Maven repos and a few other things, right?#2018-12-1301:34seancorfieldYeah, :mvn/repos and :paths (which defaults to src)...#2018-12-1307:46borkdudeis tools.deps officially supported on Windows as well? I didn’t think about this when choosing it as the thing for Advent of CLJC but I didn’t hear any complaints yet#2018-12-1309:13dominicmNot yet it isn't#2018-12-1310:36borkdudeI hear a lot of people are using WSL on Windows now#2018-12-1313:02seancorfieldYup I do nearly all my clojure development on windows via wsl because of that. {:tag :div, :attrs {:class "message-reaction", :title "heavy_plus_sign"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("➕")} " 4")}
#2018-12-1316:15normanIs there a reliable non-brew way to get clj on OS X?#2018-12-1316:26Alex Miller (Clojure team)we are only providing brew as a means of install, but you could basically use the linux instructions too I think - they’re both basically doing the same thing#2018-12-1316:26Alex Miller (Clojure team)it’s essentially just putting files in the right places#2018-12-1318:21delaguardoHi there. I have some problems with pulling git deps from private repo. My ssh-agent is configured and running but I'm getting USERAUTH fail every time. Is there some corner cases where it might fail that I must be aware of?#2018-12-1318:28seancorfield@delaguardo Using https? If so, that's a known limitation of tools.deps. This is the uber-ticket for possibly solving that https://dev.clojure.org/jira/browse/TDEPS-91#2018-12-1318:28seancorfieldNote that it links to three related git/auth tickets.#2018-12-1318:28delaguardoNo, I'm using /cdn-cgi/l/email-protection#2018-12-1318:29Alex Miller (Clojure team)it’s a star-shaped room, so mostly corners#2018-12-1318:29Alex Miller (Clojure team)so if you do ssh-add -l - do you see your identity?#2018-12-1318:30delaguardoYes, and I can clone that repo. But for some reason jgit throwing an exception#2018-12-1318:30Alex Miller (Clojure team)other common issues are having stuff you don’t need in ~/.ssh/config#2018-12-1318:30Alex Miller (Clojure team)and issues with host key types#2018-12-1318:30Alex Miller (Clojure team)the former being more common in my experience#2018-12-1318:31Alex Miller (Clojure team)if you can dump your config here, I can eyeball it (can’t remember off-hand the offending things)#2018-12-1318:31Alex Miller (Clojure team)but I would know it if I saw it#2018-12-1318:32delaguardoHm.. everything is working on another laptop. Only one difference - my key has passphrase and it is added to agent via osx keychain#2018-12-1318:33Alex Miller (Clojure team)that should be ok. can you dump your config file?#2018-12-1318:34Alex Miller (Clojure team)I think it’s IdentityFile that causes problem in your config (and you should delete that line)#2018-12-1318:34Alex Miller (Clojure team)so something like this should be sufficient:#2018-12-1318:34Alex Miller (Clojure team)Host *
AddKeysToAgent yes
UseKeychain yes
#2018-12-1318:39delaguardoWill try to do that. Thanks!#2018-12-1318:40Alex Miller (Clojure team)the macos version you’re using on the two machines can also make a difference - they have changed the keychain integration a couple times#2018-12-1322:26manderson@alexmiller RE: the AUTHFAIL issue, I added a patch to https://dev.clojure.org/jira/browse/TDEPS-49 that fixes the problem without having to remove the IdentityFile line in your config. Not sure if you saw it as it was just before Conj, but wanted to mention it.#2018-12-1322:39Alex Miller (Clojure team)Didn’t see it, thx#2018-12-1406:22didibusSomeone brought up that tools.deps can't transitively resolve project.clj dependencies. Is that true?#2018-12-1406:28orestisAFAIK, yes. #2018-12-1406:32didibusOh, so until all lib upgrade, it could cause some issues there#2018-12-1406:32didibusto add deps.edn#2018-12-1406:33didibusCan it handle pom.xml ?#2018-12-1406:33didibusOr only deps.edn ?#2018-12-1407:00orestisI think Pom is fine, but not 100% sure. I believe Pom is used for maven-things so presumably it works fine. #2018-12-1413:47Alex Miller (Clojure team)That’s correct#2018-12-1407:00orestisA usual trick is to fork a git library, at a deps.edn and use your fork with updated dependencies until it’s fixed upstream. #2018-12-1407:36dominicmJust to be clear, this only affects git dependencies. Nobody would depend on a git repo with only a project.clj.#2018-12-1407:38dominicmIt is absolutely fine if you depend on a git repo, which depends on a leiningen-built project on clojars or maven central.#2018-12-1723:01martinklepschWhen passing some serialized EDN to clj is that when people use commas instead of spaces to get it regarded as a single argument?#2018-12-1723:01martinklepschclj -m some.ns '{:my "data"}'
#2018-12-1723:14dominicmYeah. But not for the above case. #2018-12-1723:14dominicmIt's inside of deps.edn files.#2018-12-1723:42aaelonyHi - longtime leiningen user here. I’m on a mac, I am using home brew, but the clj that gets installed stacktraces miserably#2018-12-1723:42aaelonyHere is the stacktrace: $ clj -Sdescribe
Error building classpath. EOF while reading
java.lang.RuntimeException: EOF while reading
at clojure.lang.Util.runtimeException(Util.java:221)
at clojure.lang.EdnReader.readDelimitedList(EdnReader.java:746)
at clojure.lang.EdnReader$MapReader.invoke(EdnReader.java:680)
at clojure.lang.EdnReader.read(EdnReader.java:145)
at clojure.lang.EdnReader.read(EdnReader.java:111)
at clojure.edn$read.invokeStatic(edn.clj:35)
at clojure.edn$read.invoke(edn.clj:14)
at $slurp_edn.invokeStatic(io.clj:29)
at $slurp_edn.invoke(io.clj:22)
at clojure.tools.deps.alpha.reader$slurp_edn_map.invokeStatic(reader.clj:38)
at clojure.tools.deps.alpha.reader$slurp_edn_map.invoke(reader.clj:35)
at clojure.tools.deps.alpha.reader$slurp_deps.invokeStatic(reader.clj:58)
at clojure.tools.deps.alpha.reader$slurp_deps.invoke(reader.clj:54)
at clojure.core$map$fn__5587.invoke(core.clj:2745)
at clojure.lang.LazySeq.sval(LazySeq.java:40)
at clojure.lang.LazySeq.seq(LazySeq.java:49)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.boundedLength(RT.java:1785)
at clojure.lang.RestFn.applyTo(RestFn.java:130)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$apply.invoke(core.clj:652)
at clojure.tools.deps.alpha.reader$merge_deps.invokeStatic(reader.clj:73)
at clojure.tools.deps.alpha.reader$merge_deps.invoke(reader.clj:70)
at clojure.tools.deps.alpha.reader$read_deps.invokeStatic(reader.clj:78)
at clojure.tools.deps.alpha.reader$read_deps.invoke(reader.clj:75)
at clojure.tools.deps.alpha.script.make_classpath$combine_deps_files.invokeStatic(make_classpath.clj:47)
at clojure.tools.deps.alpha.script.make_classpath$combine_deps_files.invoke(make_classpath.clj:43)
at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:68)
at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:64)
at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:109)
at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:84)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
#2018-12-1723:42aaelonyany ideas welcome#2018-12-1723:42aaelony(or I’ll go back to leiningen#2018-12-1723:42aaelony)#2018-12-1723:43seancorfieldWhat does cat ~/.clojure/deps.edn show?#2018-12-1723:43dpsuttonwhen i run clj -Sdescribe it gives me some output that includes config-files ["/usr/local/lib/clojure/deps.edn" "/home/dan/.clojure/deps.edn" ]. There are some deps.edn files and it seems like one of these is broken. can you check your equivalent ones#2018-12-1723:43aaelonyThat’s likely the culprit… $ cat ~/.clojure/deps.edn
;; The deps.edn file describes the information needed to build a classpath.
;;
;; When using the `clojure` or `clj` script, there are several deps.edn files
;; that are combined:
;; - install-level
;; - user level (this file)
;; - project level (current directory when invoked)
;;
;; For all attributes other than :paths, these config files are merged left to right.
;; Only the last :paths is kept and others are dropped.
{
{:aliases
{:new {:extra-deps {seancorfield/clj-new
{:mvn/version "0.5.5"}}
:main-opts ["-m" "clj-new.create"]}
}
;; Paths
;; Directories in the current project to include in the classpath
;; :paths ["src"]
;; External dependencies
;; :deps {
;; org.clojure/clojure {:mvn/version "1.9.0"}
;; }
;; Aliases
;; resolve-deps aliases (-R) affect dependency resolution, options:
;; :extra-deps - specifies extra deps to add to :deps
;; :override-deps - specifies a coordinate to use instead of that in :deps
;; :default-deps - specifies a coordinate to use for a lib if one isn't found
;; make-classpath aliases (-C) affect the classpath generation, options:
;; :extra-paths - vector of additional paths to add to the classpath
;; :classpath-overrides - map of lib to path that overrides the result of resolving deps
;; :aliases {
;; :deps {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.5.460"}}}
;; :test {:extra-paths ["test"]}
;; }
;; Provider attributes
;; :mvn/repos {
;; "central" {:url ""}
;; "clojars" {:url ""}
;; }
} #2018-12-1723:44hiredmanwhat does clojure -Srepro -Sdescribe show?#2018-12-1723:44dpsuttonit looks like aliases is malformed#2018-12-1723:44aaelonyyeah#2018-12-1723:44dpsutton@hiredman s/he can't get any output due to this being unbalanced#2018-12-1723:44seancorfieldThat's where your EOF is coming from @aaelony -- you're missing a }#2018-12-1723:45hiredman@dpsutton right, clojure -Sdescribe would error if either the system or user level deps.edn had a problem, clojure -Srepro -Sdescribe would error if it was the system level#2018-12-1723:45aaelonyCan I nuke the ~/.clojure/deps.edn and try again?#2018-12-1723:45seancorfield{:aliases
{:new {:extra-deps {seancorfield/clj-new
{:mvn/version "0.5.5"}} ; NEED EXTRA } HERE
:main-opts ["-m" "clj-new.create"]}
}{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-12-1723:45dpsuttonah thanks for explaining @hiredman#2018-12-1723:46aaelonygot it - that is now fixed#2018-12-1723:46aaelonythanks{:tag :div, :attrs {:class "message-reaction", :title "100"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("💯")} " 4")}
#2018-12-1723:48aaelonyyay - all set now… phfew.#2018-12-1723:48aaelonybig thanks all around#2018-12-1723:50seancorfieldHappy to help! At work we drive everything with clojure CLI -- we started with Leiningen back in 2011, switched to Boot at the end of 2015, and switched completely to the new CLI stuff this year.#2018-12-1723:51aaelonyGreat - I’ve been mostly using R for the past year++ and getting back into Clojure and updating my toolset from lein…#2018-12-1723:52seancorfieldCool. And you can get started with Clojure 1.10 since the "gold" release of that dropped today which greatly improves error reporting in a lot of cases!#2018-12-1723:53aaelonyyeah - ironically, I didn’t feel the error reporting pain (until just now). The clj tool is still on 1.9 until it gets released to home brew.#2018-12-1723:56Alex Miller (Clojure team)This particular error isn’t going to be any different with 1.10 afaik #2018-12-1723:57aaelonyfor me, this was a “not knowing what you don’t know” kind of error#2018-12-1723:57Alex Miller (Clojure team)The error reporting differences are primarily in the repl and this is just throwing an exception out of the main#2018-12-1723:58aaelonyperhaps if there was a spec check on the ~/.clojure/deps.edn file#2018-12-1723:58Alex Miller (Clojure team)It would probably be reasonable to better trap around config file reading and give a better semantic error here#2018-12-1723:59Alex Miller (Clojure team)Well, we’ve been supporting back to Clojure 1.8 in tools.deps, which is pre spec#2018-12-1723:59Alex Miller (Clojure team)I will probably bump that up soon#2018-12-1723:59aaelonyI see#2018-12-1723:59Alex Miller (Clojure team)But in this case, it didn’t even read so would never be able to do a spec check anyhow#2018-12-1723:59aaelonyThough I read about it in the docs, I was pretty much unaware that the ~/.clojure/deps.edn file was involved#2018-12-1800:00aaelonyI should have known better#2018-12-1800:01Alex Miller (Clojure team)Error could be more informative for sure#2018-12-1800:01seancorfieldhttps://github.com/seancorfield/dot-clojure updated now that 1.10 has dropped: (! 528)-> clj -A:master
Clojure 1.11.0-master-SNAPSHOT
user=>
🙂#2018-12-1800:02aaelony🙂#2018-12-1820:16kennyDoes :deps/root work for both local and git deps?#2018-12-1820:21kennyIt appears it doesn't.#2018-12-1821:12dominicmWhich doesn't it work for? It doesn't make sense for local dependencies really. #2018-12-1821:51Alex Miller (Clojure team)should work for both{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2018-12-1821:51Alex Miller (Clojure team)in the case of a local dep it’s a directory relative to :local/root#2018-12-1821:57kennyIt doesn't appear to work for local/root. I get the following message:
Error resolving /home/kenny/compute_software/app-ions: Manifest type not detected when finding deps for compute/alerts-api in coordinate {:local/root "../alerts", :deps/root "api", :sha "ca8f741dd264fa13a900cb1d283e32a21f4c21c8"}
#2018-12-1821:58kennyIf I change :local/root to ../alerts/api it works as expected.#2018-12-1822:03Alex Miller (Clojure team)the message is saying that it’s looking for a manifest file (deps.edn or pom.xml) at the specified location and not finding one#2018-12-1822:04dominicmWhich location is it looking in? Does it consider deps/root when searching? #2018-12-1822:04Alex Miller (Clojure team)it’s looking at <:local/root>/<:deps/root>#2018-12-1822:05Alex Miller (Clojure team)so it should be looking for ../alerts/api/deps.edn here#2018-12-1822:05dominicmThat wouldn't explain it working with local/root alone? #2018-12-1822:06Alex Miller (Clojure team)well, the question is where .. is relative to#2018-12-1822:06Alex Miller (Clojure team)currently, it’s always relative to the current directory of wherever you are running#2018-12-1822:06Alex Miller (Clojure team)not the relative directory of whatever is depending on it#2018-12-1822:06Alex Miller (Clojure team)this is a known issue with local roots and relative dirs#2018-12-1822:07Alex Miller (Clojure team)I feel like I’m missing important parts of the problem description here so just kind of guessing#2018-12-1822:09kenny#2018-12-1822:12kennyActually that's not correct.
pwd
/home/kenny/compute_software/app-ions
clj -A:dev -Spath
Error building classpath. Manifest type not detected when finding deps for compute/alerts-api in coordinate {:local/root "../alerts", :deps/root "api", :sha "ca8f741dd264fa13a900cb1d283e32a21f4c21c8"}
#2018-12-1822:13kenny> so it should be looking for ../alerts/api/deps.edn here
And that file exists:
ls ../alerts/api/deps.edn
../alerts/api/deps.edn
#2018-12-1822:21hiredmanI am not sure, but I suspect that coordinate results in it looking for /home/kenny/compute_software/app-ions/api/../alerts/#2018-12-1822:25hiredmanno I am not right, just confused#2018-12-1822:35hiredmanunless I am confused again https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/local.clj#L27-L31 would need to include :deps/root in path (instead of the empty string) in order to detect the deps.edn in :deps/root#2018-12-1822:57Alex Miller (Clojure team)you’re not using that function#2018-12-1823:03hiredmanoh, yep, further confusion, that looks like it does that kind of resolution, but then doesn't appear to actually be called anywhere in tools.deps#2018-12-1823:05Alex Miller (Clojure team)the manifest is being found in https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/local.clj#L33-L38 which will fall into the last case and call https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions.clj#L21-L29#2018-12-1823:06Alex Miller (Clojure team)but I will actually go back and say it’s not combining the two paths - it’s really just using :local/root#2018-12-1823:06Alex Miller (Clojure team)I was misremembering the code that’s in the git extension#2018-12-1917:20seancorfieldUpgraded clojure to the 1.10 release via brew and then wondered why my outdated dependencies checker was no longer working... and then realized that TDEPS-82 is not yet fixed!#2018-12-1918:13Alex Miller (Clojure team)I am working through some tdeps stuff right now…#2018-12-1918:13Alex Miller (Clojure team)will try to hit that one#2018-12-1918:14Alex Miller (Clojure team)planning to get to a release by end of week{:tag :div, :attrs {:class "message-reaction", :title "star-struck"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 16")}
#2018-12-1918:25seancorfieldThank you!#2018-12-2022:56geraldodevIt would be nice if clj show invalid edn parsing errors#2018-12-2023:00dominicmI think there is a ticket for that, it's been mentioned. I think one would be welcome if you can't find one, if you can, a vote helps prioritize. #2018-12-2023:06Alex Miller (Clojure team)I was planning to look at that for the next release#2018-12-2023:06Alex Miller (Clojure team)but there are limits to what we can do other than say “the edn is invalid”#2018-12-2023:07Alex Miller (Clojure team)since the edn file is a map, it either will read the whole thing, or it will fail (if you miss a brace for example)#2018-12-2023:07dpsuttonand bang for the buck i suppose. EOF encountered while ... is the current message, right?#2018-12-2023:07Alex Miller (Clojure team)yeah#2018-12-2023:08dpsuttonit seemed straight forward but maybe that is stockholm syndrome#2018-12-2023:08dominicm@alexmiller is this the same behaviour as for Clojure? Could we look at something like the figwheel error tool? #2018-12-2023:08Alex Miller (Clojure team)I’m planning to catch this case and make the error message better (and skip the stack trace) but it’s probably not going to actually help you in a lot of cases#2018-12-2023:08dominicmProbably good questions for JIRA 😊 #2018-12-2023:08Alex Miller (Clojure team)sorry, don’t have context on figwheel#2018-12-2023:09dominicmI think figwheel can print out a snippet and give you an ASCII arrow pointing at the error. #2018-12-2023:09Alex Miller (Clojure team)the errors I see reported are almost always invalid edn (usually a missing brace), not a valid map that is invalid#2018-12-2023:10Alex Miller (Clojure team)I think figwheel is doing that with spec-like validation on invalid values, but that’s not the error I’m describing#2018-12-2023:10Alex Miller (Clojure team)we have specs and (if we bump the min clojure version) can check the read map against the specs to get that, but again, that’s not where I see most issues#2018-12-2023:11geraldodevI was bitten by Map literal must contain an even number of forms. Even on a modest file size it's very pratical to be pointed to the exact spot. I've downloaded diogo149/lein-edn-validator to help me find .#2018-12-2023:13geraldodev@dominicm tons of issues. ppl are using tools.dep. That's good!#2018-12-2023:13dominicm@alexmiller figwheel also does something with the data coming out of cljs compiler, and can information related to this.
I suppose in theory tdeps can be even smarter if the data was available from clojure.edn#2018-12-2023:24dominicmhttps://github.com/diogo149/lein-edn-validator/blob/master/src/leiningen/edn_validator.clj interesting approach.#2018-12-2023:56Alex Miller (Clojure team)Well, I’m not planning to do that#2018-12-2108:06dominicm@alexmiller out of curiosity, would you accept it as a contribution via a third party library or similar? Assuming startup time didn't hurt too much. #2018-12-2201:42dominicmIs there any reason :default-deps shouldn't be available as a top-level key? My reasoning being that I actually want any transitive dependencies that are newer than what I specify to be used instead. Maybe default-deps doesn't quite fill that niche. But if I depend on clojure 1.8, and cognitect-aws depends on clojure 1.10, I'm happy for 1.10 to be used. Maybe a "linter" could inform on this :thinking_face:#2018-12-2221:16borkdudehow does one force download of dependencies in a deps.edn project?#2018-12-2221:36ghadi-Sforce#2018-12-2222:11borkdudeI figured, but that also opens a repl …
$ clojure -Sforce
Clojure 1.10.0
user=> ^D
#2018-12-2222:12borkdudealso, that forces recomputation of the classpath, I only want to force the initial downloading of stuff, else return fast without doing anything extra#2018-12-2222:13borkdudeit seems the problem solved itself on circleci, just do stuff and it will take care of itself 😉#2018-12-2222:22ghadi-Sforce -Spath @borkdude #2018-12-2222:22ghadiThat will just printout the computed classpath#2018-12-2222:25borkdudethanks#2018-12-2916:28borkdudeis there something to update a deps.edn file to the latest git sha’s ?#2018-12-2916:29borkdudecool: https://github.com/Olical/depot#2019-12-3115:51jmvhi all, do deps.edn git refs support repos with multiple subprojects? i'm wondering if i can specify a path into the repo to find the deps.edn file along with the sha#2019-12-3118:06richiardiandrea@jmv we use :local/root for that, not sure about sha#2019-12-3118:12seancorfield@jmv305 :deps/root should let you navigate into a subproject within a git repo.#2019-12-3118:50jmvthank you, i will give that a try!#2019-01-0221:33Alex Miller (Clojure team)tools.deps.alpha 0.6.474 and clj 1.10.0.408 are now available.
Changes:
* TDEPS-79, TDEPS-109 Fix infinite loop in resolving recursive deps tree (will often improve resolution perf in other cases as well)
* TDEPS-50, TDEPS-109 Fix handling of Maven classifiers during resolution
To explicitly specify a Maven classifier in a dep, specify it in the lib name as follows: groupId/artifactId$classifier
Note: the :classifier attribute in Maven dep coordinates is no longer supported and will now produce an error.
* TDEPS-107 Canonicalize exclusion groupIds in pom gen
* TDEPS-82 Ensure -Sdescribe doesn't trigger resolution
* Error handling improvements when Maven artifact is not found
* Error handling improvements for invalid syntax in deps.edn
{:tag :div, :attrs {:class "message-reaction", :title "clj"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "clj", :src "https://emoji.slack-edge.com/T03RZGPFR/clj/acf97abdead7f2d9.png"}, :content nil})} " 40")}
#2019-01-0221:34seancorfieldTDEPS-82 ❤️ ❤️ ❤️#2019-01-0307:39ChrisIs it necessary to reinstall from http://clojure.org to upgrade to these releases?#2019-01-0313:31Alex Miller (Clojure team)Yes? How else would you get them?#2019-01-0314:55borkdudesome packages have an upgrade command (but this would still get the updates from http://clojure.org)#2019-01-0318:24ChrisYes, I was wondering if there was an update command or it even automatically upgraded the tools deps lib.#2019-01-0311:49geraldodevI tried the last tools.deps and I have a fulcro depified project that has nubank/workspaces {:mvn/version "1.0.2" :exclusions [com.cognitect/transit-java]}
With the new version it does not loads transit-java. The problem is that it is excluded from workspace but its a dependency of transit-clj (of fulcro) and shadow-cljs . It looks like the exclusion is for good (affect other packages). clj -Stree does not show transit-java. When I revert to 1.9.0.397 the error is gone#2019-01-0313:31Alex Miller (Clojure team)Can you be more explicit with a repro to make sure I understand?#2019-01-0314:11Alex Miller (Clojure team)@geraldodev ^^#2019-01-0314:11Alex Miller (Clojure team)in particular, would love to have your full deps.edn#2019-01-0316:11geraldodevMy deps.edn, good_tree.txt was produced with linux-install-1.9.0.397.sh
bad_tree.txt was produced with linux-install-1.10.0.408.sh. Look that there are no transit-java with the latter version. It was supposed to be there because it is a dependency of fulcro and shadow-cljs#2019-01-0316:13geraldodevThe only deps missing is a variation of sean corfield's ~/.clojure/deps.edn, but I think it does not play any role on this problem#2019-01-0316:23Alex Miller (Clojure team)what is the command you’re running to get the trees?#2019-01-0316:23Alex Miller (Clojure team)just clj -Stree ? or using any aliases?#2019-01-0316:27geraldodevclj -Stree on the project directory#2019-01-0316:27geraldodevno aliases.#2019-01-0316:28geraldodevThe resolution appears to be different on the most recent version#2019-01-0316:28Alex Miller (Clojure team)thx, I will take a look#2019-01-0316:28geraldodevnp, thnx for your work on tools.deps#2019-01-0317:23Alex Miller (Clojure team)@geraldodev that’s an interesting case. there are basically multiple dependency paths to transit-clj 0.8.313 (all same version). The path in which it is first found and selected (nubank/workspaces -> com.cognitect/transit-clj) happens to exclude its transitive dep, com.cognitect/transit-java.#2019-01-0317:24geraldodevI checked the version. It was strange to me because it is the same like you saw#2019-01-0317:25Alex Miller (Clojure team)in simplest form:#2019-01-0317:25Alex Miller (Clojure team)tree:
A (excl C) -> B -> C
D -> B -> C
selected:
A, D, B
#2019-01-0317:26Alex Miller (Clojure team)whereas if B happened to be encountered via D first, C would be included#2019-01-0317:27Alex Miller (Clojure team)I know why what you’re seeing happened to change, but I think this problem likely existed on the prior version too, but via some other arbitrary ordering#2019-01-0317:28Alex Miller (Clojure team)I will need to think about this a bit#2019-01-0317:28geraldodevBut you are ble to get transit-java on prior version#2019-01-0317:28Alex Miller (Clojure team)you happened to get transit-java on prior version#2019-01-0317:29geraldodevvia arbitrary ordering. I get it. It's a coincidence...#2019-01-0317:29Alex Miller (Clojure team)but I don’t actually think you’re guaranteed that - it’s probably a result of the order the deps are being encountered#2019-01-0317:29Alex Miller (Clojure team)I’m not certain of that, but that’s my suspicion#2019-01-0317:30Alex Miller (Clojure team)I think what this dependency tree is saying is not self-consistent#2019-01-0317:31Alex Miller (Clojure team)that is, if you treated the deps as logical propositions, there is no solution that satisfies both excluding C under A and including it under D in that simple version above#2019-01-0317:32Alex Miller (Clojure team)given that it’s the same B in both cases#2019-01-0317:33Alex Miller (Clojure team)for example, if you run this same example in Maven, you get a match of the “bad” tree (no transit-java included at all)#2019-01-0317:34timgilbertWouldn't you want it to be like:
tree:
A1 (excl C1) -> B1 -> C1
D2 -> B2 -> C2
selected:
A1, D2, (resolve-duplicate #{B1 B2}), C2
?#2019-01-0317:34Alex Miller (Clojure team)in this particular case, the versions of B and C in both branches are identical#2019-01-0317:35Alex Miller (Clojure team)so I omitted the versions#2019-01-0317:35Alex Miller (Clojure team)if the two B’s are identical, then it shouldn’t matter which one you choose#2019-01-0317:36Alex Miller (Clojure team)and exclusions are never version-specific#2019-01-0317:36geraldodevThere are no explicity dependency of transit-java on lein's fulcro template. It appears to include it.#2019-01-0317:37Alex Miller (Clojure team)tree:
A1 (excl C) -> B1 -> C1
D1 -> B1 -> C1
selected:
A1, D1, B1
#2019-01-0317:38Alex Miller (Clojure team)is a version-specific example (and the exclusion intentionally does not have a version, as exclusions are not version-specific)#2019-01-0317:38timgilbertI guess I was thinking the semantics of exclusion were more "omit C from the entire transitive tree of A", not "globally exclude C from the tree"#2019-01-0317:39Alex Miller (Clojure team)it is the former#2019-01-0317:39Alex Miller (Clojure team)but I’m saying that omit C1 can’t be consistent with also saying include D1 which depends on B1 which depends on C1#2019-01-0317:40Alex Miller (Clojure team)you’re saying both omit C1 and include C1 - we can’t do both#2019-01-0317:40timgilbertI see the contradiction, yeah...#2019-01-0317:43timgilbertI guess I'd just intuitively expect that in order to get the behavior where C is excluded, you'd have to do
tree:
A1 (excl C) -> B1 -> C1
D1 (excl C) -> B1 -> C1
selected:
A1, D1, B1
#2019-01-0317:43Alex Miller (Clojure team)yes, that is what you have to do in either clj or Maven#2019-01-0317:44Alex Miller (Clojure team)there have been requests for global exclusions as a feature but that’s not currently a thing#2019-01-0317:45Alex Miller (Clojure team)if anything were to be done on this, it would be to make it non-arbitrary#2019-01-0317:46Alex Miller (Clojure team)bascially if you have B and B (excl C), do not treat those as interchangeable - prefer one over the other#2019-01-0317:46Alex Miller (Clojure team)this gets tricky when combined with version comparisons though#2019-01-0317:46Alex Miller (Clojure team)and I’m not sure which one you would want to prefer#2019-01-0317:48Alex Miller (Clojure team)seems like geraldodev’s expectation is that you would prefer including it#2019-01-0317:50geraldodevyes, like lein does#2019-01-0317:50Alex Miller (Clojure team)lein relies on Maven’s selection, which is also order-dependent#2019-01-0317:51Alex Miller (Clojure team)it wouldn’t surprise me if the ordering affected what you got in lein too#2019-01-0317:51geraldodevif workspace is not picking a version for you and another package is so be it. because it needs to be there. the version being the same is irrelevant to a greater proposal of having the lib#2019-01-0317:52Alex Miller (Clojure team)this is not about version picking though, it’s about inclusion vs exclusion#2019-01-0317:52geraldodevbut the exclusion was caused because of the version being the same#2019-01-0317:55Alex Miller (Clojure team)there is no set of dependencies that can satisfy the constraints stated, so the most correct answer here is actually to fail#2019-01-0317:55Alex Miller (Clojure team)we’re just pouring semantics over the top of a logically inconsistent case#2019-01-0317:55Alex Miller (Clojure team)we do that all the time when selecting deps though wrt versions#2019-01-0317:56hiredmanthe question is kind of, are exclusions global, or are exclusions local. are exclusions applied while determining what the transitive dependencies of a given dependency are, or are they applied later after all the dependencies have been determined an a graph has been created#2019-01-0317:56Alex Miller (Clojure team)that’s not the question#2019-01-0317:56Alex Miller (Clojure team)exclusions are not global#2019-01-0317:56Alex Miller (Clojure team)they are always contextual#2019-01-0317:58geraldodevlein deps :tree on template project after lein ancient :upgrade. It uses workspaces on 1.0.2#2019-01-0317:59Alex Miller (Clojure team)the selection algorithm is attempting to satisfy (often contradictory) constraints while also balancing performance concerns#2019-01-0317:59geraldodevtransit-java is being picked by transitive dependency of shadow-cljs#2019-01-0317:59Alex Miller (Clojure team)clj is intentionally not using the same algorithm as either lein or Maven#2019-01-0318:00Alex Miller (Clojure team)those both prefer versions that are closer to the top or first encountered at a sibling level#2019-01-0318:00Alex Miller (Clojure team)clj prefers the newest#2019-01-0318:00hiredmanmaybe I misstated it, but it sounds like the difference between the current observed behavior, and the behavior @geraldodev is suggesting is between applying exclusions before or after it has been determined that the B both A and D depend on is the same, which maybe isn't possible, but I described it as global because it seems like in order to know A and D depend on the same B you need a global view of the dependency graph#2019-01-0318:02geraldodev@alexmiller it's the newest and forbbidenn by one dependency#2019-01-0318:02Alex Miller (Clojure team)I think this discussion is missing too much context of what’s being considered to be correct#2019-01-0318:04Alex Miller (Clojure team)like when choosing versions, you have to consider what versions you’ve seen, whether a version is under an exclusion of a selected version, etc#2019-01-0318:05geraldodev@alexmiller I'm in no place to argue about this. I just expected the lein behaviour.#2019-01-0318:05Alex Miller (Clojure team)and for performance reasons, we’re doing this in one pass, which limits information and when you know things#2019-01-0318:06Alex Miller (Clojure team)I’m still contending that the overall dependency tree here is logically inconsistent and there is no solution that meets all constraints#2019-01-0318:07Alex Miller (Clojure team)I would not be surprised if lein / Maven both have order-dependent answers to this and depending how you order things, you could see transit-java either being included or not here#2019-01-0318:08Alex Miller (Clojure team)if that’s not true, then their selection algorithm is effectively placing some preference, either intentionally or accidentally#2019-01-0318:08Alex Miller (Clojure team)clj’s selection algorithm is different, and may or may not make the same choice#2019-01-0318:09Alex Miller (Clojure team)I would prefer it not to be order-dependent though as that’s generally a source of confusion#2019-01-0318:09Alex Miller (Clojure team)Maven’s algorithm relies on order as means of selection, so it has to be#2019-01-0318:11hiredmansome of it is maybe ergonomic, having the exclusion listed on a particular dependency leads to the expectation (as incorrect as it may be) that the exclusion is somehow limited in scope to that dependency and won't effect dependency info from other deps#2019-01-0319:55kennyIn order to use Neanderthal on a Mac you need to specify these exclusions: https://github.com/uncomplicate/neanderthal/blob/7cf295119527843a6d1e9c17161b8a7b2f741309/examples/hello-world/project.clj#L6-L7. I cannot figure out a way to do this in a deps.edn. I tried this:
{:deps {uncomplicate/neanderthal {:mvn/version "0.19.0"
:exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
[org.jcuda/jcublas-natives :classifier "apple-x86_64"]]}}}
which results in
Error building classpath. Could not find artifact org.jcuda:jcuda-natives:jar:apple-x86_64:0.9.0 in central ()
I also tried
{:deps {uncomplicate/neanderthal {:mvn/version "0.19.0"
:exclusions [org.jcuda/jcuda-natives org.jcuda/jcublas-natives]}}}
which results in the same error. Is there a way to get this to work with a deps.edn?#2019-01-0320:24Alex Miller (Clojure team)exclusions in Maven can’t use classifiers afaik (certainly the deps.edn version does not support it)#2019-01-0320:28Alex Miller (Clojure team)I would expect something like this to work {:deps {uncomplicate/neanderthal {:mvn/version "0.19.0" :exclusions [org.jcuda/jcuda-natives org.jcuda/jcublas-natives]}}}#2019-01-0320:31kennyThat results in
Error building classpath. Could not find artifact org.jcuda:jcuda-natives:jar:apple-x86_64:0.9.0 in central ()
#2019-01-0320:31kennyHaving the exclusions that are in project.clj works as expected.#2019-01-0320:31kenny… in a project.clj.#2019-01-0320:31Alex Miller (Clojure team)yeah, the error there is accurate - there is no such jar#2019-01-0320:32Alex Miller (Clojure team)I guess this could be the exclusion code not being classifier aware#2019-01-0320:32Alex Miller (Clojure team)so it doesn’t know to try to exclude the os-dependent dependency#2019-01-0320:33Alex Miller (Clojure team)could I trouble you to file a ticket at https://dev.clojure.org/jira/browse/TDEPS ?#2019-01-0320:33Alex Miller (Clojure team)assuming that’s the issue, it shouldn’t be hard to fix#2019-01-0320:45Alex Miller (Clojure team)yeah, that was it. I have a fix for it#2019-01-0321:09kennyGreat! Should I still file a ticket?#2019-01-0321:38Alex Miller (Clojure team)yes please#2019-01-0321:38Alex Miller (Clojure team)or I can do it if you want#2019-01-0322:59kennyOk will do in a bit. Is there a workaround for now?#2019-01-0323:49Alex Miller (Clojure team)No, don’t have one, sorry. I can rev a new version tomorrow with the fix though#2019-01-0401:10kennyThat'd be great. What would I need to do on my end to get that to work?#2019-01-0414:39Alex Miller (Clojure team)I filed a ticket at https://dev.clojure.org/jira/browse/TDEPS-112 and am working on a release. You’ll need to either “brew upgrade clojure” if on mac or re-install if on linux once it’s out (will take a few hours)#2019-01-0416:32kennySorry, meant to file the ticket - yesterday was quite busy. Thank you Alex!#2019-01-0416:58kennyI assume the brew upgrade will be available once this PR goes through? https://github.com/Homebrew/homebrew-core/pull/35692#2019-01-0417:41Alex Miller (Clojure team)yes{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 5")}
#2019-01-0417:42Alex Miller (Clojure team)usually takes up to 24 hrs#2019-01-0722:49kenny@alexmiller I am still getting the same exception after running brew upgrade clojure.#2019-01-0722:53kennyActually I needed to open a new terminal. It appears to be working now.#2019-01-0803:19Alex Miller (Clojure team)👍#2019-01-0323:06kennyI am getting this error when using clojure on a different computer than my usual one.
Error resolving /Users/Mark1/Provisdom/math:
I can run git clone successfully. Why would the Clojure CLI not be picking this up?#2019-01-0323:49Alex Miller (Clojure team)It uses jgit, so is a different pile of stuff#2019-01-0323:50Alex Miller (Clojure team)Can’t say I have enough to go on there#2019-01-0323:50Alex Miller (Clojure team)What os? Ssh?#2019-01-0323:50Alex Miller (Clojure team)Private or public repo?#2019-01-0323:53kennyMac OS. SSH. Public.#2019-01-0400:27Alex Miller (Clojure team)run ssh-add -l to see if the ssh agent is running and has an identity in it (you should get probably 1 line back)#2019-01-0400:28Alex Miller (Clojure team)given that it’s public, you could use the https url too, which would probably just work#2019-01-0400:29Alex Miller (Clojure team)if you have an ssh identity though, the likely problem areas are your /.ssh/config, or something with /.ssh/known_hosts or the host key types#2019-01-0401:10kennyI'll just switch the https for now and deal with it in the future if I need to access private repos.#2019-01-0410:56rickmoynihanFYI peeps there is a new version of lein-tools-deps that uses 0.6.474.
The main improvement is the TDEPS-82 fix which significantly improves runtime performance as we no longer resolve deps twice (Thanks Alex!). Also we no longer hardcode how we locate the clojure command line tool.
https://github.com/RickMoynihan/lein-tools-deps#2019-01-0412:52jeroenvandijkThank you!#2019-01-0413:11rcustodioDo you guys use depstar more or boot with this?#2019-01-0413:46jeroenvandijkI haven't used it on our major projects (leiningen, still too much hassle to switch). On new projects I'm working on boot and juxt.pack. expect for uberjar-ing boot works as expected#2019-01-0413:57rcustodioI see, good to know, thanks#2019-01-0417:05seancorfieldWe switched from Boot a few months ago and our entire dev/test/build pipeline is now based on clj/`deps.edn` and uses my fork of depstar (which has bug fixes and also the ability to create "thin" JAR files such as you'd push to Clojars).#2019-01-0417:11rcustodioIs that fork public?{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-0419:31seancorfieldYes, and deployed to Clojars for easy usage. #2019-01-0419:32seancorfieldhttps://github.com/seancorfield/depstar#2019-01-0419:33rcustodioThanks!!#2019-01-0413:59rcustodioI’ve added javax.xml.bind/jaxb-api {:mvn/version "2.3.1"} but I’m having this Error building classpath. Could not find artifact javax.xml.bind:jaxb-api:jar:2.3.1 in central ()#2019-01-0413:59rcustodioCan anyone help me?#2019-01-0413:59rcustodio:deps {br.com.vikingmakt/augustus {:mvn/version "0.1.12"}
br.com.vikingmakt/njord {:mvn/version "0.3.0"}
cheshire {:mvn/version "5.8.1"}
clj-http {:mvn/version "3.9.1"}
clj-time {:mvn/version "0.15.0"}
com.novemberain/langohr {:mvn/version "5.0.0"}
com.novemberain/monger {:mvn/version "3.5.0"}
javax.xml.bind/jaxb-api {:mvn/version "2.3.1"}
org.apache.logging.log4j/log4j-core {:mvn/version "2.11.1"}
org.clojure/clojure {:mvn/version "1.10.0"}
org.clojure/tools.logging {:mvn/version "0.4.1"}}
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
#2019-01-0414:00rcustodioNo idea what is wrong with it#2019-01-0414:04Alex Miller (Clojure team)that leading quote on “2.3.1” looks weird - maybe a smart quote? “2.3.1"#2019-01-0414:04Alex Miller (Clojure team)I would retype that :)#2019-01-0414:05rcustodiowoooow…. thanks @alexmiller for some reason emacs was not showing any diff on quotes to me#2019-01-0414:06Alex Miller (Clojure team)when I copy pasted, it came over as quote-space in my terminal#2019-01-0414:06Alex Miller (Clojure team)¯\(ツ)/¯#2019-01-0414:06rcustodioI see.. next time I will try that, thanks#2019-01-0417:05seancorfieldWe switched from Boot a few months ago and our entire dev/test/build pipeline is now based on clj/`deps.edn` and uses my fork of depstar (which has bug fixes and also the ability to create "thin" JAR files such as you'd push to Clojars).#2019-01-0417:08orestisI’m using @seancorfield ‘s depstar and it works just fine. I might want to do AOT eventually but for now it does the job. #2019-01-0417:09seancorfieldAOT is evil. Stay away! 😇#2019-01-0417:11orestisI’ve seen you saying that before, but I’m not sure what the downsides are for things that go to production? Upside is less than half of the startup time (we went from 8s to 23s when switching from cambada to depstar). #2019-01-0419:28seancorfieldAOT of an entire app as the last step before building an uberjar for production deployment is fine if you care about startup time. #2019-01-0420:16orestisThanks!#2019-01-0417:11orestisI’d love to know what I’m missing :)#2019-01-0417:19aviI’m finding AOT useful for CLI tools#2019-01-0417:28ghadidepstar and AOT are not incompatible#2019-01-0418:39trevorCan I have a java (maven) project in github as a dependency with tools-deps?#2019-01-0418:45dominicmNo#2019-01-0418:45Alex Miller (Clojure team)no#2019-01-0418:45Alex Miller (Clojure team)tools.deps / clj don’t do “builds” so there is no opportunity to compile the java source#2019-01-0418:45Alex Miller (Clojure team)you will need to build and deploy the Java source as an artifact somewhere and depend on it that way{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-0514:36mkvlrnot sure if that's something worth considering but the new $classifier syntax doesn't play nice with http://jitpack.io.#2019-01-0515:02Alex Miller (Clojure team)Why? If there’s an issue, then please provide details or a ticket. I don’t get how it could affect jitpack#2019-01-0516:44mkvlrok, will do that Monday morning#2019-01-0514:46mkvlrand something seems not to be quite right yet for our use case (TDEPS-50) as we're seeing java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider. Not sure yet if it's related to the uberjar or what because it works in development. Will dig deeper on monday.#2019-01-0515:01Alex Miller (Clojure team)That sounds like a native lib issue. There is a separate ticket to consider support around native libs - tools like lein do a bunch of extra stuff to make that work and I’m not sure yet if we need to take that on inside deps#2019-01-0517:55mkvlr@alexmiller oh sorry, please forget both things I've written above, it was entirely my fault: in our production build script I was explicitly requiring an old tools.deps version that didn't understand the $classifier syntax yet 🙈 Now that I've fixed that it's all working great 👌 Sorry again, thanks for your pointer and your great work. Enjoy the rest of your weekend!#2019-01-0606:13duncanmakwhere’s the source code to the clj tool?#2019-01-0606:32lilactownnot sure if it’s the source of truth, but I found the code here: https://github.com/clojure/brew-install/tree/1.10.0/src/main/resources#2019-01-0606:32lilactownclj calls clojure with rlwrap#2019-01-0609:02pesterhazy@duncanmak main code here https://github.com/clojure/tools.deps.alpha#2019-01-0609:03pesterhazywrapper scripts in the brew-install repo (not only for macOS but also Linux0#2019-01-0611:44Alex Miller (Clojure team)Those are the correct places#2019-01-0618:11duncanmakmorning @alexmiller#2019-01-0618:11duncanmaki’ve been clicking around, trying to understand how clj and tools.deps work#2019-01-0618:12duncanmakafaict, it’s all java code, so my question is, where / when does it change the classpath? does it start the jvm twice? once to calculate the right classpath, and once to actually run the instance?#2019-01-0618:13duncanmakand also, i’ve been looking for the code of the procurers#2019-01-0618:30Alex Miller (Clojure team)Yes, it runs the jvm twice - once to compute the classpath then again to launch your program#2019-01-0618:30Alex Miller (Clojure team)Caching is used to avoid the first step when possible#2019-01-0618:31Alex Miller (Clojure team)The procurer code is all under tools.deps.alpha.extensions#2019-01-0623:39duncanmak@alexmiller ahh, okay, so it’s not as magical as I thought, great!#2019-01-0623:39duncanmak@alexmiller where can i see it running twice, is that in the shell script wrapper?#2019-01-0623:39duncanmakhttps://github.com/clojure/brew-install/blob/1.10.0/src/main/resources/clojure#2019-01-0623:40duncanmakso I’m guessing, this is the first invocation, to resolve-tags: https://github.com/clojure/brew-install/blob/1.10.0/src/main/resources/clojure#L179#2019-01-0623:40duncanmakand then this is the 2nd and final run: https://github.com/clojure/brew-install/blob/1.10.0/src/main/resources/clojure#L346#2019-01-0700:07seancorfield@duncanmak Line 179 is only run if you ask clj to resolve tags for you. Optional. But required before building the class path from git repos.#2019-01-0700:08seancorfieldThis is where the classpath is built (unless it is already cached): https://github.com/clojure/brew-install/blob/1.10.0/src/main/resources/clojure#L301#2019-01-0700:10duncanmakCool#2019-01-0700:10duncanmakVery clever!#2019-01-0700:10duncanmakI'm working with Kawa Scheme, I'm hoping to build a similar thing for it also #2019-01-0715:10borkdudeit it me or is tools.deps missing from this question in the Clojure Survey?
22. Which tools do you use to compile/package/deploy/release your ClojureScript projects?#2019-01-0715:13Alex Miller (Clojure team)that’s ClojureScript-specific - is that a thing?#2019-01-0715:14Alex Miller (Clojure team)I just added it, so it’s there if people want to pick it#2019-01-0715:15borkdudewell, I do a lot of things with cljs.main, which uses tools.deps?#2019-01-0715:15borkdudelike compiling. I deploy via scp.#2019-01-0715:15dpsuttonbe nice if cljs-test-display was added to the cljs testing question as well#2019-01-0715:19borkdudeso maybe cljs.main should be the option instead of tools.deps, dunno 🙂#2019-01-0715:26Alex Miller (Clojure team)I changed it to “clj / cljs.main”#2019-01-0715:26Alex Miller (Clojure team)what’s cljs-test-display?#2019-01-0715:27Alex Miller (Clojure team)that seems like a visualization tool rather than the test runner?#2019-01-0715:30dpsuttoni suppose. then credit would go to figwheel and cljs.test for running the actual tests. but i feel like cljs-test-display is more than that. i just mention because that's my interface to all of our cljs tests and i was looking for it in the list#2019-01-0715:30dpsuttonand this is the worst room for this sorry 🙂#2019-01-0715:34Alex Miller (Clojure team)ha, no worries - there is an Other category for stuff like this - often new stuff shows up there and then I add it the following year if it’s strong enough#2019-01-0720:14duncanmakhey @alexmiller, i also started looking into Groovy’s Grape, which I see to be something very similar to tools.deps for Clojure#2019-01-0720:15duncanmakis the choice of using Ivy vs Maven (or Eclipse Aether) significant? Or do the two APIs basically do the same thing and it’s just a personal choice to use one or the other?#2019-01-0721:00Alex Miller (Clojure team)I’ve spent some quality time with Grape in the past and worked on a build system that used Ivy extensively as well, so I’m somewhat familiar with both. I have never looked at Ivy’s actual code to see if it could satisfy what we needed wrt Maven repositories (although I’d be somewhat surprised if it didn’t ultimately use the same Maven APIs we are already using).#2019-01-0721:19Alex Miller (Clojure team)I glanced at it and it’s actually not using the Maven APIs at all afaict, so would be an alternate path to investigate for downloads at some future time (could also jettison both and just do the moral equivalent directly in Clojure, which might actually be better). We do use a bunch of Maven stuff to read pom files and we’d need that regardless though.#2019-01-0721:37kennyWhen starting a REPL with clj, is there a way to set default JVM opts?#2019-01-0721:39Alex Miller (Clojure team)no, but that is a ticket in the system for consideration#2019-01-0721:40Alex Miller (Clojure team)you can pass them explicitly on the command line by prefixing with -J though like -J-Xmx1g#2019-01-0721:41kennyHmm ok. Any idea what the default -Xmx is?#2019-01-0721:41kennyOr maybe it doesn't set it at all? https://github.com/clojure/brew-install/blob/1.10.0/src/main/resources/clojure#L346#2019-01-0721:42Alex Miller (Clojure team)in latest JVMs there is no default - it’s computed as a fraction of system resources I think, or something like that#2019-01-0721:43Alex Miller (Clojure team)and more modern GCs can also release heap back to the OS#2019-01-0721:43Alex Miller (Clojure team)so no, we’re not setting any jvm settings - that’s up to you to determine#2019-01-0721:46kennyGot it, thanks.#2019-01-0721:51duncanmak@alexmiller thanks!#2019-01-0814:49johnI seem to be stuck on betas with brew#2019-01-0814:50johnI just ran brew upgrade clojure and it just updated with clojure 1.10.0.403 -> 1.10.0.411#2019-01-0814:50johnand
user=> *clojure-version*
{:major 1, :minor 10, :incremental 0, :qualifier "beta6"}
#2019-01-0814:52johnmy global deps.edn, duh :face_palm:#2019-01-0815:21borkdudeyeah, that’s what -Srepro is for 🙂#2019-01-0815:57Alex Miller (Clojure team)yeah, if you have a ~/.clojure/deps.edn, it won’t be modified (in case you’ve added your own config to it)#2019-01-0911:41gphilippLet’s say I have a repo named foo with 3 logical modules in their own subdirectory: a, b and c. Each have their own deps.edn with their required dependencies. I can also have a root deps.edn which refers to these modules using 3 {:local/root ...} deps. Now, I fail to see how I can add a dependency using the git dep scheme on say, module foo-b from brand new project bar which lives in a separate repo. Has anyone done something similar before ?#2019-01-0915:24gphilippWell, @matthias.margush just mentioned to me that actually we can add a :deps/root value to a git coordinate to specify a subrepo to use a subdirectory. That’s the answer to my issue above, but alas it’s not (yet) documented in https://clojure.org/reference/deps_and_cli or https://clojure.org/guides/deps_and_cli.#2019-01-0915:27Alex Miller (Clojure team)there’s an issue to doc it at https://github.com/clojure/clojure-site/issues/307, just haven’t gotten to it{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 20")}
#2019-01-1010:55gphilippThx @alexmiller. While I’m at it, we’ve been bitten by https://dev.clojure.org/jira/browse/TDEPS-74 (deps transitivity issue involving :local/root). Currently the scenario whereby project foo depends on module a of project bar using bar {:git/url "" :deps/root "a"} and bar.a depends on siblings module bar.b using {b {:local/root "b"}} doesn’t work. @matthias.margush has put together a patch which solves the issue (more info on the ticket), if you have some time to look at it that would be great. We’re not far from being able to switch over to tools.deps for several multi-modules projects.#2019-01-1011:41Alex Miller (Clojure team)yeah, there are several issues with the path stuff and I don’t think the change there is sufficient to handle the real scope of the issues#2019-01-1015:49matthiasThat makes sense - it did seem like the way coordinate specifiers interact could use some hammock time overall.#2019-01-1015:49matthiasWhat are the other jiras that relate?#2019-01-1015:52Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-106#2019-01-1015:52Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-96#2019-01-1015:52Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-52#2019-01-1015:53Alex Miller (Clojure team)some of those intersect with other problem areas (config file management)#2019-01-1418:59matthiasHi Alex. Reading through these in more detail, it looks fixes could potentially be bundled together with a fix for TDEPS-74... they are closely related, right? If it would be helpful at all, I'd be willing to take a look at these in context of the patch I started for TDEPS-74.#2019-01-1015:53Alex Miller (Clojure team)I’d say it’s unlikely I’m going to look at this set of things this week#2019-01-1016:28matthiasOh yeah, not expected. Just want to get it on the radar. tools.deps really opens up the ways that we can organize work across teams, and thanks for the work you all are doing on it.#2019-01-1015:55matthiasOne thing I noticed personally was that transport, location and manifest type are different things that relate and interact, but don’t map conceptually one to one to how extensions are dispatched.#2019-01-1016:00Alex Miller (Clojure team)transport and location aren’t uniform dimensions for procurers#2019-01-1016:01Alex Miller (Clojure team)manifest types are something that extensions dispatch on#2019-01-1016:23matthiasAre procurers things like maven, local, and git (and not pom, deps)?#2019-01-1016:55Alex Miller (Clojure team)Yeah - there are two layers of multimethods#2019-01-1017:36matthiasI see.#2019-01-1017:43matthiasI think possibly related to your comment about uniformity of dimensions - a capability I’d like to be able to support for our engineers is to checkout a git/deps dependency locally while working on a cross cutting feature. External tooling could support this well enough for example by transforming a git coordinate to a local one in the local work area.
But the differences in the set (& maybe the semantics) of dimensions make the creation of tooling more bespoke than needed. I think specifically in this use case it would help for the :deps/root to work the same between git coordinates and local coordinates. (Unless that’s already the case and I misread the code.)
So such a tool needs to know the dimensions available and how they are used for each kind of coord.#2019-01-1023:40duncanmak@alexmiller i’m interested in learning more about how to download jars without using maven/ivy - do you know of anything that talks about how that works?#2019-01-1023:52seancorfield@duncanmak If you know the URL of the JAR in the repository, you can just download it via any means -- such as curl at the command line, or programmatically using URI and byte input streams etc.#2019-01-1023:53seancorfield(unless the repo is secured and then you'll need to factor in auth credentials)#2019-01-1100:01hiredmanmaven has different plugins for talking to repos, but the most common transport for talking to repos is http, so a maven repo can just be directory containing files on a webserver#2019-01-1100:02hiredmanthe directory tree structure in that case is actually the same structure as ~/.m2, so you can upload your m2 to s3, make it publicly accessible, then use it as a maven repo#2019-01-1100:03hiredmanthe complicated part of maven is parsing and understanding pom files#2019-01-1100:04hiredman(for example this is the clojars maven repo https://clojars.org/repo/)#2019-01-1100:12duncanmaki see#2019-01-1112:18eval2020Is dependency resolution in any way influenced by the contents of /.m2? Ie I’m considering caching /.m2 on CI but was wondering whether CI-runs without a local repository might actually yield helpful early warnings/errors (eg library updates that cause problems).#2019-01-1112:24jeroenvandijkNot caching would tell you if a dependency suddenly disappeared (which shouldn't happen for at least clojars), but otherwise I cannot think of a reason to do it. If the library is present locally it will not do an additional fetch unless you use -SNAPSHOT versions, which is probably not desirable anyway. Having said that i'm curious for other answers to this question#2019-01-1112:34eval2020that’s indeed what I would expect. I see there’s also .cpcache/ which seems to contain the actual result of resolution.#2019-01-1113:49jeroenvandijkActually now I come to think about git deps, where force pushes and deletion of repo's are a possibility. This might be a valid concern#2019-01-1113:50jeroenvandijkBut git deps don't really interact with ~/.m2 deps i think#2019-01-1115:29eval2020indeed - these end up in ~/.gitlibs and it seems you can’t set this path (as opposed to mvn/local-repo) so not easily cacheable on gitlab-ci.#2019-01-1116:05j0nidepending on your CI solution, you could add a full dependency pull to a step which depends on tests passing, so you don't waste time on it as part of your regular feedback cycle#2019-01-1113:38mike_ananev@alexmiller Hi. Is there any updates with cli tools for Windows? very expected feature#2019-01-1113:39Alex Miller (Clojure team)no update, sorry#2019-01-1115:45borkdude@mike1452 how is the WSL support in Windows nowadays?#2019-01-1116:56seancorfieldclj/`deps.edn` works really well on WSL. The problem you then get is editor integration of you use a windows editor with your repl running on Linux. #2019-01-1116:57seancorfield(You can't "load file" from windows because Linux won't understand the path the editor sends to it)#2019-01-1116:59seancorfieldThere are workarounds but it's a less than ideal process. I tend to select-all + evaluate (and I have symlinks set up in WSL so the file paths match windows, modulo the drive colon prefix). #2019-01-1120:51timgilbertSay, anyone executing their applications via clojure inside of docker images? Want to share any findings? I'm wondering how to handle caching (ie, whether to clojure -Sforce -Spath to download deps to the image, then cache them for faster startup)#2019-01-1121:23mike_ananev@borkdude corporate windows policies prohibits using WSL. so, half of our team, with BYOD devices, using clj cli tools, and another half with corporate PCs (Windows OS only) should use leiningen, which is available for Windows and allowed in our network . it is very inconvenient. we are still waiting for the windows version of Clj cli tools. Well, "First rule of Open Source, never ask when something will be done". And it is sad. 😞#2019-01-1121:44duncanmakhmm, interesting#2019-01-1121:44duncanmakwhat is making clj Unix-dependent?#2019-01-1121:46hiredmanit needs a powershell port I guess, which is an issue lein had a long for time, basically the people writing the tools don't use windows much and don't use powershell, so it is difficult for them to do the port or review it#2019-01-1121:53dottedmagI had to port a bunch of shell code to PS recently, and it was not so difficult, but annoying as hell. Favourite tidbits: 1) downloading a file with a progress indicator enabled is awfully slow (it looks like this indicator gets updated for every byte, and terminal on Windows is quite slow), and this progress indicator is enabled by default; 2) extracting a file from Zip using PS standard library cmdlet is ~10 times slower than directly invoking .NET code this cmdlet is wrapping.#2019-01-1121:55dottedmagOTOH one does not have to worry about differences between supported grep options or check if the tool is called shasum or sha256sum.#2019-01-1122:15Alex Miller (Clojure team)I have a not quite working port of clj to powershell. In the journal I just posted I asked people interested in actually working on dev to ping me here. If someone(s) are interested in really digging in, let me know and I will do what I can to guide.#2019-01-1303:24ajoberstar@alexmiller I've got some PowerShell experience and would be interested in helping this along. What still needs to be done?#2019-01-1203:53kenji_signifierHi, I was wondering if you have thought about adding clojure CLI to SDKMAN https://sdkman.io. I had a conversation with the main author Marco Vermeulen in https://gitter.im/sdkman/cli-dev and he needs a tarball that is usable by just untar. Leiningen added the support while ago. Plz check out https://github.com/technomancy/leiningen/issues/2204#2019-01-1204:29Alex Miller (Clojure team)one tricky thing in the clj script is that the script gets modified on install to point to the absolute location of the lib dir#2019-01-1204:29Alex Miller (Clojure team)not sure what the best way to handle that is#2019-01-1204:30Alex Miller (Clojure team)but feel free to file an issue at https://dev.clojure.org/jira/browse/TDEPS#2019-01-1204:31Alex Miller (Clojure team)would be happy to add it but I’m not going to have time to work on it, some whatever help you could provide would be useful#2019-01-1204:33Alex Miller (Clojure team)the current install is basically a tar file plus an install script that mostly just moves files around, but also makes the jar location replacement mentioned above#2019-01-1206:22kenji_signifier@alexmiller Created https://dev.clojure.org/jira/browse/TDEPS-113 I’m happy to help this out. Should I enhance https://github.com/clojure/brew-install by creating a PR to add another package.sh for SDKMAN under script dir, or following the contribution workflow and sign the CA?#2019-01-1206:29Alex Miller (Clojure team)We don’t do PRs in this repo so patch on the jira would be better#2019-01-1206:30Alex Miller (Clojure team)And ca is required#2019-01-1206:35seancorfieldSee https://clojure.org/community/contributing for the process @kenji_signifier if you haven't already seen it...#2019-01-1207:30kenji_signifierthx @alexmiller and @seancorfield I’ll follow the workflow once I create a patch for it.#2019-01-1308:59dottedmagIs there a way to have global :jvm-opts?#2019-01-1309:00dottedmagBesides making a shell wrapper around clj.#2019-01-1313:54Alex Miller (Clojure team)Currently no, that’s an enhancement request#2019-01-1419:32borkdudesince there is no cognitect channel and this is a tools.deps project, I’ll post it here. there’s a trivial PR lingering for a while now. I wonder why there’s no interesting in accepting it from Cognitect Labs:
https://github.com/cognitect-labs/test-runner#2019-01-1419:33borkdudeI even forgot about the previous PR and made a duplicate one 🙂#2019-01-1419:36Alex Miller (Clojure team)Sorry, Luke was minding that project and he’s not at Cognitect anymore#2019-01-1419:36Alex Miller (Clojure team)I’ll take a look when I get a chance#2019-01-1419:37borkdudecool 🙂#2019-01-1419:37borkdudeI’ve been using it more and more. There’s also a CLJS clone of it#2019-01-1419:40borkdudeghadi made the first PR. he is at cognitect right?#2019-01-1419:57dominicm@U050ECB92 is this any you became a space cat wizard? #2019-01-1419:48Alex Miller (Clojure team)He is#2019-01-1419:51ghadimerged @borkdude#2019-01-1419:51ghadi209b64504cb3bd3b99ecfec7937b358a879f55c1 is master now{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 20")}
#2019-01-1421:03aviAre there any examples out there of libraries that have both a project.clj and also a deps.edn, for compatibility with both Leiningen and tools.deps, but don’t duplicate the dependency specifications? I’m interested in possibly contributing such a feature to a library I use.#2019-01-1421:05dpsutton:plugins [[lein-tools-deps "0.4.1"]] is in one of our project.clj files. This allows the lein project to read deps out of the deps.edn file#2019-01-1516:06rickmoynihan:plugins [[lein-tools-deps "0.4.3"]] is the latest and should be a bit quicker at resolution.#2019-01-1520:27aviThanks!#2019-01-1521:04thosmosI've been using that for quite a while, and it works great. but now Cursive supports deps.edn directly so I mostly use that{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-1609:52rickmoynihanlein-tools-deps is not without its issues… I should probably make them clearer on the repo.#2019-01-1613:52aviOnce I read the docs, I was up and running with no clear issues, at least not at first. Maybe some of the steps could be moved to the top of the readme, but no big deal#2019-01-1613:56rickmoynihanGood to know… but it can cause problems if you’re not aware of how it works… and it’s not made clear yet.
Essentially it currently flattens the whole transitive dependency graph. It does this to guarantee resolution is the same in both tools.deps and leiningen — but it does mean that downstream mvn resolution semantics can change.#2019-01-1613:57rickmoynihane.g wrt to :exclusions#2019-01-1614:45aviThat sentence seems very important… if it’s not in the readme already maybe just add it as-is?#2019-01-1614:50rickmoynihanI think it warrants a small section covering the implications tbh — will write it at some point.#2019-01-1614:51avi💪#2019-01-1614:51aviI’d be happy to review if that’d be helpful#2019-01-1614:53rickmoynihanyou’re welcome to comment on what is written — but it’ll probably be ok. Too busy to do it just now though.#2019-01-1614:53rickmoynihanthere are github issues that touch on it already#2019-01-1614:54avigot it!#2019-01-1421:05dpsuttonMakes it very simple to use lein for building/deploying but deps.edn for dev and running#2019-01-1421:39aviWill check it out, thank you!#2019-01-1421:42hiredmanI just pushed https://git.sr.ht/%7Ehiredman/dset/tree up over the weekend, it has both a deps.edn and a project.clj, with all the deps coming from deps.edn#2019-01-1500:14lilactownhow do you like using http://sr.ht?#2019-01-1500:21hiredmanso far all I have done is create that repo and push to it, seems fine, I appreciate that I can browse the site without letting it execute js#2019-01-1520:27aviGreat, thanks for sharing!#2019-01-1500:13richiardiandreawe are doing like the above, I have also a deps.edn only project but having lein makes it easy for colleagues to use Cursive, which still has some quirks with deps.edn#2019-01-1500:43cfleming@richiardiandrea It does? I’d like to hear about them.#2019-01-1500:44richiardiandreaah ah for sure 😄 one of my colleagues opened a couple of issues so you are aware of everything I think, don't remember which ones....wherever you see monorepo is us I guess!#2019-01-1521:46cflemingOk, I’ll check those out, thanks - AFAIK most things should be fixed these days.#2019-01-1515:16vemv👋
I asked this yesterday in off-topic, wondering what you think about it https://clojurians.slack.com/archives/C03RZGPG3/p1547483045535400
One thing I would fear if people widely adopted "SHAs for every dependency" is that two different libraries (let's call them A and B) could independently declare a dependency on C, but at different SHAs.
Is that a scenario that tools.deps can gracefully handle?#2019-01-1515:16Alex Miller (Clojure team)yes#2019-01-1515:16Alex Miller (Clojure team)well depends what part you’re asking about#2019-01-1515:16Alex Miller (Clojure team)if you declare two sha’s in the transitive dep tree, it will use the newer one#2019-01-1515:21vemvthank you! I'll look into using tools.deps at work then 🙂#2019-01-1515:25Alex Miller (Clojure team)Where newer = more descendant. If there is no descendant relationship (independent branches) then it’s an error{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 20")}
#2019-01-1515:17Alex Miller (Clojure team)(same as we do for Maven artifacts)#2019-01-1520:37aviSo in this library I’m hoping to help out with, I’ve moved the dependencies from project.clj to deps.edn and added the plugin lein-tools-deps to project.clj. lein deps seems to work just fine, but when I run lein test I get:
$ lein test
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main
Tests failed.
I tried changing the clojure version specified in deps.edn to 1.9.0 and 1.10.0 but that seemed to have no effect. Anyone know what’s happening here, what I’m missing?#2019-01-1520:49ghadiseems like your tests are not picking up the dependencies#2019-01-1520:50ghadiI wonder if your lein-tools-deps plugin isn't working in the test profile#2019-01-1520:57aviHmm, OK, I’ll look into that — thanks!#2019-01-1521:07aviAha, I needed to add :middleware and :lein-tools-deps/config to project.clj — as explained clearly in the docs of lein-tools-deps! (docs ftw)#2019-01-1521:07thosmos@aviflax I had that same problem with Cursive and the only thing I found to get around it was to downgrade lein-tools-deps to "0.4.1"#2019-01-1521:09aviHave you tried adding those keys to your project.clj ?#2019-01-1523:42aaelonyI have a codebase that I’ve instrumented with tools-deps and can run via clj -m something.main <args>
from the command line. What do folks typically do or what guidance is out there to make it easier for other (non clj) users to run it. In the past, I’d have done a lein uberjar and wrapped it in a bash script. Is there a better way?#2019-01-1600:05seancorfieldYou can still produce an uberjar and/or wrap it in a bash script https://github.com/seancorfield/depstar{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-1603:14souenzzoAbout depstar/tools-deps jar'ing:
I need to deliver a jar that will run with java -jar my.jar. It's a client requirement.
There is some way to create it using tools-deps eco-system??
PS: without use things like clj -A:lein -m lein.uberjar#2019-01-1604:53richiardiandreaI am using https://github.com/juxt/pack.alpha but I am not exactly sure it does the same as leiningen#2019-01-1618:41dominicmPack works differently to leiningen, intentionally. #2019-01-1618:51seancorfieldI started to look at adding manifest support to depstar and realized I'd have to deal with AOT/`compile` unless you just set Main-Class: in the manifest to clojure.main... and I didn't have any enthusiasm for fixing that part 🙂 I may take another run at that at some point (it seems like it needs to create a temp folder, add it to the classpath, bind *compile-path* to that temp folder path, and run clojure.core/compile for the named main namespace … which just seems like a lot of work, just to enable java -jar and not need -m entry.point!).#2019-01-1618:55dominicm@seancorfield can depstar take a flag to add a folder? That's how pack makes AOT work #2019-01-1618:56seancorfielddepstar deliberately takes no options at the moment. It's "simple", rather than "easy" 🙂#2019-01-1700:31souenzzoMaybe someone will make this simple thing easy... 😜#2019-01-1601:26ajoberstar@alexmiller Still looking for community contributions on the Windows clj script? I'm interested and have PowerShell experience.
(Sorry for the dupe if you already saw my previous post and just haven't had a chance to reply.)#2019-01-1601:44Alex Miller (Clojure team)I saw it, thx, just been busy on other stuff{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-1603:14souenzzoAbout depstar/tools-deps jar'ing:
I need to deliver a jar that will run with java -jar my.jar. It's a client requirement.
There is some way to create it using tools-deps eco-system??
PS: without use things like clj -A:lein -m lein.uberjar#2019-01-1603:27seancorfield@souenzzo yes some of the tools.deps libraries add a manifest when building an uberjar. I had it in depstar at one point but preferred fewer options. {:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-1603:29seancorfieldOpen an issue and I'll put that code back in. Or you can try the other tools. I'm on my phone so I can only point you at the wiki in the tools.deps repo for a list. #2019-01-1606:19seancorfield(It would help if Issues were enabled for my project I guess… fixed!)#2019-01-1618:42dominicm@seancorfield any advice for running clj on WSL? Gotchas to be aware of? Preparing for a training course, I seem to have edge working, but I'm expecting real life issues. #2019-01-1618:44seancorfieldThe only annoying thing I've run into is that if you edit code on Windows, connected to a REPL of any sort running on WSL, any file-level ops you attempt will fail due to the mismatch of filepaths between the two environments. For example, a hot key to "load file" in your editor will generally send a full file path to the REPL, which in this case will start with C:\… when the WSL REPL is expecting /…#2019-01-1618:45seancorfieldMy workaround for that is select-all, eval-selection (but I may try to "fix" it in Chlorine directly).#2019-01-1618:46seancorfieldOh, caveat for that: I use a project repo path on Windows of something like C:\projects\myrepo and have /projects symlinks on WSL to /mnt/c/projects so the file paths are almost the same.#2019-01-1618:47seancorfield(that way, just dropping C: and using / instead of \ will work on both sides -- modulo the editor file path issue)#2019-01-1618:48seancorfieldAlso, since WSL is headless, you can't run REBL on it (so you end up having to hack that together with Leiningen on the Windows side.#2019-01-1618:48seancorfield(I haven't made any serious attempt to try to debug/workaround that)#2019-01-1618:49dominicmThat's a useful tip, thanks! #2019-01-1821:42Christian Johansenis there a special trick to deploying ClojureScript libraries to clojars with tools.deps? I tried doing clojure -Spom and then mvn deploy. I certainly did put a jar in clojars, but trying to use it I discover that it doesn't contain any sources :thinking_face:#2019-01-1822:23eval2020@christian767 The pom needs build > resources. For a cljs-lib I use this pom: https://gitlab.com/eval/otarta/blob/master/pom.xml. Check contents of jar with jar tf foo.jar.#2019-01-1822:24Christian JohansenThanks!#2019-01-1822:24Christian JohansenSeems my mistake was not building the jar, D'OH!#2019-01-1822:24Christian Johansenthought it would be auto-built. and it was, but the jar maven built was useless, unsurprisingly, as it doesn't know about cljs files I guess#2019-01-1822:25Christian Johansenused mvn deploy:deploy-file with a prepackaged jar instead{:tag :div, :attrs {:class "message-reaction", :title "+1::skin-tone-2"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 5")}
#2019-01-1914:19Christian JohansenI've had various minor issues figuring out how to combine tools.deps, figwheel-main, devcards, emacs/CIDER and Clojars, and got a lot of help in here, and now I've summarized my experience in a howto on configuring this combination: https://cjohansen.no/tools-deps-figwheel-main-devcards-emacs/{:tag :div, :attrs {:class "message-reaction", :title "raised_hands"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🙌")} " 10")}
#2019-01-1921:01richiardiandreathanks this is nice#2019-01-1921:02richiardiandreathe only difference I have there is deploying with deps-deploy...also sooner or later I would like to be able to sign the jar.#2019-01-1914:19Christian Johansenhope it can be of help to someone#2019-01-1921:00richiardiandrea@christian767 I have recently added deps.edn-only deploy in https://github.com/elasticpath/fonda - actually taking inspiration from @eval2020's work#2019-01-1921:03richiardiandreasorry I read I am late...too late 😄#2019-01-1921:28Christian JohansenYeah @slipset tipped me about deps-deploy on Twitter, I'm going to update the article to recommend it in place of maven 👍#2019-01-1921:35slipset@richiardiandrea jar signing just landed on master for deps-deploy. Comments welcome.#2019-01-1921:36Christian Johansen👌#2019-01-1921:37slipsetHad to read a bit too much to figure out how to do that.#2019-01-1921:39slipsetThe lein code for that is somewhat b0rken (reading the gpg passphrase is a bit unstable at times), so I had to reimplement that…#2019-01-1921:39slipsethttps://github.com/slipset/deps-deploy/blob/master/src/deps_deploy/gpg.clj#L19#2019-01-1921:41Christian JohansenUgh, gpg. Can't think of anything that makes me feel more like a moron #2019-01-1921:42richiardiandreaWow timely! Thank you!#2019-01-1921:43slipsetInspired by @christian767’s blog 🙂{:tag :div, :attrs {:class "message-reaction", :title "champagne"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🍾")} " 5")}
#2019-01-1921:45Christian Johansen😊#2019-01-1921:55richiardiandreaThat blog is really comprehensive, I am thinking about making it a template now 😺 #2019-01-1922:00Christian Johansenhehe, please do. I considered making it a minimal github repo, but time didn't allow for it between shipping kids around, walking the dog and putting together IKEA furniture 🤷 🙂{:tag :div, :attrs {:class "message-reaction", :title "scream"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😱")} " 5")}
#2019-01-2317:22rschmuklerHey all - does anyone know if anything became of the add-lib function described here http://insideclojure.org/2018/05/04/add-lib/ ?#2019-01-2317:23rschmuklerI noticed a branch (`add-lib`) on the github repo, but it's ~20 commits behind master. Is it still preferable to use it?#2019-01-2318:15Alex Miller (Clojure team)we haven’t integrated it yet#2019-01-2318:16Alex Miller (Clojure team)I can merge up to master though if you want a master-compatible version of the branch#2019-01-2318:17Alex Miller (Clojure team)actually I did that recently, and add-lib branch is currently only like 1 commits behind master#2019-01-2318:22Alex Miller (Clojure team)merged, you can use sha e160f184f051f120014244679831a9bccb37c9de for latest#2019-01-2318:43rschmukler@alexmiller Thanks a ton! Heads up that I found a bug where a transient dependency wasn't reloaded into the classpath. I will see if it's fixed by the latest, and if not, open an issue.#2019-01-2318:43rschmuklerUnless you'd prefer me not opening issues for it, since it's not even part of a release yet#2019-01-2319:48Alex Miller (Clojure team)Did you mean transient or transitive?#2019-01-2319:49Alex Miller (Clojure team)add-lib won’t do re-loading - it’s for additive use#2019-01-2321:36ghadi@seancorfield have you run into any issues with depstar and multi release jars?#2019-01-2321:38seancorfield@ghadi What do you mean by “multi release”?#2019-01-2321:39Alex Miller (Clojure team)as of java 9 you can make jars with different per-jdk sets of stuff in them#2019-01-2321:40Alex Miller (Clojure team)https://www.javaworld.com/article/3184029/java-language/java-9s-other-new-enhancements-part-4-multi-release-jar-files.html#2019-01-2321:40seancorfieldI haven’t tried to do anything with Java 9+ yet — we have a bunch of legacy code that won’t run on it.#2019-01-2321:42ghadiI hadn't considered MRJars and was curious if it panned out#2019-01-2321:44seancorfieldGiven that depstar doesn’t add a manifest — and filters out some manifest-related stuff as I recall — I don’t know how it would even go about that.#2019-01-2321:45ghadiI forgot whether it filtered anything or not from the Meta-Inf#2019-01-2321:45seancorfieldI started looking at adding basic manifest support but that seemed to lead me down the AOT path, at least for the Main-Class: entry when it wasn’t clojure.main … and I decided I didn’t have the stomach for that (yet).#2019-01-2321:45ghadiIf it works it's by accident#2019-01-2321:46seancorfield(defn excluded?
[filename]
(or (#{"project.clj"
"LICENSE"
"COPYRIGHT"} filename)
(re-matches #"(?i)META-INF/.*\.(?:MF|SF|RSA|DSA)" filename)
(re-matches #"(?i)META-INF/(?:INDEX\.LIST|DEPENDENCIES|NOTICE|LICENSE)(?:\.txt)?" filename)))
#2019-01-2321:47seancorfield(which seems pretty arbitrary — and I haven’t rationalized why it excludes all of that lot)#2019-01-2322:21dominicmDoes anyone have a firm reason that pack should implement a depstar like implementation? #2019-01-2322:21ghadiwhat does that mean?#2019-01-2322:26dominicmRather, is there a reason pack should adopt "jar Smashing"? #2019-01-2322:31hiredmanI've got a thing I worked for a little bit, but I don't think I have the time to go anywhere with it: https://gist.github.com/hiredman/d68cafb6aa8cea563c7b77d54f522421 instead of building uberjars (which can end up taking forever to transfer) it uploads individual jars/git repos/local deps only if they haven't already been uploaded and then deploys a file containing a built classpath pointing to those things{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 5")}
#2019-01-2322:32hiredman(inspired by a talk rich gave somewhere about datomic ions)#2019-01-2407:22dominicmThat's really neat.#2019-01-2407:54plexushas anyone already come up with a thing that reloads deps.edn, adding any newly added libraries to the classpath?#2019-01-2408:19dominicmAlex's add-lib would probably be pliable to that. If you build it, I would be interested. #2019-01-2409:36thhellerFWIW I have done a few experiments with dynamically adding dependencies at runtime for shadow-cljs#2019-01-2409:37thhellerthe only case where this works reliably is when adding dependencies that don't depend on anything different than what is already loaded#2019-01-2409:37thhelleras soon as there are any version conflicts things get icky#2019-01-2409:38thhellereg. lib-a depends on X version 1.0.0 and is loaded. lib-b wants to be added but would cause X version 1.2.0 to be loaded instead of 1.0.0 if constructed normally#2019-01-2409:38thhellersince we can't easily unload version 1.0.0 you might now not be able to load properly#2019-01-2409:39thhellerunfortunately that scenario is quite common for bigger libs#2019-01-2409:39thhellerworks well enough for simple libs without deps though#2019-01-2409:39thhellergets even more complicated when accounting for npm deps 😉#2019-01-2409:39plexusI'm guessing pomegranate/alembic and friends also have this issue then#2019-01-2409:40thhellerdeps resolves version conflicts slightly differently so it is more prone to this#2019-01-2409:40thhellerbut yes same issue is present everywhere#2019-01-2409:40plexusI see#2019-01-2516:26Alex Miller (Clojure team)I don’t think deps is any “more prone” to this than pomegranate. it’s a fundamental issue with how libraries and classes are loaded on the jvm.#2019-01-2516:26Alex Miller (Clojure team)if you’re interested in clj on Windows, I have created #clj-on-windows as a working channel for anyone that wants to help with it#2019-01-2601:36matthiasHi @alexmiller. I uploaded a couple of patches for discussion on the tools.deps jira board. We are keen to get a few of those ironed out.
https://dev.clojure.org/jira/browse/TDEPS-52
https://dev.clojure.org/jira/browse/TDEPS-74
{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-2601:38Alex Miller (Clojure team)thx, I prob won’t have time till next Friday#2019-01-3001:21lilactownis there a way to just get whatevers at master with git deps?#2019-01-3001:21lilactowntry to write a quick-try command for a lib#2019-01-3002:34Alex Miller (Clojure team)No#2019-01-3002:35Alex Miller (Clojure team)By design, we want to use stable versions for caching purposes#2019-01-3002:35Alex Miller (Clojure team)But you can use the gitlibs lib to look up what sha master is#2019-01-3002:43Alex Miller (Clojure team)you can even one-line it - clj comes with a built-in :deps alias that includes tools.deps (and tools.gitlib):#2019-01-3002:43Alex Miller (Clojure team)$ clj -A:deps -e "(require 'clojure.tools.gitlibs)" -e '(println (clojure.tools.gitlibs/resolve "" "master"))'
d86fec647b705b5d01d27c32322c9ccc9f65b77c
$ clj -Sdeps '{:deps {org.clojure/tools.gitlibs {:git/url "" :sha "d86fec647b705b5d01d27c32322c9ccc9f65b77c"}}}'
Checking out: at d86fec647b705b5d01d27c32322c9ccc9f65b77c
Clojure 1.10.0
user=> (require '[clojure.spec-alpha2 :as s])
nil
user=> (s/conform (s/spec int?) 10)
10#2019-01-3002:48Alex Miller (Clojure team)rub some scripting on that and you’ve got something#2019-01-3003:45seancorfieldCould you combine that with the add-lib branch and dynamically load the master version into your REPL? (I can't remember whether add-lib works with git libs)#2019-01-3003:56Alex Miller (Clojure team)sure#2019-01-3003:57Alex Miller (Clojure team)add-lib works with any dep supported by tools.deps#2019-01-3004:07seancorfieldYeah, nice user=> (require '[clojure.tools.deps.alpha.repl])
nil
user=> (require '[clojure.tools.gitlibs])
nil
user=> (defn load-master [lib] (let [git (str "" lib ".git")] (clojure.tools.deps.alpha.repl/add-lib lib {:git/url git :sha (clojure.tools.gitlibs/resolve git "master")})))
#'user/load-master
user=> (load-master 'clojure/tools.trace)
Checking out: at fbba1a5012bae0ac4ed5cd704babf23e9e3fbc44
Downloading: org/clojure/clojure/1.4.0/clojure-1.4.0.pom from
Downloading: org/clojure/clojure/1.4.0/clojure-1.4.0.jar from
true
user=> {:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 5")}
#2019-01-3004:11seancorfield^ @lilactown#2019-01-3004:13lilactownClever! Thanks 😁#2019-01-3004:46seancorfieldI ❤️ Clojure 🙂{:tag :div, :attrs {:class "message-reaction", :title "cider"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ({:tag :img, :attrs {:alt "cider", :src "https://emoji.slack-edge.com/T03RZGPFR/cider/44d3c1a93e44ea40.png"}, :content nil})} " 5")}
#2019-01-3004:47seancorfield(interesting that clojure/tools.trace has a dependency on Clojure 1.4.0!)#2019-01-3004:56Alex Miller (Clojure team)a lot of the contribs depend on the oldest version they can support#2019-01-3004:56Alex Miller (Clojure team)CI no longer tests anything older than 1.7 iirc#2019-01-3017:19jvtriguerosI haven't taken the plunge into using tools.dep yet, but absorbing via osmosis through Sean's code snippets! Thanks!#2019-01-3105:45seancorfieldI just updated my .clojure/deps.edn file on GitHub to add REBL support etc https://github.com/seancorfield/dot-clojure/commit/66f2b8fe1d1fd44ba42c79cd9489e7ac620e5fef#2019-01-3121:01borkdudeis it possible to access environment variables in deps.edn like $HOME? probably not..#2019-01-3121:01dominicmNope! Why do you ask? #2019-01-3121:02borkdudebecause I want to say ~/git/... in a local dep. just a convenience#2019-01-3121:04dottedmagIs it a transient thing for working on a local copy of a library? clj -Sdeps "...$HOME..."?#2019-01-3121:05borkdudeyup, can do it like that#2019-01-3121:55Lone Rangerhi all... so I've been attempting to transition over to the deps.edn -- first of all, is there a name for this style of build tools? aka lein vs boot vs ... what... deps?. In general it's great but there seems to be "something else going on". I'm seeing blog posts about a build.clj file but I don't see anything official on it. Can anyone explain what that's all about or have a good reference for it?#2019-01-3122:02lilactown@goomba leiningen and boot are fully-featured toolbelts for doing things like fetching and loading dependencies, building uber jars, running tests, etc.#2019-01-3122:03Lone Rangernod#2019-01-3122:04lilactowntools.deps is only about fetching and dependencies and building a classpath to run your JVM program. it doesn't handle any of the other things#2019-01-3122:04Lone Rangerokay, maybe I'm in the wrong place? is the clj tool not part of this and deps.edn?#2019-01-3122:05Alex Miller (Clojure team)it is#2019-01-3122:05Alex Miller (Clojure team)clj is about two things: making classpaths from transitive deps, and launching programs#2019-01-3122:05Alex Miller (Clojure team)deps.edn describes the deps and ways to modify the making of classpaths#2019-01-3122:05Alex Miller (Clojure team)all other tasks are programs, so you launch a program to do the task#2019-01-3122:05Lone Rangerso far it's been great at those 😄#2019-01-3122:06Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools#2019-01-3122:06Lone Rangeris that where these build.clj files I'm seeing come in?#2019-01-3122:06Alex Miller (Clojure team)^^ some tools#2019-01-3122:06Alex Miller (Clojure team)no one idea what build.clj is#2019-01-3122:06Alex Miller (Clojure team)but it’s not deps.edn / clj#2019-01-3122:07lilactownbecause clj doesn't do all those other things, some people have decided to write their own script for e.g. building a JAR file, or bundling the CLJS code. instead of using a leiningen feature or plugin#2019-01-3122:07Lone Rangerhere's the only example I can find#2019-01-3122:07Lone Rangerhttps://github.com/clojure/clojurescript/blob/master/samples/string-requires-npm-deps/build.clj#2019-01-3122:07lilactownone might decide to call that script build.clj 🙂#2019-01-3122:08Lone Rangerokay so that stuff is essentially home-rolled scripting that some decide to call build.clj#2019-01-3122:08lilactownyeah. in this case, they wrote a script to bundle their CLJS code for deploying#2019-01-3122:09Lone Rangerinteresting. That is the one area in which I'm hurting right now -- I've been running lein in parallel with clj because I can't find a reliable way to do uberjars#2019-01-3122:09lilactownin leiningen, I believe there's a cljsbuild plugin to do that for you#2019-01-3122:09lilactownah yep. same for uberjars#2019-01-3122:09Lone Rangermhm. Are uberjars not really a thing for clojure? I assumed that was the defacto standard for deployment#2019-01-3122:10Alex Miller (Clojure team)they are very common. there are at least 3 tools in that tools page above that build them#2019-01-3122:12Lone Rangerindeed. Just trying to consolidate methodologies as much as possible.#2019-01-3122:13Lone Rangerthanks everyone for fielding questions 😄#2019-01-3122:13lilactownI think some people are moving away from them for Clojure, since there's not a ton of benefit you get other than improved startup time in most cases#2019-01-3122:14Alex Miller (Clojure team)they are certainly convenient as a unit of deployment#2019-01-3122:16Lone Rangerwhat do they otherwise deploy with then? clj -m mything.core?#2019-01-3122:20lilactownah I was also conflating it with AOT. but currently I throw it all in a docker container because I already have infrastructure for that ¯\(ツ)/¯#2019-01-3122:21Alex Miller (Clojure team)AOT is orthogonal{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-01-3122:59seancorfield@goomba FWIW, at work we use (my fork of) depstar to build all-source uberjars for pure convenience of shipping an "artifact" to a server; to run those we do java -cp path/to/the.jar clojure.main -m entry.point ... where entry.point is the namespace containing the -main we wish to run (our artifacts often contain several, related -main functions in different namespaces).#2019-01-3123:00seancorfieldAll of our dev/test process uses clojure (the core of clj) wrapped in a small bash script to simplify handling of command line options.#2019-01-3123:00Lone Rangergotcha (worth a lot, as always!)#2019-01-3123:01seancorfieldWe do not AOT anything.#2019-01-3123:03seancorfieldIt's also worth mentioning, as background, that we started with lein back in 2011 (ish -- that was when we first took Clojure to production), and we switched completely to boot at the end of 2015 (I think, I'd have to go consult my blog to be sure), and then we switched completely to clj/`deps.edn` last year.#2019-02-0113:01mishais there good bare-bones template for clojure+clojurescript monorepo with tools.deps?#2019-02-0118:03dominicmEdge isn't quite bare bones, but is the only thing I know of without having looked much#2019-02-0200:21mishareading through the docs now, thanks#2019-02-0113:14misha{:deps {com.bhauman/figwheel-main {:mvn/version "0.2.0"}}
opieop#2019-02-0119:53eraadHi! I’m trying to use this library (originally built with Leiningen) from a clojure-cli project:
https://github.com/datil/xmldsig
I added a deps.edn but I have a couple of questions:
1. The lib has a src/main/java source folder. Do I need to add this as a “src” folder in the deps.edn?
2. I’m trying to import it using git deps, it adds to the classpath but I cannot seem to be able to import it.
Any recommendations on where to look?#2019-02-0119:54hiredmanyou won't be able to use it#2019-02-0119:54Alex Miller (Clojure team)libraries with Java source can’t be used this way#2019-02-0119:54Alex Miller (Clojure team)clj is designed for libs with no “build” step#2019-02-0206:36matthiasThis is going to be game changing too - we're applying the idea in some really interesting ways.#2019-02-0206:42dominicmCan you share more about how you're able to leverage this to great effect? #2019-02-0119:55Alex Miller (Clojure team)which is most Clojure libraries, but not those with Java source#2019-02-0119:55Alex Miller (Clojure team)you can use the built version as an artifact via Maven or via a locally built jar by pointing to it with a :local/root coordinate#2019-02-0120:07eraadCool @alexmiller @hiredman thanks!#2019-02-0121:49viestithis reminds me of https://dev.clojure.org/jira/browse/CLJ-2343#2019-02-0121:50viestiif gen-class would work at the repl, wouldn’t this mean a build-free support of libs using gen-class? 🙂#2019-02-0121:50viestidon’t know how many those would be though#2019-02-0121:54viestiOddly enough, I think I found a puzzle for Alex: https://github.com/metasoarous/oz/issues/44#issuecomment-459877612#2019-02-0121:56dominicm@viesti you have conflicting files in the class path, nothing can save you now. The ordering is pretty much random, oz is working by coincidence.#2019-02-0121:57dominicmFwiw, you could resolve this if you are a bit crazy by iterating over all of the matches.#2019-02-0121:57viestiyeah, the moment when you realise that the floor starts moving#2019-02-0121:58dominicmI'd recommend that public/index.html isn't used, and that a namespaced prefix is used instead.#2019-02-0121:59viestihum, that might be it#2019-02-0121:59viestisince might be difficult to fish out the conflicting dependency#2019-02-0122:01dominicmThe other library, json-to-html should also not use public/index.html#2019-02-0122:04viestiyeah, could point this out to the other lib#2019-02-0122:04viestimakes me think on using public/index.html in my own apps… 🙂#2019-02-0122:26dominicmYou should only use it there !#2019-02-0122:30viestiyeah, should report to the other dep actually, I think#2019-02-0122:34viestiseems that markdown-to-hiccup-0.6.1.jar might have unnecessary resources in it’s artifact#2019-02-0122:35dominicmIt's both really. Unless oz is an app.#2019-02-0122:35viestispend the day pulling a hairball of Ansible dependencies, now wondering how I ended up to another ball of strings in the evening#2019-02-0122:35viestithinking I’m good at catching balls of strings#2019-02-0206:43dominicmI've only learned this stuff from getting tangled too#2019-02-0206:44dominicmSome of us are destined for tangling#2019-02-0206:46dominicmI just discovered this also quite by accident: https://github.com/replikativ/hasch/blob/master/resources/public/index.html#2019-02-0206:46dominicmnow figwheel main doesn't work...#2019-02-0207:34viestiThe case of the creeping index.html... Trying to hold off thoughts of hijacking webapps by malicious classpath resource injection 🙀 #2019-02-0207:48dominicma lot of people unknowingly serve a /license.html as part of jetty iirc#2019-02-0207:48dominicmoh, no, it's an about.html#2019-02-0207:48dominicmI found out when I discovered the page on our site#2019-02-0210:40viestiheh, there really is such a resource, didn’t find it from served from couple of our apps#2019-02-0216:04dominicmDepends how you set it up, some setups serve html from the root :)#2019-02-0217:57viestiyup, duct template has nice default, which creates a folder with the app name for the resources#2019-02-0122:37viestiprobably both, although oz is higher in the hierarchy, but it can be a dependency itself#2019-02-0123:22viestialmost forgot, thanks for the advice! :) {:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-02-0206:36matthiasThis is going to be game changing too - we're applying the idea in some really interesting ways.#2019-02-0411:33borkdudeIn a mono-repo is it possible to use a common deps.edn file with aliases that can be used from all sub repos?#2019-02-0411:33borkdude(without hacking $CLJ_CONFIG)#2019-02-0412:11dominicmNot that I've discovered.#2019-02-0412:11dominicmI don't believe it's possible.#2019-02-0412:24Alex Miller (Clojure team)No, not currently#2019-02-0412:29borkdudeI’m experimenting with porting https://github.com/borkdude/boot-bundle to deps.edn, so in the future, we could migrate our boot stuff to deps.edn#2019-02-0412:30borkdudeThe idea is that you can define a bunch (bundle) of deps in a common file and then refer to a common name for that bundle in your deps.edn files’ :deps and/or :extra-deps.#2019-02-0413:24gphilipp@borkdude not sure if that covers your need, but @matthias.margush has created this tool: https://github.com/matthias-margush/aka#2019-02-0413:29borkdudethanks#2019-02-0417:53seancorfieldWe have a monorepo with dozens of subprojects -- and we use a small shell script to wrap clojure and use CLJ_CONFIG to point to the monorepo's deps.edn file (and then each subproject has its own deps.edn file). Works great. Any reason you don't want to do that @borkdude?#2019-02-0417:53seancorfieldBasically the shell script does cd path/to/subproject.clj; CLJ_CONFIG=../versions/deps.edn clojure ...args...#2019-02-0418:12richiardiandreadatapoint -> we have a monorepo and looking for a clean solution as well#2019-02-0418:12richiardiandreaor hints at how to contribute a solution#2019-02-0418:20borkdude@seancorfield the reason I thought I wanted to avoid that is that it messes with people’s own configs in their .clojure dir?#2019-02-0418:24Alex Miller (Clojure team)some possible answers, curious if these would help:
1) a clj flag to pass one or more config files that go “in the chain”#2019-02-0418:24Alex Miller (Clojure team)2) an env property to list or more additional config file locations#2019-02-0418:25Alex Miller (Clojure team)3) some kind of config file in .clojure/ (I’m guessing probably not ideal as this is for a group of projects but may not extend to everything you’re doing)#2019-02-0418:37dominicm1 makes this manual opt in everywhere, so users have to inform their tooling of this flag.
2 would require something like direnv, and would also require extensive editor integration.#2019-02-0418:38borkdudeanother way: :include "../parent.deps.edn"#2019-02-0418:38dominicm1 seems more tenable than 2 for something like cursive which has the most extensive analysis of deps.end files.#2019-02-0418:39borkdudeThat, together with a way of including the deps of aliases into another alias would be helpful to me#2019-02-0418:40ghosttoasterHey all. I'm having trouble getting a very simple deps.edn going. I keep on getting a message that opencv can't be found in mvn central, but I've added my custom repo where the jar can be found.#2019-02-0418:41eval20201 with less magic: clj --deps-file foo/deps.edn would only take foo/deps.edn into account. The current situation could be seen as clj --deps-file ~/.clojure/deps.edn --deps-file deps.edn ....#2019-02-0418:42seancorfield@borkdude when you're working on a monorepo project, I'd argue you want a more reproducible set of processes so I'd say you want .clojure to be omitted from deps.edn.#2019-02-0418:42dominicm@seancorfield I don't think that's any more true than a normal repo is it? #2019-02-0418:42borkdude@seancorfield people tend to put their cider-nrepl config into .clojure/deps.edn. I don’t want to disturb that{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-02-0418:43borkdudeSince deps.edn is “just data”™️ I’m inclined to rewrite a template to the deps.edn that will be used#2019-02-0418:43dominicmI also put Rebel, library overrides for environment specific bugs, benchmarking and more. All depending on the context I'm in.#2019-02-0418:44seancorfield@dominicm It's a matter of opinion 🙂 We've generally added stuff to our versions/deps.edn file that individual devs find useful. And we have a couple of "dev" namespaces that include REPL startup stuff. So there's really nothing in our .clojure/deps.edn files that would be useful in addition (we already have nrepl/socket REPL/REBL stuff in our project deps.edn files).#2019-02-0418:44dominicm@borkdude be aware that updating a deps.edn like that will erase comments. I wrote ednup to be a crappy solution to that, but it's not prime ready yet.#2019-02-0418:45borkdude@dominicm I’m exploring rewrite-clj using zippers, that will leave everything intact. https://github.com/Olical/depot does this too.#2019-02-0418:45seancorfieldThat said, I wouldn't object to an enhancement to tools.deps/`clj` that allows additional deps.edn files to be specified 🙂#2019-02-0418:45dominicm@borkdude that's what ednup does#2019-02-0418:46borkdude:extra-deps.edns#2019-02-0418:46borkdude@dominicm this one? https://github.com/SevereOverfl0w/ednup#2019-02-0418:46dominicm@seancorfield I work on a few monorepos (they're mono to their context), and updating my dev config in each of them would be a pain. I'd feel inclined not to commit my dev hacking and clutter git history, especially in projects where we have stringent reviews on code.#2019-02-0418:47dominicm@borkdude yup. You can see usage in my rewrite of resolve tags.#2019-02-0418:48borkdudeI’ve started experimenting with supporting deps.edn for boot-bundle here, also using rewrite-clj: https://github.com/borkdude/boot-bundle/blob/tools.deps/src/boot_bundle/rewrite.clj
but since it’s hard to settle down on an approach, I’m first looking if deps.edn itself could support this better#2019-02-0418:48dominicmIt's not a great library, but I plan to iterate on it intensely at some point, and it's great for simple things.#2019-02-0418:49dominicmOh, if you want examples, pack has an injector which uses rewrite directly.#2019-02-0418:51borkdudeI meant: figuring out the format/API is hard, not the actual programming 🙂#2019-02-0418:51borkdudeI want to support existing users of boot-bundle (including myself) and support a migration path from boot to deps.edn#2019-02-0418:52borkdudeanyways, being able to specify multiple extra deps.edns from the cmd-line is a good thing in my opinion#2019-02-0418:52borkdudeyou can always use -Srepro to exclude .clojure so that functionality is already there#2019-02-0419:02Alex Miller (Clojure team)it is still a pending todo to finalize what -Srepro means, esp in context of -Sdeps and other features. some of that is pending config file rework#2019-02-0418:53hiredmananother option, instead of having a super deps.edn for the monorepo, or generating dep.edns for subprojects might be something like a dependency linter, you give it some rules (deps on project x most version y) and run it over a directory and it checks all the deps.edn files it finds#2019-02-0418:54hiredmana linter could even be run against projects not in your monorepo to see how some other library might integrate#2019-02-0418:54borkdude@hiredman meh, then you would still have to update all those occurrences, or do you mean the linter would actually update the files?#2019-02-0418:54hiredmanno, you would have to update the files#2019-02-0418:55borkdudethe reason I wrote boot-bundle was to prevent having to do that, it became annoying#2019-02-0418:55borkdudeI would want one source of the truth and the include that in another project#2019-02-0418:56borkdudesome call it “managed dependencies” but I didn’t realize that when I wrote the lib#2019-02-0418:56hiredmanhow much dependency churn do you have?#2019-02-0418:57borkdudeso what I’m searching for is managed deps for deps.edn. Another way to accomplish this maybe is to have “empty” projects that declare their deps and then make a dependency on those with local/root.#2019-02-0418:58borkdudebut that gets out of hand too, I want it to be declarative and easy to modify#2019-02-0418:59borkdude@hiredman we have a couple of local libraries which all depend on subsets of the same dependencies. in some of those we use AOT. it’s nasty if you get the versions wrong, so we want them to be the same#2019-02-0418:59hiredmansure#2019-02-0419:03Alex Miller (Clojure team)we always intended to support something like managed deps and I think that is a valuable use case#2019-02-0419:04Alex Miller (Clojure team)we have not put as much thought into how that should look as with other stuff yet#2019-02-0419:04seancorfieldOur versions/deps.edn file is mostly :override-deps to "pin" dependencies for the whole monorepo. Each subproject declares its own dependencies as group/artifact {}. And we have dev/test/build aliases in versions/deps.edn.#2019-02-0419:05borkdude@alexmiller with being able to specify extra deps.edn configs from the cmd line + the ability to merge multiple aliases into one (:foo has :bar and :baz like with -R:bar:baz) I think we would be 99% there#2019-02-0419:05borkdudemaybe even 100%#2019-02-0419:05seancorfieldHaving four deps.edn in the chain instead of three would solve all our issues 🙂 (system, user .clojure, project root, subproject root).#2019-02-0419:07borkdudeproject/subproject seems arbitrary to me (different projects use different directory structures), being able to specify this yourself and number of times you can do this should be flexible#2019-02-0419:08hiredmanaot adds whole other wrinkle on top of things for monorepos, because you are depending on build output of a local project, not the local project (with clj doesn't directly support so much)#2019-02-0419:16hiredmando you aot as low down the deps tree as possible, or as high up as possible?#2019-02-0419:17borkdude@hiredman We wrote a lib that has some Java code which we compile up front. We don’t want to have the situation that we’re going to use different deps than what that was compiled against.#2019-02-0419:18hiredmanah interesting, built java not aot'ed clojure?#2019-02-0419:18seancorfield@borkdude Perhaps I'll rephrase that as "monorepo" and "project" level then instead of "project" and "subproject"? 🙂#2019-02-0419:19borkdudeWe also ran into trouble once with a service that was running clojure 1.8 which did different edn serialization than 1.9, that was also a problem. we just don’t want to think about this stuff too much and exclude any such possibility.#2019-02-0419:20borkdude@seancorfield boot-bundle supports loading bundles (managed deps) even from Clojars, it doesn’t matter how you organize this.#2019-02-0419:21borkdude@hiredman yes#2019-02-0419:28seancorfield@borkdude and each "bundle" would map to a separate deps.edn file? So you could have a folder for each "bundle" and then access them via multiple :local/root deps?#2019-02-0419:29borkdude@seancorfield how this is done currently in boot bundle is one keyword maps to one or more deps (or recursively, to another bundle). this is just EDN which is expanded dynamically in the scope of a build.boot file.#2019-02-0419:29seancorfieldFor us, the problem isn't really the deps themselves, but the aliases we want across the whole monorepo -- sort of like requiring every dev to have consistent .clojure/deps.edn files but just for the projects in the monorepo.#2019-02-0419:29borkdudewhere this bundle.edn file comes from is up to you, could be from clojars, a directory, etc.#2019-02-0419:30seancorfieldI haven't looked at anything in Boot for quite a while now. I hadn't heard of boot-bundle. Sounds useful in a deps.edn world.#2019-02-0419:32borkdudein boot this is easy to implement because you can actually program in the build file. in deps.edn I currently see no other way than to rewrite a deps.edn.template and expand it into a deps.edn. but with the options we mentioned in this conversation, I think it’s very close at hand#2019-02-0419:33borkduderelying on add-lib could also work, but I’m not too familiar with that and I think that’s alpha-alpha?#2019-02-0419:36Alex Miller (Clojure team)alpha-omega#2019-02-0419:42seancorfieldI recently added some aliases to my .clojure/deps.edn file that pull in that branch and the comments in my EDN file show how to write a load-master function that can resolve master from git repos and then add that library 🙂#2019-02-0419:42seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L94-L107#2019-02-0423:57richiardiandrearead the whole conversation, our team is looking for the same answers @borkdude, maybe we can join forces 😄 Share the pain err your thoughts with me ok?#2019-02-0505:05borkdude@richiardiandrea Totally!{:tag :div, :attrs {:class "message-reaction", :title "grinning"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😀")} " 5")}
#2019-02-0508:22Christian Johansen>Error building classpath. Manifest type not detected when finding deps for myapp/myapp in coordinate #:local{:root "/Users/christian/projects/myapp"}#2019-02-0508:23Christian JohansenI'm getting this on a project that used to work fine. The local dependency is a leiningen based project. Am I doing something wrong?#2019-02-0508:29Christian Johansenhmm, maybe I used to have a deps.edn file in that project? :thinking_face:#2019-02-0512:57Alex Miller (Clojure team)Probably. You can force it with :deps/manifest :deps (and it’s tolerant of a missing deps.edn)#2019-02-0513:05Christian JohansenOk, good to know! Solved it by extracting project.clj deps into deps.edn. This is work in progress, and I've had a deps.edn file that wasn't commited to git that I probably git reset away 🙈#2019-02-0509:57borkdude@richiardiandrea Here’s an idea I have:
https://gist.github.com/borkdude/32d486805aabb1d8926270d4f12ff3dd
Feel free to shoot at it.
(cc @alexmiller @seancorfield @dominicm)#2019-02-0510:19dominicmIs there a jira ticket for managed deps?#2019-02-0510:21dominicmhttps://github.com/juxt/edge/issues/37 I was cleaning up issues, and realized this issue was related to what we were discussing last night 🙂#2019-02-0510:42borkdudeI’m willing to post my proposal into a JIRA ticket, but I thought it would be good to have a little screening first#2019-02-0510:44dominicmSure :) I actually wanted somewhere to direct the above issue to.#2019-02-0618:08timgilbertIs there anything like an offline mode for tools.deps? I'm finding my CI builds take 2-2.5 minutes per alias per build step just downloading maven-metadata.xml files from my S3 repo and clojars / maven-central, and I can't seem to cache the results#2019-02-0618:09timgilbertThis is separate from the actual .jar / .pom files which I'm able to cache with no problem (in ~/.m2)#2019-02-0618:13borkdudeWe use s3 wagon private in boot and I notice it downloads metadata more often than it should, which also slows down things there#2019-02-0618:14timgilbertAlso, these specific packages have become my arch-enemies as every time I run clojure I see this block of text:
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-dynamodb/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from
Downloading: com/amazonaws/aws-java-sdk-kinesis/maven-metadata.xml from
#2019-02-0618:16timgilbertI don't know if there's a TTL set to a ridiculously low value by AWS or what, but I just cannot avoid seeing that every single time if I've got those dpes#2019-02-0618:24Alex Miller (Clojure team)file a ticket with as much info as possible{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-02-0618:25aviI suspect the answer is no, but just to confirm: is there any way to have tools.deps compile Java code when consuming a Clojure library via git that also includes some Java code?#2019-02-0618:27Alex Miller (Clojure team)no#2019-02-0618:27aviAlrighty then, thanks!#2019-02-0623:03kennyIs there a way to see why a classpath refresh takes a long time with clj?#2019-02-0623:08Alex Miller (Clojure team)there is a super secret verbose knob you could try#2019-02-0623:09Alex Miller (Clojure team)clj -Sdeps '{:aliases {:v {:verbose true}}}' -A:v#2019-02-0623:10Alex Miller (Clojure team)the most likely culprit is slow I/O of course, which you can really spot only by observing where that pauses#2019-02-0623:10kennyThe weird part is it only occurs in one particular project.#2019-02-0623:11Alex Miller (Clojure team)the next most likely is that tools.deps is doing more work than it should, which you may also be able to spot by seeing it redoing work#2019-02-0623:14kennyHard to tell what's expected and whats not. Expanding git deps takes a good 3-5s for this project.#2019-02-0623:16kennyThis project also has a fairly large dependency tree. The ~20s refresh gets a bit painful though.#2019-02-0623:19Alex Miller (Clojure team)git deps are stored in two parts - the git objects, and the working tree (per sha)#2019-02-0623:20kennyIt wouldn't be checking out anything new - everything is all on disk. Just changing the SHAs back and forth.#2019-02-0623:20Alex Miller (Clojure team)the git objects are mostly append-only so a fetch should only need to be new stuff, small diffs#2019-02-0623:20Alex Miller (Clojure team)working tree could be slower, but it’s all in the objects#2019-02-0623:21Alex Miller (Clojure team)I guess I would compare doing a git clone or fetch separately and see if it’s substantially different#2019-02-0623:22Alex Miller (Clojure team)there’s caching in many places here - git objects, git working tree per sha, maven deps in maven repo, and classpath in cpcache#2019-02-0623:23Alex Miller (Clojure team)so I guess I’m curious what part of that isn’t cached when it’s slow (and why)#2019-02-0623:23kennyFor example, each one of these steps takes 3-5s:
Expanding compute/logger {:git/url
There are several of those.#2019-02-0623:24Alex Miller (Clojure team)is it correct that there is a newer sha?#2019-02-0623:25kennyYes, 319 is newer than 9f9f.
Just ran time on the -Spath command. It actually took 52s for a CP refresh.#2019-02-0623:27kennyI should clarify - that is one example of the messages that takes a while. AFAICT, there are no duplicates in the log.#2019-02-0623:55Alex Miller (Clojure team)so it should be fetching the git repo, then cloning the working tree. Are there maybe big binary objects in the working tree? One idea I had with gitlibs was to have the ability to narrow the working tree to just a subset of the full dir (so you could exclude like an images directory or something was irrelevant to the Clojure source)#2019-02-0623:56Alex Miller (Clojure team)still not entirely sure that’s the slow bit from your description but maybe a testable hypothesis#2019-02-1117:24borkdudeRe managed deps:
https://dev.clojure.org/jira/browse/TDEPS-116#2019-02-1117:26borkdudeThat proposal introduces the ability to have managed deps with minimal change to the deps.edn spec (don’t know about the impl)#2019-02-1209:34martinklepschWhen using :local/root with a .jar will that resolve transitive dependencies of the jar by inspecting the pom.xml inside that jar?
And is it standard that pom.xml is included in the jar?#2019-02-1209:37borkdudeIs that supposed to work? I’m surprised#2019-02-1209:38borkdudeYeah, it’s documented: Local jar: {:local/root "/path/to/file.jar"}. TIL{:tag :div, :attrs {:class "message-reaction", :title "clap"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👏")} " 5")}
#2019-02-1209:39martinklepschYeah, it's pretty neat. I just wasn't aware that pom.xml is always(?) part of the jar so thought that this couldn't work with transitive deps.#2019-02-1213:46Alex Miller (Clojure team)It is normally part of the jar.#2019-02-1214:22stijnwhat might be the reason that it takes a very long time to get a REPL up? I just upgraded my ArchLinux and before it took a few seconds, now, minutes. I believe the only change is the upgrade of the jdk 8.u192-1 -> 8.u202-1.#2019-02-1214:24stijnIf I don't include the 'dev' profile, which has some machinery with tools.namespace (and a logback.xml) in it, it goes back to seconds.#2019-02-1214:33Alex Miller (Clojure team)well the main thing it’s doing is network I/O so that would be my default area of concern#2019-02-1214:34Alex Miller (Clojure team)do some stack traces with ctrl-\ while it’s working and see what it’s doing#2019-02-1215:07Alex Miller (Clojure team)8.u201 (and 202) include new buffer overrun checks in Linux, not sure if that would affect visible perf: “Additional safeguards to protect against buffer overruns in native code have been enabled on Linux. If a buffer overrun is encountered the system will write the message “stack smashing detected” and the program will exit. Issues of this type should be reported to your vendor.”#2019-02-1216:28stijnif I switch to java 11 it's back to 'seconds'#2019-02-1216:30stijnit looks like the compilation of namespaces that are required from user is slow. when these are uncommented, it's fast on 8.u202 too#2019-02-1217:20Alex Miller (Clojure team)hmm, still not clear to me what the difference is#2019-02-1217:21bronsacould you post your user.clj maybe?#2019-02-1307:26stathissiderisdoes the Clojure runtime have access to the aliases that were passed? Can my code access this information somehow?#2019-02-1307:31dominicmThere are some system properties set. I think only clojure.libfile#2019-02-1307:31dominicmWhy do you need to know? #2019-02-1307:50stathissiderisI’d like to know whether its a figwheel run or not#2019-02-1307:59dominicm@stathissideris everyone is going to name their Figwheel alias differently. I'd check if the namespace is loaded/can be required, and depending on context, check if the state var is bound.#2019-02-1307:59dominicmYour code might work with some lein setups too that way#2019-02-1308:04stathissiderisgood point about about naming, but I need this for my specific setup, I’m not making a generic tool, so hardcoding it would be ok#2019-02-1308:06stathissiderisIs is possible to combine -e and -m to inject a var maybe?#2019-02-1308:13dominicmI do this in edge using :jvm-opts#2019-02-1308:14dominicmJvm opts compose, main opts do not.#2019-02-1308:15dominicmUsing -D in jvm opts that is.#2019-02-1308:17stathissiderisoh thanks, I’ll check it out#2019-02-1313:40dominicmI think there's a NPE in t.d.a:
Error building classpath. nil
java.lang.NullPointerException
at clojure.tools.deps.alpha.extensions.maven$eval638$fn__641.invoke(maven.clj:41)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.alpha$canonicalize_deps$fn__1027.invoke(alpha.clj:74)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
at clojure.core$reduce.invokeStatic(core.clj:6827)
at clojure.core$reduce.invoke(core.clj:6810)
at clojure.tools.deps.alpha$canonicalize_deps.invokeStatic(alpha.clj:73)
at clojure.tools.deps.alpha$canonicalize_deps.invoke(alpha.clj:71)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:189)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:170)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:236)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:218)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invokeStatic(make_classpath.clj:59)
at clojure.tools.deps.alpha.script.make_classpath$create_classpath.invoke(make_classpath.clj:52)
at clojure.tools.deps.alpha.script.make_classpath$run.invokeStatic(make_classpath.clj:70)
at clojure.tools.deps.alpha.script.make_classpath$run.invoke(make_classpath.clj:64)
at clojure.tools.deps.alpha.script.make_classpath$_main.invokeStatic(make_classpath.clj:109)
at clojure.tools.deps.alpha.script.make_classpath$_main.doInvoke(make_classpath.clj:84)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:491)
at clojure.main$main_opt.invoke(main.clj:487)
at clojure.main$main.invokeStatic(main.clj:598)
at clojure.main$main.doInvoke(main.clj:561)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:37)
this may be time sensitive as I'm using a just-deployed coordinate.#2019-02-1313:40dominicmorg.webjars.bower/react {:mvn/version "15.4.2"}
org.webjars.bower/es6-promise {:mvn/version "4.0.5"}
org.webjars.bower/fetch {:mvn/version "0.9.0"}
org.webjars.bower/graphiql {:mvn/version "0.11.11"}
org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"}
org.webjars.npm/subscriptions-transport-ws {:mvn/version "0.8.3"}
these are the deps I just added#2019-02-1313:42dominicm org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"} this appears to be the exact problem dep#2019-02-1313:51Alex Miller (Clojure team)runtime does not have access to the aliases that were used @stathissideris#2019-02-1313:53stathissideris@alexmiller thanks, I ended up using -D to indicate the alias via properties as @dominicm suggested #2019-02-1313:53Alex Miller (Clojure team)I can repro the npe, will take a look. shouldn’t be timing-related#2019-02-1313:54Alex Miller (Clojure team)there’s a version range in the transitive deps, wouldn’t surprise me if that’s involved#2019-02-1313:55dominicmthat seems more likely, yeah. I was worried that we'd caught maven in an inbetween state and it was causing an NPE.#2019-02-1313:56Alex Miller (Clojure team)I think it’s finding no matches in the version range#2019-02-1313:56Alex Miller (Clojure team)so code could definitely be tightened up around that to not npe but it will just not find the dep then#2019-02-1313:57dominicmyeah. makes sense. webjars is a bit weird like this I guess 🙂#2019-02-1313:57dominicmI can add explicit graphiql dep here.#2019-02-1313:58Alex Miller (Clojure team)org.webjars.npm/graphiql-subscriptions-fetcher {:mvn/version "0.0.2"} depends on org.webjars.npm/graphql {:mvn/version "[0.9.1,0.10)" but I don’t see anything older than 0.10.1 in maven central#2019-02-1313:58Alex Miller (Clojure team)https://repo1.maven.org/maven2/org/webjars/npm/graphiql/#2019-02-1313:58Alex Miller (Clojure team)I’ll fix the npe#2019-02-1313:59dominicmwebjars is a bit weird. You basically are deploying npm things. So if nobody has deployed graphiql 0.10, then it won't be there.#2019-02-1313:59dominicmadding an explicit graphiql dep hasn't helped though, annoyingly. So I guess I will deploy 0.10 via http://webjars.org#2019-02-1313:59Alex Miller (Clojure team)does npm actually have jars in it?#2019-02-1314:00dominicmwebjars takes stuff from npm, and puts it in a jar#2019-02-1314:00Alex Miller (Clojure team)tdeps could have an extension to use webjars directly#2019-02-1314:00dominicmdo you mean npm directly? I don't know how you'd use webjars directly 🙂#2019-02-1314:01dominicmbut either way, that would be a huge +1 from me#2019-02-1314:02Alex Miller (Clojure team)well, you would need to tell me more things for me to understand what’s possible, but anything that can be a source of named, versioned artifacts is a potential target#2019-02-1314:08dominicmgood question, admittedly I don't fully understand how webjars works.#2019-02-1314:08Alex Miller (Clojure team)well, webjars seems to have a a cdn? whatever step you’re doing seems like some sort of mindless wrapping task#2019-02-1314:08Alex Miller (Clojure team)programs are good at those#2019-02-1314:09dominicm> All of the WebJar contents are available on the public jsDelivr CDN. Just prefix //cdn.jsdelivr.net/webjars/{groupId} in front of your static asset URLs. For instance, if using the org.webjars : jquery WebJar and your local URL to jquery.js is /webjars/jquery/2.1.0/jquery.js then the CDN URL would be: //cdn.jsdelivr.net/webjars/org.webjars/jquery/2.1.0/jquery.js
So I guess the contents of the jars are available via the CDN, rather than the jars themselves.#2019-02-1314:09dominicmthe actual jars go to maven.#2019-02-1314:09dominicmthen you can request a new jar to go to maven via clicking buttons on http://webjars.org - that triggers an automated process#2019-02-1314:13Alex Miller (Clojure team)“clicking buttons” :)#2019-02-1314:14dominicmis your proposal that instead of deploying to maven, a webjars-like thing would build the jars and stick them in ~/.local/cache/webjars/foo.bar.1.jar ?#2019-02-1314:15Alex Miller (Clojure team)why do you even need jars? they’re just files on the classpath right?#2019-02-1314:16Alex Miller (Clojure team)it would be helpful to do something that actually helped people down the line as they build and deploy stuff#2019-02-1314:17Alex Miller (Clojure team)I am too ignorant about cljs / npm / node / etc to not know what would be useful#2019-02-1314:17dominicmyou're right. jars not needed.#2019-02-1314:17Alex Miller (Clojure team)npm is a central repository. it has stuff you want in your app. tools.deps has an extension point to plug in code to get stuff and put it on your classpath (and determine its dependencies)#2019-02-1314:17dominicmfwiw, this is pretty unrelated to cljs.#2019-02-1314:18Alex Miller (Clojure team)see how ignorant I am? :)#2019-02-1314:18dominicmI think npm is a good target for t.d.a. So is bower.#2019-02-1314:18dominicmmaybe not bower, it's kinda old now.#2019-02-1314:19Alex Miller (Clojure team)yeah, anything that didn’t come out this month in js is old :)#2019-02-1314:20Alex Miller (Clojure team)I guess coming back to a problem statement - this is like js stuff that you want to serve as static resources from your app?#2019-02-1314:24Alex Miller (Clojure team)does npm really not have any kind of namespacing?#2019-02-1314:27Alex Miller (Clojure team)oh, it has scopes, I see#2019-02-1314:28Alex Miller (Clojure team)well, seems like the info necessary exists if you want to file a ticket defining what’s useful#2019-02-1314:29dominicm> I guess coming back to a problem statement - this is like js stuff that you want to serve as static resources from your app?
Exactly!#2019-02-1314:30dominicmAlthough, admittedly, I'd personally be quite happy with cdn integrations.#2019-02-1314:30dominicmbower predates npm afaik. It was a big step forward at the time, but it's not really used. npm got more traction and was dominant by the time a majority of companies turned their attention to js.#2019-02-1314:31dominicmbower doesn't recommend you use bower so ¯\(ツ)/¯#2019-02-1314:31dominicmI'll write down that I want to create this, and have a note for myself to think a little more about what I want 🙂#2019-02-1314:32dominicm@alexmiller while I have you, is there a workaround for this graphiql issue? bintray appears to be having issues, and I can't get an older graphiql to deploy. :exclusions hasn't worked, nor has an explicit graphiql dependency on 0.11.11.#2019-02-1314:35Alex Miller (Clojure team)not sure I have a good option for you#2019-02-1314:36Alex Miller (Clojure team)if you made webjars deploy version 0.9.1 to maven central that would let it proceed#2019-02-1314:37dominicmUnfortunately, that's failing 🙂 I thought you might say that though. It's okay#2019-02-1314:40Alex Miller (Clojure team)kind of an interesting failure mode. would be better if it could defer long enough to be overridden#2019-02-1314:40Alex Miller (Clojure team)in this particular case, the only reason it’s resolving early is due to the version range I think, trying to narrow that down to single version#2019-02-1316:43dominicm#2019-02-1316:43dominicmgraphiql vs graphql#2019-02-1317:17dominicmYou hit an NPE whenever you have a transitive dependency which doesn't exist. I'm seeing it with a version pinned to [1.0.3] (maybe that is a range though?)#2019-02-1317:19dominicmActually, no. I'm confused.#2019-02-1317:19dominicmhttps://search.maven.org/artifact/org.webjars.npm/graphql/0.9.1/jar depends on https://search.maven.org/artifact/org.webjars.npm/iterall/1.0.3/jar but I'm still hitting the exact same NPE.#2019-02-1317:20dominicmI've rm -rf .cpcache, and used -Sforce, just in case.#2019-02-1317:20dominicm❯ clj -Sdeps '{:deps {org.webjars.npm/graphql {:mvn/version "0.9.1"}} :aliases {:v {:verbose true}}}' -A:v -Sforce
Initial deps to expand:
{org.clojure/clojure #:mvn{:version "1.10.0"},
org.webjars.npm/graphql #:mvn{:version "0.9.1"}}
Expanding org.clojure/clojure #:mvn{:version 1.10.0}
=> include, pin top dep
Expanding org.webjars.npm/graphql #:mvn{:version 0.9.1}
Error building classpath. nil
java.lang.NullPointerException
at clojure.tools.deps.alpha.extensions.maven$eval638$fn__641.invoke(maven.clj:41)
#2019-02-1317:35dominicm[ERROR] Failed to execute goal on project my-app: Could not resolve dependencies for project : Failed to collect dependencies at org.webjars.npm:graphql:jar:0.9.1 -> org.webjars.npm:iterall:jar:[1.0.3]: No versions available for org.webjars.npm:iterall:jar:[1.0.3] within specified range -> [Help 1] maven is struggling too#2019-02-1317:36dominicmOh. Interesting. It is fine with it as a top-level dependency, but not as a transitive dep.#2019-02-1318:17Alex Miller (Clojure team)[1.0.3] is a version range. from Maven’s perspective just 1.0.3 by itself is a specific version#2019-02-1318:20Alex Miller (Clojure team)I don’t think there’s any difference with top vs transitive - you probably just have it in your local Maven repo at ~/.m2/repository now#2019-02-1318:22Alex Miller (Clojure team)it works for me as a transitive dep, and as a top-level dep with specific version, and as a top-level dep with version range [1.0.3]#2019-02-1318:22Alex Miller (Clojure team)(clearing all caches before each one)#2019-02-1318:24Alex Miller (Clojure team)$ rm -rf ~/.m2/repository/org/webjars
$ clj -Sdeps '{:deps {org.webjars.npm/graphql {:mvn/version "0.9.1"}}}' -Sforce
Downloading: org/webjars/npm/graphql/0.9.1/graphql-0.9.1.pom from
Downloading: org/webjars/npm/iterall/maven-metadata.xml from
Downloading: org/webjars/npm/iterall/1.0.3/iterall-1.0.3.pom from
Downloading: org/webjars/npm/graphql/0.9.1/graphql-0.9.1.jar from
Downloading: org/webjars/npm/iterall/1.0.3/iterall-1.0.3.jar from
Clojure 1.10.0
user=>#2019-02-1318:27Alex Miller (Clojure team)when you deploy to Maven central it can take 10 minutes to an hour to show up for download, so maybe it just wasn’t there yet#2019-02-1318:37dominicmI'm a bit suspicious that perhaps version ranges are searched somewhere different than specific versions.#2019-02-1318:41Alex Miller (Clojure team)they aren’t#2019-02-1318:41Alex Miller (Clojure team)but there is an extra piece of code to explicitly resolve them in tdeps#2019-02-1318:42Alex Miller (Clojure team)if you haven’t, you should try selectively deleting part of your ~/.m2/repository/org/webjars/…#2019-02-1318:42Alex Miller (Clojure team)there are metadata files about versions that are cached there and only updated once every 24 hrs#2019-02-1318:42Alex Miller (Clojure team)(this is a setting we don’t currently surface - if you’re using mvn, you can pass -U to force an update)#2019-02-1318:43dominicmIs it possible that metadata would store [1.0.3] separately from 1.0.3?#2019-02-1319:01Alex Miller (Clojure team)it just stores known versions and then uses that to resolve#2019-02-1319:01Alex Miller (Clojure team)but if you had already filled that cache for the day and the repository said that version didn’t exist, it’s going to keep telling you that#2019-02-1319:01Alex Miller (Clojure team)it will not go re-check the repository#2019-02-1319:02Alex Miller (Clojure team)this file in particular: ~/.m2/repository/org/webjars/npm/iterall/maven-metadata-central.xml#2019-02-1319:03Alex Miller (Clojure team)last check timestamps are in ~/.m2/repository/org/webjars/npm/iterall/resolver-status.properties#2019-02-1319:59stijn@alexmiller on the slow REPL startup of the latest java 8 (u202) on Linux, looking at the stack traces, it spends a lot of time compiling some java-time things (https://github.com/dm3/clojure.java-time). One CPU goes to 100% during this and after that stack trace is gone, the REPL starts.#2019-02-1320:08hiredmanjava-time does lot of meta programming stuff, finding types, doing a graph traversal to find paths between types#2019-02-1320:08hiredmanso I could imagine types changing between jdk versions#2019-02-1320:13dominicmblugh, after all this work, one of my transitive deps doesn't work because it has a custom gql license which webjars won't automatically repackage because it doesn't recognize. If we were a direct npm client this would be avoided. Very tempting proposition really.#2019-02-1320:13dominicm@alexmiller if I wrote an npm client with an EPL license in clojure, would it be candidate for use in t.d.a? Or would you prefer it was under the clojure/core umbrella?#2019-02-1320:42Alex Miller (Clojure team)could be an external lib (we use others already). I think there are some out there already. but maybe the amount of access needed by tda is so small to not even be worth that, not sure.#2019-02-1320:43Alex Miller (Clojure team)I guess you probably want a cache ala tools.gitlibs though and that’s something we’d want outside tda#2019-02-1320:46dominicmI haven't really looked at the npm client details yet. Might be worth seeing if we can share the yarn cache. Plus I'm a bit uncertain about how to handle http. #2019-02-1320:47Alex Miller (Clojure team)well I think you have the same options as gitlibs - shell out to npm or make a direct implementation#2019-02-1320:49Alex Miller (Clojure team)pros and cons. might be worth gathering some more information about the things you need to implement:
1) given a name/version, determine that lib’s deps (don’t need transitive, just shallow)
2) given a name/version, download the bits somewhere#2019-02-1320:49dominicmNpm is far more amenable I think. Yarn is a reimplementation of the npm client. Plus, npm doesn't have a global cache, and because of the hierarchical nature, you probably want more control.#2019-02-1320:49dominicmI suppose you have to figure out auth, which is the recurring issue.#2019-02-1320:53Alex Miller (Clojure team)well public stuff is fine I assume#2019-02-1320:53Alex Miller (Clojure team)for creds, we already have a means to do this via the maven settings.xml on a per-repo basis#2019-02-1320:53Alex Miller (Clojure team)admittedly kind of weird to put npm auth there#2019-02-1320:19stijn@hiredman, the weird thing is that it's only a minor version upgrade (from 8.u192 to 8.u202). With 192 the repl starts in seconds, with 202 it takes minutes#2019-02-1320:21ghadiI know it's aside from your problem @stijn but I wouldn't use that library, to be perfectly honest.#2019-02-1320:21ghadijava.time is perfectly approachable#2019-02-1320:22dominicmhttps://juxt.pro/tick/docs/index.html makes java.time quite nice, and it works in cljs :face_with_rolling_eyes: (I work for JUXT)#2019-02-1320:29seancorfieldThat's a matter of opinion @U050ECB92 🙂 I find clojure.java-time really does produce cleaner-looking code in a lot of situations.#2019-02-1321:17ghadiclojure.java-time seems to have a steep cost in classloading and metaprogramming#2019-02-1321:19seancorfieldIt's a one-off cost, the first time you use certain constructors. It's fast after that.#2019-02-1321:20seancorfieldThe readme even shows how you can "warm up" that stuff.#2019-02-1321:20seancorfieldI was a bit surprised that it caused such a "blip" on 202(?) since it's fine on 192 (and on subsequent builds) according to the OP.#2019-02-1408:47stijnI will investigate some more when I have the time. For now, went back to 192.#2019-02-1320:21ghadithe actual JDK package java.time#2019-02-1320:24hiredmanmaybe check to see if you have org.threeten.extra.Temporals and or org.joda.time.DateTime on your classpath, and if calling (Class/forName ...) with those class names changes behavior on the different jdk versions#2019-02-1408:50stijnyes org.joda.time.DateTime is on the classpath. will check later what the impact is#2019-02-1320:46Alex Miller (Clojure team)tools.deps.alpha 0.6.488 and clj 1.10.0.414 now available - https://groups.google.com/forum/#!topic/clojure/5EWvlgC4C9U#2019-02-1320:46Alex Miller (Clojure team)just a couple small enhancements, no biggie#2019-02-1320:46dominicmWoah, very quick, thanks! #2019-02-1320:49Alex Miller (Clojure team)hadn’t released in a while#2019-02-1321:25seancorfieldThat Groups post hasn't gone out via email again, as far as I can see @alexmiller#2019-02-1321:27seancorfield(I just replied to it on the web which should cause it to be seen via email 🙂 )#2019-02-1322:12Alex Miller (Clojure team)I sent that one from a different account#2019-02-1419:40avilooks like the brew PR has been merged — I just ran brew upgrade and my Clojure install got upgraded to 1.10.0.414#2019-02-1419:52Alex Miller (Clojure team)yep, it's out there now#2019-02-1420:00avi🎉 thanks Alex! aw_yeah#2019-02-1420:08Alex Miller (Clojure team)just noticed there are some homebrew download stats at https://formulae.brew.sh/formula/clojure#2019-02-1420:51aviThat’s nice! Looks like you can get the stats as data in JSON format too: https://formulae.brew.sh/api/formula/clojure.json#2019-02-1502:56mfikesRegarding the slowdown in REPL startup times when going from 1.8.0_192 to 1.8.0_201, I encountered this on macOS. Details here (with essentially a trivial repro) https://clojurians-log.clojureverse.org/cljsrn/2019-02-05#2019-02-1716:34ChrisHave you seen this?
http://insideclojure.org/2019/02/16/journal/#2019-02-2014:53zetafishDoes anyone know how to change the local maven repo folder when running clj so that jars are downloaded and searched for in a different folder than ~/.m2#2019-02-2015:08Alex Miller (Clojure team)in deps.edn, set :mvn/local-repo "/foo/bar" at top level{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-02-2016:23zetafishThanks @alexmiller that works. Sadly enough it seems not possible to put this option in the :aliases. My problem is that I want the maven repo to be local to my source code when I run unit tests on my CI server. While when developing locally I want to just use the default ~/.m2. Are there plans to allow the key also in the :aliases? Or maybe I should revert to tricks like rewriting the deps.edn file on the CI server prior to running tests etc (ugh)#2019-02-2016:25Alex Miller (Clojure team)you can use -Sdeps '{:mvn/local-repo "whatever"}' just on CI#2019-02-2016:25zetafishCool! I am just a beginner with tools-deps 🙂#2019-02-2016:59dominicmMy friend got this error when using brew on his mac, I don't use mac, so don't really understand why gcc is related to clojure at all?
Error: An exception occurred within a child process:
CompilerSelectionError: clojure cannot be built with any available compilers.
Install GNU's GCC
#2019-02-2017:00Alex Miller (Clojure team)seems bad#2019-02-2017:00Alex Miller (Clojure team)brew doctor is the generic "why is brew broken?"#2019-02-2017:00Alex Miller (Clojure team)brew install clojure should not require anything to be compiled#2019-02-2017:01Alex Miller (Clojure team)brew update updates brew itself#2019-02-2017:02dominicmDoes brew generally work without xtools?#2019-02-2017:07dominicminstalling xtools solved it, so I guess that's a dependency? /shrug#2019-02-2017:11Alex Miller (Clojure team)I never understood how xtools is exactly involved, I guess to allow brew to compile stuff#2019-02-2017:13dominicmyeah, but, unless it compiles rlwrap from source, I have no idea what it would be building.#2019-02-2017:28Alex Miller (Clojure team)Me neither#2019-02-2116:39stathissiderisdoes this look wrong somehow? It’s an alias in my deps.edn
{:main-opts ["-e" "(require 'dev) (dev/seed-db)"]}
and it throws a
Exception in thread "main" clojure.lang.LispReader$ReaderException: java.lang.RuntimeException: EOF while reading, starting at line 1#2019-02-2116:59Alex Miller (Clojure team)yeah, the inner string is not going to end up getting quoted right#2019-02-2117:02stathissiderisso it just needs more quoting? {:main-opts ["-e" "\"(require 'dev) (dev/seed-db)\""]}#2019-02-2117:03stathissideristhat doesn’t work either#2019-02-2117:03Alex Miller (Clojure team)no, that probably won't work either due to how this gets propagated. this is a known issue#2019-02-2117:03Alex Miller (Clojure team)you can try the "corfield comma" (comma instead of spaces)#2019-02-2117:04Alex Miller (Clojure team){:main-opts ["-e" "(require,'dev)(dev/seed-db)"]}#2019-02-2117:04stathissiderisyes, that does work, thank you very much — I don’t think I would have found it myself!#2019-02-2117:39seancorfieldsomeone summoned me 🙂 @stathissideris It's usually around this time I point people at my dot-clojure file for examples of :main-opts https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#2019-02-2117:42stathissiderisI’ve been slowly figuring out many of these myself, so many thanks for this treasure trove!#2019-02-2208:13ChrisThis is great! For your rebl-11 alias, does that remove the need to install OpenJFX? #2019-02-2209:17ChrisWell, it does! I just tried 🙂#2019-02-2402:55maxpHow to make uberjar like depstar does but with aot and manifest, optionally excluding sources?#2019-02-2413:09ChrisCambada might be able to do what you need. It can do no sources and AOT, not sure about manifests.
https://github.com/luchiniatwork/cambada/blob/master/README.md#2019-02-2416:45borkdudeYou might also get away with this approach, using leiningen as a lib?
https://github.com/oakes/full-stack-clj-example#2019-02-2500:56albert.laihttps://github.com/EwenG/badigeon#2019-02-2509:05ChrisWow Badigeon looks great, nice. #2019-02-2514:01maxpCambada looks great but there was some critical issue - it failed to make jar with cheshire 5.8.1 that comes with new Jackson, with cheshire 5.8.0 it wokrs well.#2019-02-2514:05maxpOk, I'll try badigeon later.
At that time I just do (compile ') and depstar classes into jar 🙂#2019-02-2521:04aviCambada has been working well for my project.
Here’s my deps.edn alias: https://github.com/FundingCircle/fc4-framework/blob/master/tool/deps.edn#L68#2019-02-2609:19maxpIt works well when I use cheshire 5.8.0 in project dependencies but fails with cheshire 5.8.1!
The difference is in version of Jackson used by Cheshire.#2019-02-2512:08souenzzofew days ago i saw it on twitter a project that use's tools-deps + lein as library to generate uberjar but I lost the link 😞#2019-02-2512:08dominicmNot lein-tools-deps do you mean?#2019-02-2512:08dominicmor pack.alpha / depstar maybe?#2019-02-2512:11borkdudeI just mentioned it in the thread above: https://github.com/oakes/full-stack-clj-example{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 5")}
#2019-02-2720:20pbrownHey there! I have a small question about intended use/behavior:
Should libraries declare [:paths ["src"]] in their deps.edn when they use the default?
If my deps.edn includes a top-level [:paths ["bogus/sources"]] entry, and I bring a lib in via :git/url that does not declare :paths in its own deps.edn, its :paths-derived classpath entry becomes ~/.gitlibs/libs/$GROUP_ID/$ARTIFACT_ID/$SHA/bogus/sources, which doesn't exist.
For example:
clj -Srepro -Sdeps '{:paths ["bogus/sources"] :deps {com.bhauman/figwheel-core {:git/url "" :sha "840f05547935f65a28c5a3c640a9473961da5636"}}}' -Spath | tr : '\n' | grep figwheel
# => /home/phil/.gitlibs/libs/com.bhauman/figwheel-core/840f05547935f65a28c5a3c640a9473961da5636/bogus/sources
#2019-02-2720:20pbrownOr is this something tools.deps could/should do (would it be safe to assume we all use the same /usr/local/lib/clojure/deps.edn)?#2019-02-2720:24Alex Miller (Clojure team)This a known bug#2019-02-2720:24Alex Miller (Clojure team)Where the current project paths are used with dependency projects on the file system#2019-02-2720:28Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-52#2019-02-2720:28pbrownah ty, missed it the first time through. It has the same repro!#2019-03-0202:47nbenketafHi there, can I mix clojure and java files in the same tools deps project ?
Like :path ["src" "java-src"] where src contains clojure source files and java-src java source files, and then import the classes from java source files directly in a clj source file ?#2019-03-0202:51hiredmanno#2019-03-0202:59nbenketafok thanks, are there some external tools/libs that can achieve that or is it planned in the tools deps roadmap ?#2019-03-0203:01hiredmanas far as I know it is positively not on the road map, java requires compilation before you can use it, a build step, and tools.deps is positioned as "not a build tool"#2019-03-0203:02hiredmanI would be kind shocked if there isn't a tool someone wrote for it, but I don't know of one off hand#2019-03-0203:08nbenketafMaybe I can handle that with the “full-stack-clj-example” above, integrating leiningen as a lib, thanks anyway 🙂#2019-03-0203:52seancorfield@nbenketaf Or you could, you know, run javac on your Java code, and then run clj...#2019-03-0204:01seancorfieldAnother option is this: {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "ec9f898d828faca8a9bfa7b8a3bcc1bdb9deb9a6dda2"}, :content ("[email protected]")}#2019-03-0205:00nbenketaf@seancorfield thank you for the inputs, here is the larger problem I am trying to solve : I have a forked project A initially managed with lein which includes some java files. I tried to convert this project to a tools deps one, basically I added a deps.edn file. Then I have another project B that declares a deps to project A with a git/url coordinate, and it’s in project B where I need the compiled java code. I guess I should package my forked project A, and use the mvn coordinate instead#2019-03-0205:09seancorfieldYeah, if a dependent project has code that needs to be compiled, either you need to package it as an "artifact", or you'll need to bring it down as source and compile stuff yourself.#2019-03-0205:10seancorfieldJava source projects pretty much require compiled artifacts for use. Clojure is trying to get away from that by allowing source projects to be used as-is.#2019-03-0210:18nbenketafOk i see, thank you for taking the time answering 🙂#2019-03-0421:00rschmuklerIs anyone using https://github.com/clojure/tools.deps.alpha/tree/add-lib effectively? I'm getting a NullPointerException on clojure.tools.deps.alpha.util.maven/make-session (maven.clj:141) - seemingly since around the time the zero-day vulnerabilities in the JVM got fixed (but I might be conflating things). No worries if the answer is "this branch isn't supported, don't use it"#2019-03-0421:01Alex Miller (Clojure team)could you be more specific about what you're doing?#2019-03-0421:01rschmuklerCalling add-lib causes that to be thrown#2019-03-0421:02rschmuklerHmm actually though, not all the time#2019-03-0421:02Alex Miller (Clojure team)so, that's a branch, what commit are you using, what is the full call, etc. give me a repro.#2019-03-0421:03rschmuklercalling it with clojure.java-time {:mvn/version 0.3.2} seems to be the issue.#2019-03-0421:03rschmukler@alexmiller will do - give me a moment to try and create a minimal repro repo#2019-03-0421:04Alex Miller (Clojure team)you don't need a repo - you can put it all in an clj -Sdeps call to start, then whatever expression you're doing#2019-03-0421:16Alex Miller (Clojure team)$ clj -Sdeps "{:deps
> {org.clojure/tools.deps.alpha
> {:git/url \""
> :sha \"e160f184f051f120014244679831a9bccb37c9de\"}}}"
Checking out: at e160f184f051f120014244679831a9bccb37c9de
Clojure 1.10.0
user=> (use 'clojure.tools.deps.alpha.repl)
nil
user=> (add-lib 'clojure.java-time {:mvn/version "0.3.2"})
true
user=> (require 'java-time)
Syntax error (IllegalAccessError) compiling at (temporal.clj:324:1).
class clojure.lang.PersistentUnrolledVector$Card1 tried to access field clojure.lang.APersistentVector._hash (clojure.lang.PersistentUnrolledVector$Card1 is in unnamed module of loader clojure.lang.DynamicClassLoader @7a0e1b5e; clojure.lang.APersistentVector is in unnamed module of loader 'app')#2019-03-0421:17rschmukler@alexmiller I'm not even getting that far, although I can't seem to reproduce the issue in a new repo so now I'm trying to recreate more and more of the state in my user.clj as to what's happening before I invoke add-lib.#2019-03-0421:18Alex Miller (Clojure team)I get that defaulting to Clojure 1.10. Not sure what clojure.java-time or the dependency clj-tuple assume with clj or java version#2019-03-0421:19rschmuklerInteresting. I've got clojure.java-time working w/ a 1.10 repo, although I added it via deps.edn, not dynamically via add-lib - so I haven't tested it.#2019-03-0421:21Alex Miller (Clojure team)clj-tuple seems to be compiled Java classes, certainly possible that doesn't play nice with add-lib#2019-03-0421:21Alex Miller (Clojure team)that error doesn't match what you described though re NPE#2019-03-0421:21rschmuklerI'm seeing it w/ other libs as well - but I'm hitting the error while invoking add-lib, not after (and trying to require it)#2019-03-0421:22rschmuklerWhile I try and reproduce, here's a stack trace if it helps: https://gist.github.com/rschmukler/8ae09c46ede6bab8d003dc4fe8cccef8#2019-03-0421:22Alex Miller (Clojure team)what sha are you using for tools.deps?#2019-03-0421:22rschmuklere160f184f051f120014244679831a9bccb37c9de#2019-03-0421:23rschmuklerAlso it seems to raise the error w/ about 50% of the deps.#2019-03-0421:23rschmuklerie. I wrote a wrapper that calls add-lib on my project for everything in deps.edn#2019-03-0421:23rschmukler(defn sync-libs
"Dynamically refreshes all libs in `deps.edn` and installs any missing"
([] (sync-libs nil))
([?alias]
(let [edn (-> "deps.edn" slurp read-string)
libs (if ?alias
(get-in edn [:aliases ?alias :extra-deps])
(get edn :deps))]
(doseq [[lib-sym lib-spec] libs]
(try
(add-lib lib-sym lib-spec)
(catch NullPointerException e
(println "Error raised for:" lib-sym lib-spec)))))))
#2019-03-0421:24rschmuklerBut again, this doesn't appear to be an issue in a different project. Still trying to build back up to it happening.#2019-03-0421:28Alex Miller (Clojure team)add-lib doesn't re-add any deps it already has btw#2019-03-0421:28rschmuklerThat's fine, I was presuming it to be a no-op#2019-03-0421:29rschmukler(ie. my workflow is I have a file watcher watching deps.edn and it syncs the libs on change. I'm not doing any diffing)#2019-03-0421:29Alex Miller (Clojure team)the error seems to imply you can't make a maven system, but not sure why you wouldn't get an error at that point#2019-03-0421:30Alex Miller (Clojure team)(use 'clojure.tools.deps.alpha.util.maven)
(make-system)
#2019-03-0421:31rschmuklerThat worked (ie. no exception)#2019-03-0421:33Alex Miller (Clojure team)well, not really sure. you're doing this under nrepl and it's possible the classloader hierarchy there is having some impact, don't really know.#2019-03-0421:33Alex Miller (Clojure team)I haven't seen this reported before (although obviously not many people using this)#2019-03-0421:35rschmuklerI'm actually seeing it even if I don't use nrepl (ie. if I just use a plain ol' clojure repl)#2019-03-0421:35Alex Miller (Clojure team)well, that's useful in ruling things out#2019-03-0421:35rschmuklerBut yeah, I appreciate that I'm beyond the bleeding edge w/ this. I'll see if I can reproduce it...#2019-03-0421:35Alex Miller (Clojure team)is there any chance you have this happening in multiple threads a the same time?#2019-03-0421:36rschmuklerI don't think so. I thought it might be some sort of race condition but it seems to be consistent in the dependencies that it fails on#2019-03-0421:36rschmukler(and since it's just a clojure repl, I can't imagine anything spawning threads)#2019-03-0421:38Alex Miller (Clojure team)you could try (dotimes [_ 20] (assert (not (nil? (make-session (make-system) default-local-repo)))))#2019-03-0421:39Alex Miller (Clojure team)see if you get any nils#2019-03-0421:40rschmuklerAlright, will try momentarily#2019-03-0421:43Alex Miller (Clojure team)I guess would also be good to know which java version you are on#2019-03-0421:45rschmuklerI found the library that causes it#2019-03-0421:46rschmuklerSo, if you have an existing deps.edn w/ metasoarous/oz {:mvn/version "1.5.6"} and then try and invoke add-lib the errors happen#2019-03-0421:46rschmuklerconfirming in my other repo#2019-03-0421:47rschmuklerYep! That does it. https://github.com/rschmukler/clojure-add-lib-exception#2019-03-0421:48rschmuklerNote that the sync-libs is commented out. Digging as to why that lib causes it.#2019-03-0421:50rschmuklerλ java --version
openjdk 11.0.2 2018-10-16
OpenJDK Runtime Environment (build 11.0.2+7)
OpenJDK 64-Bit Server VM (build 11.0.2+7, mixed mode)
#2019-03-0421:53rschmuklerMight be that library's dependency on https://github.com/clojure-goes-fast/lazy-require#2019-03-0421:53rschmuklerYep, this looks like it#2019-03-0421:54rschmukler> Another problem might happen if you mix lazy-require with some other code that does dynamic namespace loading. lazy-require itself makes sure to serialize the loading of namespaces so that nothing breaks because of concurrent loading (e.g., see clojure-emacs/cider#2092). However, if other code also loads namespaces in the background, something might happen. If you ever experience such issues, please file a ticket.#2019-03-0421:55rschmuklerAlthough, it's not like I'm requiring that lib, or even using it. ie. simply having it in the project (oz, not confirmed that it's this library specifically) causes the issue.#2019-03-0421:56Alex Miller (Clojure team)well, certainly sounds likely#2019-03-0421:58Alex Miller (Clojure team)oz is using in it's notebook namespaces#2019-03-0421:58Alex Miller (Clojure team)well, I guess specifically https://github.com/metasoarous/oz/blob/master/src/clj/oz/notebook/iclojure.clj#2019-03-0421:59rschmuklerHeh, we are doing exactly the same thing 😛#2019-03-0421:59Alex Miller (Clojure team)actually that loads it but doesn't use it#2019-03-0421:59rschmuklerAlthough, I don't think it's actually using it. If you look at the lib itself, it's pure#2019-03-0421:59rschmuklerhttps://github.com/clojure-goes-fast/lazy-require/blob/master/src/lazy_require/core.clj#2019-03-0421:59rschmuklerie. it's not like it's doing anything to the run-time or anything#2019-03-0522:21aaelonyI happen to need to use an old (but productionized) box that I’m not allowed to update java on. It’s running java 6. Both lein and clj complain about Exception in thread "main" java.lang.UnsupportedClassVersionError: clojure/main : Unsupported major.minor version 52.0
#2019-03-0522:21aaelonyrunning on linux: #66~precise1-Ubuntu#2019-03-0522:23hiredmanjava 6 isn't support on recent clojure versions if I recall#2019-03-0522:23hiredmanI think 8 is the new minimum#2019-03-0522:24hiredmanhttps://dev.clojure.org/jira/browse/CLJ-2363#2019-03-0522:25hiredmanand the change log for 1.10 says java 8 or above#2019-03-0522:25aaelonythat’s probably what I’m up against. Thanks.#2019-03-0522:26aaelonymaybe there’s an old version of lein I can use…#2019-03-0522:28aaelonyupdate… found a work-around unrelated to clojure… thanks#2019-03-0522:32Alex Miller (Clojure team)yes, clj and clojure 1.10 both require Java 1.8+#2019-03-0522:32Alex Miller (Clojure team)I assume "productionized" means "completely insecure" :)#2019-03-0523:37aaelonylol. It means I didn’t have the option of considering any change there… haha#2019-03-0523:43Alex Miller (Clojure team)Yeah, I get it#2019-03-2622:52borkdudeI’m getting a stale classpath for an alias with only one local/root dependency. I added a dependency to the local/root, but now I removed it. Yet clj -A:clj-kondo -Spath still reports it being there.#2019-03-2622:52borkdudeHow/where can I clean this?#2019-03-2622:53borkdude-Sforce did the trick#2019-03-2622:56Alex Miller (Clojure team)Yeah, currently dep changes to local deps do not cause a recalc - need a force for that#2019-03-2622:56Alex Miller (Clojure team)I have a solution for this but it’s a bit involved#2019-03-2720:19borkdudeis there an install script for MacOS like on linux? using brew on CI takes a bit long#2019-03-2720:36borkdudecaching brew probably also works#2019-03-2720:37hiredmanare you saying your ci server is running osx?#2019-03-2720:37borkdudeyeah#2019-03-2720:37hiredmanwild#2019-03-2720:37borkdudeCircleCI 🙂#2019-03-2720:38borkdudeit’s pretty cool. I can build a binary for linux and MacOS that way#2019-03-2721:04Alex Miller (Clojure team)You can use the Linux script on mac I believe#2019-03-2914:20sundbpI'm trying to use a git/url dependency behind a proxy (for an https repo). Any idea how I can set the proxy info that jgit uses under the hood?#2019-03-2914:26Alex Miller (Clojure team)I think that is an open issue#2019-03-2914:27Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-124#2019-03-2914:27Alex Miller (Clojure team)haven't had a chance to look at it yet#2019-03-3020:58sundbpThanks. That’s indeed the relevant issue. bootstrapping the JGit proxy settings to be consistent with either maven settings, or with git settings (https://gist.github.com/evantoli/f8c23a37eb3558ab8765) sounds logical.#2019-04-0110:39mpenetany news on https://dev.clojure.org/jira/browse/TDEPS-51 ?#2019-04-0110:40mpeneta comment says it's fixed but I couldn't confirm it, it still seems broken#2019-04-0111:59mpenetmy bad it does work now#2019-04-0119:41sundbp@alexmiller I'd be willing to do some lifting on tdeps-124 (git proxies) if it is of help and we can agree a direction of where to take proxy info from. It is quite a blocker for my adoption of tools.deps so it is high prio for me.#2019-04-0120:23Alex Miller (Clojure team)go for it! I think the two main options are to leverage the existing maven info (you can look at how that's accessed in the maven extension) or to look at leveraging whatever is native for git. this probably ends up flowing down into tools.gitlibs too.#2019-04-0120:37sundbpI'm torn between using the git config as more logical, and the cost of 2 config actions required for the user. It is probably still my slight preference.#2019-04-0121:05Alex Miller (Clojure team)make a list of the tradeoffs#2019-04-0121:05Alex Miller (Clojure team)in the ticket#2019-04-0219:38sundbp@alexmiller i’m confusing myself. signed up for a JIRA account and was going to comment on that ticket but don’t seem to see any Reply type button anywhere. Is it permissioned?#2019-04-0219:43Alex Miller (Clojure team)yeah, gimme a sec#2019-04-0219:44Alex Miller (Clojure team)should work now#2019-04-0220:36sundbpthanks#2019-04-0221:51sundbpput down the 3 ways I’ve been thinking about in that ticket.#2019-04-0513:09Laverne SchrockOpinion:
Of the options presented on the ticket, all except for the "Use maven proxy" seem reasonable. A single source for the proxy config is nice, but Maven settings should only be for fetching things via Maven. If we want centralization I'd vote for the "environment variable" or the "JVM flags" options (though I'm unclear on what the latter would end up looking like).#2019-04-0222:06sundbp@alexmiller another thing I’ve been running into is the monorepo story. I’ve so far adopted the convention to “hijack” CLJ_CONFIG via wrapper tools in my monorepo. E.g. I make it impossible for the user to control CLJ_CONFIG, and use it as an escape hatch to merge in one more deps.edn into the mix. This is obviously not ideal, and leads to discussions like this one in terms of tooling etc: https://github.com/juxt/pack.alpha/pull/34. Similarly to -Sdeps, is there any milage in a similar flag taking an actual additional deps.edn file? e.g. -Sdeps-file which can be given 0 or more times. That’d make at least the parts of the monorepo cases I’ve considered more manageable. Has it already been considered? If not any initial thoughts?#2019-04-0222:07Alex Miller (Clojure team)interested in doing more, haven't had a chance to think seriously about it#2019-04-0303:26lilactownAny examples using deps.edn + packr?#2019-04-0313:01sundbpI seem to run into some classloader issue when compiling gen-class using clj.#2019-04-0313:01sundbpI'm trying to use compile with a gen-class. I get a class not found exception when the constructor is defined taking a class from a lib I'm using. I can import that same class, and changing the type to eg String makes compile work. Any caveat about types for constructor args and class path etc?#2019-04-0313:01sundbpThe same setup with lein works OK.#2019-04-0313:03sundbpIt is like the classloader is out of whack. If I start a rebel repl I can import the class etc, and I can compile from the repl. But when passed via the -e flag the same expression gives class not found exception.#2019-04-0313:10sundbpIf I wrap it into something simple that does the same thing via a -m main method it is OK. But the -e version with same code fails.#2019-04-0521:11rafaelI'm seeing an Error building classpath message when running clj. I'm running with -Sverbose but it doesn't help to identify the problematic dependency.#2019-04-0521:13rafael$ clj -Sverbose -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.9.0"} midje {:mvn/version "1.9.7"}}}'
version = 1.9.0.329
install_dir = /nix/store/i2zsj4qknknpfrvfhpx0z69c44b0c12j-clojure-1.9.0.329-prefix
config_dir = /home/rafael/.clojure
config_paths = /nix/store/i2zsj4qknknpfrvfhpx0z69c44b0c12j-clojure-1.9.0.329-prefix/deps.edn /home/rafael/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/2657530762.cp
Refreshing classpath
Error building classpath. In the dependency graph, library org.clojure/clojure specifies version range "[1.2.1],[1.3.0]" - please specify a version in your configuration.
#2019-04-0521:13rafaelShould I open an issue? I'm having a hard time finding out how to proceed debugging the problem.#2019-04-0521:14Alex Miller (Clojure team)Update your clj version#2019-04-0521:15Alex Miller (Clojure team)Some older versions like this one didn’t handle version ranges#2019-04-0521:15Alex Miller (Clojure team)Newer ones do#2019-04-0521:17rafaelNewbie question: is there a way to test a new clj version without updating the binary installed in the system?#2019-04-0521:27lilactownI think it's literally a shell script, you could try downloading it and executing it in a local dir?#2019-04-0521:28Alex Miller (Clojure team)It’s both a script and an uberjar#2019-04-0521:29Alex Miller (Clojure team)They need to be matched up#2019-04-0521:31Alex Miller (Clojure team)There isn’t an easy way to do what you’re asking but you could always rollback to the version you’re on - it’s just the same url with the version you want#2019-04-0521:31rafaelThanks!#2019-04-0813:15stathissiderisdeps has made such a big difference to my Clojure workflow! I’m eternally grateful for Leinignen for taking us this far, but tools-deps has made things even better#2019-04-0813:21Alex Miller (Clojure team)thx!#2019-04-0814:31stathissiderisI’ll list some specific benefits that I’ve seen:
1. Less mono-repo oriented development due to effortless git deps (this was huge for me).
2. I’m more likely to fork open source libraries (and attempt to fix them) also because of git deps.
3. Casually adding custom automated project-related “tasks” (DB migrations, linting, testing etc) because of the combination of aliases and Clojure “scripts” (where with lein I would need a plugin)
4. Stopped using project generation tooling, I now make a deps.edn, an src folder and I suddenly have a new Clojure project!{:tag :div, :attrs {:class "message-reaction", :title "tada"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("🎉")} " 4")}
#2019-04-0814:39kirill.salykin> 3.
cant you do same with lein alias?#2019-04-0814:49souenzzoyeah, but lein culture says to develop a plugin.#2019-04-0815:05stathissideristo the point where I was not aware of lein alias 😄#2019-04-0815:12stathissiderisoh I did know it, but it requires a main method#2019-04-0916:15borkdudehow can I add compiled .class files to the classpath with tools.deps?#2019-04-0916:17dominicmPut them in a folder, add it to your paths.#2019-04-0916:17borkdudeso I put it in target/classes and then :paths ["src" "target/classes"]?#2019-04-0916:18Alex Miller (Clojure team)yep#2019-04-0916:18borkdudeI tried this from the command line: -J-cp target/classes but that didn’t work#2019-04-0916:18Alex Miller (Clojure team)no, that won't work as clj is building the classpath#2019-04-0916:19Alex Miller (Clojure team)probably should actually make that an error#2019-04-0916:19borkdudeand if target/classes doesn’t exist, it won’t bail out?#2019-04-0916:19Alex Miller (Clojure team)that's the one java property you can't set :)#2019-04-0916:19Alex Miller (Clojure team)missing dirs are fine#2019-04-0916:19borkdudecool#2019-04-0916:19Alex Miller (Clojure team)java doesn't care#2019-04-0916:19Alex Miller (Clojure team)(but it also won't pick up stuff that gets added later, I don't think)#2019-04-0916:19borkdudethat’s fine#2019-04-0916:20borkdudeI only need to compile a class if I’m using GraalVM compilation, for normal JVM usage it’s unnecessary#2019-04-0916:20Alex Miller (Clojure team)yeah, should be fine#2019-04-0916:20borkdudeso this dir is going to be in my gitignore#2019-04-0916:20Alex Miller (Clojure team)makes sense#2019-04-0917:16rschmuklerIs it possible to set environment variables via the deps.edn file, similar to the :env section of a project.clj? If not, what's the recommended way to handle that when using deps.edn?#2019-04-0917:19hiredman:env as a way to set env variables is something some lein plugin provides#2019-04-0917:20hiredmanI don't think that plugin actually sets environment variables, it provides its own kind of shadow system that you get access to as long as you use that plugin's functions for reading env variables#2019-04-0917:21rschmuklerAh! I misunderstood. There was a github issue that made it seem like it did something better than that -namely they were overwriting LD_PRELOAD and it was working it seemed#2019-04-0917:22rschmukler(I'm hooking into some C FFI with an old relic of a library)#2019-04-0917:23hiredmanit looks like environ dumps the environment stuff to .lein-env, which is maybe a standard thing? I am not sure#2019-04-0917:23hiredmanbut like, if you have environment variables, why not just set them?#2019-04-0917:24hiredmanLD_PRELOAD=... clj ...#2019-04-0917:24rschmuklerI don't want to overwrite them outside of the project, and I typically invoke clojure via cider#2019-04-0917:24rschmuklerso, adding project specific environment variables to my emacs config feels quite dirty#2019-04-0917:24rschmukler(and requires all engineers on the project solve it on their own)#2019-04-0917:33rschmuklerI was able to solve it via dir-locals w/ emacs, for anyone interested...#2019-04-0917:33rschmukler((nil . ((eval . (setenv "LD_PRELOAD" "libcurl.so.3")))))
#2019-04-0917:55Alex Miller (Clojure team)on clj command line, you can use prefix -J to pass env vars - like -J-Dmy.prop=val#2019-04-0917:56Alex Miller (Clojure team)or you can put those in an alias in deps.edn under :jvm-opts#2019-04-0917:56Alex Miller (Clojure team){:aliases
{:foo {:jvm-opts ["-Dmy.prop=val"]}}}#2019-04-0917:57Alex Miller (Clojure team)clj -A:foo#2019-04-0917:57rschmuklerIs there a default equivalent?#2019-04-0917:57Alex Miller (Clojure team)no, but that has been requested multiple times and I expect will be added when I next get around to it#2019-04-0917:57Alex Miller (Clojure team)there's a ticket for it#2019-04-0917:58rschmuklerCool cool. It's completely understandable that there wouldn't be one (yet, or even ever). Thanks for the response!#2019-04-0917:58Alex Miller (Clojure team)oh, you're asking about env vars#2019-04-0917:59Alex Miller (Clojure team)I'm talking about jvm opts#2019-04-0917:59Alex Miller (Clojure team)those are different things#2019-04-0917:59Alex Miller (Clojure team)sorry for the confusion#2019-04-0917:59Alex Miller (Clojure team)I consider that out of scope for clj right now#2019-04-0918:00rschmuklerMakes complete sense. Frankly, I think it probably introduces more nasty things than it's worth (eg. people using it as a config store for default environment variables, etc)... This is definitely an edge-case#2019-04-0918:00Alex Miller (Clojure team)yeah, I'll resist adding that as long as I can :)#2019-04-0918:00rschmukler:thumbsup: I think that's a good call#2019-04-0918:02seancorfieldThat begs an interesting question: when folks are providing "environment" to processes, do they a) use only environment variables b) use only JVM properties or c) use a combination with one overriding the other? And, for c) which overrides which?#2019-04-0918:04rschmukler@seancorfield As someone who came to Clojure (and the JVM) after heavily integrating docker + kubernetes, I can tell you I use environment variables for almost everything, and, if possible, don't intend to ever use or learn JVM properties#2019-04-0918:06rschmuklerI think it's mostly that containers (and golang especially) harbored in a lot of workflow / tooling around the idea of using environment variables, and it's something that can apply to outside any one run-time. ie. I can write a service and guarantee environment vars as a config option... If I switch away from JVM for any reason, I now need to provide a config migration story too#2019-04-0918:07rschmuklerregarding (c) - I would expect environment variables to be the last authority... but I think that's likely because of the esteem that I hold them in above#2019-04-0918:09Alex Miller (Clojure team)usually in an app I load config from an edn file, but allow override via env, and don't use jvm props at all{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-04-0918:10Alex Miller (Clojure team)I find that gives you a lot of flexibility and options for local dev, CI, prod#2019-04-0918:10Alex Miller (Clojure team)but then I don't work on many apps :)#2019-04-0918:11rschmukler@alexmiller that's exactly what I've done as well. I will sometimes have multiple configs corresponding to an environment (`prod.edn`, dev.edn, test.edn) which might overwrite certain keys in config.edn. I then use ENV to determine which env-specific config to use, while still allowing overwrites for specific vars via environment variables#2019-04-0918:12rschmuklerElixir also does something similar to the above#2019-04-0918:14Alex Miller (Clojure team)it's not well-known, but I like many things about https://github.com/levand/immuconf#2019-04-0918:16Alex Miller (Clojure team)if you're willing to ignore the ENV stuff above and use only a stack of appropriately created config files#2019-04-0918:17rschmuklerBrilliant! Similarly, duct does EDN-based config quite well. The docs for it are a bit lacking... But they let you do coercion and environment variables directly via EDN reader macros#2019-04-0918:17rschmuklerhttps://github.com/duct-framework/core/blob/master/src/duct/core/env.clj#2019-04-0918:18Alex Miller (Clojure team)yeah, I like duct too from what I recall#2019-04-0918:18Alex Miller (Clojure team)I've used environ quite a bit, but can't say I love it#2019-04-0918:20rschmuklerDuct + Integrant have quickly become my favorite way to build / prototype applications. I feel like I'm building an arsenal of components that I can use to build almost any system that I have seen in my career. They integrate the config map w/ spec so you get good error messages, while still having a single map that shows the entire system. It's basically stuartsierra's component extended to a map / graph, rather than relying on manual instantiation from the programmer#2019-04-0918:20Alex Miller (Clojure team)yeah, weavejester is great at maximizing the Clojure Way ™️#2019-04-0918:21rschmuklertrade mark#2019-04-0918:21rschmukler😉#2019-04-0920:01jcfHi all. I'm looking into deploying a small toy project to Heroku inside a Docker container and I'm running into an interesting problem with my use of tools.deps.
When I build the Docker container locally I can see all the dependencies being downloaded. When I push the container to Heroku and run the thing the same dependencies are installed all over again, which takes so long Heroku kill my container and the app fails to start.
Is there anything container-specific to look out for with tools.deps? My first guess was there was a missing Maven repo somewhere but when I run the container locally all the dependencies are there. Naturally I'm now wondering if Heroku do something different on their platform that might prevent tools.deps from noticing the Maven repo.#2019-04-0920:03jcfI'm running clojure -Srepro -e '(println "Dependencies installed.")' to install things in my Dockerfile, and then I run the app with exec clojure -Srepro -A:measure -m foo.bar.main. The measure alias only adds a single JVM opt.#2019-04-0920:03jcf:measure {:jvm-opts ["-Dio.pedestal.log.defaultMetricsRecorder=foo.bar.report/reporter"]}
#2019-04-0920:05jcfI can reproduce the artefact-download behaviour if I run the container on Docker and invoke my CMD by hand.#2019-04-0920:05dominicmIs it the same user?#2019-04-0920:06jcfAnd I think I might have cracked it. It looks like ~/.m2 is /app/.m2/repository when I run the container but the Docker build step will install things in /root/.m2.#2019-04-0920:06jcf@dominicm I think you've cracked it too!#2019-04-0920:07jcfI'm guessing Heroku run things as some other user, which messes with $HOME. I wonder if I can tell tools.deps where to put the Maven repo with the Docker invocations only. Don't really want to mess with that in the project deps.edn as it's kinda up to the dev where they want to keep Maven artefacts.#2019-04-0920:08jcfDoes :maven/local-repo not work in an alias? 🙂#2019-04-0920:09dominicm@jcf try -Sdeps perhaps? But remember permissions.#2019-04-0920:10jcfI'll give that a stab.#2019-04-0920:20jcfI'm giving -Dmaven.local.repo=/app/.m2 a try. Let's see if that does the trick. #docker 🙂#2019-04-0920:24jcfThat didn't work. Because it's maven.repo.local. 😕#2019-04-0920:24Alex Miller (Clojure team)you can set it in deps.edn{:tag :div, :attrs {:class "message-reaction", :title "eyes"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👀")} " 4")}
#2019-04-0920:24Alex Miller (Clojure team):mvn/local-repo "/app/m2" as a root level entry#2019-04-0920:24Alex Miller (Clojure team)or whatever you need#2019-04-0920:25jcf@alexmiller I was hoping to only specify that for the Docker side of things to enable the default behaviour for local development.#2019-04-0920:25Alex Miller (Clojure team)that's not something you can set in an alias#2019-04-0920:25Alex Miller (Clojure team)but you could append by using clj -Sdeps '{:mvn/local-repo "..."}'{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-04-0920:26dominicmThat's what I was trying to say ^^ but with the brevity that mobile prefers :)#2019-04-0920:26jcfI tried that originally and the running container still installed everything. I'll bring back the -Sdeps ... approach and poke around inside the container to see if the user in the running container maybe can't access the :mvn/local-repo.#2019-04-0920:39jcfThe -Sdeps {:mvn/local-repo "/app/.m2"} did the trick! Just had to blow away the container on Heroku and push it up again.
Now I need to make sure we stick within allocated memory and start up fast enough despite not installing deps. Might have to uberjar yet.
Thanks for the help, @dominicm @alexmiller!#2019-04-0921:09Alex Miller (Clojure team)np#2019-04-1106:17borkdudeif I check some files into a resources folder in my git repo, will someone who uses my repo as a git dep be able to see these resources? I remember there was an issue with this in tools.deps?#2019-04-1106:17borkdudeor was this only when you’re using it as a local/root?#2019-04-1106:25borkdudeafter some testing, both seem to work#2019-04-1106:25borkdudeI remember what the issue was with this. maybe a caching issue#2019-04-1115:00borkdudewith leiningen it seems that lib/tools.jar is automatically on the classpath. with tools.deps it doesn’t seem so. what’s the best way to do it without using a cmd line like:
clj -Sdeps "{:deps {tools.jar {:local/root \"$JAVA_HOME/lib/tools.jar\"}}}"
#2019-04-1115:12Alex Miller (Clojure team)you should not have to do this#2019-04-1115:13Alex Miller (Clojure team)it should automatically be included if you are using the JDK (not the JRE) afaik#2019-04-1115:13Alex Miller (Clojure team)are you accidentally using the JRE?#2019-04-1115:16Alex Miller (Clojure team)might also be some differences from Java 8 vs Java 9+ (post-module system) in the jdk file structure. I'm not sure there even is a tools.jar anymore?#2019-04-1115:18Alex Miller (Clojure team)oh yeah, as of java 9, tools.jar does not exist anymore#2019-04-1115:19Alex Miller (Clojure team)needing to include it directly is weird, and probably wrong#2019-04-1115:30ghadiif the original question was about resources, as long as that directory is listed in the :paths on the dependent repo, then the depending repo will be able to see it#2019-04-1115:31borkdudeI’m using java 8#2019-04-1115:31borkdudea quick test if it works in any of your REPLs:
(import 'com.sun.tools.javadoc.Main)
#2019-04-1115:32borkdude$ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home
#2019-04-1115:32Alex Miller (Clojure team)that stuff is dead#2019-04-1115:32borkdudeis there something newer I should use?#2019-04-1115:33Alex Miller (Clojure team)yes - javax.tools.DocumentationTool#2019-04-1115:34Alex Miller (Clojure team)I don't know the coords for it, but you should just be using it as normal Maven artifact#2019-04-1115:34borkdudethat seems to work out of the box:
$ clj
Clojure 1.10.0
user=> (import 'javax.tools.DocumentationTool)
javax.tools.DocumentationTool
#2019-04-1115:36Alex Miller (Clojure team)prob doesn't in newer jdks#2019-04-1115:37Alex Miller (Clojure team)actually, it does on Java 11#2019-04-1115:37borkdudeso I can do something like this as with the older Doclet API? I want to generate EDN#2019-04-1115:38Alex Miller (Clojure team)I believe it is the replacement for all that old sun stuff, yes#2019-04-1115:38Alex Miller (Clojure team)I'm sure there are buckets of docs to read about it, I've never used it#2019-04-1115:38borkdudethanks#2019-04-1116:03borkdudeIt seems I do need the “dead” Doclet class if I’m using Java 8, since the jdk.javadoc stuff is only on Java 9#2019-04-1116:24borkdudeso I’m seeing this in the REPL:
(System/getProperty "java.home")
"/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre"
and the same with leiningen. But tools.jar is in ../jdk/lib/tools.jar.#2019-04-1116:27borkdudefunny, it seems in 2012 people wrote a special plugin for this:
https://github.com/pallet/lein-jdk-tools/blob/develop/src/lein_jdk_tools/plugin.clj#L11#2019-04-1116:44borkdudeI wonder that if I’m going to migrate to JDK 11 if my code will still be able to run for Java 8 users if I’m going to use those new classes, so I’m hesitant about it#2019-04-1116:54Alex Miller (Clojure team)it should be fine. that "new" doclet stuff has been around since at least java 7#2019-04-1116:55Alex Miller (Clojure team)seems better than building on something that has been pronounced dead and unsupported#2019-04-1116:56borkdudebut the new doclet stuff is only there since Java 9, or maybe I’m not finding the right stuff on google…#2019-04-1116:56borkdudeo wait: https://docs.oracle.com/javase/8/docs/jdk/api/javadoc/doclet/com/sun/javadoc/Doclet.html#2019-04-1116:57borkdudeOK yeah, I’m going to use that.#2019-04-1117:02dominicmWhat are you using doclet for?#2019-04-1117:07dominicmOh, it's for reading docs in java. I see.#2019-04-1117:07dominicmI thought it was for generating them.#2019-04-1117:05Alex Miller (Clojure team)everything com.sun is on the removal path#2019-04-1117:07ghadihttps://docs.oracle.com/en/java/javase/11/docs/api/jdk.javadoc/jdk/javadoc/doclet/package-summary.html#2019-04-1117:07ghadimake sure you read the migration parts of ^#2019-04-1117:08ghadi@borkdude#2019-04-1117:43borkdude@ghadi thanks!#2019-04-1117:44borkdudeOld type: SeeTag
New type: com.sun.source.doctree.LinkTree
com.sun.source.doctree.SeeTree
Hmm..#2019-04-1117:46borkdudeCrap, it does seem that this new stuff is only available since 9#2019-04-1117:47borkdudeAnyway, thanks for the help.#2019-04-1117:51nopromptDo we have tool for releasing to Clojars in deps land?#2019-04-1117:52borkdudeThere are a bunch of these tools on the wiki#2019-04-1117:52nopromptI’m still lein deploy clojars. 😐#2019-04-1117:52borkdudeMe too.#2019-04-1117:52nopromptI reckon if any of those were as convenient as that you’d have mentioned it. 🙂#2019-04-1117:53borkdudeHaven’t tried them yet#2019-04-1117:54nopromptI just want a microwave.#2019-04-1117:54nopromptA jar cooking microwave.#2019-04-1117:55borkdudedepstar is one that I’ve heard of#2019-04-1117:55Alex Miller (Clojure team)you can call lein fns from clj#2019-04-1117:55borkdudehttps://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging#2019-04-1117:57borkdude@noprompt this is an example of calling lein using tools.deps: https://github.com/oakes/full-stack-clj-example
nice hack#2019-04-1117:59Alex Miller (Clojure team)I added that to the wiki page :)#2019-04-1119:28mpenetit doesnt work with git deps, it uses lein for dependency resolution when building the uberjar https://github.com/oakes/full-stack-clj-example/blob/master/prod.clj#2019-04-1119:28mpenetI'd still favor pack/depstar#2019-04-1119:29mpenetI guess it could be improved, the concept is quite interesting#2019-04-1117:58nopromptCool. I’ll check that out. pack looks pretty nice. The Juxt folks tend to make decent things.{:tag :div, :attrs {:class "message-reaction", :title "heart"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("❤️")} " 4")}
#2019-04-1120:50dominicmI appreciate the complement. Let me know if you have any issues.#2019-04-1209:55sooheonDoes the capsule method of uberjaring not support AOT compilation?#2019-04-1210:06dominicmIt works with AOT'd assets#2019-04-1210:06dominicmWhy do you think it doesn't?#2019-04-1210:10sooheonI’m coming from lein uberjar where you explicitly give it a ns to AOT compile. Also saw that Capsule main didn’t care about :gen-class. Do you mean I should AOT compile with something else, and provide the output of that as extra-paths?#2019-04-1210:11dominicmIf you really want to AOT, yeah, I just use clojure -e#2019-04-1210:12sooheonOk thanks#2019-04-1117:58borkdude(it’s not an off the shelve solution)#2019-04-1118:13martinklepschI found that there's a thing for every step of the process but none that puts all those bits together sadly#2019-04-1118:50seancorfieldWe're using (my fork of) depstar at work for building uberjars. I'm also using it to build "thin" jars for deploying to Clojars (and I use mvn deploy for that, so I run clj -Spom, clj -A:uberjar (alias for depstar's uberjar entry point), and then mvn deploy:deploy-file ...)#2019-04-1118:50ghadi^ @noprompt#2019-04-1118:50ghadithat will get you going#2019-04-1118:51seancorfield(well, I have a little shell script that combines that -- and also updated the pom.xml with the current SHA of the project so http://clojars.org can point to the commit on GitHub)#2019-04-1118:53nopromptNice. I’ll give this a try.#2019-04-1119:22borkdudeDarn.
> This class must contain the start(Root) method.
This is in Java 11 documentation for javadoc which is also used by those new ToolProvider thingies.
But the start(Root) method is from the ancient com.sun code. So this buys me nothing… The documentation for the new Doclet interface doesn’t even mention this though.
https://docs.oracle.com/en/java/javase/11/tools/javadoc.html#2019-04-1200:42cflemingIs anyone using deps with polyglot JVM projects, e.g. containing Java/Kotlin/etc?#2019-04-1200:42cflemingI’m interested in recommendations for approaches if so.#2019-04-1200:43cflemingI see there are some lightweight build tool options listed in the wiki but I don’t know what the state of any of them is.#2019-04-1201:20Alex Miller (Clojure team)I probably wouldn’t use deps as the controller for that, I’d use it under make, or whatever#2019-04-1211:52sundbpCan’t recall if I brought this up before, but can’t find any traces (tickets etc). So we have -Sdeps to add “extra stuff”. For my purposes it’d be very useful to have the equivalent to add extra deps.edn files to be merged in, e.g. -Sdepsfile path/to/extra/deps.edn. As it is logically the same as -Sdeps "$(< path/to/extra/deps.edn)", just packaged in a different way - would there be any pushback to adding such an option? @alexmiller#2019-04-1212:42Alex Miller (Clojure team)There is a ticket for this and I’m not opposed to it, but am still thinking about various aspects#2019-04-1212:56orestisWait, -Sdeps adds extra stuff? Can it be combined with -A then?#2019-04-1213:46dominicmIt can, yeah.#2019-04-1213:47dominicmTo make sure I don't clobber things, I use a namespaced alias name in scripts, sometimes with generated extra paths (e.g. In tmp)#2019-04-1213:47Alex Miller (Clojure team)-Sdeps adds an additional "deps.edn" on the end of the merge chain#2019-04-1214:16orestisOhhh that’s so nice. #2019-04-1219:49sundbp@alexmiller great to hear.#2019-04-1308:19borkdudeif I make an alias, I should be getting the extra paths of that alias when doing -R:the-alias right?#2019-04-1308:19borkdudesomehow it’s not working for me, but it is working with the -A option#2019-04-1308:22borkdudeoh, -C… 🙂#2019-04-1520:35borkdudeis there a way to let tools.deps print only the paths that you specified in your deps.edn file and not the entire classpath?#2019-04-1520:37Alex Miller (Clojure team)No - why would that be useful?#2019-04-1520:37borkdudelinting 🙂#2019-04-1520:38Alex Miller (Clojure team)Why?#2019-04-1520:38borkdudewhy linting you mean?#2019-04-1520:40borkdudeone use case: I want to lint the entire classpath, but only report errors from my own code. but it’s not too hard to do some manipulation on the output of -Spath or hard-code it. the amount of paths if fairly limited.#2019-04-1520:45borkdudeto answer your question more generally: it could in theory be useful to list all the files from your own paths for whatever reason#2019-04-1520:45borkdudebut I guess not high priority for now#2019-04-1605:54orestisShouldn’t be that complicated to read the file and get the paths out, right?#2019-04-1711:54orestisIs there an invocation of Clojure that will give the current version number?#2019-04-1711:56mpenet*clojure-version* I think#2019-04-1711:56mpenet*clojure-version*
{:major 1, :minor 10, :incremental 0, :qualifier nil}
#2019-04-1712:03orestisAh, sorry - I meant about the Clojure CLI tools#2019-04-1712:04orestisWhen installing in Linux, I don’t want to download curl -O every time if it’s already present and at the correct version.#2019-04-1712:08dominicmclojure -Sdescribe?#2019-04-1712:10dominicm❯ clojure -Sdescribe | grep -o ':version ".*"' | sed 's/:version //' | sed 's/"//g'#2019-04-1712:14orestisFantastic! Thanks 🙂#2019-04-1712:38orestisOK, I’m having trouble running the Clojure CLI tools in Elastic Beanstalk, and the AWS support this time isn’t very helpful.#2019-04-1712:38orestisIn my logs, I see this error: mkdir: cannot create directory '/.clojure': Permission denied#2019-04-1712:39orestisWhich indicates to me that something is trying create a .clojure directory at the root of the filesystem, which is weird.#2019-04-1712:42dominicmAre you installing during user data?#2019-04-1712:42dominicmDoes -Sdescribe try and create .clojure unconditionally?#2019-04-1712:48orestisIt seems so — it’s a mess because Elastic Beanstalk is just duct tape under the hood.#2019-04-1712:48orestisSo you run your Java command with ruby foreman and supervisord…#2019-04-1712:50orestisGot it, they were setting HOME=/tmp whereas the user actually running the clojure command didn’t have permissions there.#2019-04-1712:50orestisSetting CLJ_CONFIG to /home/webapp/.clojure seemed to have fixed the problem.#2019-04-1712:59Alex Miller (Clojure team)there's a ticket about working in read-only fs btw#2019-04-1712:59Alex Miller (Clojure team)https://dev.clojure.org/jira/browse/TDEPS-119#2019-04-1713:08orestis.ebextensions/setup.config file that works:#2019-04-1713:08orestisoption_settings:
- option_name: CLJ_CONFIG
value: /home/webapp/.clojure
commands:
01_install_clojure:
command: |
curl -O
chmod +x linux-install-1.10.0.442.sh
sudo ./linux-install-1.10.0.442.sh
#2019-04-1713:12Alex Miller (Clojure team)currently, clj does need some writable path to work as it always drives at least the cp through the cached cp file#2019-04-1713:17orestisPosted a quick guide to https://clojureverse.org/t/running-a-clojure-application-on-aws-elastic-beanstalk-with-clojure-cli-tools/4161#2019-04-1713:17orestisThe project .cpcache works fine, it’s the $CLJ_CONFIG that was giving me trouble…#2019-04-1718:19tanzoniteblackis there an equivalent of lein's :init-nsconfig, where you specify which namespace to load and switch into by default when starting a repl?#2019-04-1718:37dominicm-e "(in-ns 'foo)"#2019-04-1721:15kennyAny ideas why I'd hit this? I'm running on a CI that has permission to git clone via HTTPS.
Error building classpath. ...: Authentication is required but no CredentialsProvider has been registered
#2019-04-1721:17kennyShoot. Looking at tools.gitlibs, it seems like it may not support cloning private repos via HTTPS 😞#2019-04-1721:22ghadiIt doesn't. I've worked on that.#2019-04-1721:22ghadiHave two different approaches for tackling that.#2019-04-1721:25kennyAh, I see TDEPS-91 and TDEPS-33. I assume there's no workaround for this right now?#2019-04-1721:27kennyI suppose I could try using the shell-git branch here: https://github.com/ghadishayban/tools.gitlibs/tree/shell-git.#2019-04-1721:27kenny... if you could override the tools-deps version the CLI uses.#2019-04-1723:22ghadi@kenny GitHub or otherwise?#2019-04-1723:23kennyWhere my git deps are coming from?#2019-04-1723:39Alex Miller (Clojure team)you can override it if you just hack your clojure script :)#2019-04-1723:40Alex Miller (Clojure team)just prepend the source dir to tools_cp#2019-04-1800:00kennyInteresting... Thinking I'll attempt the path of adding an SSH key to the CI. If too painful, will try the clojure script hacking.#2019-04-1808:34jeroenvandijk@kenny You can also play with gitconfig and for instance switch between ssh and https without code changes # [url ""]
# insteadOf =
#2019-04-1808:34jeroenvandijkpart of my ~/.gitconfig#2019-04-1814:42kenny@jeroenvandijk But this won't let tools-deps support https auth, correct?#2019-04-1814:44jeroenvandijkMy guess is that you can even achieve that https://git-scm.com/docs/gitcredentials#2019-04-1814:45jeroenvandijki did only the opposite as the https auth for github is limited#2019-04-1814:45jeroenvandijkwe're using circleci and they even have a feature to add an ssh key of another user (over a deploy key) to the project automatically#2019-04-1819:09dominicm@alexmiller what do you think about gitlibs making clones read only so that they aren't accidentally modified by user error?#2019-04-1819:11Alex Miller (Clojure team)you mean the working tree I presume#2019-04-1819:11Alex Miller (Clojure team)seems reasonable#2019-04-1819:12ghadithere's no such thing#2019-04-1819:13ghadireadonly directory means you cannot add or remove to it. modifying an existing file is still fair play#2019-04-1819:13ghadiyou'd have to ensure that every file was r/o too#2019-04-1819:14Alex Miller (Clojure team)I assumed that's what we were talking about#2019-04-1819:14dominicmI'll lodge a JIRA#2019-04-1819:15Alex Miller (Clojure team)the dirs under ~/.gitlibs/libs are sha-pegged and shouldn't change#2019-04-1819:16Alex Miller (Clojure team)does changing those properties affect whether it looks like the correct working tree to git?#2019-04-1819:17ghadihuh?#2019-04-1819:19Alex Miller (Clojure team)I guess I'm asking if file attributes affect the sha, but looks like no#2019-04-1822:00dominicmSome do, like +x#2019-04-2508:56jeroenvandijkWould it be a good idea to add the :inclusions to a maven entry? I'm updating the aws-java-sdk library and i'm really only interested in a few of it's dependencies, but it is exploding#2019-04-2510:00delaguardoyou have :exclusions key defined for maven entry. And it has much less abstract (or opinionated) meaning than :inclusions#2019-04-2510:06jeroenvandijkYep. So :exclusions would become unreasonably big with aws-java-sk therefore whitelisting seems like a better approach. Especially since an update means new exclusions#2019-04-2510:09delaguardoas far as I understand aws-java-sdk is just a collection of packages grouped bu functionality. If you are interested only in one particular package - add just this package and do not add aws-java-sdk#2019-04-2510:11delaguardolike this {com.amazonaws/aws-java-sdk-s3 {:mvn/version "1.11.539"}}#2019-04-2510:40jeroenvandijkOk i guess you have never experienced this problem. I have#2019-04-2510:40jeroenvandijkThanks for trying to help though#2019-04-2511:06delaguardoYou might be surprised) maybe you can describe you pain more? it could be helpful for anyone how is willing to help you#2019-04-2511:20jeroenvandijke.g. when I try to use Amazonica (`[amazonica "0.3.142"]`) for specific resources I would need to know what to exclude whereas I actually know what to include if I just want to do some s3 things#2019-04-2511:46delaguardoOk, so consider this https://github.com/mcohen01/amazonica#for-the-memory-constrained
I also made a gist with dependency trees for two cases: amazonica without any classpath changes and amazonica with just s3 added
https://gist.github.com/DeLaGuardo/4e32e950fa7ee7b594c9df2e97a10fd4#2019-04-2511:46delaguardoMain idea - exclude everything, then add parts you know#2019-04-2511:59delaguardoand btw - there is a bit nicer library to work with aws s3 - https://github.com/cognitect-labs/aws-api#2019-04-2512:08jeroenvandijk> Main idea - exclude everything, then add parts you know
Yeah I think you are right. Maybe I was just being impatient#2019-04-2512:31Alex Miller (Clojure team)There are split out versions of the stuff under aws-java-sdk#2019-04-2512:35Alex Miller (Clojure team)See https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/setup-project-maven.html#2019-04-2512:42jeroenvandijkthank you, i guess i was just being impatient.#2019-04-2514:30Lone Rangeralright... trying to fork/repair an out dated library (https://github.com/benashford/redis-async) which includes some local Java classes#2019-04-2514:31Lone Rangerso I converted the project.clj to deps.edn for starters#2019-04-2514:32Lone Ranger#2019-04-2514:32Lone Rangerthe directory structure is as follows:#2019-04-2514:32Lone Ranger#2019-04-2514:33Lone Rangerthe problem comes when running the tests, the Java classes do not seem to be importing properly#2019-04-2514:33Lone Ranger#2019-04-2514:34ghadi.java files need to be compiled into .class files#2019-04-2514:34ghadithey cannot be used raw#2019-04-2514:34Lone Rangerahh interesting#2019-04-2514:34ghadiI would suggest separating the java part of the project into a different artifact, and then including that in the clojure deps.edn project#2019-04-2514:35Lone RangerI think that's a good suggestion. Do you include it with {:local/root "../local-path"} as one would with a clojure project?#2019-04-2514:36Lone Rangeror is there a different way to include local java?#2019-04-2514:36Lone Rangeror should I upload it to maven or something?#2019-04-2514:36ghadiupload to maven or clojars#2019-04-2514:37Lone Rangeryou can upload java to clojars?#2019-04-2514:37ghadii think so#2019-04-2514:37ghadi"it's just a jar"#2019-04-2514:37Lone Rangercool... will look into it. Ah true story#2019-04-2515:08Lone Rangerare you sure you can't use .java files without compiling first...? I thought there was some kind of implicit compilation as long as class path was set up properly#2019-04-2515:08Lone Rangerbecause it worked fine for leiningen#2019-04-2515:08Lone Rangervia the key :java-source-paths ["jresp/src/main/java"]#2019-04-2515:10ghadiyeah, I'm sure#2019-04-2515:11ghadilein calls javac#2019-04-2515:15Lone Rangerah so that's fundamentally different for the deps.edn approach#2019-04-2515:54Alex Miller (Clojure team)yes#2019-04-2516:33seancorfieldYou can always use the -e option to shell out to a call to javac 🙂#2019-04-2516:33seancorfield(no, don't, it's horrible... but it does work 🙂 )#2019-04-3009:33yuhanIs there any way to use a lein plugin in a tools.deps project? E.g. Codox, lein-ns-dep-graph#2019-04-3009:57delaguardo@qythium most of lein plugins is just a normal projects. You can include them in your dependencies list and execute their -main function if exists or make your own custom -main. E.g. https://github.com/weavejester/codox/blob/master/codox/src/codox/main.clj#L116 there is main function to generate codox documentation. It requires some arguments that you have to assemble by yourself but there is nothing impossible.
clj -Sdeps '{:deps {codox {:mvn/version "0.10.6"}}}' -e "(do (require 'codox.main) (codox.main/generate-docs))"
Oneliner as a prove of concept#2019-04-3016:50seancorfieldAnd remember, if you're on Clojure 1.10, you can use requiring-resolve so that becomes -e "((requiring-resolve 'codox.main/generate-docs))"#2019-04-3010:08yuhanwow, I can't believe that just worked out of the box 😮#2019-04-3010:10yuhanfully formatted HTML docs of all the namespaces in a large project with zero setup, how awesome is that#2019-04-3010:10yuhanthanks!#2019-04-3010:32Christian Johansenwhen changing the sha on git deps, I frequently run into a variation of this error:#2019-04-3010:32Christian JohansenCloning:
#2019-04-3010:32Christian Johansenis this a known problem, or should I report it in jira?#2019-04-3010:34dominicmI'm not aware of any similar reports in this channel, nor have I encountered it itself.#2019-04-3010:34Christian Johansenhmm, I wonder if it could be a false message. I removed the lib from ~/.gitlibs/_repos, and then I see that it errors on auth, as I haven't sun ssh-add after a recent reboot#2019-04-3010:34Christian Johansenso maybe that error above also was really caused by an authentication failure?#2019-04-3010:41dominicmMaybe. Could be that there's more to be done with catching it correctly too though.#2019-04-3011:43Alex Miller (Clojure team)I haven’t seen anything like this from anyone#2019-04-3013:02Christian Johansenok. I'll see if I can reproduce it and if so, report#2019-05-0107:17luposlipdeps.edn newbie here: Anyone's got some intelligent input to the following issue, where a working lein based project, using a java library hosted on jitpack, doesn't work when used via deps.edn (dependency tree doesn't get resolved)?
https://github.com/luposlip/json-schema/issues/1#2019-05-0107:44dominicm@luposlip you need to put jitpack in your :mvn/repos#2019-05-0107:47luposlipeven when it's already referred to by the json-schema library?
The user wants to use json-schema, which already refers to jitpack, from a project using deps.edn.#2019-05-0107:59dominicmI'm not sure that deps can pick those up, there might be a jira.#2019-05-0108:01dominicmhttps://dev.clojure.org/jira/browse/TDEPS-46#2019-05-0108:01luposlipHmm.. When a library is packaged and deployed to clojars it normally works out of the box when using it from another lein project, regardless of the dependencies and where they are stored.
If this is not the case with clj + deps.edn it seems like a showstopper to me.#2019-05-0108:01luposlipAhh, thanks for the link!#2019-05-0108:02luposlipI'll follow that jira! 🙂#2019-05-0108:02dominicmYour vote will help with setting priority#2019-05-0108:03luposlipI will upvote it!#2019-05-0123:02kszabohey, moving from boot to tools-deps + kaocha, which of the many uberjar building tools do you recommend to replace boot’s capabilities? I see several but see no consensus from the community on which is the most stable / can expect future maintenance.#2019-05-0123:21kennyWe've been using https://github.com/seancorfield/depstar in production for several months now. Does the job.#2019-05-0123:43kszabothanks, but I just noticed that it doesn’t do AOT, guess I’ll need to add that to the list of required features so that the build tooling change does not change the produced artifact#2019-05-0123:56kszaboIt seems badigeon will fit our needs, thanks!#2019-05-0123:58seancorfieldAOT is evil 🙂{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 4")}
#2019-05-0200:04kennyYep - we stopped AOT'ing our code when we moved over from Boot. Too many headaches.#2019-05-0200:07seancorfieldWe stopped AOT'ing long before that 🙂 Hence depstar does exactly as much as we need: build a JAR from the classpath. In production we do java -cp /path/to/the.jar clojure.main -m our.entry.point#2019-05-0201:01donaldballWhat is the current reason to consider AOT evil? We’ve just started using it for our service uberjars to shave a modest but meaningful time off of startup time.#2019-05-0203:00seancorfieldIf you're using it as the very last stage, to produce an uberjar, and you really need it to improve startup time, that's "fine". But the Leiningen default where the main namespace is :gen-class'd and AOT is the default in project.clj just gets beginnners into a mess -- and it is completely unnecessary. AOT overall is completely unnecessary -- unless you really, really care about startup time for a final production artifact.#2019-05-0203:01seancorfieldAOT's transitive compilation issues and the problems you can run into when you end up with a mix of .class and .clj versions of the same file can be hell to debug. Much better to just avoid all of that, unless you know what you're doing and you see AOT as purely an end-of-the-line optimization.#2019-05-0203:01seancorfieldThe biggest overhead for us at startup time is New Relic's Java agent instrumenting classes (since Clojure generates so many).#2019-05-0203:02seancorfieldAnd since nearly all of our processes are long-running, we just don't care about startup time.#2019-05-0215:03kennyI could see it mattering more if you need to scale your service quickly due to large demand spikes.#2019-05-0217:19kszaboIn the end I took your advice and did not AOT before uberjaring. Still I stayed with badigeon because it allowed me to exclude some cljs related files I don’t need from my resources, which sadly depstar doesn’t support, having a static exclude function.#2019-05-0217:24seancorfieldFeel free to create an issue for depstar to exclude files -- I just refactored some of that code recently so it'll be easier to extend to support options (although, right now, depstar has no command-line options at all beyond the JAR filename).#2019-05-0215:56avi👋 I just searched this channel, the Web, and the Clojure JIRA for anything related to git dependencies that use submodules, but didn’t turn up anything… I’m trying to include a dependency via git, and when I run clojure -Stree it fails with org.eclipse.jgit.api.errors.JGitInternalException: Checkout conflict with file: resources/devtools-protocol which is caused by org.eclipse.jgit.errors.MissingObjectException: Missing unknown 40a14030 (sha truncated) — that’s the sha of the submodule.
Anyway I’m not looking for free troubleshooting, just wondering whether this is a known issue and whether it’s on anyone’s radar? If not I could create a ticket…?#2019-05-0215:57dominicmI don't think this has come up 🙂 a jira is best I think#2019-05-0215:57avialrighty — thanks!#2019-05-0216:05avibug report: https://dev.clojure.org/jira/browse/TDEPS-126#2019-05-0320:01vlaaadI really want to have add-lib branch into master, dynamic dependency loading removes so much friction!#2019-05-0320:29dominicmIt doesn't need to be on master#2019-05-0320:41seancorfield@vlaaad you can depend on the SHA for it -- that's what I do. Alex generally keeps that branch up-to-date with master.#2019-05-0320:42seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L106-L119#2019-05-0320:56vlaaadI know I can depend on shas, but that branch is 18 commits behind#2019-05-0320:58vlaaadoh, latest commit was merging master into it, nice#2019-05-0320:59vlaaadanyway, being unofficial and partially supported is concerning: I want newest stuff as well#2019-05-0321:19dominicmDevelopment workflows can be a little more alpha, especially something low risk like this. If it doesn't work, you restart the repl instead.#2019-05-0409:20yuhanhow do I add jvm-opts via the clj command?#2019-05-0409:21yuhanTrying to get https://github.com/clojure-goes-fast/clj-async-profiler working#2019-05-0409:21yuhanclj -Djdk.attach.allowAttachSelf does not work#2019-05-0409:28yuhanright, I figured it out - clj -J-Djdk.attach.allowAttachSelf#2019-05-0409:32yuhanbut to add it to my deps.edn file I can't just insert :jvm-opts ["-Djdk.attach.allowAttachSelf"] at the top level?#2019-05-0410:43Alex Miller (Clojure team)No, that’s an outstanding issue. You will need to put it in an alias and use the alias#2019-05-0614:10tianshuI want use -Jopt to add a jar file into classpath. what is the correct syntax looks like? clojure -J-cpjarpath ... seems to be no effect and clojure -J-cp jarpath ... will cause an error.#2019-05-0614:22dominicm@doglooksgood use -Sdeps#2019-05-0614:27Alex Miller (Clojure team)clj builds the classpath so you must use your deps.edn for any modifications to the classpath. in this case, you'll want to use a local jar lib dependency - like my/lib {:local/root "/path/to.jar"}#2019-05-0614:28Alex Miller (Clojure team)if you just want to add it sometimes, either put it in an alias with :extra-deps or declare it dynamically on the command line with -Sdeps#2019-05-0618:43sparkofreasonGetting this error running clojureL com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available. This is on WSL running under Windows 10. Seem to recall having this issue before, but I don't remember the fix. git commands work fine, I recall this being something specific with the Java ssh implementation.#2019-05-0618:51sparkofreason@seancorfield This was encountered while attempting to build an uberjar with depstar. Tried on Windows instead of WSL, where I get Exception in thread "main" java.nio.file.NoSuchFileException: C:\Users\dave\AppData\Local\Temp\uberjar8947452084826452117\uncomplicate\neanderthal\aux.clj, which seems weird, like there was a file in the manifest that didn't get copied.#2019-05-0618:54seancorfield@dave.dixon If you can repro easily, feel free to create an issue on GitHub with details and I'll take a look -- but I suspect something AOT-related#2019-05-0622:30seancorfield@dave.dixon Thanks for the repro case. It's definitely a weird one. Files/copy is failing on the target file during the copy which suggests the parent folder is missing (it isn't) or there is a permissions problem (there isn't). I've tried various changes to depstar locally but cannot figure out what the issue is.#2019-05-0622:32seancorfieldTracing through the JDK source, it seems to hinge on the last Windows filesystem error triggering an exception so I thought maybe it was a sync'ing issue -- but using SYNC or DSYNC for StandardOpenOption made no difference either.#2019-05-0622:36sparkofreasonI cloned and ran with :local/root and it works. :thinking_face:#2019-05-0622:40sparkofreasonWait - never mind, I was commenting out the neanderthal dependency 😬 Been that kind of day.#2019-05-0623:16sparkofreason@seancorfield If I exclude that file explicitly, it "works", in the sense of not throwing that exception. So it seems to be that single aux.clj file that's an issue.#2019-05-0623:17seancorfieldWeird.#2019-05-0623:46seancorfieldAnd it gets weirder. So I modified depstar so that if a copy operation fails, it tries again, just copying the file to the top of the tree in the temp folders (so uncomplicate/neanderthal/aux.clj copies to thing.clj) and then it "succeeds" and copies the files that follow aux.clj in the Neanderthal JAR. However, I got a weird NoSuchFile exception trying to write test.jar after that (where it shouldn't matter if the file exists or not). I ran touch test.jar and re-ran the command -- and it worked just fine (with aux.clj copied to the top of the tree).#2019-05-0700:17Alex Miller (Clojure team)are there two aux.clj files somehow?#2019-05-0700:17Alex Miller (Clojure team)such that you're hitting a locked file or something?#2019-05-0700:18Alex Miller (Clojure team)jars can contain two files of the same name or a file and a directory of the same name#2019-05-0700:20seancorfieldNope. Just the one aux.clj#2019-05-0700:22seancorfieldIt's really odd because the exception (no such file) comes from a copy operation for the target file -- which doesn't exist because that's the file you're trying to copy to#2019-05-0700:23seancorfieldand it succeeds in copying additional files to that same folder location after failing on this one file.#2019-05-0700:24seancorfieldAnd you can copy the file to some other location in the tree...#2019-05-0700:24Alex Miller (Clojure team)well, file systems are a new fangled technology, wouldn't expect them to be foolproof yet#2019-05-0700:24Alex Miller (Clojure team)if only they were statically typed, we wouldn't have these bugs{:tag :div, :attrs {:class "message-reaction", :title "rolling_on_the_floor_laughing"}, :content ({:tag :span, :attrs {:class "emoji"}, :content nil} " 8")}
#2019-05-0700:29seancorfielduncomplicate/neanderthal/internal/device/common.clj
uncomplicate/neanderthal/internal/device/cublas.clj
uncomplicate/neanderthal/internal/device/clblock.clj
uncomplicate/neanderthal/aux.clj
java.nio.file.NoSuchFileException C:\Users\sean\AppData\Local\Temp\uberjar11308132828377820906\uncomplicate\neanderthal\aux.clj
[u n c o m p l i c a t e / n e a n d e r t h a l / a u x . c l j]
[117 110 99 111 109 112 108 105 99 97 116 101 47 110 101 97 110 100 101 114 116 104 97 108 47 97 117 120 46 99 108 106]
java.nio.file.NoSuchFileException C:\Users\sean\AppData\Local\Temp\uberjar11308132828377820906\_random.txt -> C:\Users\sean\AppData\Local\Temp\uberjar11308132828377820906\uncomplicate\neanderthal\aux.clj
failed to copy back over
uncomplicate/neanderthal/block.clj
uncomplicate/neanderthal/linalg.clj
uncomplicate/neanderthal/real.clj
uncomplicate/neanderthal/core.clj
uncomplicate/neanderthal/opencl.clj
uncomplicate/neanderthal/internal/device/vect-math.cl
uncomplicate/neanderthal/internal/device/blas-plus.cl
It successfully copies files before and after. It can copy that one file to _random.txt happily but still can't copy it to the uncomplicate\neanderthal\aux.clj location after that.#2019-05-0700:44seancorfieldI can even copy it to uncomplicate/neanderthal/not.aux.clj ... is a filename of aux.* somehow magical on Windows?#2019-05-0700:53seancorfieldYes! aux, con, and prn are special filenames on Windows! https://superuser.com/questions/206423/windows-7-can-not-rename-a-file-to-aux-svg-the-specified-device-name-is-inva{:tag :div, :attrs {:class "message-reaction", :title "joy"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😂")} " 4")}
#2019-05-0700:54seancorfieldSo I bet that is what is happening here. Wow...#2019-05-0700:55seancorfieldPS C:\Users\sean\clojure\depstar-test> touch aux.clj
C:\Program Files (x86)\Gow\bin\touch.exe: cannot touch `aux.clj': No such file or directory
PS C:\Users\sean\clojure\depstar-test> touch con.clj
C:\Program Files (x86)\Gow\bin\touch.exe: setting times of `con.clj': Invalid argument
PS C:\Users\sean\clojure\depstar-test> touch prn.clj
C:\Program Files (x86)\Gow\bin\touch.exe: cannot touch `prn.clj': No such file or directory
PS C:\Users\sean\clojure\depstar-test> touch prn1.clj
PS C:\Users\sean\clojure\depstar-test>
:rolling_on_the_floor_laughing:{:tag :div, :attrs {:class "message-reaction", :title "disappointed"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("😞")} " 4")}
#2019-05-0707:25tianshu@dominicm @alexmiller thanks!#2019-05-0707:25tianshuvery clear#2019-05-0713:04sparkofreason@seancorfield @alexmiller Since I seem to be completely blocked on building on windows, any idea on my WSL issue above?#2019-05-0713:39mhuebertif one specifies a tools.deps version i deps.edn, is that version somehow used to resolve other deps?#2019-05-0713:41Alex Miller (Clojure team)no#2019-05-0713:42Alex Miller (Clojure team)there isn't any way "official" way to change the tools.deps version you're using#2019-05-0713:43Alex Miller (Clojure team)that said, you can edit your clojure script and modify the tools_cp to point to a different source version of tools.deps (and that's how I test a lot of stuff)#2019-05-0713:44Alex Miller (Clojure team)so instead of tools_cp="$install_dir/libexec/clojure-tools-1.10.0.442.jar", something like
tools_cp="/path/to/my/tools.deps.alpha/src/main/clojure:$install_dir/libexec/clojure-tools-1.10.0.442.jar"#2019-05-0713:45Alex Miller (Clojure team)eventually I'll provided an env var for that so you can just set the var rather than hack the script#2019-05-0713:50mhueberthmm. i’m configuring a circleci build which has some git deps that need the recent local/root changes, currently it installs clojure via the Manifest type not detected when finding deps for chia/reactive in coordinate #:local{:root "/home/circleci/chia/reactive"}
& there is nothing in the /home/circleci/ dir, except form the main “/project” working directory#2019-05-0714:06Alex Miller (Clojure team)yeah, so it can't detect what kind of project that is as there is no manifest file (deps.edn or pom.xml)#2019-05-0714:07mhuebertso chia is in /home/circleci/.gitlibs#2019-05-0714:08mhuebertthe clj -Spath command is executed in the ~/project/editor dir#2019-05-0714:08mhuebertwhich has a deps.edn pointing to the :git dep#2019-05-0714:09Alex Miller (Clojure team)both local deps and git deps try to detect their project type by looking for well known files at the project root#2019-05-0714:09mhuebertthe git dep is downloaded#2019-05-0714:10mhuebertit appears to be finding the root deps.edn at ~/.gitlibs/libs/chia/chia/da6ceb2989b712ab59c654679e1eb7b2e4c6ef02#2019-05-0714:11mhuebertor one of the sub-dirs.. have to check#2019-05-0714:11Alex Miller (Clojure team)the error you have above indicates something is referencing a local dep at /home/circleci/chia/reactive#2019-05-0714:12Alex Miller (Clojure team)that may be the resolved path of a relative dir in the referring deps.edn#2019-05-0714:13mhuebertthe root dep has chia/reactive {:local/root "reactive"}, and some sibling deps have chia/reactive {:local/root "../reactive"}#2019-05-0714:13mhueberti have no idea how it is navigating so far up#2019-05-0714:14Alex Miller (Clojure team)its based on the working directory#2019-05-0714:16mhueberthm i thought that with the recent changes, one could use relative paths within a git dep, and those would be resolved not from the current working directory, but relative to the path into the git dep where the relevant deps.edn is being read?#2019-05-0714:21Alex Miller (Clojure team)yes, should work that way. I don't have enough context to understand where things are going awry for you#2019-05-0714:21mhuebertok. i will keep digging, thank you for the pointers#2019-05-0714:22Alex Miller (Clojure team)clj -Stree might help, not sure#2019-05-0717:10dwhite44i have a project that is using a git reference to a private repo. everything works fine locally (mac) but when I try anything on our build server (CentOs), I get “Error building classpath. connector is not available:
com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available”. this happens just by running /usr/local/bin/clojure. If I comment out the dependency to that git repo, it works fine and brings in all the maven dependencies. I can clone that repo without issue so doesn’t seem like it would be a git access issue. any ideas?#2019-05-0717:57Alex Miller (Clojure team)it's trying to use the ssh agent, which does not seem to be running#2019-05-0717:59tanzoniteblack@dwhite44 a number of people in our company have had this issue since we switched to tools-deps. If you're on a mac, make sure you've explicitly run ssh-add -K in the terminal. If that still doesn't work, try removing your ssh config (i.e. mv ~/.ssh/config ~/.ssh/config.bak). Some of the people in our company have had something in their config that jgit chokes on. Never have figured out exactly what#2019-05-0718:00dwhite44ok. just not sure how the same repo would clone fine if the ssh agent isn’t running#2019-05-0718:02Alex Miller (Clojure team)there are several possible reasons due to the differences between how git finds credentials vs how jgit+jsch does it#2019-05-0718:02tanzoniteblack^^^ That has been a major source of headaches#2019-05-0718:04sparkofreasonI have ssh-agent running, added the key, no config file, same error.#2019-05-0718:13Alex Miller (Clojure team)yeah, Ghadi Shayban has done a full rewrite of tools.gitlibs that just shells out to git, which has different tradeoffs but at least generally "does what git does"#2019-05-0719:59seancorfieldI just pushed depstar "0.2.0" to clojars -- it's a rewrite to leverage the ZipFileSystem, as a way to get around Windows' reserved filename issues, which means a single copy pass so it's also much faster. Would love feedback from anyone already using depstar.#2019-05-0720:00seancorfield(although it now produces slightly different ownership/permissions on the resulting JAR file, I think because of the FileSystemProvider approach)#2019-05-0910:18kszabohey, how do you manage compilation of Clojure :gen-class files with tools.deps at development time? Do you call clojure.core/compile manually?#2019-05-0916:52dominicmI do, yeah#2019-05-0917:53seancorfield{:deps {org.clojure/clojure {:mvn/version "1.10.0"}}
:paths ["classes" "src"]
:aliases
{:compile {:main-opts ["-e" "(compile,'cfml.struct)"]}
:build {:extra-deps {seancorfield/depstar {:mvn/version "RELEASE"}}
:main-opts ["-e" "(compile,'cfml.struct)"
"-m" "hf.depstar.jar" "cfml-interop.jar"]}}}
is an example from one of my projects. I run it with clj -A:compile to get a REPL or clj -A:build to create a JAR for deployment. It duplicates the compile call in both. I could just have it in :compile but then I'd have to remember clj -A:compile:build whenever I wanted to build a JAR.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-05-1009:48kszaboThanks! We have a bit more complicated project with shadow-cljs + deps.edn with docker images. Since we already had a Makefile to run docker-based integration tests locally, we just added a new target to it, essentially doing what you are doing above with a manual clojure invocation.#2019-05-0914:51hlolliin leiningen there's a possibility of defining native paths with :native-path, that causes all files on this path to be spit'ed out into the current working directory, which suites well for jna/jni apps using .so files, I'm not seeing this behaviour in tools.deps, is it missing, not implemented or am I missing an argument?#2019-05-0915:44kszaboThis is implemented in badigeon:
https://github.com/EwenG/badigeon/blob/master/sample/badigeon/sample.clj#L121#2019-05-1015:02hlollivery nice, thanks!{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-05-0914:59Alex Miller (Clojure team)Not implemented#2019-05-0915:02hlolliprima 🙂 no stress, then I'll continue to advise to Overtone users to stick with leiningen until then#2019-05-0917:53seancorfield{:deps {org.clojure/clojure {:mvn/version "1.10.0"}}
:paths ["classes" "src"]
:aliases
{:compile {:main-opts ["-e" "(compile,'cfml.struct)"]}
:build {:extra-deps {seancorfield/depstar {:mvn/version "RELEASE"}}
:main-opts ["-e" "(compile,'cfml.struct)"
"-m" "hf.depstar.jar" "cfml-interop.jar"]}}}
is an example from one of my projects. I run it with clj -A:compile to get a REPL or clj -A:build to create a JAR for deployment. It duplicates the compile call in both. I could just have it in :compile but then I'd have to remember clj -A:compile:build whenever I wanted to build a JAR.{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-05-0918:32borkdudeis this kind of thing possible with tools.deps? https://stackoverflow.com/a/24949419/6264
I imagine it’s just a -e expression?#2019-05-0918:41seancorfield(! 1084)-> clj -e '(set! *assert* false)' -r
false
user=> (assert false)
nil
user=> (assert true)
nil
user=> ^D
Thu May 09 11:40:55
(sean)-(jobs:0)-(~/clojure)
(! 1085)-> clj -e '(set! *assert* true)' -r
true
user=> (assert false)
Execution error (AssertionError) at user/eval138 (REPL:1).
Assert failed: false
user=> (assert true)
nil
user=>
#2019-05-0918:41seancorfieldWhy "imagine" when you can just "repl"? 🙂#2019-05-0918:43borkdudetrue. I’m just wondering if there was a java property for it#2019-05-0918:43borkdudeI’ll just put an assert in my code and see if it fails, thanks#2019-05-0918:43seancorfieldThere's a JVM property for spec asserts.#2019-05-0918:44seancorfield(but not regular Clojure asserts I believe)#2019-05-0918:44borkdudeyeah, that’s why I imaged there would be one for regular asserts as well#2019-05-0918:48borkdudeworks 🙂#2019-05-0919:14borkdude:jvm-opts are only supported in aliases, not on the top level right? I don’t see this in the documentation and I remember asking this before{:tag :div, :attrs {:class "message-reaction", :title "heavy_check_mark"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("✔️")} " 4")}
#2019-05-0919:14ghadiright ^#2019-05-1006:59mpenetanother uberjar lib I discovered today : https://github.com/tonsky/uberdeps#2019-05-1010:22rickmoynihanCool. I’ve added it to the Tools wiki page here: https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging{:tag :div, :attrs {:class "message-reaction", :title "+1"}, :content ({:tag :span, :attrs {:class "emoji"}, :content ("👍")} " 4")}
#2019-05-1021:33timgilbertSlightly off-topic, but this new GitHub package repo thing looks potentially interesting for tools.deps enthusiasts, it supports Maven packages: https://help.github.com/en/articles/configuring-maven-for-use-with-github-package-registry#2019-05-1110:40luposlipHey @alexmiller - any chance this will get fixed soon?
https://dev.clojure.org/jira/browse/TDEPS-46#2019-05-1110:40luposlipAnother example of the issue:
https://github.com/luposlip/json-schema/issues/1#2019-05-1110:51Oliver GeorgeAre there any tips for debugging TransportException "Auth fail" errors? I swear I haven't changed anything and I'm suddenly getting this error. I can manually git clone the repo via ssh just fine.
Error building classpath. : Auth fail
org.eclipse.jgit.api.errors.TransportException: : Auth fail
Full stack trace here: https://gist.github.com/olivergeorge/17105466d16021269415ff66423a91ab#2019-05-1314:14borkdudeI’m running a project which calls the tools.deps API on Windows, but this crashes (clojure on Windows installed in PowerShell per wiki):
https://www.dropbox.com/s/1ouyqublibscp3x/Screenshot%202019-05-13%2016.12.23.png?dl=0
https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/reader.clj#L21#2019-05-1314:20borkdudeMore info here: https://github.com/taylorwood/clj.native-image/issues/6#2019-05-1314:37Alex Miller (Clojure team)can you file a jira on TDEPS?#2019-05-1314:51borkdudehttps://dev.clojure.org/jira/browse/TDEPS-128#2019-05-1315:50timgilbertYou might want to take a look at https://dev.clojure.org/jira/browse/TDEPS-120 @borkdude, I was seeing the same error message when my classpath was too long#2019-05-1315:52timgilbertHmm, actually looking more closely I think your issue is something different, never mind#2019-05-1413:47stathissideriswhen using a git dependency, do I also get access to the resources of the dependency or just the source?#2019-05-1414:40dominicm@stathissideris what's the distinction?#2019-05-1414:40dominicmIf its in :paths, you get it#2019-05-1414:41stathissideris@dominicm yeah, I just figured that out by experimenting 🙂 thanks!#2019-05-1514:32ChrisIs anyone aware of a tools-deps dependency visualizer? (i.e. the equivalent of https://github.com/walmartlabs/vizdeps)#2019-05-1516:32ChrisNevermind! It turned out to be relatively easy to write my own 🙂#2019-05-1517:54richiardiandreaA little rusty with Clojure on JVM...if I want to work against a Java code base, do I need to include taarget in order to import classes in the REPL using clojure?#2019-05-1517:55Alex Miller (Clojure team)yes, assuming target is the root of your compiled classes#2019-05-1517:56richiardiandreaok cool that's what I was missing then, I guess it goes without saying that I then need to compile the java code every time the file changes#2019-05-1517:59richiardiandreathank you Alex that works#2019-05-1520:18Jakub Holý (HolyJak)Hello, what do people use to watch for file changes to trigger actions (e.g. clojure.core/compile)? Lein certainly has a watch functionality/plugin but what about clj CLI? Thanks!#2019-05-1520:20Alex Miller (Clojure team)as above, clj doesn't cover compilation#2019-05-1520:20Alex Miller (Clojure team)it does cover "running programs"#2019-05-1520:20Alex Miller (Clojure team)the compiler is a program#2019-05-1520:20Alex Miller (Clojure team)watching is a program#2019-05-1520:21Alex Miller (Clojure team)compose programs, run with clj if you like#2019-05-1520:38Jakub Holý (HolyJak)thank you, Alex. My question is essentially: Does anyone made a clj-friendly plugin for watching files and executing actions upon change?#2019-05-1813:42orestisI like hawk for this. #2019-05-1520:40Alex Miller (Clojure team)I'm not aware of something specific to tools.deps, but there several watcher type things out there#2019-05-1520:40Alex Miller (Clojure team)last time I needed one, I just wrote the 60 lines of java interop to use the one in the jdk#2019-05-1520:41Alex Miller (Clojure team)admittedly, it's a weird java api#2019-05-1520:44Alex Miller (Clojure team)https://www.clojure-toolbox.com/ has a bunch of filesystem watcher libs#2019-05-1520:53souenzzo@holyjak I use (load "core") to load/reload files
But usually it's a "built-in" feature from your IDE/Plugin/Editor/..#2019-05-1521:07Jakub Holý (HolyJak)well, I need to watch for changes to run compile automatically...#2019-05-1521:08seancorfieldPersonally, I don't like watcher-based workflows. I usually find the watch fires when I don't want it to and/or gets into a bad state because it fires while there are still errors in the code.#2019-05-1521:09seancorfieldI'd much rather have an explicit compile command in a (comment ...) form that I can eval from my editor without even worrying about what the terminal is doing.#2019-05-1521:12seancorfield(also, why do you need to run compile at all? what features are you relying on that require Clojure code to be compiled?)#2019-05-1522:23Jakub Holý (HolyJak)https://github.com/holyjak/clj-concordion - integration with Java Specification by Example test framework that expects me to give it Fixture objects with test-specific methods. So you do `(deffixture "class.Name" [fns2expose as methods]) for each specification document.#2019-05-1522:24Jakub Holý (HolyJak)good points about the downsides of watchers, thank you!#2019-05-1522:25seancorfieldAh... so you're relying on code gen / gen-class rather than reify for this?#2019-05-1522:26Jakub Holý (HolyJak)exactly ,as there is no interface to reify#2019-05-1521:10Jakub Holý (HolyJak)Wied problem: while this works clj -J-javaagent:bin/springloaded-1.2.5.RELEASE.jar -J-noverify
this fails: ; deps.edn: [...]
:aliases {:agent {:jvm-opts ["-J-javaagent:bin/springloaded-1.2.5.RELEASE.jar" "-J-noverify"]}}
; CLI:
$ clj -O:agent
Unrecognized option: -J-javaagent:bin/springloaded-1.2.5.RELEASE.jar Why?#2019-05-1521:11seancorfield-J passes options through. They don't start with -J inside :jvm-opts#2019-05-1521:11seancorfieldYou want :aliases {:agent {:jvm-opts ["-javaagent:bin/springloaded-1.2.5.RELEASE.jar" "-noverify"]}}#2019-05-1521:12Jakub Holý (HolyJak)oh I see, please ignore 🙂 I have to omit -J#2019-05-1521:20Jakub Holý (HolyJak)Given a .clj file path, what is the simples way of finding out the namespace of the file? I could read the first line and check for #"\(ns (\w+)" but there is surely a cleaner way?#2019-05-2810:21rickmoynihanThe thing is, I need to merge edn files#2019-05-2810:22rickmoynihane.g. data_readers.edn but in particular duct_hierarchy.edn#2019-05-2811:03rickmoynihanok actually avoiding the conflicts and using a classloader would probably work… I guess I can just walk the resources and merge them myself.
Which of the uberjar options will use a classloader? I’m guessing capsule?#2019-05-2811:05rickmoynihanhmmm… infact I probably don’t need to merge them myself… duct must be doing this already to work at the repl.#2019-05-2811:07rickmoynihanindeed it does: https://github.com/duct-framework/core/blob/c5fb68ac3539248942c50cdbba5a2ca63735ee3c/src/duct/core.clj#L42-L44 I’d literally just written those same three lines.#2019-05-2811:17rickmoynihanok I can confirm pack.alpha capsule just works ™️ thanks#2019-05-2811:33rickmoynihanalso capsule is awesome 😀#2019-05-2812:54dominicmCapsule is very cool :)#2019-05-2812:55dominicmPack currently has a no conflict policy so that you don't have an option to shoot yourself in that respect.#2019-05-2813:54rickmoynihan:thumbsup: a good policy to have#2019-05-2813:56rickmoynihanwhat is the purpose of -e in pack.alpha? Why not just an alias with the extra-paths there? Or is it different somehow?#2019-05-2813:56rickmoynihanIs there any extra meaning in
> Add directory to classpath #2019-05-2814:23dominicm@rickmoynihan no extra meaning. That flag will continue to exist so that I can put directories generated with mktemp into the class path.#2019-05-2814:24rickmoynihanahh makes sense thanks#2019-05-2814:25dominicmThe "for building" is probably a bias in my writing for my particular use case.#2019-05-2814:27rickmoynihanyeah — but that use case makes sense 🙂 the docs could possibly benefit from a clarification as to that purpose… but no biggie 🙂#2019-05-2814:27rickmoynihanit is essentially the same as :extra-paths but more suited for scripting/api usage in wider build processes#2019-05-2814:36dominicmI'm terrible at writing documentation :)#2019-05-2817:33rickmoynihan@dominicm: incidentally it seems that pack.alpha doesn’t seem to honor :extra-paths in other aliases you supply during the pack phase. I’m guessing this is a bug? Though I could have my wiring wrong.#2019-05-2817:33dominicmHmmm, that would be a bug, yeah.#2019-05-2817:34dominicmI thought I tested that though#2019-05-2817:35rickmoynihanok I’ll open an issue for it — and also check it’s a real problem.
Issue filed here: https://github.com/juxt/pack.alpha/issues/36#2019-05-2818:05Daniel HinesReposting noob cli question from #beginners:
Is there a way to create a “default profile” for the clj cli? Such that whenever I run clj, certain deps are always merged in?#2019-05-2818:05Daniel Hineslike a ~/.deps.edn file or something?#2019-05-2818:11rickmoynihanyou can include custom (private) aliases in ~/.clojure/deps.edn but as far as I know you always have to opt in to aliases#2019-05-2818:11rickmoynihantools.deps prefers to be explicit#2019-05-2818:12didibusYes, put a deps.edn file here $HOME/.clojure#2019-05-2818:16didibusNah, you can just add normal deps#2019-05-2818:16didibusAnd they'll always be included#2019-05-2818:16didibusThat's how Clojure itself gets loaded#2019-05-2818:17Daniel HinesDumb question from a non-Java person: what is an “alias”? https://clojure.org/reference/deps_and_cli#_aliases doesn’t really explain that.#2019-05-2818:17Daniel HinesAlso, -S seems to be missing from that list of possible aliases.#2019-05-2818:17didibusThere are 3 deps.edn. The install one, the one in your home directory. And optionally, your project one. The :deps map gets merged from install to home to project. In case of conflicts, the later ones override the prior ones.#2019-05-2818:19didibusAliases are like dynamic modifications#2019-05-2818:20didibusNormally, the classpath, that is, the path to all libraries available to your program is defined by the :deps map, and the :paths vector, and a few others.#2019-05-2818:21didibusBut, maybe on occasions, you want to start a program or a repl with more or less libraries included#2019-05-2818:21didibusThat's what aliases let you do#2019-05-2818:21didibusSo you can define sets of dependencies and name them with an alias.#2019-05-2818:22didibusNow, from the command line, you can choose to include some of those additional sets by adding the alias for it#2019-05-2818:23Daniel HinesThat’s a very helpful explanation @didibus, thanks. So including aliases merges the options contained in those aliases?#2019-05-2818:23didibusYa#2019-05-2818:24Daniel HinesYou said it’s possible exclude as well? How would that be accomplished, since it’s a merge?#2019-05-2818:24didibusPretty much. Now the way things are "merged" can vary depending in the type of alias and config.#2019-05-2818:25didibusWell, you can't exactly exclude. So you'd be better off thinking of them more as inclusions#2019-05-2818:26didibusBut for example, you can specify :override-deps which will override versions#2019-05-2818:26didibusOr :classpath-overrides which overrides the path to a lib#2019-05-2818:28seancorfield@d4hines As an example of what you can do with ~/.clojure/deps.edn, here's mine https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#2019-05-2818:29seancorfieldThat has some usage notes as comments as well. Plus the README in that repo explains in more detail...#2019-05-2819:25Daniel HinesThis is super helpful, thanks!#2019-05-2818:33didibusI'm not sure if you could classpath-override to a null dir#2019-05-2818:33didibusOr a dir which doesn't have the dependency#2019-05-2818:33didibusAs a weird way to exclude#2019-05-2818:33didibusBut I wouldn't recommend it#2019-05-2818:34didibusBetter design it so aliases are additions or overrides, but not removals#2019-05-2818:39didibusAlso, I'm not sure what you mean by -S, that's just a prefix for other cli options. It's not related to aliases.#2019-05-2917:55rickmoynihanwhen using :git/deps does :deps/root need to specify a directory containing a deps.edn file or can it be a path to a deps.edn file named something arbitrary, like foo.edn?#2019-05-2917:55rickmoynihanI’m guessing it must be a directory#2019-05-2918:08dominicmYeah, directory#2019-05-2918:28rickmoynihanI’m assuming it’s not possible to override just the :deps/root for a :git/dep in aliases? Use case is sharing a single :git/url and :sha across multiple aliases that parameterise just the :deps/root.
I’m guessing that to do this I would need to duplicate the :sha and :git/url as :extra-deps in each alias.#2019-05-2919:15dominicmYeah, duplication#2019-05-2919:15dominicmI'd write a tool to manage it for me#2019-05-2919:35Jakub Holý (HolyJak)Is there a trick to not need to repeat the version number here: {:deps {org.apache.lucene/lucene-core {:mvn/version "8.1.1"}
org.apache.lucene/lucene-queryparser {:mvn/version "8.1.1"}
org.apache.lucene/lucene-analyzers-common {:mvn/version "8.1.1"}} ?#2019-05-2919:47seancorfieldNope.#2019-05-2919:47seancorfieldUnless there's a bundled artifact you could depend on that would bring in all those pieces.#2019-05-3009:42dm3How do I specify repository mirrors with deps.edn?#2019-05-3009:49rickmoynihanHow are people deploying artifacts to private s3 buckets with tools.deps? Are people using mvn deploy, or scripting up pomegranate aether with s3-wagon-private somehow?#2019-05-3009:56rickmoynihanpomegrante’s aether deploy looks like it might be easy enough… though not sure how to wire in the s3 wagon…#2019-05-3009:57rickmoynihanIt looks like it might be automagically done#2019-05-3010:32rickmoynihanok it looks like it’s pretty easy to get pomegranate deploy with s3 wagon working in a simple clj script#2019-05-3011:00dominicm@rickmoynihan write that down somewhere for others!#2019-05-3011:04rickmoynihanWill do. It looks like it basically just works. I’m sure it’d be easily extractable into an aliasable task, though I haven’t thought too much about how it should work in that form.#2019-05-3011:05rickmoynihanRight now I essentially want to programmatically generate a bunch of very similar artifacts that differ only in terms of one or two files and their artifact id.#2019-05-3011:08rickmoynihanThe higher level problem I’m looking to solve is we have a repo that contains our own css framework, and we want to package client themes from a single place but consume them downstream as essentially webjars. So each clients theme will typically differ only by their theme.css#2019-05-3011:09rickmoynihanso I’ll likely solve that problem first, then look at extracting something reusable.#2019-05-3019:55mnewtIs there a way to specify a dependency for a leiningen project? (i.e. project.clj is the only manifest)#2019-05-3020:41Frank HenardIs this what you're looking for: https://github.com/RickMoynihan/lein-tools-deps ?#2019-05-3020:49mnewt@ballpark Thanks for that. That is useful but unfortunately not quite what I’m looking for. I was hoping that there was a way to pull in an existing leiningen project without modifying it. Is there a library or something that can resolve dependencies from a project.clj file or is the only way to do that to run leiningen itself? My use case is I would like to include specific versions of projects directly from github.#2019-05-3021:10seancorfield@mnewt Petition the maintainer to add deps.edn 🙂 You could always override the deduction, specify the manifest as :deps and then provide the transitive dependencies yourself in your deps.edn file.#2019-05-3021:12mnewt@seancorfield Thanks, that makes sense#2019-05-3112:48dharriganIf I have a configuration in my $HOME/.clojure/deps.edn file, thus: {:extra-deps: {foo-bar {:local/root "libs/foo}}}, it doesn't appear that resolution happens relative to the $HOME/.clojure" directory, I have to put the full path in, i.e., ....{:local/root "/home/foo/.clojure/libs/foo"#2019-05-3112:50dharriganIs my thinking correct on this? No way for the resolution to happen relative to where the deps.edn file is?#2019-06-0312:41rickmoynihanI think that’s correct, as things will get more complex when you have transitive git deps etc.#2019-06-0312:54dharriganthanks for the confirmation 🙂#2019-05-3122:15sparkofreasonI still have this exception running clojure from WSL. I've confirmed that ssh-agent is running and configured. com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available: ssh-agent:jna,ssh-agent:nc Does anybody know the solution?#2019-05-3123:06hiredmanYou need some native code to talk over Unix sockects on the jvm, which is how you interact with ssh-agent. The jsch library might not be able figure out the native code required for wsl#2019-06-0100:28sparkofreasonI assume people have this working, and there's just something about my particular WSL install (pengwin) that is an issue.#2019-06-0102:06TravisDoes anyone know if there is a way to override where tools-deps looks for the .m2 directory ? For instance with Maven you can do something like
MAVEN_OPTS=-Dmaven.repo.local=/cache/.m2
#2019-06-0111:01Alex Miller (Clojure team)You can set :mvn/local-repo at the root in deps.edn#2019-06-0222:17Travis@U064X3EF3 is there a way to set this at the command line for override in some build env?#2019-06-0222:22Alex Miller (Clojure team)You can use -Sdeps with an in-line deps.edn to add it on the command line#2019-06-0111:08pezHello. I am trying to get the Calva jack-in story to be told nicely on Windows. Can I assume that powershell is used for cli/deps?#2019-06-0111:09pezAlso I am a bit confused about when the command line tool is named clj and when it is named clojure.#2019-06-0112:19sogaiu@pez in *nix and macos isn't one just a wrapper to use the other along with rlwrap? have a look inside the files?#2019-06-0112:23pezYes, checked the contents on my Mac now. clj wraps clojure in rlwrap. So it is clojure I should use for jack-in then. Thanks!#2019-06-0117:49yuhanI tried unsuccessfully to install the cli on Windows a while back too, to my understanding the official guide at https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows only installs a PowerShell module in ~/Documents#2019-06-0117:49yuhanbut I could not figure out how to get from that to a clj / clojure executable on the PATH#2019-06-0317:11rickmoynihanI’m running a clojure CLI tool inside travis, and one of the deps is in a private github repo; and it dies with an error like:
Error building classpath.
The build itself is itself inside a private repo too, and travis seems to be able to clone that ok.
Any ideas what I might need to do?#2019-06-0317:16dominicm@rickmoynihan I would guess ssh-add or similar#2019-06-0317:16rickmoynihanyeah#2019-06-0317:16dominicmIs an agent running?#2019-06-0317:17rickmoynihanI would’ve expected travis to already have that setup though, given that one of the first things it does is clone a private GH repo#2019-06-0317:17dominicmYeah. I wouldn't be certain it's using an agent#2019-06-0317:17rickmoynihanfair#2019-06-0317:18dominicmThe other thing is, are you using gitlab?#2019-06-0317:18rickmoynihangithub#2019-06-0317:36dominicmOkay, then the other bug I'm thinking of doesn't apply#2019-06-0317:48Alex Miller (Clojure team)clj uses jgit/jsch (not git) so it's definitely possible to have git work on the initial clone and then tools.deps fail to clone a dep repo#2019-06-0317:50Alex Miller (Clojure team)there are a number of deficiencies in the jgit/jsch stuff (we're on a somewhat old version now too). definitely needs some attention and we're considering either using a newer version or calling out to git or possibly making both available#2019-06-0320:16Alex WhittDoes tools.deps support loading native .so libs via JNI? With leiningen, I had:
:resource-paths ["lib"]
:native-path "%s/lib"
:jvm-opts [~(str "-Djava.library.path=lib/:" (System/getenv "$LD_LIBRARY_PATH"))]
And that did the trick. I've tried a few different things but can't get this working with tools.deps (I get an UnsatisfiedLinkError.)#2019-06-0320:17Alex Miller (Clojure team)no, this is not yet supported#2019-06-0320:18Alex Miller (Clojure team)tools.deps can add paths to the classpath and use jvm opts, but it doesn't have support for unpacking or loading native libs#2019-06-0320:19Alex WhittDang, alright. Thanks.#2019-06-0320:29dominicm@alex.joseph.whitt what lib is this? Most of the ones I've seen unpack themselves.#2019-06-0320:31Alex Whittit's built from this: https://github.com/entropia/libsocket-can-java/blob/master/jni/cansocket.cpp#2019-06-0320:33Alex Miller (Clojure team)lein has a thing I think that unpacks native libs#2019-06-0320:33Alex Miller (Clojure team)I don't know the details#2019-06-0320:36Alex WhittFor now I'll probably try to follow this approach: https://stackoverflow.com/a/28012737/2015090#2019-06-0320:37Alex WhittIt would be real nice if this was just built in#2019-06-0322:01jeroenvandijk@rickmoynihan Almost bedtime, but you can add a user's git keys with circleci and this allows you to use private libs (assuming they are accessible with those keys). I can have a look tomorrow if necessary#2019-06-0409:55martinklepschcurl -O
clj -Sdeps '{:deps {org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu #:local{:root "mxnet-full_2.11-linux-x86_64-cpu-1.4.1.jar"}}}'
Error building classpath. 1 problem was encountered while building the effective model for org.apache.mxnet:mxnet-core:[unknown-version]
[FATAL] Non-resolvable parent POM for org.apache.mxnet:mxnet-core:[unknown-version]: Could not find artifact org.apache.mxnet:mxnet-parent:pom:INTERNAL in central () @
Is this supported and a bug or am I doing something wrong here?#2019-06-0411:17Alex Miller (Clojure team)You might need to add a maven repo if that parent pom isn’t in maven central#2019-06-0414:51martinklepschThe weird thing is that this works:
clj -Sdeps '{:deps {org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu {:mvn/version "1.4.1"}}}'
#2019-06-0414:53Alex Miller (Clojure team)we do have a known issue that repos defined in the pom don't get used and that may be the issue here#2019-06-0414:54martinklepschbut then the above should also not work, right?#2019-06-0414:56Alex Miller (Clojure team)the last example there is a different path and relies on the maven repo to resolve artifacts (https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-46 is the issue I'm thinking of)#2019-06-0414:56Alex Miller (Clojure team)when you load a jar dep, tools.deps is reading the model and resolving the deps#2019-06-0414:56Alex Miller (Clojure team)I think it's in the tools.deps pom reader where more needs to be done#2019-06-1009:11martinklepschI'm not sure if TDEPS-46 s whats going on here. Just let me know if you want a separate JIRA to document this failure case#2019-06-1012:40Alex Miller (Clojure team)you could leave a comment on that ticket so it can be checked and spun out if needed#2019-06-1016:14martinklepschIt seems I can no longer comment: could you give permission to <mailto:/cdn-cgi/l/email-protection|/cdn-cgi/l/email-protection>#2019-06-0411:30dm3are repository mirrors supported?#2019-06-0412:13Alex Miller (Clojure team)I don't think so? I think that's still a todo#2019-06-0412:17Alex Miller (Clojure team)yeah, that's not supported atm and I don't actually see a ticket for it, although that would be fine if you wanted to file one https://clojure.atlassian.net/servicedesk/customer/portal/1#2019-06-0415:38Alex Whitt@dm3 I've got repository mirroring working, if we're talking about the same thing. My deps.edn config includes:
:mvn/repos {"central" {:url ""}}
And my ~/.m2/settings.xml:
https://gist.github.com/WhittlesJr/3422e3374c0393188c1321bbd7605bd5
We're using Nexus 3 for our maven server ("my-mirror"). Caches everything and works like a charm.#2019-06-0415:42Alex Whitt(I had to use http because I couldn't add a self-signed certificate to the java trust store in my particular case)#2019-06-0415:44Alex WhittI suppose that having all that config in settings.xml breaks repeatability though, so a pure tools.deps solution would be very nice indeed.#2019-06-0415:45Alex Whitt(Er, maybe it is repeatable after it's all cached? Perhaps you could drop the settings.xml after that point and it would still download what it needed from "my-mirror." I'm not sure.)#2019-06-0415:51Alex Miller (Clojure team)well that's not mirroring, that's just replacing the repo#2019-06-0415:51Alex Miller (Clojure team)that should work#2019-06-0415:52Alex Miller (Clojure team)maven actually has support for mirrors (or maybe the term is different, I can't remember)#2019-06-0415:52Alex Miller (Clojure team)mirrors being duplicate repos that can serve the same content#2019-06-0416:02Alex WhittYeah I think I've seen the word "mirror" used for both meanings so I'm not sure what term to use. It is at least called "mirrors" in settings.xml (the <mirrors> section). On first request, Nexus will download and cache the needed artifacts from maven central or clojars, and then the client downloads everything from Nexus. Is that different from what you're talking about?#2019-06-0416:03Alex WhittI think I've also heard it called "proxying"#2019-06-0416:05Alex WhittBut that's also confusing because "proxying" usually refers to using an http proxy#2019-06-0416:13Alex WhittMaybe what I'm doing is just one species of mirroring: https://maven.apache.org/guides/mini/guide-mirror-settings.html
(third bullet point)#2019-06-0419:06dm3yeah, you’re overriding the repo#2019-06-0419:06dm3mirror is another layer#2019-06-0513:40sveri@alexmiller Hey, I just came back from a longer break and noticed tools-deps is finally available on windows, which is pretty awesome, thanks 🙂#2019-06-0513:50Alex Miller (Clojure team)give it a whirl#2019-06-0513:55sveriYea, I just did and after I allowed myself to run unsigned scripts in the powershell installation worked and I can run it#2019-06-0513:55sveriJust about to experiment with it finally#2019-06-0517:44markxHi all, why is sha required for a :git/url? Is it possible to automatically use master branch?#2019-06-0517:45hiredmanyou can't cache a reference#2019-06-0517:46hiredmanclj startup performance is based building a classpath once and caching it for the same deps.edn file#2019-06-0517:50markxAh makes sense. It doesn’t have a record of the local version.
I was just hoping to achieve some function like npm or pip or other package mangers.#2019-06-0517:52seancorfield@markx If you really want to be able to add libraries from master, you can do something like this https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L106-L119#2019-06-0517:56markxI still need to wire my brain to connect the repl and the cli tool. I understand the code but don’t know how to do it on cli.#2019-06-0517:58markxWhat I really really want is something like clj install xxx then I can just run xxx from my shell.#2019-06-0517:58markxWhat I really don’t like is to build or download a jar and manually put it into PATH.#2019-06-0517:59lilactownhey markx. a thing I'm building might help#2019-06-0517:59lilactownis this for installing libraries, or installing applications?#2019-06-0517:59markxIdeally for both, but mostly for applications.#2019-06-0518:02lilactownhttps://laughing-banach-af1115.netlify.com/#2019-06-0518:03lilactownit doesn't have direct support for installing global aliases right now. but it does have a plum add command that adds a dependency to a local deps.edn#2019-06-0518:12dominicm@U4YGF4NGM do you have a plan to preserve whitespace when you do this?#2019-06-0518:13lilactownit should preserve whitespace#2019-06-0518:13lilactownwell, it formats it using cljfmt#2019-06-0518:14lilactownwhich I suppose is not the same 😅#2019-06-0518:14dominicmI would not use this, because that will lose comments.#2019-06-0518:14lilactownno, it will preserve comments. that is why I moved to using cljfmt / rewrite-clj in the first place#2019-06-0518:15lilactowngo ahead and give it a shot#2019-06-0518:15dominicmOh, you're using rewrite-clj directly too :)#2019-06-0518:15dominicmGood. That's what I was going to suggest if you didn't have an idea.#2019-06-0518:15dominicmBut I was hoping you'd found a novel approach I didn't think of :)#2019-06-0518:15dominicmI'll look through the code to compare.#2019-06-0518:16lilactownyep 😄 cljfmt is there for formatting, and then rewrite-clj for actually doing the addition:
https://github.com/Lokeh/plum/blob/master/clj/src/plum/add.clj#L56#2019-06-0518:17dominicmI had some users get quite annoyed that they lost their carefully aligned arguments when I first added this, just fyi#2019-06-0518:18lilactownthat makes sense. I wish there was a way to do preserving modifications of EDN better#2019-06-0518:19dominicmI spent some time on it in ednup#2019-06-0518:19dominicmThe goal was to make a clojure map you could assoc without losing formatting#2019-06-0518:20lilactownI wonder if we could format the modification and then assoc that using rewrite-clj to not mess with the rest of the file?#2019-06-0518:57dominicmWhat I did in ednup was look at the previous key, and match the whitespace based on that#2019-06-0518:11markxThis is a nice tool. I read about it on clojureverse.
IIUC, it provides cli commands to avoid manually editing the deps.edn file. But it doesn’t solve all my pains. I hope it can go one step further.
Do you plan to support installing global aliases?#2019-06-0518:12lilactownit could be added. if it would help you, create an issue on github.#2019-06-0518:13lilactownI would also look over a PR if you're interested in contributing 😄#2019-06-0518:40markxYeah I’m interested. I’ll try. But don’t wait for me. 😛#2019-06-0607:07sveriHow do I start a repl in a given namespace? I have not seen anything in the documentation about it.#2019-06-0608:21sveriI figured I could use something like this:
:aliases
{:cursive {:main-opts ["-e" "(load \"de/sveri/getless/user\")"
"-e" "(de.sveri.getless.user/reset)"]}}
It works, but is it the right way to use tools-deps?#2019-06-0612:10Alex Miller (Clojure team)You would need to do something like this#2019-06-0612:11Alex Miller (Clojure team)You could simplify to one expr by using requiring-resolve (added in 1.10)#2019-06-0612:13Alex Miller (Clojure team)((requiring-resolve ‘de.sveri.getless.user/reset))#2019-06-0612:32sveri@alexmiller Thanks, that works and looks better#2019-06-0612:41Alex Miller (Clojure team)although double parens always looks weird#2019-06-0622:38Jakub Holý (HolyJak)Is :local/root supposed to work for pointing to a local lein project (with a pom.xml)? Or only deps projects? Thanks! (It doesn't for me)#2019-06-0622:58Alex Miller (Clojure team)it should work for deps.edn and pom.xml#2019-06-0622:58Alex Miller (Clojure team)in what way does it not work?#2019-06-0709:44Jakub Holý (HolyJak)E.g. REPL does not see the code from that project: $ clj -A:test
Clojure 1.10.1-beta2
user=> (require '[clj-concordion.core :as cc])
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate clj_concordion/core__init.class, clj_concordion/core.clj or clj_concordion/core.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.
with deps.edn including {:aliases
{:test
{:extra-deps {clj-concordion {:local/root "/Users/me/Projects/clj-concordion"}
;clj-concordion {:local/root "/Users/me/Projects/clj-concordion/target/clj-concordion-0.0.5.jar"} (the .jar works) and $ head ~/Projects/clj-concordion/pom.xml
<?xml ...><project ...>
<modelVersion>4.0.0</modelVersion>
<groupId>clj-concordion</groupId>
<artifactId>clj-concordion</artifactId>
...
<build>
<sourceDirectory>src</sourceDirectory>
...
and $ head -n1 ~/Projects/clj-concordion/src/clj_concordion/core.clj
(ns clj-concordion.core
(Could there be a problem with using src/ instead of src/main/clojure/?)#2019-06-0716:05mssis there a function that pretty prints a deps.edn dependency tree a la lein deps :tree? are people just using maven to accomplish that?#2019-06-0716:06zaneYou mean like clojure -Stree?#2019-06-0716:06mssexactly what I was looking for! thanks for the help#2019-06-0716:07zaneYou're welcome!#2019-06-0717:34NolanI’ve got a java class on the classpath that looks like this:
public final class com.whatever.Main {
public static void main(java.lang.String[]);
}
Is there any way to use :main-opts to invoke com.whatever.Main.main()? I tried:
:aliases
{:some-alias
{:main-opts
["-m" "com.whatever.Main"]}}
but that looks for com/whatever/Main__init.class. I realize I’m probably stretching how aliases are meant to be used, but it would be mega convenient to be able to run this using the cli.#2019-06-0717:48zaneYou could create a Clojure namespace with a -main that just delegates to that Java method, maybe? :thinking_face:#2019-06-0717:51Nolanyeah, that’s probably the appropriate way to go about it.#2019-06-0717:54Alex Miller (Clojure team)clj is a Clojure launcher, not a general purpose Java launcher#2019-06-0717:57NolanI subsequently realized how silly it was to be trying to use clj before there was a clojure component. Everything fell into place once I realized that the whole problem went away as soon as I started writing what I initially set out to write.#2019-06-0822:47kommenran into a an unexpected situation#2019-06-0822:48kommenthis is ok:#2019-06-0822:48kommen$ clj -Sdeps '{:deps {org.clojure/clojurescript {:git/url "" :tag "r1.10.520" :sha "0c353f1947089cb8b1f010b4294b94ac109d4ef6"} org.clojure/tools.reader {:mvn/version "1.2.2"}}}'
Checking out: at 0c353f1947089cb8b1f010b4294b94ac109d4ef6
Clojure 1.10.0
user=> (require 'clojure.tools.reader)
nil
user=> (doc clojure.tools.reader/read+string)
nil
user=> #2019-06-0822:48kommenbut this should not work, as tools.reader 1.2.2 has no read+string:#2019-06-0822:49kommen$ clj -Sdeps '{:deps {org.clojure/clojurescript {:mvn/version "1.10.520"} org.clojure/tools.reader {:mvn/version "1.2.2"}}}'
Clojure 1.10.0
user=> (require 'clojure.tools.reader)
nil
user=> (doc clojure.tools.reader/read+string)
-------------------------
clojure.tools.reader/read+string
([] [reader & args])
Like read, and taking the same args. reader must be a SourceLoggingPushbackReader.
Returns a vector containing the object read and the (whitespace-trimmed) string read.
nil
user=>
#2019-06-0822:49kommenturns out clojurescript is an uberjar which contains tools.reader 1.3.0, which takes precedence over the specified version in deps.edn#2019-06-0822:50kommenis that to be expected?#2019-06-0823:12Alex Miller (Clojure team)Yes, I think so#2019-06-0823:13Alex Miller (Clojure team)They vendor a couple things iirc, but asking in #cljs-dev would be better#2019-06-0906:45dominicmI think that clojurescript vendors an AOT version of t.r, which takes priority.#2019-06-1009:07martinklepschI've found another potential bug/issue, let me know if it makes sense to file an issue.
With the code below we'll end up with two versions of Clojure on the classpath:
(tdeps/resolve-deps {:deps {'org.clojure/clojure {:mvn/version "1.9.0"}}
:mvn/repos default-repos}
{:extra-deps {"org.clojure/clojure" {:local/root "/Users/martinklepsch/.m2/repository/org/clojure/clojure/1.10.0-beta2/clojure-1.10.0-beta2.jar"}}})
It can be easily fixed by changing the "org.clojure/clojure" to be a symbol but I would expect tools deps to handle that(?)#2019-06-1012:46Alex Miller (Clojure team)you can file a ticket for that#2019-06-1014:42dominicmFeels like strings in that position should blow up#2019-06-1014:44dominicmHowever, strings would make a pretty neat reference to the canonical upstream for git repos, even when you're using a separate branch#2019-06-1015:09Alex Miller (Clojure team)it should be an error, libs are symbols#2019-06-1214:12Lone Rangerdoes anybody happen to know if there's a lein deps :tree equivalent for the command line tool?#2019-06-1214:15bronsaclj -Stree#2019-06-1214:24Lone Rangeryou're awesome, thanks#2019-06-1214:25Lone Rangerdoes this also show conflicts? My datomic-pro seems to be cranky and I'm trying to track down why#2019-06-1214:26Alex Miller (Clojure team)it doesn't - you may want to post the message in #datomic#2019-06-1214:26Alex Miller (Clojure team)there are some common cases and someone might just know hte answer#2019-06-1214:27Lone Rangeraha. Thank ya sir#2019-06-1215:56Lone Rangeralright question... in deps.edn is there a way to specify username/password for servers? i.e., without doing this in ~/.m2/settings.xml:
<servers>
<server>
<id>my.datomic.com</id>
<username>username</username>
<password>password</password>
</server>
</servers>
#2019-06-1216:00Lone Rangeror perhaps another way to say this is, if I have :mvn/repos {"" {:url ""}} in my deps.edn and the ☝️ snippet in ~/.m2/settings.xml, and yet I'm still getting a 401 on my download for com.datomic/datomic-pro {:mvn/version "0.9.5786"} ... is there another way to accomplish the same thing or am I just doing it wrong?#2019-06-1216:01Alex Miller (Clojure team)that is the right way to go about it, so something must not be lined up correctly#2019-06-1216:06Lone Rangerokay, I'm stupid#2019-06-1216:07Lone RangerI didn't realize in the examples that was just a snippet of settings.xml#2019-06-1216:08Lone Rangerneed the full thing... working now#2019-06-1216:10Alex Miller (Clojure team)cool, it's a balance on how much to put without it being overwhelming in those docs#2019-06-1216:21dominicmI saw something really neat recently where you could expand an example into the fully working program, this would be a good use case for that#2019-06-1217:26sveriYea, or a link to a full code example on github.#2019-06-1216:14Lone Rangerindeed#2019-06-1319:53jmvis it possible to set the location of downloaded artifacts? the CI we use treats each step in the build pipeline as a clean slate and so it re-downloads all of the dependencies for each step.#2019-06-1402:31Alex Miller (Clojure team)@jmv305 yes, set the :mvn/local-repo attribute at the root of deps.edn (or in -Sdeps on command line) - takes a string that is a path to the root#2019-06-1414:51jmvthank you alex, that worked!#2019-06-1402:32Alex Miller (Clojure team)clj -Sdeps '{:mvn/local-repo "/right/here"}'#2019-06-1408:14borkdudeI’m getting an error when I put clojure 1.10.1 in my extra-deps. I wanted the dependency to be there in my maven repo during test time, since I’m analyzing the source code.
;; org.clojure/clojure {:mvn/version "1.10.1"}
#2019-06-1408:15borkdudeThe error being:
Running tests in #{"test"}
Syntax error (ClassNotFoundException) compiling at (clj_kondo/core_test.clj:1:1).
clojure.core$with_meta__5405
#2019-06-1408:16borkdudeI didn’t want to declare that clojure version at my top level, since I don’t want to force that version on other people#2019-06-1408:17borkdudetrying a repro#2019-06-1408:22borkdudeI can force the download in another way, never mind#2019-06-1409:43Jakub Holý (HolyJak)Does anyone know how to publish a deps.edn-based library to Clojars? Its docs do not mention it https://github.com/clojars/clojars-web/wiki/Pushing#2019-06-1409:51delaguardoI'm using maven for that. clj -Spom and then mvn deploy.#2019-06-1409:52delaguardoSo you probably need to read through https://github.com/clojars/clojars-web/wiki/Pushing#maven#2019-06-1416:01seancorfieldYup that's how I do it too. With depstar to create the JAR. #2019-06-1416:03seancorfieldThe pom generated by clj is minimal -- you'll need to add a bunch of stuff and edit things. See the pom.xml file in the depstar repo. Without all the extra stuff Clojars won't know how to refer back to your source repo and you won't be able to import your library on http://cljdoc.org either. #2019-06-1416:03seancorfieldnext.jdbc is also published this way. #2019-06-1416:09borkdudethis is good to know. I currently still mirror my deps in a project.clj and then call lein deploy#2019-06-1416:29jmvdo you update the version in the pom by hand?#2019-06-1416:31borkdudegood question. I wonder where the version comes from since that’s not present in deps.edn?#2019-06-1416:36jmvhttps://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/gen/pom.clj#L76#2019-06-1417:28seancorfieldYes, I update the pom.xml <version> tag manually. But the the SCM <tag> is updated by my deploy shell script when I make a release.#2019-06-1417:28seancorfieldSo my process the first time is clj -Spom and add all the SCM stuff etc into the skeletal pom.xml (by copying from another project and editing all the URLs etc).#2019-06-1417:31seancorfieldSubsequent times: do the edits to prepare for the release, push to GH, create a new release on GH with release notes, pull (to ensure I have the latest master locally), then clj -Spom && clj -A:jar project.jar && git status (to verify the pom.xml didn’t change again). Then I usually re-run the test suite at this point. Then deploy project, which updates the pom.xml from the HEAD SHA, commits & pushes it, then uses mvn deploy to push it up to http://clojars.org#2019-06-1417:32seancorfieldThis is how I update the pom.xml in my deploy script sha=`git rev-parse HEAD`
version=`fgrep '<version>' pom.xml | head -1 | sed 's;.*<version>\(.*\)</version>.*;\1;'`
echo "Setting tag to ${sha} for version ${version}..."
cp pom.xml /tmp/${1}.pom.xml
sed "s;<tag>[0-9a-f]*</tag>;<tag>${sha}</tag>;" < /tmp/${1}.pom.xml > pom.xml
#2019-06-1417:32seancorfieldThis is the deploy line: mvn deploy:deploy-file -Dfile=${1}.jar -DpomFile=pom.xml -DrepositoryId=clojars -Durl=
#2019-06-1417:40jmvthanks sean, this is helpful to see#2019-06-1417:43seancorfieldThe automated commit looks like this https://github.com/seancorfield/next-jdbc/commit/43f74637996ffeaef51bb1965374c784c9ec5b28 but the actual release — whatever was tagged by GH — is what Clojars points to, e.g., https://github.com/seancorfield/next-jdbc/commit/9280b9ab4753a62e14b564df0797c124af14e4ad (which is one before that).#2019-06-1417:44seancorfieldHowever, it’s the <tag> in the pom.xml that drives http://cljdoc.org when it fetches the project from GH to analyze — and that means it gets the deployed version (which is one commit ahead of the release). Just something to bear in mind.#2019-06-1417:45jmvright that makes sense#2019-06-1509:06Jakub Holý (HolyJak)Thanks!#2019-06-2608:43dominicmAn example that doesn't use mvn: https://github.com/juxt/pack.alpha#uploading-to-clojars-or-maven#2019-06-1511:20sogaiuis anyone else using m4 or something similar to generate their deps.edn files?#2019-06-1515:27gerred👋 if I'm working on a dependency from a local source, is there any way to re-load that dependency from the repl?#2019-06-1518:02orestis@gerred if you have a local source, can’t you just reevaluate the code in the REPL?#2019-06-1716:28gerredlocal source, but different repo. for example, working on an upstream lib but want to see the effects on whatever my downstream project is. 🙂#2019-06-1716:28gerredotherwise, totally#2019-06-1716:28gerred(say, adding a feature to someone's upstream before I PR it)#2019-06-1717:28orestisShouldn’t matter which repo the code is from, as long as your editor picks up the correct namespace (most do)#2019-06-1610:45borkdude@gerred yeah, you can do (require '[my.lib] :reload)#2019-06-1611:11minhnnhow to auto reload new lib into repl when I add It to my deps.edn, It depend on IDE or I must do itt manualy with deps/resolve-deps and deps/make-classpath#2019-06-1611:22borkdude@minhnhat10bk there is add-lib in tools.deps, google for it#2019-06-1611:43minhnnI have check but It seem like no add-lib in clojure.tools.deps.alpha#2019-06-1611:46sogaiudid you check the add-lib branch? https://github.com/clojure/tools.deps.alpha/tree/add-lib#2019-06-1611:46sogaiuhttp://insideclojure.org/2018/05/04/add-lib/#2019-06-1716:21TravisHey guys, I might be missing it in the docs but is there away with tools-deps to include java src in the project ?#2019-06-1716:52ghaditools.deps doesn't compile Java @camechis#2019-06-1716:54Travisthanks, thats what I figured#2019-06-1717:12souenzzoclj idea is awesome. it should be available in every language.#2019-06-1812:41Lone RangerIs there a way to download the dependencies without running the app? (use case, building Docker image)#2019-06-1812:44Lone Rangerclj -i?#2019-06-1812:58Lone Rangerahh looks like clj -version will do the trick#2019-06-1812:59kirill.salykinclj -Stree?#2019-06-1813:01Lone Rangeryeah that's better ... clj -version isn't a thing#2019-06-1813:01Lone RangerI was doing clj -e "1" but that felt stupid#2019-06-1813:26ghadiclojure -Spath > /dev/null#2019-06-1912:01denik@alexmiller just read your post on add-lib (http://insideclojure.org/2018/05/04/add-lib/). I'm working on a project where this would be very useful. What's the status?#2019-06-1912:12vlaaadI heard some rumors Rich is thinking whether it should be in clojure core instead of tools-deps 😱#2019-06-1912:14vlaaadI wish it is more official then a separate branch because it removes SO MUCH friction#2019-06-1912:36Alex Miller (Clojure team)We’ll get back to it at some point. Some part of it may migrate into core, hopefully that would be for 1.11#2019-06-2001:36rgmCan I pull in new code from a :local/root dependency without restarting the repl? I’m trying to extract a lib and want to have my test runner on watch mode in the main project.#2019-06-2001:37rgmer, I guess now that I think about it that’s not quite the repl. I guess I mean is there something related to add-lib that I could run from the repl or a test namespace that can reload a known :local/root dep#2019-06-2003:29seancorfield@rgm Is that code you already have on the classpath, and it just got updated? Or is this a genuine addition of new code/dependencies?#2019-06-2003:56rgmIt’s already on the class path. The plot twist is that there’s a fair bit of cljc and cljs.#2019-06-2003:58rgmLet’s imagine I have editor buffers open for both the main proj, and namespaces in the lib I’m extracting, nrepl’d or prepl’d in on the editor, and if I can avoid restarting the REPL jvm to get new lib code that’d be amazing and happy.#2019-06-2004:00seancorfieldOn the JVM, just eval code as you change it. No need for any reloading there. On the browser (cljs) side, that's what Figwheel/Shadow's hot reloading is all about. Or am I missing something @rgm?#2019-06-2004:01rgmOh, you’re probably not. The idea that it might just work had genuinely not occurred to me.#2019-06-2004:01seancorfield(I keep a JVM REPL up and running for days, sometimes weeks, and never use a watcher or reloader process)#2019-06-2004:01rgmI didn’t imagine that figwheel would observe outside the project directory.#2019-06-2004:03seancorfieldHmm, if you're editing code outside the project as well, I don't know how Figwheel et al would cope. I was just reading about some editor/REPL setup tho' where you can have both JVM and browser REPLs running and when you eval clj it goes to the JVM, cljs goes to the browser, and cljc goes to both.#2019-06-2004:04seancorfieldSo with that workflow, you'd still follow a good REPL-Driven Development workflow of edit-eval, edit-eval, without even needing to save your files.#2019-06-2004:04seancorfield(I work for fairly long times across multiple files without saving sometimes)#2019-06-2004:06rgmHuh, yeah. Maybe if I evaluate the editor buffer via prepl that could work.#2019-06-2006:48OlicalIf you're still on Conjure that's <localleader>eb :)#2019-06-2004:06seancorfieldI haven't touched cljs for over four years but I've made a public pledge to go back and spend this weekend getting familiar with modern tooling -- including Figwheel and Shadow -- although I don't know what editor I'll use for this learning project. I've moved away from nREPL (to "just" a Socket REPL or in-process REPL/REBL) but I think cljs will force me to use nREPL?#2019-06-2004:48rgmI’m having a pretty fun time with prepl and conjure (I’m a vimmer) … my experience in vim has been than nrepl + piggieback + fireplace.vim is still pretty rough. Colleagues using emacs/cider laugh at me and my struggles.#2019-06-2005:15seancorfieldI've gone through a bunch of setups over the last nine years. I think I started with TextMate and a plain REPL...#2019-06-2009:14Alex Miller (Clojure team)Me too :)#2019-06-2005:16seancorfield...tried a bunch of editor+repl combos, settled on Eclipse/CCW for a while, then moved (back) to Emacs -- I used it a lot 20-30 years ago, then switched to Atom/ProtoREPL and more recently Atom/Chlorine, which means I can get away from nREPL.#2019-06-2005:17seancorfieldAnd, lately, I've been very enamored with REBL as part of my workflow so my normal work setup is clj/`deps.edn`, Atom/Chlorine, REBL with a Socket REPL.#2019-06-2005:17seancorfieldBut I've always had a soft spot for vi so I've been keeping an eye on Liquid, which is basically an in-process vi-like editing experience with built-in evaluation (and no REPL needed).#2019-06-2005:19seancorfieldI have Liquid + REBL (started from clj/`deps.edn`) working now and will try to use that quite a bit going forward (but it does lack some parinfer/paredit/global-find'n'replace niceties).#2019-06-2005:19seancorfieldI don't know how any of that works for cljs stuff tho'. Part of this weekend's exercise will be exploring tooling and figuring all that out.#2019-06-2005:20felipebarrosYou should totally be a streamer @seancorfield... I would love to watch you experimenting with that stuff and analyze your workflow.#2019-06-2005:22seancorfield@anantpaatra Yeah, I was very interesting in the #livestream channel when it started ages ago but the problem is I mostly work on a large (86K+ line) proprietary code base and couldn't stream that.#2019-06-2005:22seancorfieldI am considering doing more screencasts (or even streaming) my OSS work tho'.#2019-06-2005:23seancorfield@anantpaatra did you see the 9 minute screencast I posted on YouTube of me working with Atom/Chlorine/REBL?#2019-06-2005:39felipebarrosIt was definitely interesting to watch that as it is a completely different workflow from everything I've seen so far. But the pace is like trying to appreciate a super-bike when it passes in front of you at 350km/h. You can hear it is loud, you can see it is expensive, it's probably beautiful and you quickly think "damn, I would love to ride that" but its gone before you even realize and you're left with your regular pedestrian day ahead of you. haha#2019-06-2005:42felipebarrosI understand it was made for experienced developers though, but I would really like that clj/deps.edn details being slowly crunched.#2019-06-2005:42felipebarrosAnd after I get that part, the REBL.#2019-06-2005:50seancorfieldOK, good feedback. I didn't realize it came across like that. I thought it was slow and gentle. I'll try harder next time.#2019-06-2005:23felipebarrosFor example, watching someone going from zero to a complex project by using clj/`deps.edn` would be awesome. I truly need something like that.#2019-06-2005:24felipebarrosI haven't! Going to check it now.#2019-06-2005:24didibus+1, not to mention you've got a great voice for it as well!#2019-06-2005:24seancorfieldYou'll be telling me next that I have an accent... 🙂#2019-06-2005:24didibus😛#2019-06-2005:26seancorfield(I'm from the South of England. I don't have an accent. Everybody else does! 🙂 )#2019-06-2005:28seancorfieldBTW, I'm going to be on DEFN Sunday morning (Pacific) and I believe they're livestreaming that. But it's just chat, not coding.#2019-06-2005:29didibusOh, cool. I'll be sure to tune in#2019-06-2005:30seancorfieldWhen I fixed an issue on core.memoize last week, I realized that it would have been a great opportunity to stream or screencast. I used REBL and tap> specifically to hone in on the bug and fix it, all live-eval'ing in the editor.#2019-06-2005:31seancorfield(and we're a bit off-topic... -> #off-topic if we want to continue this!)#2019-06-2007:44sogaiuwhere are our black boxen when we need them?#2019-06-2009:38minikomihttps://twitter.com/ryanbigg/status/1139053250244562944
➜ ~ clj -e "(clojure-version)"
"1.10.0"
#2019-06-2009:38minikomiis there a better way? 🙂#2019-06-2009:41Alex Miller (Clojure team)Not currently. There’s a ticket for it#2019-06-2015:32ghadido you show the version of clj or the default version of clojure? or the requested version of clojure in the current context?#2019-06-2019:33orestisEverything? If it’s reasonably structured it can be greppable for tools to extract more. #2019-06-2019:34orestisI think Elixir’s mix and also npm (to which clj is more alike) show the version of the tool and the version of the language (and also the host runtime version). I’ll verify at a computer tomorrow. #2019-06-2019:35ghadiall i mean is the version of the language is context dependent#2019-06-2019:35ghadi(but the version of tool + jdk aren't)#2019-06-2019:37orestisYeah for me the version of clj is more important - right now in my ansible scripts I’m looking at some some specific file to check a number. #2019-06-2019:37orestisWould prefer to invoke the tool and get an easily comparable version string without starting the JVM#2019-06-2019:38orestis(Not a big deal currently)#2019-06-2021:02Alex Miller (Clojure team)these questions are tbd, imo. I'm not sure that the subtle difference between the clojure version used when building classpaths vs the clojure version being used in your project is obvious to most people. I don't think it's even clear to people that these are independent things.#2019-06-2021:18vlaaadI just want to share how much I love tools-deps. It's so simple in what it does, you really know what's going on when you use it, and it's so easy to just specify git dep and immediately try something: no need to publish for library authors, no need to setup a project just to try stuff out for library users... I wish we had build tools that are that simple!#2019-06-2021:18Alex Miller (Clojure team)thanks!#2019-06-2110:19stathissiderisdoes :jvm-opts work only within aliases or at the top-level too?#2019-06-2110:20kszaboonly aliases, it’s a known issue#2019-06-2110:20stathissiderisok, thanks! (glad to hear it’s a bug and not a feature)#2019-06-2117:45misha1) can I add a dependency to library so that it can be seen from library (e.g. when run as alias), but is invisible to client library? I assume deps in :aliases behave like that. and if so:
2) is there a way to depend on an alias of such a library?
Trying to have specs in my lib on one hand (for lib internal tests, and for lib's clients if they need them),
and as a lib client to be able to exclude them from dependency (in case I use another spec version, etc.)#2019-06-2117:47hiredmanno#2019-06-2117:48mishano for both 1 and 2?#2019-06-2117:48hiredmanyes#2019-06-2117:49mishawhat is the preferred way to go about it? separate repo with specs?#2019-06-2117:49mishado :aliases extra deps appear in client's class path?#2019-06-2117:50mishaif no, that makes extra repo option not that bad#2019-06-2117:50hiredmanI maybe have misunderstood 1#2019-06-2117:51mishaI want to have specs for my lib. And want you to be able to depend on my lib excluding those specs (and clojure.spec)
both at the same time#2019-06-2117:51seancorfield:aliases are not transitive across dependencies, if that's what you're asking @misha?#2019-06-2117:52hiredmanso 1, the dep on clojure.spec is going to be there, clojure pulls it in#2019-06-2117:53mishaso:
specs-repo +
lib-repo + specs-repo-as-alias-extra-dep for tests
is an ok solution#2019-06-2117:53hiredmanor just put your specs in a namespace and don't load that namespace except from tests#2019-06-2117:54mishaoh, right. then: can I not depend on clojure in my lib repo? )#2019-06-2117:54seancorfieldIn separate repos, yes, I guess that could work -- but I don't know why anyone would bother?#2019-06-2117:54seancorfieldIf your specs are part of yours tests (only) just put them in the test folder that would be added by your :test alias. Done.#2019-06-2117:55mishatrying to avoid any transitive deps you'll get by using my lib#2019-06-2117:55seancorfieldWhat transitive deps would those be? Clients of your lib need whatever your src depends on. They don't need what your test code depends on.#2019-06-2117:56mishaSean, those specs might be useful for you as a client. But if you dont care, I dont want you to get clojure.spec through my lib#2019-06-2117:56hiredmanspec comes with clojure these days, but is technically a separate artifact so you can use a newer version if you want#2019-06-2117:56seancorfieldYou can't avoid clojure.spec.#2019-06-2117:56seancorfieldClojure itself pulls in clojure.spec and core.specs. The client is always going to get those.#2019-06-2117:57mishaI need specs for tests only, but I know those might be useful for clients as well. so core does not depend on those for functionality.#2019-06-2117:57seancorfieldYou can avoid test.check (by making it a :test-only :extra-deps).#2019-06-2117:57seancorfield@misha I think you're overthinking this. If you want clients to have access to your specs as an option, put them in src. If you don't want clients to have access to them, put them in test.#2019-06-2117:58mishaok. do I have to depend on clojure itself to distribute a lib? if yes, is it better to depend on as old version as possible?#2019-06-2117:58seancorfieldThe Clojure dependency will be overridden by the client.#2019-06-2118:00mishaI am absolutely overthinking it, but on the other hand I had bad experience with different versions of direct and transitive deps before. trying to not mess things up for anyone.#2019-06-2118:12seancorfieldYeah, I hear that... If you have test-only dependencies, put them under an alias as :extra-deps. If you have source dependencies, those are going to end up as transitive dependency for your clients.#2019-06-2118:29mishaYou are right, clojure includes spec, I have to include clojure, so it boils down only to "make specs public or not". thanks, guys.#2019-06-2118:30seancorfieldAs an example, both clojure.java.jdbc and next.jdbc have all their specs in a dedicated namespace that clients can require if they wish but otherwise they are not activated.#2019-06-2118:32seancorfieldFor clojure.java.jdbc, I primarily made them optional like that so it could continue to run on older Clojure versions (that didn't support Spec). For next.jdbc, I added them after the fact as an option for folks who want better error messages (since next.jdbc sort of follows Clojure core's principle of "do the right thing as fast as possible; potentially explode in arbitrary ways if asked to do the wrong thing" (i.e., NPE or ClassCast or whatever).#2019-06-2118:48mishathis is exactly how I am approaching it now. Docstrings – contract, validation – optional, in a separate ns#2019-06-2119:45andy.fingerhutFun fact. This works: clj -Sdeps "{:deps {org.clojure/clojure {:mvn/version \"1.0.0\"}}}"#2019-06-2119:47andy.fingerhutAnd if you are wondering, I tried it because it is helpful in tracking down when some functions were added to Clojure that do not (yet) have :added metadata, but should soon.#2019-06-2119:50seancorfieldI didn't try going back any further than 1.2.1 https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L4-L15 but that's good to know @andy.fingerhut#2019-06-2119:53seancorfieldInteresting little glitch in 1.0.0: (! 563)-> clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.0.0"}}}'
Clojure 1.0.0-
user=> (clojure-version)
"1.0.0-"
user=> *clojure-version*
{:major 1, :minor 0, :incremental 0, :qualifier ""}
user=> ^D
Fri Jun 21 12:53:28
(sean)-(jobs:0)-(~/clojure)
(! 564)-> clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.1.0"}}}'
Clojure 1.1.0
user=> (clojure-version)
"1.1.0"
user=> *clojure-version*
{:major 1, :minor 1, :incremental 0, :qualifier ""}
user=>
#2019-06-2119:55andy.fingerhutI'll file a ticket for a 1.0.1 release 🙂#2019-06-2120:04Alex Miller (Clojure team)Right :)#2019-06-2222:21seancorfieldA screencast about the CLI, deps.edn, and aliases https://youtu.be/CWjUccpFvrg (this time in 1080p instead of 360p!).#2019-06-2306:53felipebarrosThank you so much for that! The pace is elegant, the scope is healthy and the perspective is intimate and clarifying. Excellent!#2019-06-2305:29steveb8nQ: I can’t get :override-deps to work with :local/root deps. e.g. I have a default :git/sha dep and then for local dev I want to override that with a :local/root dep. Has anyone else made this work?#2019-06-2305:34steveb8nstarting a separate thread to stop annoying others#2019-06-2305:35seancorfieldThis sounds like it should work, but I’d have to see the relevant parts of the deps.edn file I think…#2019-06-2305:35steveb8nmy specific problem is that the the :git/sha dep is not reachable in local dev so I’m trying to override it. however, with the override, it still tries to pull the :git/sha dep and that breaks#2019-06-2305:36steveb8nI agree. it should work. however it appears that with an :override-deps, the overridden dep must still be accessible#2019-06-2305:37steveb8nlet me see if I can make a simple test case deps.edn to demo it#2019-06-2305:39seancorfieldI tried from the command line with an alias to provide override-deps and it seems to work…#2019-06-2305:41seancorfieldYup, deleted my ~/.gitlibs folder and the :dev alias that overrides the dep with :local/root works fine. No attempt is made to clone the git repo.#2019-06-2305:44steveb8nok, here’s a repro (I think)#2019-06-2305:44steveb8n{:deps {reifyhealth/specmonstah {:mvn/version "2.0.0-alpha-1"}}
:aliases {:local {:override-deps {reifyhealth/specmonstah {:git/url ""
:sha "8e47ffc60d3177cea90f2f657e73c48971d8a1e4"}}}}}
#2019-06-2305:45steveb8nyuck formatting#2019-06-2305:45steveb8nthen clojure -R:local -Spath | tr ‘:’ ‘\n’ | sort#2019-06-2305:46steveb8nshows the alpha version, not my fork#2019-06-2305:46steveb8nnot quite the same as what I described but I think it might show how I am using the CLI opt wrong#2019-06-2305:50steveb8nhmm, I wonder if my version of tools.deps is too old and this is a fixed bug. I’ll check that now#2019-06-2305:51seancorfield(! 640)-> clj -Spath | tr ':' '\n' | sort| fgrep specmon
/Users/sean/.m2/repository/reifyhealth/specmonstah/2.0.0-alpha-1/specmonstah-2.0.0-alpha-1.jar
Sat Jun 22 22:50:18
(sean)-(jobs:0)-(~/clojure/steve)
(! 641)-> clj -A:local -Spath | tr ':' '\n' | sort | fgrep specmon
/Users/sean/.gitlibs/libs/reifyhealth/specmonstah/8e47ffc60d3177cea90f2f657e73c48971d8a1e4/src
Sat Jun 22 22:50:24
(sean)-(jobs:0)-(~/clojure/steve)
(! 642)-> cat deps.edn
{:deps {reifyhealth/specmonstah {:mvn/version "2.0.0-alpha-1"}}
:aliases {:local {:override-deps {reifyhealth/specmonstah {:git/url ""
:sha "8e47ffc60d3177cea90f2f657e73c48971d8a1e4"}}}}}
works for me#2019-06-2305:51seancorfield-R:local behaves the same, I checked.#2019-06-2305:51seancorfield(! 644)-> clj -Sdescribe
{:version "1.10.1.447"
#2019-06-2305:53steveb8nI’m 1.9.0.391#2019-06-2305:53steveb8ndoesn’t seem likely it’s a bug but I guess I should rule that out#2019-06-2305:54seancorfieldYeah, I think this is a bug that was fixed (ages ago).#2019-06-2305:54steveb8nyeah. I just ran the same commands you pasted in and local does not change the result#2019-06-2305:55steveb8nI’ll trying upgrading#2019-06-2305:55steveb8nnot sure how 😉#2019-06-2305:55steveb8nI’ll go find the docs for upgrading#2019-06-2305:55seancorfieldMac or Linux?#2019-06-2305:56steveb8nmac. I think it’s “brew upgrade clojure”#2019-06-2305:56steveb8ntrying that now#2019-06-2305:57seancorfieldYup, that’s the right command.#2019-06-2305:57steveb8nwaiting on Aussie broadband 🙂#2019-06-2305:57seancorfieldSorry. Mine’s about 150M down and 7.5M up.#2019-06-2305:58steveb8nmines 94m down which is really good for Aussie. I can’t complain. I suspect the latency hurts despite the good bandwidth#2019-06-2305:59seancorfieldYah, we have global monitors for our work stuff and the latency from Asia/Australia is pretty bad.#2019-06-2305:59seancorfield(our main data center is US East Coast and our primary AWS region is also US East Coast)#2019-06-2306:02steveb8nok. that fixed it in the repro case! brilliant#2019-06-2306:03steveb8nlet me check my original#2019-06-2306:03steveb8nyep, fixed there too#2019-06-2306:03steveb8nthanks for being a sounding board/tester 🙂#2019-06-2306:03steveb8nI’ll try to pay it forward#2019-06-2306:03seancorfield1.9 is super old. So many changes, fixes, enhancements since then!#2019-06-2306:04seancorfield(but then I’m the guy who always runs Clojure alpha/beta builds in production! LOL!)#2019-06-2306:04steveb8nyeah. I’m using 1.10 for my project, just not for the CLI that kicks it all off#2019-06-2306:04steveb8nnow I am 🙂#2019-06-2306:05steveb8nhave a good day/night. and thanks again for the videos, I’m gonna go watch the rest of them now#2019-06-2306:14seancorfieldG’nite! It’s 11:15pm here and definitely bedtime.#2019-06-2305:30steveb8nbtw: @seancorfield I just watched your video on deps.edn. really useful. thanks.#2019-06-2305:33seancorfieldGlad it was helpful!#2019-06-2305:34seancorfieldCan you share your deps for the git/url and local/root deps in the alias?#2019-06-2319:34didibusIs there a way to specify jvm options but not for an alias?#2019-06-2319:35didibusAnd not at the command line#2019-06-2319:35didibusSo have them specified in deps.edn for when I run main#2019-06-2320:01seancorfieldRight now they only work inside an alias @didibus — it’s a known issue/limitation. Sounds like the team may change that at some point…#2019-06-2320:02didibusI see, thx !#2019-06-2321:41andy.fingerhutDoes anyone happen to know when you make a text file beginning with #! /usr/bin/env clojure on a Unix/Linux system, once the JVM begins and the Clojure reader starts reading the file, is the Clojure reader also given that first line to read as well?#2019-06-2321:41andy.fingerhutOr is there some code somewhere that somehow skips over it, and thus the Clojure reader never sees it?#2019-06-2321:42andy.fingerhutAnd if so, do you know where that "skipping" code is?#2019-06-2321:49Alex Miller (Clojure team)In LispReader#2019-06-2321:50Alex Miller (Clojure team)It skips lines that start with #!#2019-06-2321:50Alex Miller (Clojure team)Or rather, those act as single line comments#2019-06-2321:51andy.fingerhutthx. Someone was asking how this script worked, and I think I knew all of the magic, except that part. Now my magical knowledge is complete! https://gist.github.com/ericnormand/6bb4562c4bc578ef223182e3bb1e72c5#2019-06-2321:54Alex Miller (Clojure team)It was certainly a future looking escape in the reader :)#2019-06-2321:57andy.fingerhutAnd I have a vague feeling that I have asked this and/or discovered it before, and was also then surprised that it has been around since 2009#2019-06-2321:59andy.fingerhutDo you think it belongs in the Reading Clojure Characters guide, or better to keep it obscure? 🙂#2019-06-2322:09Alex Miller (Clojure team)Well I don’t think it belongs there as it’s not really part of the Clojure language#2019-06-2322:10Alex Miller (Clojure team)Really more syntax intentionally ignored by Clojure #2019-06-2322:10Alex Miller (Clojure team)Could be on the reader page#2019-06-2323:30andy.fingerhutI haven't followed the myriad ways people have used deps and clj/clojure, but does the approach on that page: https://gist.github.com/ericnormand/6bb4562c4bc578ef223182e3bb1e72c5 pretty much a good way to go, if you want to embed the JVM and/or clj/clojure options within the same file as the Clojure code? As in, have people found other ways to combine those options together into the same file as the Clojure code, so that those options do not end up in a separate file/script somewhere?#2019-06-2412:08futuroI love the use of #_ to hide the shell opts from the Clojure reader. #2019-06-2416:18andy.fingerhutIt does restrict you a little bit, in that those first 16 lines must be valid shell syntax, as well as legal Clojure syntax, but yeah, I am pretty sure it is not the first language where someone has found a handful of dual-language lines of code.#2019-06-2417:33futuroHmm, that’s a good point. #2019-06-2401:33sogaiuas another data point: https://gitlab.com/eval/ohmyclj/#2019-06-2423:22rgmSo I’m running macOS 10.15 betas and seem to have gotten bit by https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-91 … it manifests as a failure to clone a private lib from a :git/url, whether using an id_rsa created by under macOS 10.15 or earlier.#2019-06-2423:23rgmis the new jira taking new commenters yet? Wanted to update TDEPS-91#2019-06-2423:33andy.fingerhutThe new JIRA does allow adding comments, if you have a free account created, and if that account's permissions have been bumped up by a Clojure JIRA admin.#2019-06-2423:34rgmoh, cool… is that someone less busy than Alex?#2019-06-2423:34andy.fingerhutAt least, the older Clojure JIRA instance required someone to bump up your permissions. My permissions allow me to do so with the new JIRA, but I believe Alex bumped up most/all people who have created or commented on issues in the old JIRA, in the new one.#2019-06-2423:35andy.fingerhutI had permission to do that on the old JIRA system, but not sure whether I do on the new one or not. Let me take a look. Otherwise, I suspect it would not be someone less busy than Alex.#2019-06-2423:36rgmno rush; I have enough of a handle on what’s going on in ~/.gitlibs that I’ve just manually set it up so clj et al don’t complain any more, and I can do manual git pulls for a while.#2019-06-2423:38rgmjust wanted to bump this since it’s gonna bite a lot of people come 10.15 release in September.#2019-06-2423:39andy.fingerhutTo be clearer, I mean that my account on the new Clojure JIRA gives me permission to add comments on issues. I do not know how on the new JIRA to give permission to others to do so, and/or my account on the new JIRA doesn't have authorization to do so.#2019-06-2423:39rgmah, gotcha. I’ll ask Alex.#2019-06-2501:00ghadi@rgm an id_rsa key should work just fine. Are you running under an ssh agent?#2019-06-2501:01rgmI am.#2019-06-2501:01ghadiis the key inside your agent?#2019-06-2501:01ghadissh-add -l#2019-06-2501:02ghadishould verify ^#2019-06-2501:05ghadiI've edited TDEPS-91 to be problem-oriented rather than leading with a solution#2019-06-2501:14rgmyep, it is.#2019-06-2501:14rgmthe same key file works under 10.14 and fails under 10.15, same clj version#2019-06-2501:15rgmunder an agent both places.#2019-06-2501:21ghadiif you can find out what the difference is, I'd be much obliged#2019-06-2501:21ghadiis SSH_AUTH_SOCK set?#2019-06-2501:50rgmhm, good question. It appears yes.#2019-06-2501:52rgm#2019-06-2615:07credulousHi! I’m trying to convert my workflow to clj/deps.edn, and I’m having a bit of trouble with depstar, which I am trying to use to package my app.#2019-06-2615:07ghadiseancorfield/depstar @credulous?#2019-06-2615:07ghadican you describe what you expected and what you are experiencing?#2019-06-2615:08creduloushealthfinch/depstar I think?#2019-06-2615:08credulousYes I can… I added this to my ~/.clojure/deps.edn:#2019-06-2615:08credulous:depstar {:extra-deps {com.healthfinch/depstar {:git/url ""
:sha "4aa7b35189693feebc7d7e4a180b8af0326c9164"} }
:main-opts ["-m" "hf.depstar.uberjar"]
}#2019-06-2615:08ghadiswitch to the fork (I originally wrote healthfinch/depstar but now sean has the canonical repo)#2019-06-2615:08credulousOK!#2019-06-2615:09credulousI’ll try that before I describe anything more.#2019-06-2615:09credulousThanks#2019-06-2615:09ghadihis README is good, too#2019-06-2615:09ghadibetter#2019-06-2615:09credulousIn his blog post, Sean is still linking to yours#2019-06-2616:00seancorfieldI'll update that blog post when I get to my desk -- thanks for catching that!#2019-06-2617:01seancorfieldUpdated. Make take a few minutes to clear the cache!#2019-06-2615:09ghadiwhich blog post?#2019-06-2615:10credulousIt’s over a year old. I should have been paying more attention. https://corfield.org/blog/2018/04/18/all-the-paths/#2019-06-2615:13credulousYup, the fork works. Thanks @ghadi!#2019-06-2615:15credulousAnd may I also say: @seancorfield is everywhere! What a contributor.#2019-06-2615:18ghadihe is a treasure#2019-06-2615:20Alex Miller (Clojure team)MVP!#2019-06-2616:00seancorfieldI'll update that blog post when I get to my desk -- thanks for catching that!#2019-06-2617:01credulousThanks! Also, I’m floundering a bit trying to include my public folder in the uberjar, using depstar. I created an alias with {:extra-paths ["public"]}, but that includes the contents of the public folder into the root of the uberjar. Any advice?#2019-06-2617:01seancorfieldUpdated. Make take a few minutes to clear the cache!#2019-06-2617:02seancorfield@credulous So you have resources/public -- would {:extra-paths ["resources"]} do what you need?#2019-06-2617:03credulousNo, in my project folder I have /resources and /public#2019-06-2617:03seancorfieldDon't do that 🙂#2019-06-2617:03credulousOK!#2019-06-2617:03seancorfieldPut public inside resources -- then your "source" paths are going to be "src" "resources"#2019-06-2617:04seancorfieldAnd in your code, specify that pages etc come from "public" rather than the root of your classpath.#2019-06-2617:05credulousSounds good. I was cribbing from a complete project on github… boodle, I think.#2019-06-2617:05seancorfield{:root "public"} in Compojure's resources call (assuming you're using that?)#2019-06-2617:07credulousI’m probably using compojure wrong too… I’m not using :root anywhere#2019-06-2617:07credulous(compojure/defroutes app
(-> (compojure/routes
(route/resources "/")
api/routes )
reload/wrap-reload))
#2019-06-2617:07credulouswhere route/resources defaults to “public” as the source directory, I think#2019-06-2810:36vlaaadCan someone explain how do I build a classpath properly?
(deps/make-classpath
(deps/resolve-deps {:deps {'re-find {:git/url ""
:sha "c27af619a9d9e12832b7446e127d68ab916ce78c"}}}
nil)
nil
nil)
produces this classpath:
- .../tools.cli-0.4.1.jar
- .../math.combinatorics-0.1.4.jar
- .../clojure-1.8.0.jar"
but there should also be .gitlibs/libs/refind/...sha/src, no?#2019-06-2811:54vlaaadApparently I had to add :paths ["src"] to map, which works, but feels a bit wrong... shouldn't it be the default? if I'll set :paths to ["src/clj"] in my ~/.clojure/deps.edn, will it break dependencies because I overridden their source path?#2019-06-2811:54vlaaadApparently I had to add :paths ["src"] to map, which works, but feels a bit wrong... shouldn't it be the default? if I'll set :paths to ["src/clj"] in my ~/.clojure/deps.edn, will it break dependencies because I overridden their source path?#2019-06-2812:54Alex Miller (Clojure team):paths ["src"] is in the install-level deps.edn, but when you're using the library directly, you're in full control of what you include#2019-06-2812:55Alex Miller (Clojure team)setting to "src/clj" only affects your lib, not transitive deps - they set their own paths in their own deps.edn#2019-06-2812:59vlaaadthat's the thing, re-find does not set :paths explicitly, that's why I need to specify ["src"]#2019-06-2813:00vlaaadit all works until everyone's system :paths property is the same#2019-06-2813:00vlaaadsetting "src/clj" affects dependent libs too#2019-06-2813:03vlaaad(run! println
(str/split
(deps/make-classpath
(deps/resolve-deps
{:deps {'re-find {:git/url ""
:sha "c27af619a9d9e12832b7446e127d68ab916ce78c"}}
:paths ["src/clj"]}
nil)
nil
nil)
#":"))
=>
/home/vlaaad/.gitlibs/libs/re-find/re-find/c27af619a9d9e12832b7446e127d68ab916ce78c/src/clj
/home/vlaaad/.m2/repository/org/clojure/tools.cli/0.4.1/tools.cli-0.4.1.jar
/home/vlaaad/.m2/repository/org/clojure/math.combinatorics/0.1.4/math.combinatorics-0.1.4.jar
/home/vlaaad/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar
#2019-06-2813:05vlaaadre-find sources are implicitly on src, not src/clj, but it gets overridden#2019-06-2813:15Alex Miller (Clojure team)sorry, I don't understand what you're saying#2019-06-2813:16Alex Miller (Clojure team)it looks like answer above is what I would expect to see#2019-06-2813:16Alex Miller (Clojure team)it looks like answer above is what I would expect to see#2019-06-2813:18vlaaadI am surprised that I have to specify :paths to get code from git dependency#2019-06-2813:19vlaaadI expect gitlibs to have default :paths ["src"]#2019-06-2813:22Alex Miller (Clojure team)oh, I see what you're saying#2019-06-2813:23Alex Miller (Clojure team)it's up to libs to specify their own paths and re-find could (probably should) set :paths ["src"], and if it did, that would be used#2019-06-2813:23Alex Miller (Clojure team)you're seeing src/clj there due to a bug, where if no paths are used, the paths in the root deps.edn are used#2019-06-2813:26Alex Miller (Clojure team)https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-96 (may be other tickets that cover this too)#2019-06-2813:31Alex Miller (Clojure team)https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-52 is same thing#2019-06-2813:36vlaaadah, it's already reported, nice#2019-06-2813:39Alex Miller (Clojure team)yeah, the cause is bound up in some other issues so I've had a long mull about how to fix, but ready to move forward on it when I have some time#2019-06-2814:22vlaaadclojure.tools.deps.alpha.specs seem wrong#2019-06-2814:22vlaaad(s/def :local/root string?)
(s/def :local/coord (s/keys :req-un [:local/root] :opt-un [::path]))
#2019-06-2814:22vlaaadshouldn't it be :req instead of :req-un?#2019-06-2814:35Alex Miller (Clojure team)yes, I think you're right#2019-06-2814:35Alex Miller (Clojure team)fixed#2019-06-2821:33Drew VerleeI'm attempting to build an uberjar using deps + uberdeps Everything works as expected until i try to run the uberjar and it tells me it cant find clojure.main.
➜ cat deps.edn
{:aliases {
:uberjar {:extra-deps {uberdeps {:mvn/version "0.1.4"}}
:main-opts ["-m" "uberdeps.uberjar"]}}}
➜ clj -A:uberjar
[uberdeps] Packaging target/frog.jar...
[uberdeps] Packaged target/frog.jar in 2 ms
➜ cat src/core.clj
(ns frog.core
(:gen-class))
(defn -main [& args]
(println "HIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII"))
➜ java -cp target/frog.jar clojure.main -m frog.core
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main
As a starting point, im not sure why the instructions from uberdeps have me look for clojure.main, if its an uberjar isn't it included?#2019-06-2821:35hiredmanif you don't have clojure in your deps#2019-06-2821:35hiredmansome of it depends on how smart the uberdeps library is#2019-06-2821:36hiredmanif you don't include a dep on clojure explicitly, you will fallback to whatever is in the system (or whatever it is called) deps.edn file that was installed when you installed clj#2019-06-2821:37hiredmanbut uberdeps might not be processing your deps.edn file using the tools.deps code, so it might not know to do that fallback#2019-06-2821:37hiredmanso if you don't get include a clojure version in your deps.edn, there wouldn't be clojure in your jar#2019-06-2821:37hiredmanyep#2019-06-2821:38hiredmanit just slurps in the edn and reads it#2019-06-2821:38hiredmana lot of tools built on top of deps.edn are super immature and very broken, and they largely duplicate each other#2019-06-2821:39hiredmanthis is the second broken uberjar building library I have diagnosed today (the other tool seemed to end up aot compiling things twice)#2019-06-2821:41hiredmanalso, if you are not aot compiling you don't need the :gen-class#2019-06-2822:08Drew VerleeI thought it was necessary to for the rest of the tooling to convert the clojure code to java.#2019-06-2822:16Drew VerleeOh, your saying that if I don't specify that, the compiling still happens, just not ahead of time?#2019-06-2822:19hiredmanclojure code is not converted to java, it is compiled to jvm bytecode#2019-06-2822:20hiredmanit is always compiled to jvm bytecode, either when the code is loaded and right before it is run, or you load the code ahead of time and save the compiled bytecode and run the compiled bytecode (aot compilation)#2019-06-2822:20hiredman:gen-class only does anything when you are aot compiling#2019-06-2822:20hiredmanand it is only needed in specific circumstances#2019-06-2822:20hiredmanyou are not aot compiling so you do not need it#2019-06-2822:31Drew VerleeAlmost understand what your saying, I'll need to read a bit. Thanks!#2019-06-2821:44Alex Miller (Clojure team)a new version of tools.deps is out now that probably makes it much harder to do the wrong thing with the install deps.edn#2019-06-2821:45hiredmanI dunno, this code literally just slurps the deps.edn and calls read-string on it#2019-06-2821:45seancorfield@drewverlee Try using depstar instead of uberdeps -- unless there's some specific feature you need from the latter?#2019-06-2821:49Drew VerleeThere isn't! I did try both, I think I got a similar error. For one, It's possible I have an old version of deps. I'll have to check when I'm at my computer again. I'm always blown away by how fast everyone responds.#2019-06-2821:50seancorfieldHappy to help with any issues you have with depstar since I maintain it.#2019-06-2823:46Drew VerleeSo it resulted in a FileNotFoundException:
➜ cat deps.edn
{:deps {org.clojure/clojure {:mvn/version "1.10.0"}}
:aliases {:depstar
{:extra-deps
{seancorfield/depstar {:mvn/version "0.2.1"}}}}}
➜ clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
Building uber jar: MyProject.jar
➜ frog java -cp MyProject.jar clojure.main -m frog.core
Exception in thread "main" java.io.FileNotFoundException: Could not locate frog/core__init.class, frog/core.clj or frog/core.cljc on classpath.
at clojure.lang.RT.load(RT.java:466)
at clojure.lang.RT.load(RT.java:428)
at clojure.core$load$fn__6824.invoke(core.clj:6126)
at clojure.core$load.invokeStatic(core.clj:6125)
at clojure.core$load.doInvoke(core.clj:6109)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5908)
at clojure.core$load_one.invoke(core.clj:5903)
at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
at clojure.core$load_lib.invokeStatic(core.clj:5947)
at clojure.core$load_lib.doInvoke(core.clj:5928)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$load_libs.invokeStatic(core.clj:5985)
at clojure.core$load_libs.doInvoke(core.clj:5969)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$require.invokeStatic(core.clj:6007)
at clojure.main$main_opt.invokeStatic(main.clj:491)
at clojure.main$main_opt.invoke(main.clj:487)
at clojure.main$main.invokeStatic(main.clj:598)
at clojure.main$main.doInvoke(main.clj:561)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:37)
➜ clojure -A:depstar -m hf.depstar.uberjar MyProject.jar
Building uber jar: MyProject.jar
➜ java -cp MyProject.jar clojure.main -m frog.core
Exception in thread "main" java.io.FileNotFoundException: Could not locate frog/core__init.class, frog/core.clj or frog/core.cljc on classpath.
at clojure.lang.RT.load(RT.java:466)
at clojure.lang.RT.load(RT.java:428)
at clojure.core$load$fn__6824.invoke(core.clj:6126)
at clojure.core$load.invokeStatic(core.clj:6125)
at clojure.core$load.doInvoke(core.clj:6109)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5908)
at clojure.core$load_one.invoke(core.clj:5903)
at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
at clojure.core$load_lib.invokeStatic(core.clj:5947)
at clojure.core$load_lib.doInvoke(core.clj:5928)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$load_libs.invokeStatic(core.clj:5985)
at clojure.core$load_libs.doInvoke(core.clj:5969)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$require.invokeStatic(core.clj:6007)
at clojure.main$main_opt.invokeStatic(main.clj:491)
at clojure.main$main_opt.invoke(main.clj:487)
at clojure.main$main.invokeStatic(main.clj:598)
at clojure.main$main.doInvoke(main.clj:561)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:37)
➜ cat src/core.clj
(ns frog.core)
(defn -main [& args]
(println "Hello World"))
➜ clj -Spath
src:/home/drew/.m2/repository/org/clojure/clojure/1.10.0/clojure-1.10.0.jar:/home/drew/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/drew/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
I assume because as src/ is on my class path (which is default right?). Then it would pick up the core.clj file with the frog.core ns. But either it doesn't pick it up, or i'm not specifying it correctly.
It seems to be in the jar file:
➜ cat core.clj
(ns frog.core)
(defn -main [& args]
(println "Hello World"))
#2019-06-2900:02seancorfieldIs that really src/core.clj or src/frog/core.clj?#2019-06-2900:03seancorfield(it should be the latter: the file path must match the namespace segments!)#2019-06-2900:03seancorfieldA good sanity check is to see whether clj -m frog.core works before you try to build the JAR.#2019-06-2900:07Drew Verleeah ok. i guess i didn't realize the relationship between the files and the ns was more then convention.#2019-06-2900:22seancorfieldWhen you ask Clojure to run frog.core, how is it going to find it? The classpath is all file paths, so the ns has to be mapped to a file path to perform that search if it can't already find the compiled class in memory... so that's frog/core.clj (or frog/core.cljc) relative to each folder on the classpath, e.g., src.#2019-06-2821:45Alex Miller (Clojure team)well, if you're not using tools.deps at all, then yes, that's prob broken#2019-06-2821:45seancorfieldSome of these packaging libs try to create a classpath which is not what you would have when running your code 😞#2019-06-2821:46Alex Miller (Clojure team)but I would encourage whomever to file tickets on these libs so they can be better, rather than raging about brokenness#2019-06-2821:46seancorfield(I hadn't even heard of uberdeps before just now)#2019-06-2821:47hiredmanI could be wrong too, I am just guessing at the problem#2019-06-2821:51hiredmanI am just suggesting, like, be aware of that state of maturity of a lot of these libraries(tools.deps is still new, so tooling built on it is even newer), and maybe vet them a little more before using them.#2019-06-2821:52andy.fingerhutWell, vetting includes using, at least a little bit 🙂#2019-06-2908:31dominicmI'm fairly confident that pack does the right thing™. We had some early issues due to the system deps.edn (although it sounds like I'm able to remove that now! 🎉). I think pack is the oldest library in this space, and I've been incredibly careful about things like classpath collisions, which other tools have been more unconcerned about.#2019-06-2911:37joseI just installed clojure 1.10.1.455, using https://download.clojure.org/install/linux-install-1.10.1.455.sh. Running clj command starts a REPL with clojure 1.9, while if I install the previous version, 1.10.447, clj command starts a REPL with clojure 1.10.1.
I think the reason is that in the latest released jar, at clojure/tools/deps/deps.edn, clojure version is set to 1.9 and that I don't require a clojure version on my ~/.clojure/deps.edn
Is this the intended behavior?#2019-06-2912:20Alex Miller (Clojure team)Oh, that is a bug#2019-06-2912:20Alex Miller (Clojure team)I know exactly why that’s happening#2019-06-2912:20Alex Miller (Clojure team)Thx#2019-06-2912:38Alex Miller (Clojure team)@jlle 1.10.1.458 is now available, should fix that#2019-06-3008:38joseTested it, now clj start a REPL with 1.10.1. Thanks for fixing it!#2019-06-2914:47Alex Miller (Clojure team)For the toolsmiths out there, I'm going to deprecate the reader/clojure-env call in tools.deps.alpha (and eventually remove it). Afaik, this was being used exclusively to obtain the list of config files used by clj. As of the latest tools.deps.alpha, 0.7.516, there is a new call reader/default-deps that will return the list of deps.edn config files. The install-level deps.edn is now embedded and this function replicates the logic from clj to find the user-level deps.edn and project-level deps.edn without shelling out (so also OS-safe). I sent a few PRs to places where I could find people doing this. @dominicm @olical @seancorfield @taylor#2019-06-2915:08OlicalWonderful, thank you very much!#2019-06-2915:05dominicmI think that I embedded my own copy of the system deps.edn a while ago. I'll make an issue to remove it and switch to whatever I'm supposed to be doing now.#2019-08-0320:51seancorfieldThat also has an example of how to add master from GitHub 🙂#2019-08-0320:51dharriganthanks 🙂#2019-08-0519:52seancorfieldseancorfield/depstar {:mvn/version "0.3.1"} was just released -- includes a new -m / --main option so that if you have a pom.xml and do your own AOT/compile step to create a "main" entry point, you can now override the Main-Class in the generated manifest (overriding the default of clojure.main).#2019-08-0519:56ghadidepstar @seancorfield?#2019-08-0519:56ghadithe coord might be missing a piece#2019-08-0520:11seancorfieldUpdated! Thanks!#2019-08-0612:24dharriganSo#2019-08-0612:25dharriganPulling in tools.deps.alpha via a git/url hash, to access the branch that allows me to load in a jar from the repl#2019-08-0612:25dharriganI'm also using slf4j#2019-08-0612:25dharriganMy logs stopped working (i.e., output)#2019-08-0612:26dharriganSLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/david/.m2/repository/org/slf4j/slf4j-nop/1.6.2/slf4j-nop-1.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/david/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
#2019-08-0612:26dharriganaccording to the doc, the choice is totally random#2019-08-0612:26dharriganwhy does deps include the nop static binding?#2019-08-0612:26dharriganI can't exclude it, like I can if I'm doing "mvn/version"#2019-08-0612:27dharrigan(using :exclusions)#2019-08-0612:27dharriganorg.clojure/tools.deps.alpha d0b33e0
...
...
chopped for brevity
...
org.slf4j/slf4j-nop 1.6.2
#2019-08-0612:28dharriganFrom the slf4j doc#2019-08-0612:28dharriganThe warning emitted by SLF4J is just that, a warning. Even when multiple bindings are present, SLF4J will pick one logging framework/implementation and bind with it. The way SLF4J picks a binding is determined by the JVM and for all practical purposes should be considered random. As of version 1.6.6, SLF4J will name the framework/implementation class it is actually bound to.
#2019-08-0612:29Alex Miller (Clojure team)I think we have a ticket for this, probably need to stop including that#2019-08-0612:30dharriganYes please! Having a library link in the static binding ain't right 🙂#2019-08-0612:30Alex Miller (Clojure team)Why did the exclusion not work?#2019-08-0612:30dharrigannot supported, according to the docs:#2019-08-0612:30dharriganhttps://clojure.org/reference/deps_and_cli#_dependencies#2019-08-0612:31dharriganyou can for maven coordinates, but not for git coordinates#2019-08-0612:31Alex Miller (Clojure team)Did you try it?#2019-08-0612:31dharriganYup#2019-08-0612:31dharriganorg.clojure/tools.deps.alpha {:git/url ""
:sha "d0b33e0d346736aa985c150145b332f97b92135e"
:exclusions ["org.slf4j/jcl-over-slf4j"
"org.slf4j/slf4j-nop"] }}}
#2019-08-0612:31Alex Miller (Clojure team)Should be a symbol, not a string#2019-08-0612:32Alex Miller (Clojure team)I think it will work#2019-08-0612:32dharriganah good spot#2019-08-0612:32dharriganretrying#2019-08-0612:33dharriganthat worked, after rm -rf'ing my .cpcache dir#2019-08-0612:34dharriganta very much, but still, I think the nop should be excluded totally, may catch others out 😉#2019-08-0612:34Alex Miller (Clojure team)Yeah#2019-08-0612:34dharriganparrot#2019-08-0613:15rickmoynihanHey a new colleague is getting the following error when resolving a git dep in a private github repo. Not sure what to do… I’ve seen the ticket https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-91 which looks relevant.#2019-08-0613:15rickmoynihan#2019-08-0613:17rickmoynihanHer ssh key appears to be the same version as mine which works…
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,xxxx
#2019-08-0613:18rickmoynihanshe’s also running ssh-agent and has added the key#2019-08-0613:44Alex Miller (Clojure team)so ssh-add -l lists her identity?#2019-08-0613:45Alex Miller (Clojure team)and I assume she does actually have access to the git repo (can access via git)#2019-08-0613:48Alex Miller (Clojure team)I've found on mac that I had a setting in my ~/.ssh/config under "Host *" that had a setting for IdentityFile or something like that which I had to delete#2019-08-0613:48rickmoynihanyes to both of those#2019-08-0613:48rickmoynihan👀#2019-08-0613:49Alex Miller (Clojure team)my ssh config looks like:#2019-08-0613:49Alex Miller (Clojure team)Host *
AddKeysToAgent yes
UseKeychain yes
#2019-08-0613:49rickmoynihanthanks @alexmiller dropping IdentityFile works#2019-08-0613:49Alex Miller (Clojure team)ssh ninja#2019-08-0613:50rickmoynihan:punch:#2019-08-0613:51Alex Miller (Clojure team)I added that to that jira issue in a comment#2019-08-0613:52rickmoynihan:thumbsup: my future self with thank you again for that#2019-08-0613:56Alex Miller (Clojure team)ditto, I had to look it up (again)#2019-08-0614:21rgm@alexmiller 🎉 … private deps seem to be working for me now#2019-08-0614:25rgm(to be clear, this is just a general man-are-there-ever-a-lot-of-moving-parts-to-auth gripe, not a tools.deps gripe 🙂)#2019-08-0618:24zaneI'm curious how y'all are switching between different Java versions when using clj and clojure.#2019-08-0618:25Alex Miller (Clojure team)hint: the answer has nothing to do with clj and clojure :)#2019-08-0618:25Alex Miller (Clojure team)they just use whatever java is on the path#2019-08-0618:26Alex Miller (Clojure team)so to rephrase, how are y'all switching between Java versions on your OS of choice?#2019-08-0618:28zane> they just use whatever java is on the path
Got it. So that means that clj and clojure will ignore the version set by, for example, jenv?#2019-08-0618:30zaneI suppose since jenv works by providing a java shim it ought to still work provided that the jenv shim is the first java on the path. :thinking_face:#2019-08-0618:31Alex Miller (Clojure team)I don't know what jenv does but that seems right#2019-08-0618:40zaneThank you! 🙂#2019-08-0619:17seancorfield@zane I set JAVA_HOME to point to the JDK installation that I want to use... (! 586)-> JAVA_HOME=$OPENJDK11_HOME clj
Clojure 1.10.1
user=> (System/getProperty "java.version")
"11.0.1"
user=> ^D
Tue Aug 06 12:16:46
(sean)-(jobs:0)-(~/clojure)
(! 587)-> JAVA_HOME=$OPENJDK8_HOME clj
Clojure 1.10.1
user=> (System/getProperty "java.version")
"1.8.0_192"
user=>
#2019-08-0621:26zaneHey Sean, thanks for sharing your process! This only changes the JAVA_HOME for the command that follows, right? (Or for the shell that does the export if export is included.)#2019-08-0621:27seancorfieldJust for that command, yes. Which is why it's nice for testing.#2019-08-0621:29zaneMakes a ton of sense for testing. I've done that myself.#2019-08-0621:30zaneThe particular use case I'm solving for is when I have a long-running Emacs process that occasionally launches clojure processes via cider-jack-in.#2019-08-0621:30zaneIn that case setting JAVA_HOME in the shell won't result in it getting picked up by the already-running Emacs process.#2019-08-0622:47seancorfieldBut you could get Emacs to set the env var for the invocation of clojure by changing the command line it uses?#2019-08-0622:59zaneIf I'm understanding you correctly, probably? Would require delving deeper into Cider than I want to do right now. 🙂#2019-08-0714:04Ahmed Hassan@zane you can connect to running clojure process by using cider-connect-clj#2019-08-0714:05Ahmed HassanSimilarly for ClojureScript you can use cider-connect-cljs#2019-08-0714:15zaneYes, I'm aware. I prefer to let Cider launch Clojure because choosing compatible versions of all the middleware can be difficult. But thank you! 🙂#2019-08-0716:04seancorfield"because choosing compatible versions of all the middleware can be difficult" -- a big part of why I stopped using Emacs/CIDER/nREPL #2019-08-0716:11Ahmed Hassan@U04V70XH6 What do you use now?#2019-08-0716:11seancorfieldI switched from Emacs to Atom/ProtoREPL and then to Atom/Chlorine. I can connect Atom/Chlorine to any Clojure process since it only requires a Socket REPL.#2019-08-0716:12seancorfield(so I can, and do, connect Atom/Chlorine to QA and production processes that were configured to start a Socket REPL using JVM options -- no dependencies needed in the project, no logic to start a custom REPL)#2019-08-0716:13Ahmed HassanHow is experience of Atom/Chlorine as compared with Cider/Emacs?#2019-08-0716:17seancorfieldI find it a lot smoother and a lot less "fiddly". I talked a bit about why I switched from Emacs in this blog post https://corfield.org/blog/2018/12/19/atom-chlorine/#2019-08-0717:19zaneYeah. I can understand why people make that switch, though for me it wouldn't be worth it.#2019-08-0620:01dominicmI think the way voidlinux does it is neat. JAVA_HOME points at a symlink, and xbps-alternatives changes the symlink.#2019-08-0621:08dharriganI use arch, so I do sudo archlinux-java set java-12-openjdk#2019-08-0621:08dharriganor sudo archlinux-java set java-8-openjdk#2019-08-0621:08dharriganeasy peasy#2019-08-0701:12cflemingIf I want to use a fork of Clojure with deps, can I exclude the clojure version pulled in from the system deps.edn somehow?#2019-08-0704:13seancorfield@cfleming If your local deps.edn file has that (local?) Clojure in :deps directly, it should override the system deps anyway?#2019-08-0705:19Alex Miller (Clojure team)correct ^^#2019-08-0705:19Alex Miller (Clojure team)assuming you have built and installed that fork locally in Maven so that it's artifact id is still org.clojure/clojure#2019-08-0705:20Alex Miller (Clojure team)if you want to call it something else (or use as a git or local dep), then no, that is not currently supported#2019-08-0705:22Alex Miller (Clojure team)that's not really intentional just kind of a consequence of various assumptions#2019-08-0705:23cflemingYeah, I changed the artifact ID to reduce confusion.#2019-08-0705:23cflemingHmm#2019-08-0705:23Alex Miller (Clojure team)top-level exclusions would possibly help with this but not there yet#2019-08-0705:23cflemingOk, I think it would be kind of a pain to use due to… right, the lack of that.#2019-08-0705:24cflemingI’ll stick with Maven for now then, thanks!#2019-08-0705:25cflemingActually, if I had the forked jar in a lib dir, and used :deps {org.clojure/clojure {:local/root "lib/clojure-1.10.1.jar"}}, that should work, right?#2019-08-0705:25cflemingi.e. deps will assume that the artifact ID is whatever I tell it.#2019-08-0705:26cflemingAnd that will override the official one.#2019-08-0705:35Alex Miller (Clojure team)there are some issues with overriding a maven dep with a git / local dep#2019-08-0705:35Alex Miller (Clojure team)so I don't think it will work#2019-08-0705:36cflemingOk, thanks. I’ll ditch that idea for now, then.#2019-08-0705:36Alex Miller (Clojure team)that one kind of bugs me. I haven't had a chance to look at it yet#2019-08-0705:36Alex Miller (Clojure team)part of it is the version comparison across dep types#2019-08-0705:37Alex Miller (Clojure team)but on override that shouldn't matter#2019-08-0720:56hkuptyHi folks, has anyone successfully used tools.deps w/ git repos on private repositories?#2019-08-0720:59Alex Miller (Clojure team)Yes#2019-08-0721:12hkuptyAs the maintainer of both tools.deps and tools.gitlibs, I expected a more user-friendly answer from you.#2019-08-0721:13Alex Miller (Clojure team)Sorry, was just answering your question. Was hard to provide any additional detail without any problem description#2019-08-0721:02hkuptyI'm having issues w/ unknown host key if I use {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "a4c3cdd0e48a8a8a"}, :content ("[email protected]")} or I need to provide credentials if I use the https form, which I obviously won't do, since I'm not comiting credentials on the repo#2019-08-0721:02hkuptyIs there any docs on how to get over that?#2019-08-0721:13Alex Miller (Clojure team)can you share the error message or more info about your setup and what you're doing?#2019-08-0721:13Alex Miller (Clojure team)what os are you on?#2019-08-0721:14Alex Miller (Clojure team)have you been through the git stuff at https://clojure.org/reference/deps_and_cli#_procurers ?#2019-08-0721:15Alex Miller (Clojure team)https auth is not currently supported so you'll need to use ssh#2019-08-0721:16hkuptyI'm running from Fedora 30, very basic setup.
The error code I get is:
Execution error (JSchException) at com.jcraft.jsch.Session/checkHost (Session.java:805).\nUnknownHostKey: . RSA key fingerprint is .....
It came from a direct invocation of tools.deps respolve-deps:
(deps/resolve-deps {:deps {'project {:git/url "#2019-08-0721:17hkuptyI'm running the repl from a session where I've added the keys to ssh-agent, so I assume this step is done.#2019-08-0721:18Alex Miller (Clojure team)looks like same as https://clojure.atlassian.net/browse/TDEPS-104#2019-08-0721:18carkhdid you add the host to your .ssh/known_hosts ?#2019-08-0721:18Alex Miller (Clojure team)there's a workaround described at https://codehopper.nl/2018/09/27/clojure-gitlab-deps/#2019-08-0721:18hkuptyHum... I think I know what could be the issue.#2019-08-0721:19hkuptyI'm using ED25519 instead of RSA for gitlab.#2019-08-0721:19ghadissh-keyscan -t rsa,ed25519 >> ~/.ssh/known_hosts#2019-08-0721:20ghadithat will seed UnknownHostKeys ^ into your config#2019-08-0721:21ghadiED25519 on the user identity file is fine (when using an agent) @ingvij
the error you're pasting is an rsa host key, a different kind of key#2019-08-0721:22hkuptyI see. Fine then, I'll try both approaches and I'll post the results back here.#2019-08-0721:22ghadiboth?#2019-08-0721:23hkuptyssh-keyscan and the link @alexmiller posted#2019-08-0721:23ghadithe article @alexmiller posted seems to use the same exact thing I'm describing#2019-08-0721:23hkuptyahh, sorry. I hadn't read the article yet#2019-08-0721:23ghadisince clojure deps.edn is not terminal interactive, the .ssh/known_hosts file needs the host key#2019-08-0721:24hkuptyRight, that makes sense.#2019-08-0721:25ghadi(but this approach is a bit insecure because no one is validating the fingerprints...)#2019-08-0721:25ghadi(then again most people type Yes anyways when ssh prompts you..)#2019-08-0721:27hkuptyShould work fine as a workaround I suppose.#2019-08-0721:29hkuptyThat worked for checking the dependency, but I'm getting a MetadataNotFoundException now. I'll double-check everything, specially the dependency itself. Thanks for the help @ghadi and @alexmiller.#2019-08-0721:31Alex Miller (Clojure team)if you can share more info, might have some idea what that is, but I don't even know what layer of the stack it's from#2019-08-0721:32Alex Miller (Clojure team)just in general, providing information at the beginning lets people help you more efficiently, and we're all doing this for free, in spare time#2019-08-0721:32hkuptyExecution error (MetadataNotFoundException) at org.eclipse.aether.internal.impl.DefaultMetadataResolver/resolve (DefaultMetadataResolver.java:233).\nCould not find metadata org.clojure:clojure/maven-metadata.xml in local (/home/hkupty/.m2/repository)
#2019-08-0721:36hkuptyThe stacktrace from tools.deps:
[[org.eclipse.aether.internal.impl.DefaultMetadataResolver
resolve
"DefaultMetadataResolver.java"
233]
[org.eclipse.aether.internal.impl.DefaultMetadataResolver
resolveMetadata
"DefaultMetadataResolver.java"
191]
[org.apache.maven.repository.internal.DefaultVersionResolver
resolveVersion
"DefaultVersionResolver.java"
233]
[org.eclipse.aether.internal.impl.DefaultRepositorySystem
resolveVersion
"DefaultRepositorySystem.java"
238]
[clojure.tools.deps.alpha.extensions.maven$eval362$fn__365
invoke
"maven.clj"
40]
[clojure.lang.MultiFn invoke "MultiFn.java" 239]
[clojure.tools.deps.alpha$canonicalize_deps$fn__761
invoke
"alpha.clj"
68]
[clojure.core.protocols$fn__8159 invokeStatic "protocols.clj" 168]
[clojure.core.protocols$fn__8159 invoke "protocols.clj" 124]
[clojure.core.protocols$fn__8114$G__8109__8123
invoke
"protocols.clj"
19]
[clojure.core.protocols$seq_reduce invokeStatic "protocols.clj" 31]
[clojure.core.protocols$fn__8144 invokeStatic "protocols.clj" 75]
[clojure.core.protocols$fn__8144 invoke "protocols.clj" 75]
[clojure.core.protocols$fn__8088$G__8083__8101
invoke
"protocols.clj"
13]
[clojure.core$reduce invokeStatic "core.clj" 6828]
[clojure.core$reduce invoke "core.clj" 6810]
[clojure.tools.deps.alpha$canonicalize_deps
invokeStatic
"alpha.clj"
67]
[clojure.tools.deps.alpha$canonicalize_deps invoke "alpha.clj" 65]
[clojure.tools.deps.alpha$expand_deps$fn__787
invoke
"alpha.clj"
184]
[clojure.tools.deps.alpha$expand_deps invokeStatic "alpha.clj" 183]
[clojure.tools.deps.alpha$expand_deps invoke "alpha.clj" 164]
[clojure.tools.deps.alpha$resolve_deps invokeStatic "alpha.clj" 231]
[clojure.tools.deps.alpha$resolve_deps invoke "alpha.clj" 213]
#2019-08-0721:38Alex Miller (Clojure team)org.clojure:clojure/maven-metadata.xml looks weird#2019-08-0721:38Alex Miller (Clojure team)maybe bad format in your deps.edn?#2019-08-0721:39Alex Miller (Clojure team)org.clojure:clojure when should be org.clojure/clojure maybe?#2019-08-0721:42hkuptyI checked the dependency, it's clojure version was RELEASE as it comes from default when you bootstrap a project w/ https://github.com/seancorfield/clj-new#2019-08-0721:43ghadiboo#2019-08-0721:44ghadiyou should choose an explicit version for clojure#2019-08-0721:45ghadi@alexmiller maybe -Sresolve can rewrite symbolic maven things, too#2019-08-0721:46hkuptyI know. this is still on very initial stages, just testing really..#2019-08-0721:46hkuptyThat's why I didn't care too much about it. Just pinned the version.#2019-08-0721:47Alex Miller (Clojure team)re RELEASE, agreed, bleh#2019-08-0721:48Alex Miller (Clojure team)RELEASE is not officially supported in deps.edn#2019-08-0721:48Alex Miller (Clojure team)I've let it continue to work as it's sometimes a good hack, but shouldn't be enshrining it#2019-08-0721:50Alex Miller (Clojure team)-Sresolve is going to go away (move into an external tool)#2019-08-0721:53hkuptyI moved past that error, now yet another one bothering me. A dependency of a dependency is not being resolved, whereas if I run clj on the library I'm trying to import, it works fine:
Syntax error (ExceptionInfo) compiling at (cli/ops.clj:5:1).\nUnable to resolve io.grpc/grpc-core version: [1.17.1]
When I run the repl directly it works, if I call this it fails:
(deps/resolve-deps {:deps {'io.etcd/jetcd-core {:mvn/version "0.3.0"}}}
{})
#2019-08-0721:55hkuptyI don't want to keep you involved in the debugging unless you want to, I just keep posting since I think those might be relevant. I'd understand if you'd rather want me to open an issue.#2019-08-0721:56hkuptywhich, btw, I'll do for clj-new so new projects are not created w/ RELEASE on clojure version.#2019-08-0721:57hiredmanthat is a syntax error, not related to deps at all#2019-08-0721:57hiredmanI would dig in to cli/ops.clj on the given line number, likely something isn't being quoted correctly#2019-08-0722:00hkuptythat particular invocation is this one:
(deps/resolve-deps {:deps {'io.etcd/jetcd-core {:mvn/version "0.3.0"}}}
{})
I believe this might be due to some parsing from pom.xml instead of deps.edn maybe#2019-08-0722:02hiredmanon pardon me, that does come from tools deps, I wonder why it is categorized as a syntax error#2019-08-0722:04hkuptyI assume it's complaining about the dependency declaration syntax, as it expects ns:symbol:jar:version where version is a string, but instead it gets a list.#2019-08-0722:05Alex Miller (Clojure team)@U0NCTKEV8 the syntax thing is a bug, I have a patch pending for that#2019-08-0722:05hiredmanclj -Sdeps "{:deps {io.etcd/jetcd-core {:mvn/version \"0.3.0\"}}}" might give you a clearer error#2019-08-0722:06hiredmanit looks like something depends on an artifact that is not in maven central#2019-08-0722:06Alex Miller (Clojure team)if you're calling this with tools.deps directly, you need to pass it the repos#2019-08-0722:06Alex Miller (Clojure team)clj provides those, but tools.deps does not (automatically)#2019-08-0722:07Alex Miller (Clojure team)there's an example at https://github.com/clojure/tools.deps.alpha/blob/master/API.md#2019-08-0722:08hkuptyI'll check that. However, the thing is I can do
(deps/resolve-deps {:deps {'io.grpc/grpc-core {:mvn/version "1.17.1"}}} {})#2019-08-0722:09hiredmanbut that is a different thing#2019-08-0722:09Alex Miller (Clojure team)b/c you have it locally#2019-08-0722:09hkuptyBut I can't do the same for io.etcd/jetcd-core, which fails on this very same dependency.#2019-08-0722:11hiredmanit does not for me, for me it fails to find io.netty:netty-codec-http2:jar:[4.1.30.Final] in central#2019-08-0722:12hkuptyRight. I got a Syntax error (ExceptionInfo) compiling at (cli/ops.clj:5:1).\nCould not find artifact io.grpc:grpc-core:jar:1.17.1 after moving my ~/.m2 to ~/.m2-bkp`.#2019-08-0722:12hkuptySo I'll make sure to add the extra repositories#2019-08-0722:13hiredmanit would be great if you could try the command line call to clj#2019-08-0722:14hiredmanit looks like the place in deps where that exception is thrown is if the dep is either a version range or an alias like LATEST or RELEASE#2019-08-0722:14hiredmanso my guess something in jetcd is changing how maven attempts to resolve an alias or a range in io.grpc's dependencies#2019-08-0722:15hkuptyI could make it work by adding "central" {...} to :mvn/repos#2019-08-0722:15hkuptythat solves everything#2019-08-0722:16hiredmanhrmm, I am still getting errors, and it has changed to different artifacts not being found in central now 😞#2019-08-0722:16hkupty😱#2019-08-0722:16hiredmanbut I am not using jetcd-core so no big deal#2019-08-0722:18hkupty😅 Thanks to you both for all the help and guidance 🙂#2019-08-0722:28seancorfieldIn light of continued complaints about the use of "RELEASE" in projects created by clj-new ... seancorfield/clj-new {:mvn/version "0.7.7"} pins these to 1.10.1 (for Clojure) and 0.10.0-RC1 (for test.check). You're welcome 🙂#2019-08-0722:29seancorfieldBear in mind that lein new, boot new, and clj-new all look for templates themselves (on Clojars/Central) using "RELEASE" under the hood unless you specify a template version to use (which almost no one does).#2019-08-0722:30ghadithanks @seancorfield#2019-08-0723:14andy.fingerhutThis is not a critical question in any way, but was curious whether folks had any recommendations on creating a deps.edn file for a library that has Clojure-specific and ClojureScript-specific flavors in the same git repo, e.g. core.rrb-vector is the one I have in mind. One thought: create a deps.edn that has almost 0 dependencies when no aliases are specified, and has a :clj alias for Clojure dependencies/paths, and a :cljs alias for ClojureScript dependencies/paths. Second thought: Two separate deps.edn files, except there is only one root directory file named deps.edn available.#2019-08-0723:15andy.fingerhutThird thought: two separate git repos, one for each variant?#2019-08-0802:08Alex Miller (Clojure team)Just fyi on kind of a tangent, core.rrb-vector is a Clojure contrib and the only build officially supported for those for build and deploy is the pom.xml - that has to be the primary#2019-08-0802:09andy.fingerhutUnderstood.#2019-08-0802:09Alex Miller (Clojure team)it's ok for it to also have a deps.edn#2019-08-0802:09Alex Miller (Clojure team)and the tool for variant classpaths is aliases#2019-08-0802:09andy.fingerhutBut the pom.xml is a no-op for ClojureScript, isn't it?#2019-08-0802:10andy.fingerhutoh, maybe not.#2019-08-0802:10Alex Miller (Clojure team)we have other contrib libs that build a single jar usable from either#2019-08-0802:10Alex Miller (Clojure team)data.xml, tools.reader, etc#2019-08-0802:10seancorfieldtools.cli https://github.com/clojure/tools.cli/blob/master/deps.edn#2019-08-0802:10andy.fingerhutI guess the pom.xml controls what ClojureScript source files are bundled up in there.#2019-08-0802:10Alex Miller (Clojure team)yes#2019-08-0802:10seancorfield(tools.cli is pure .cljc so it may not be a good example for you)#2019-08-0802:11Alex Miller (Clojure team)the trickiest part of clj/cljs contribs is testing and probably data.xml is the most advanced example out there#2019-08-0802:14andy.fingerhutI will take a look at data.xml more carefully. I know pretty much all contribs do automated build and at least a subset of their tests automated with results on http://build.clojure.org. Do you know if any of them build and test ClojureScript code?#2019-08-0802:17andy.fingerhutAh, I see in the console log for data.xml that it does run some cljs tests using nashorn JavaScript runtime.#2019-08-0802:20seancorfieldI should have a look at that too since the Maven build for tools.cli only does clj testing, and I do cljs testing manually via deps.edn...#2019-08-0802:22Alex Miller (Clojure team)those are the main ones I can think of that do cljs testing#2019-08-0802:23Alex Miller (Clojure team)"building" is really just including the cljs in the jar so not really a big deal, a bunch of contribs do that#2019-08-0802:23Alex Miller (Clojure team)in a lib like core.async there is a manual cljs test process that we run by hand, but it's not part of ci#2019-08-0802:26andy.fingerhutSo that is a great lead-in to what the pom.xml should ideally do (i.e. automate ClojureScript as well as Clojure tests), and I can look into that. Definitely nicer with a working example to start from.#2019-08-0802:26andy.fingerhutAs far as deps.edn file goes, would it be bad to have one, if its contents conflict with what pom.xml says?#2019-08-0802:27Alex Miller (Clojure team)no, not bad#2019-08-0802:28Alex Miller (Clojure team)many of the contribs also have project.clj files#2019-08-0802:29andy.fingerhutdeps.edn in a Maven-deployed JAR file should either (a) not be in the JAR, or (b) ignored by tools.deps, Maven, etc. even if one is in there?#2019-08-0802:31Alex Miller (Clojure team)I don't think it would be in the jar file#2019-08-0802:31Alex Miller (Clojure team)and it would be ignored by Maven#2019-08-0802:32Alex Miller (Clojure team)there are plenty of examples to check -a bunch have deps.edn files#2019-08-0802:32Alex Miller (Clojure team)tools.deps won't ignore it but presumably that's the point#2019-08-0802:32Alex Miller (Clojure team)if this gets too into the weeds on contrib, I would move it to #clojure-dev#2019-08-0802:33andy.fingerhutUnderstood. If I haven't exhausted you with questions yet, is it short to explain what happens when tools.deps follows a dependency to a git repo, and that git repo has both pom.xml and deps.edn files?#2019-08-0802:34andy.fingerhutIf that is involved, no worries, I can wait.#2019-08-0802:34andy.fingerhut(or experiment)#2019-08-0802:37Alex Miller (Clojure team)deps.edn is first#2019-08-0802:38Alex Miller (Clojure team)that's why I said it matters to tools.dep#2019-08-0802:39andy.fingerhutthx. Keeping some notes separate from here for future reference.#2019-08-0803:03seancorfield@andy.fingerhut When building JARs, you're mostly just going to get src and the META-INF stuff (which includes a pom.xml file) but nothing else from the repo (so no deps.edn file).#2019-08-0803:04seancorfieldWhen you depend on a JAR, the transitive dependencies are figured out from that embedded pom.xml file (as I found out recently with depstar!).#2019-08-0803:05seancorfieldWhen you depend on a git or local repo, tools.deps will use deps.edn if present else pom.xml. If neither are present, you can tell tools.deps to "assume" deps.edn (and it treats the absence as an empty {}).#2019-08-0803:06seancorfield:deps/manifest :deps#2019-08-0803:06seancorfield(see https://clojure.org/reference/deps_and_cli#_dependencies )#2019-08-0803:13andy.fingerhutthx!#2019-08-0815:05hkuptyHey, when using tools.deps on a git dependency (which is a tools.deps project itself as well), can I load an alias of this dependency?
The use-case is: I want to dynamically load a project using tools.deps+dynpath (up to this point I got it running), access its test/ folder and run the tests.
No specific project yet, just trying some ideas.#2019-08-0815:11Alex Miller (Clojure team)no, not supported#2019-08-0815:11Alex Miller (Clojure team)but something under consideration#2019-08-0815:13hkuptyI think that could be solved by adding paths and deps with resolve-deps and make-classpath then.#2019-08-0815:16hkuptyIf I got it right, then calling (make-classpath {...} ["test"] nil) could allow that, right?#2019-08-0815:20Alex Miller (Clojure team)need more words to understand what you mean#2019-08-0815:21Alex Miller (Clojure team)what is "that" and "it"?#2019-08-0816:01hkuptyMy intent of being able to run the tests from a dependency I pull dynamically as explained in the use case above.#2019-08-0816:09Alex Miller (Clojure team)then, yes that should work#2019-08-0816:09Alex Miller (Clojure team)you probably want to also include the source paths though I would assume#2019-08-0816:09Alex Miller (Clojure team)like ["src" "test"]#2019-08-0816:12hkuptyGood. Thanks for the help.#2019-08-0818:06jrwdunhamCan someone here help me to understand how I can rebuild/re-install a local dependency without quitting and restarting Spacemacs/Cider? I was originally declaring a Git dependency and then I changed my deps.edn file so that the dependency was to a local path. That is, my deps.edn started out as {:deps github-myorg/mylib {:git/url "" :sha "6c9cbfa1b1721b3d7789e6ff65a3c01065fd6c54"}} and changed to {:deps mylib {:local/root "/path/to/root/of/mylib"}}.#2019-08-0818:58Alex Miller (Clojure team)you'll have to restart your repl to pick that up#2019-08-0819:06jrwdunhamthanks @alexmiller. I was thinking that would be necessary, but was hoping not#2019-08-0823:12kennyI have a project that depends on a library A via :local/root. Library A depends on com.google.cloud/google-cloud-monitoring. google-cloud-monitoring depends on commons-logging. In library A, I have :exclusions [commons-logging] set on com.google.cloud/google-cloud-monitoring. When I run clj -Stree in my top level project, I still get commons-logging/commons-logging in the tree. If I run clj -Stree in library A, I also get commons-logging. Any idea what is going on here?#2019-08-0823:12kennyGlobal exclusions would solve this btw ^#2019-08-0823:14seancorfield@kenny commons-logging is used by a lot of things -- perhaps something else is bringing it in?#2019-08-0823:14kennyPerhaps. But it is shown under com.google.cloud/google-cloud-monitoring#2019-08-0823:14Alex Miller (Clojure team)where do you see it in the tree?#2019-08-0823:15Alex Miller (Clojure team)did you try :exclusions [commons-logging/commons-logging] ?#2019-08-0823:15kennyWell, several layers deep in there#2019-08-0823:15kennyNo, I can though. I thought they'd be the same.#2019-08-0823:15Alex Miller (Clojure team)I wonder if there's a non-canonicalized matching canonicalized somewhere#2019-08-0823:15kennycom.google.cloud/google-cloud-monitoring 1.78.0
javax.annotation/javax.annotation-api 1.3.2
com.google.cloud/google-cloud-core-grpc 1.78.0
com.google.auth/google-auth-library-credentials 0.16.1
io.grpc/grpc-protobuf 1.21.0
io.grpc/grpc-protobuf-lite 1.21.0
com.google.api/gax-grpc 1.45.0
io.grpc/grpc-alts 1.21.0
org.apache.commons/commons-lang3 3.5
io.grpc/grpc-grpclb 1.21.0
com.google.protobuf/protobuf-java 3.7.1
io.grpc/grpc-stub 1.21.0
com.google.api.grpc/proto-google-cloud-monitoring-v3 1.60.0
com.google.cloud/google-cloud-core 1.78.0
com.google.api.grpc/proto-google-iam-v1 0.12.0
com.google.protobuf/protobuf-java-util 3.7.1
com.google.api/api-common 1.8.1
com.google.http-client/google-http-client 1.30.1
com.google.j2objc/j2objc-annotations 1.3
io.opencensus/opencensus-contrib-http-util 0.21.0
org.apache.httpcomponents/httpclient 4.5.8
org.apache.httpcomponents/httpcore 4.4.11
commons-logging/commons-logging 1.2
#2019-08-0823:16Alex Miller (Clojure team)if you can, I'd try changing the exclusion#2019-08-0823:16kennyCommenting google-cloud-monitoring shows another dep bringing it in. I guess I can follow the trail.#2019-08-0823:17kennycommons-logging/commons-logging does it#2019-08-0823:18kennyGosh, I wonder how many other places have an exclusion that isn't qualified and it should be :thinking_face:#2019-08-0823:19kennyWait, now commons-logging is showing up in another location in the tree.#2019-08-0823:20Alex Miller (Clojure team)yeah, the code doesn't canonicalize the exclusions#2019-08-0823:21kennyChanging the :exclusions in a :local/root dep does not invalidate the clj cache.#2019-08-0823:38Alex Miller (Clojure team)Yeah, that’s a known issue#2019-08-0823:39Alex Miller (Clojure team)-Sforce#2019-08-0823:42Alex Miller (Clojure team)I have a plan to fix it but it’s a pain#2019-08-0823:43kennyGotcha, ok. Out of curiosity, would it be possible to resolve deps in parallel?#2019-08-0823:51Alex Miller (Clojure team)Yes, have thought about it some#2019-08-0823:53Alex Miller (Clojure team)Generally you have most stuff in local repo already so I don’t find it critical#2019-08-0900:20kennyThe piece that is more interesting to me is shortening the time it takes to get a classpath. We have some services where it takes nearly a minute to get a classpath.#2019-08-0902:22Alex Miller (Clojure team)why?#2019-08-0902:23Alex Miller (Clojure team)are you i/o bound (and if so, why?) or is the tree huge? or it something in resolution (cpu bound)?#2019-08-0902:44Alex Miller (Clojure team)a minute seems super long so would like to learn more about why#2019-08-0902:45Alex Miller (Clojure team)also, curious what version of clj you're on? Just reading the code, it seems like the exclusions should get canonicalized in this scenario, at least in current code#2019-08-0902:47Alex Miller (Clojure team)oh, I see why#2019-08-0902:56andy.fingerhutAre there transitive dependency reachability checks done using exponential time algorithms? 🙂#2019-08-0902:57andy.fingerhutProbably a completely irrelevant remark -- I just recall finding and fixing such a thing in tools.namespace years back.#2019-08-0903:03Alex Miller (Clojure team)it's designed to be iterative and should be a single pass over the tree#2019-08-0903:05Alex Miller (Clojure team)but problems like missing a cycle could cause issues#2019-08-0903:07Alex Miller (Clojure team)@kenny I found and fixed the bug with exclusion canonicalization, really a problem with exclusions in any deps.edn, not just transitive. very good catch!#2019-08-0903:07Alex Miller (Clojure team)not going to release it right now but will be in whatever the next release is#2019-08-0903:25seancorfieldI looked at our deps and we certainly have a mix of bare and canonicalized exclusions so it'll be interesting to see what affect that has on us 🙂#2019-08-0903:47Alex Miller (Clojure team)my read of current release is that all bare exclusions are being ignored#2019-08-0903:48seancorfieldThanks. I'll try to qualify all of ours and see if anything falls out...#2019-08-0903:51Alex Miller (Clojure team)might want to do an -Stree before/after#2019-08-0914:02kenny@alexmiller Thanks for the fix!
> are you i/o bound (and if so, why?) or is the tree huge? or it something in resolution (cpu bound)?
Just on a regular computer w/ ssd so not i/o bound. The tree is pretty big but I'd expect a tree this size with any enterprise product. Could be something in resolution that is slow, don't think it's cpu.#2019-08-0914:05Alex Miller (Clojure team)I mean i/o bound in talking to the network to download jars#2019-08-0914:05Alex Miller (Clojure team)not filesystem#2019-08-0914:05kennyOh, no. I think everything is already on disk#2019-08-0914:06Alex Miller (Clojure team)like during that minute, could you grab periodic stack traces, either with ctrl-\ or with jstack, and look at the top of the stack?#2019-08-0914:06kennySure#2019-08-0914:07Alex Miller (Clojure team)so parallel downloads would not help you at all if you're not downloading anything#2019-08-0914:07kennyRight#2019-08-0914:08Alex Miller (Clojure team)another debug thing to do is -Sdeps '{:aliases {:v {:verbose true}}}' -A:v#2019-08-0914:08kennyBTW, the slowness is a "full" refresh, not using the cache.#2019-08-0914:08kennye.g. adding/changing/removing a dep#2019-08-0914:08Alex Miller (Clojure team)if you see pauses in there, that would be suspicious, but otherwise, if you just have a big trace, I'd be interested in seeing that, could dm it to me#2019-08-0914:09Alex Miller (Clojure team)not using which cache? classpath cache? m2 local repo? gitlibs cache?#2019-08-0914:10kennyNot sure. Changing certain deps.edn takes a long time.#2019-08-0914:10Alex Miller (Clojure team)so you're not actively clearing the m2 repo or anything#2019-08-0914:10kennyOh, no.#2019-08-0914:10kennySame for gitlibs and the rest#2019-08-0914:10Alex Miller (Clojure team)so really classpath cache#2019-08-0914:11Alex Miller (Clojure team)that's the only thing stale when you change deps.edn#2019-08-0914:12Alex Miller (Clojure team)and you're on latest clj?#2019-08-0914:12Alex Miller (Clojure team)there were some cycle detection issues that were fixed months ago#2019-08-0914:12Alex Miller (Clojure team)clj -Sverbose for version#2019-08-0914:12kennyThis is from -Sdescribe :version "1.10.1.466"#2019-08-0914:13Alex Miller (Clojure team)yeah, that's latest#2019-08-0914:13Alex Miller (Clojure team)well, I'd love to take a look#2019-08-0914:16kennyFor starters, these messages have reappeared and they take a decent chunk of time:
Downloading: io/grpc/grpc-api/maven-metadata.xml from
Downloading: io/grpc/grpc-core/maven-metadata.xml from
Downloading: io/grpc/grpc-netty-shaded/maven-metadata.xml from
@seancorfield had noticed that a pom from one of my deps (com.google.cloud/google-cloud-monitoring "1.78.0") used a RELEASE version. He suggested explicitly specifying the deps mentioned there. I have done that and they still appear.#2019-08-0914:17Alex Miller (Clojure team)yeah, that's actually an s3 wagon issue, upstream from tools.deps#2019-08-0914:18kennyHe also mentioned it may have something to do with not correctly resolving deps from a parent pom.#2019-08-0914:18kennyOh. Well that could easily shave ~15s off the time.#2019-08-0914:18kennyThose deps are coming from google-cloud-monitoring not Datomic#2019-08-0914:26Alex Miller (Clojure team)I don't have enough info to debug this, would be useful to see deps.edn and the verbose trace above#2019-08-0914:28kennySure. I can send that over. I'll see if I can create a smaller deps.edn first.#2019-08-0914:29kennyJust this will do it:
{:deps {com.google.cloud/google-cloud-monitoring {:mvn/version "1.78.0"}}
:mvn/repos {"datomic-cloud" {:url ""}}}
#2019-08-0914:31Alex Miller (Clojure team)this is starting to ring a bell#2019-08-0914:32Alex Miller (Clojure team)there's a loop in these maven deps iirc#2019-08-0914:37kennyAdding time to the clj calls show I drastically overestimated the time it takes:
real 0m23.962s
user 0m46.126s
sys 0m1.158s
This certainly feels like an eternity when needing to do that many times a day. A big portion of that time is the "Downloading: ..." thing. It would be a huge productivity boost to get that under 5s.#2019-08-0914:38ghadiwhy does repeated downloading happen in your environment?#2019-08-0914:38Alex Miller (Clojure team)that's an issue with the s3 wagon I think#2019-08-0914:38ghadi(jumping into this conversation without reading the backscroll)#2019-08-0914:38kennyDoes it not do that for you?#2019-08-0914:38Alex Miller (Clojure team)I can repro it#2019-08-0914:39Alex Miller (Clojure team)without the datomic repo in the mix, it's about 5 seconds to build a classpath for that#2019-08-0914:40kennyThat s3-wagon thing has always been a nightmare for me. I remember always hitting issues with it back when we used an s3 maven repo. Perhaps a good use case for aws-api? 🙂#2019-08-0914:40Alex Miller (Clojure team)with it, I see about 8-9 seconds#2019-08-0914:41kennyYeah, I'd be curious what one of our large apps takes without the downloading thing.#2019-08-0914:41Alex Miller (Clojure team)time clj -Spath -Sforce#2019-08-0914:41kennyThat will still have the downloading issue.#2019-08-0914:42Alex Miller (Clojure team)yeah, that's the idea :)#2019-08-0914:42kennyOh. I already sent that above haha#2019-08-0918:31ikitommiany success stories of multi-module/mono-repo library setups with deps? have some working lein projects for that, but have now a deps project that needs to be split into parts.#2019-08-0918:46kenny@ikitommi We switched to a monorepo a month or so ago. All internal libraries are :local/root. Makes working in the REPL great. There's a few kinks with our setup though:
- We use CircleCI for CI/CD. There's no support for monorepos with CircleCI. This leads to longer build times - every project runs through its test steps with every push. We recently switched to their new unlimited parallelism plan which has been quite helpful in getting CI time down.
- CI configuration was moved into a set of clojure files because it became far too tedious messing around with YAML with the number of projects we have. This does, unfortunately, mean that CI config needs to be manually generated with a command every time you change the CI clojure files.
- We have a small service diff library that detects when a particular service's code, deps.edn, or :local/root deps have changed. That ensures a service won't get deployed on every push.
- We don't have a great way to have a common deps.edn across all projects. This would be quite useful for things like: global exclusions (when supported), overriding certain library versions, common aliases, etc. Ideally there'd be some way to just pass in N number of deps.edn files to clj and have it merge those in. I use Cursive so Cursive would also need to have some way to select which deps.edn files to use.
- We don't have a good way to run commands across all projects or only in a certain project. For example, I'd like to be able to do something like: monorepo my-service uberjar.
- Different libraries & services run tests with a different set of aliases. Every time we want to run the tests for a project, you need to go to the projects README (or check the CI config) and determine which aliases to use to run the tests. Either the aforementioned "command runner" or a way to combine aliases somehow would make this much better.
Overall, this workflow is far better than having individual repos and constantly needing to restart the REPL when working across projects.#2019-08-0919:04ikitommiThanks @kenny! was hoping for the monorepo kinda script, too lazy to start cooking up own tools right now. In my case, it's a library, going to be split into set of libraries, so would be easy to have same aliases for all. A sample repo would be super awesome.#2019-08-0919:11dominicm@ikitommi you might want to look at edge which does this#2019-08-0919:12dominicmAlso Sean has talked a lot about their setup at world singles#2019-08-0919:17seancorfieldWe have a monorepo with maybe two dozen subprojects, and 90k lines of Clojure.#2019-08-0919:18seancorfieldThe key thing we did was to have a primary deps.edn in a folder and point to that via CLJ_CONFIG (so it "replaces" the user-level deps.edn) and then each subproject has a deps.edn.#2019-08-0919:19seancorfieldWe use :override-deps in the primary file to "pin" versions of libs across the whole repo as needed, as well as provide all the common tooling via aliases.#2019-08-0919:19seancorfieldThe only "tooling" we've built on top of this is a small shell script that can execute multiple clojure commands and knows how to navigate to subprojects when running series of commands.#2019-08-0919:21seancorfieldLike @kenny we use :local/root deps for cross-module deps -- and we have an everything subproject that we can build the deps.edn into from across the monorepo and that's where we usually start our REPL/REBL from.#2019-08-0919:22kennyHmm that's a good idea! Cursive doesn't support CLJ_CONFIG unfortunately. Creating a everything project and starting a nrepl from the command line could solve that problem though! Generally it makes sense to have everything on the classpath while dev'ing.#2019-08-0919:23kennyHow does your script know which aliases to use for each project's tests?#2019-08-0919:34seancorfieldbuild set:of:aliases subproject is our shell script. But it can take multiple pairs of aliases/subprojects.#2019-08-0919:34seancorfieldand if we need arguments, we can use [ ] to wrap them, so build uberjar api [ run ci-ftp api #2019-08-0919:43kennyHow do expose build? Is it a script at the root of the repo? Do you have devs add to PATH?#2019-08-0919:55seancorfieldWe have a <repo>/build/bin folder containing scripts. Devs can either add it to their path or just run the scripts directly.#2019-08-0919:56seancorfieldI mostly work in the build folder but I have build/bin/build symlinked into my ~/bin folder for convenience. Other stuff I run with ./bin/<script>#2019-08-0919:57seancorfieldBetween docker compose and two git clone commands, a dev can be set up "immediately" (assuming they have an OpenJDK8 installed).#2019-08-0919:58seancorfieldIf they need to work on our legacy apps, there's one more git clone to run.#2019-08-0919:59seancorfield(one of those repos is for semi-static tooling, which is where we run docker compose -- for Redis, Elastic Search, Percona/MySQL, and a custom search engine we use)#2019-08-0920:11nwjsmithI’ve been using this trick to “bless” <repo>/bin path additions: https://thoughtbot.com/blog/git-safe#2019-08-0920:17seancorfieldInteresting little trick!#2019-08-0922:22kennyDoes CLJ_CONFIG have to be an absolute path?#2019-08-0922:23kennyI think it doesn't. I thought it wasn't working for a sec.#2019-08-0922:23seancorfieldNope. We use "../versions" in our script.#2019-08-0922:23kennyHow do you deal with not knowing where the build script is run from?#2019-08-0922:24seancorfieldCLJ_CONFIG=../versions clojure -A:defaults:<task> <args>
:defaults pulls in all the overrides etc from versions/deps.edn#2019-08-0922:24kennyFor example: if I have a build/bin/build and I run it from build/bin, I need to know to set CLJ_CONFIG to ../../versions. If I run it from build, I need to set CLJ_CONFIG to ../versions.#2019-08-0922:25seancorfielddirname $0#2019-08-0922:26seancorfieldThen you can work relative to that.#2019-08-0922:26kennyOooo, I didn't know about $0#2019-08-0922:27seancorfield(although we assume certain filesystem paths are the same on all dev/test/prod images so some of our scripts take advantage of that -- and devs just add a symlink to wherever they decided to put stuff)#2019-08-0922:50kennyGiven you symlink build, dirname $0 will return a path to wherever the symlink is. How do you deal with that?#2019-08-0922:51kennyPerhaps readlink?#2019-08-0923:20seancorfieldYes, you can use readlink $0 to get the actual file location (it exits with a non-zero status if the argument is not a symlink).#2019-08-0923:21kennyYou must then also deal with Linux/Mac platform differences with readlink 😵#2019-08-0923:22seancorfieldLike I said above, we also assume certainly filesystem paths to make our lives easier 🙂#2019-08-0923:23seancorfield(partly because we have a shell script in the main repo that a new dev can download and it does most of the env setup for them, including git cloneing repos to specific places and setting up symlinks and handling the initial Mac/Linux differences)#2019-08-0923:24seancorfieldBut there are plenty of ways to skin that particular kitty.#2019-08-0922:59hkuptyWhat is the difference between:
• (resolve-deps {:paths [p1 p2] ...}),
• (make-classpath ... [p1 p2]) and
• (make-classpath .. nil {:extra-paths [p1 p2]})
All three seem to produce the same result, which is to add the local project's paths p1 and p2 to the classpath.
To further exemplify my question from yesterday, I'm looking for something on the lines of:
{:deps {some/dep {:git/url ... :paths [p1 p2]}}}
Is the example above feasible?#2019-08-0923:31Alex Miller (Clojure team)Those 3 produce the same result but are semantically different#2019-08-0923:32Alex Miller (Clojure team)What’s your goal?#2019-08-0923:33hkuptyI want to add a folder (for example 'test') from a dependency I loaded w/ tools.deps to my currently running project's classpath.#2019-08-0923:34hkuptyThat folder is not declared on the main {:paths [...]} clause on dependency's deps.edn for obvious reasons, it is not the mainline for that project.#2019-08-0923:36hkuptyHowever, as I'm trying to build something like a buildscript CLI interface for a group of projects, I want to dynamically load them using tools.deps and run their tests, or whatever else I might want to do with them, given that not only I can load their dependencies but also add folders (or other aliases from that deps.edn) into this buildscript CLI classpath.#2019-08-0923:46hkuptyThere's a hacky way for me to work around that, which is to hijack the result of (result-deps) through some (update-in (result-deps ...) [dep :paths] conj '/my/hand-crafted/path/p2') before (make-classpath ...). That seems too hacky for me, but I can do that if tools.deps doesn't want to explore further the project structure of a dependency.#2019-08-0923:48Alex Miller (Clojure team)If you’re calling tools.deps programmatically then you’re already in the machine - do whatever you want with the intermediate results#2019-08-0923:52kenny@ikitommi I added a build script to our repo that is similar to the one @seancorfield described. Here's what I ended up with. Our repo is structured like with all projects under projects and this script located at bin/build.#2019-08-1000:03seancorfieldI should probably see if I can synthesize a generic version of our build script and release it as open source...#2019-08-1000:04kennyThat would be super awesome 🙂 I just realized mine doesn't support aliases very nicely...#2019-08-1000:09seancorfieldWe have a whole bunch of WS-specific alias expansions in it right now. build run thing will expand into whatever thing means in our project. So build run database-setup expands to build [ migrate migration photos dev ] which runs the :migrate alias in the migration subproject and then runs the :photos alias in the dev subproject.#2019-08-1000:12seancorfieldWe also have an expand-deps.sh script that looks for :local/root deps in a subproject's deps.edn file and recursively expands the list of subprojects to include all transitive subprojects that are depended on -- so we can say build test-all api and it will expand to build tests environment datamapper worldsingles ... api and that in turn expands to running build test <subproject> for each subproject in turn 🙂#2019-08-1000:13seancorfieldprev=""
curr="
where $clojure is the monorepo root.#2019-08-1000:15seancorfieldAll told, including the WS-specific task alias expansion and the 70 lines of help output, we're talking 350 lines of bash.#2019-08-1000:19kennyHmm, that's a lot of bash haha. I just about hit my tolerance with the wee 16 lines above 😛#2019-08-1000:24seancorfieldI've created some truly monstrous bash-based systems in my career... 1,000's of lines long. I probably wrote my first *nix shell script back in '79 tho'... neckbeard#2019-08-1000:27kennyWriting a small script to do the necessary parsing in plain ol' Clojure only adds 1s overhead 🙂#2019-08-1000:37seancorfieldTrue, I could rewrite all this in Clojure (and shell out to clojure as needed for the repeated invocations). I could make all the WS-specific stuff data driven and external too. Maybe one day :thinking_face:#2019-08-1000:59Alex Miller (Clojure team)@kenny I made some progress on your perf issue today#2019-08-1000:59Alex Miller (Clojure team)There’s some repository cache stuff built into Maven that we weren’t using so I’ve changed that and it makes a big difference.#2019-08-1001:01Alex Miller (Clojure team)There’s still something I don’t understand about why you’re getting that s3 maven download every time, but it’s now only doing it once at least. #2019-08-1001:08kennyAwesome! Is there a way for me to test it out before a release?#2019-08-1001:13seancorfield@kenny Looking at the clojure script, I think the answer is "not easily".#2019-08-1001:16seancorfieldYou'd have to build tools.deps JAR yourself and overwrite the version in the clojure install or hack the clojure script to point to that JAR elsewhere#2019-08-1001:21kennyThat doesn’t sound that bad. Perhaps writing bash makes tasks like that sound easy haha#2019-08-1001:32Alex Miller (Clojure team)You don’t even need to build it#2019-08-1001:33Alex Miller (Clojure team)Just clone the repo#2019-08-1001:35Alex Miller (Clojure team)Then modify your clojure script to put the tools.deps.alpha/src/main/clojure dir at the front of the tools_cp var#2019-08-1001:38Alex Miller (Clojure team)Or you can just wait a day or two for me to test and release it :)#2019-08-1006:05dominicmThere's one downside to not building it since the removal of the system deps.edn, you end up with templated version strings! So you need a hard clojure dependency.#2019-08-1012:39Alex Miller (Clojure team)Ah, true. Well you can build and use the target output, that should work.#2019-08-1015:54Alex Miller (Clojure team)Some tools-deps updates here: http://insideclojure.org/2019/08/10/journal/#2019-08-1204:38onetomis there a definitive guide on how to combine some java sources into your deps.edn clojure project?
like which directories should i put my java sources?
when and how to run the compilation into .class files?
where should i put the resulting class files, so i can import them from clojure?
things like that.#2019-08-1204:39onetomi have some protobuf .java files from a decompiled android app and i would like to use it via clojure repl...#2019-08-1204:42onetomi found this old lein docs:
https://github.com/technomancy/leiningen/blob/master/doc/MIXED_PROJECTS.md
so i can rephrase my question as:
what's the equivalent of
1. :java-source-paths?
2. lein javac?
i don't need clojure :aot, because im not accessing clojure code from java,
plus that feature has a lot of docs available already..#2019-08-1204:47onetomi do need to access the same maven deps from java which i specified in deps.edn though...
i know from the command line i can just do java -cp $(clj -Spaths)...,
but do i need something like this for javac compilation too?
( plus i what i really don't have an idea about is how to make symbols resolve in cursive in java sources; but that i can ask on #cursive )#2019-08-1204:48cfleming@onetom For source paths, you can just use :extra-paths. Once you have that set up, Cursive will just work.#2019-08-1204:49onetomcan i just mix the java source files into the same src/ directory as the clojure files then?#2019-08-1204:49cflemingThere’s no lein javac equivalent in deps. You’ll need to set up aliases for compilation and runtime - compilation will have your Java sources on the classpath, and runtime will have the resulting class files.#2019-08-1204:49cflemingYou can mix them, yes, but your Java source files will be on the runtime classpath then.#2019-08-1204:50cflemingI’d keep them separate if I were you, it’s trivial to do so.#2019-08-1204:50cflemingYou’ll also need a run alias to do the actual compilation, but I’ve not investigated that.#2019-08-1204:51onetomah, i see.
so is there any templates out there which showcase some "recommended/reasonable" setup?#2019-08-1204:51cflemingThere’s no concept of automatically compiling before run with deps, so you’ll need to manually recompile if anything has changed.#2019-08-1204:52cflemingYou could set that up in IntelliJ and just let IntelliJ compile for you, depending on how official or serious your project is#2019-08-1204:52onetomthe built-in Build feature would be intuitive to be used for this purpose#2019-08-1204:52cflemingYes, you can do that. You could just add a Build Project before-run step to your run configuration.#2019-08-1204:53cfleming#2019-08-1204:54cflemingSo to set that up set up Java compilation as normal, and add an alias to your project which adds the compile output path to your classpath.#2019-08-1204:54onetomthanks a lot! i will try it.
i hope i don't have to pull in some build tool for this simple java compilation 🙂#2019-08-1204:55cflemingAdd another alias with your extra source paths, then select it for project sync, that will mean that Cursive will add that path as a source path for working with the sources: https://cursive-ide.com/userguide/deps.html#working-with-aliases#2019-08-1204:56cflemingThen in your run config, run with deps and use your runtime alias to add the compilation output to the classpath.#2019-08-1204:56cflemingAnd you should be good to go 🙂#2019-08-1204:57onetomi only understood half of it, but haven't seen your deps.html documentation yet, so it will be clear afterwards, im sure#2019-08-1204:57cflemingOk, ask over in #cursive if you have problems, it’s more of a Cursive thing than a deps thing.#2019-08-1204:58onetombtw, i've just upgraded to IntelliJ 2019.2.1 Preview last night and indeed Cursive deps support works without issues with it so far!#2019-08-1204:59cflemingYes, they fixed a few problems there, but there are still more. I’m actually in the process of migrating the deps support off the IntelliJ API I was using, and instead basing it on the lein support, so I’ll be insulated from all their platform issues.#2019-08-1211:01PBI've noticed since moving to deps.edn my log lines no longer show up in my emacs repl buffer. Anybody know why this is?#2019-08-1211:05kszabohttp://insideclojure.org/2019/08/10/journal/#toolsdeps#2019-08-1211:05kszabo> And there have been some complaints that tools.deps includes slf4j-nop as a dep. slf4j is a pluggable logging framework and the nop logger is a “don’t log” plugin, which takes away the choice for library consumers to plug as they wish. tools.deps will remove that dependency but it will still be included in the clj installation.#2019-08-1211:14PBThank you! So I should just be able to add the dependency and it should log again?#2019-08-1213:21Alex Miller (Clojure team)just using clojure to build a classpath should not affect the clojure program you run with the classpath after that (as tools.deps is not included in that).#2019-08-1213:21Alex Miller (Clojure team)and that's how I'm interpreting your problem statement at the top#2019-08-1213:22Alex Miller (Clojure team)it's possible that cider is also relying on tools.deps for something and that is affecting your program somehow, but I don't know enough about how or whether cider uses tools.deps to know if that could be related#2019-08-1213:24Alex Miller (Clojure team)I would start by opening an issue or asking a question in #cider before jumping to the conclusion that this is related#2019-08-1211:05kszabohttp://insideclojure.org/2019/08/10/journal/#toolsdeps#2019-08-1411:02Lone RangerSo I'm looking for a command line solution to handling generic tooling tasks such as "bump version number", or "modify deps.edn to add something to :deps" ... do we just write our own scripts to do that sort of thing or is there a canonical solution?#2019-08-1411:05dominicmhttps://github.com/clojure/tools.deps.alpha/wiki/Tools#versioning-and-releasing should help you 🙂#2019-08-1411:06dominicm(and lots of other useful tools there too)#2019-08-1411:06Lone RangerAlso I'm still really curious what the heck a build.clj file is, such this:
(require '[cljs.build.api :as b])
(b/build "src"
{:output-dir "out"
:output-to "out/main.js"
:optimizations :none
:main 'example.core
:install-deps true
:npm-deps {:react "15.6.1"
:react-dom "15.6.1"}})
reference here java -cp ls ../../lib/*.jar | paste -sd ":" -`:../../src/main/cljs:../../src/main/clojure:src clojure.main build.clj`
with tools-deps you can do this though: clojure build.clj#2019-08-1411:11Lone Ranger2017 was rough 😬#2019-08-1411:12dominicmprobably why so many used lein#2019-08-1411:19Lone RangerIs there a place in deps.edn where we put license/version info or does that go somewhere else now?#2019-08-1411:20dominicmsomewhere else. pom.xml like before would be fine I think.#2019-08-1411:22Lone Ranger:thumbsup:#2019-08-1415:31dominicm@alexmiller in tools.deps.alpha.reader, is it worth having the user-deps locator as a separate function? In pack the issue has been raised to include part of the user deps.edn which seems acceptable. That part is just for including maven repo settings which are user-local.#2019-08-1415:33Alex Miller (Clojure team)Sure#2019-08-1415:35dominicmwill open a jira#2019-08-1506:38murtaza52is there any tooling for auto completion built on top of the tools.deps repl (started using the clj command) ?#2019-08-1506:47seancorfieldThe Rebel Read line tool might be what you're looking for. There is an alias for it in my dot-clojure repo deps.edn#2019-08-1510:11murtaza52thanks going through it - https://github.com/seancorfield/dot-clojure#2019-08-1616:18zaneThere is also lots of editor-specific tooling for auto-completion that plays well with tools.deps-based projects.#2019-08-1808:47murtaza52is there a lein new x equivalent for tools.deps#2019-08-1808:57dominicmClj-new#2019-08-1808:58andy.fingerhutSee here for a list of that and other tools.deps related tools: https://github.com/clojure/tools.deps.alpha/wiki/Tools#2019-08-1810:26murtaza52thanks guys#2019-08-1813:08murtaza52going through the docs, not able to understand the significance of the -A alias, when should it be used and when not ? Bcoz it seems it can do what all other aliases can do and vice versa, so either only it is needed or others are needed.#2019-08-1813:26Alex Miller (Clojure team)Generally you can just use -A#2019-08-1813:27Alex Miller (Clojure team)There are some weird cases where you might only parts of an alias to take effect#2019-08-1814:37murtaza52thanks, that explains it#2019-08-2010:33Lone Rangeris there any way that you can specify extra deps on the command line without using an alias?
I've developed a template to help my coworkers quick-start on clojurescript but explaining to them how to add an alias for clj-new to their .clojure/deps.edn is just one more annoyance for them to figure out. I'd like to be able to do something like
clj -m clj-new.create --middleware "[seancorfield/clj-new]" some.example
#2019-08-2017:43souenzzowhy not clj -Sdeps '{:deps,{seancorfield/clj-new,{:mvn/version,"0.5.5"}}}' -m clj-new.create some.example#2019-08-2022:01Lone Rangerthat's it!#2019-08-2010:34Lone Rangeror perhaps read a deps.edn from stdin that I could have them just paste in via file descriptor?#2019-08-2010:37Lone Rangerclj -i -m clj-new.create some.example <(cat << eof
{:aliases :new {:extra-deps {seancorfield/clj-new
{:mvn/version "0.5.5"}}
:main-opts ["-m" "clj-new.create"]}
eof
)
something like that would also work but I haven't been able to figure that one out yet either#2019-08-2010:43vlaaadit's worth explaining IMHO: there is no magic, everything is simple and configurable#2019-08-2011:28Alex Miller (Clojure team)-Sdeps lets you add an extra deps.edn on the command line#2019-08-2019:05Lone Rangerdoes anyone have an example of the -Sdeps syntax? Is a string?#2019-08-2019:07lispycloudsclj -Sdeps "{:deps {clj-http {:mvn/version \"3.10.0\"}}}" ?#2019-08-2019:07Lone Rangerthe command I'm hoping for is
clj -Sdeps '{seancorfield/clj-new {:mvn/version {"0.5.5"}}' -m clj-new.create project.name#2019-08-2019:07Lone Rangerahhh ok#2019-08-2019:07Lone Rangerthanks I wasn't sure the exact syntax#2019-08-2019:17Alex Miller (Clojure team)it is literally the text of a deps.edn file#2019-08-2021:44Lone Rangeroutstanding ... exactly what I needed, works exactly as advertised, thank you!#2019-08-2022:32cflemingI’m having a problem loading a namespace which requires t.d.a, using version 0.7.535. I’m getting the following error:
Syntax error compiling at (clojure/tools/deps/alpha/reader.clj:98:5).
No such var: io/read-edn
#2019-08-2022:32cflemingThis seems weird, because io/read-edn does exist.#2019-08-2022:33cflemingI’m just running a standard REPL, classpath looks ok to me.#2019-08-2022:35cfleming( "clojure/tools/deps/alpha/reader.clj")
=>
#object[java.net.URL
0x2bf95d38
"jar:file:/Users/colin/.gradle/caches/modules-2/files-2.1/org.clojure/tools.deps.alpha/0.7.535/11de58e45ac07ea473f2322f335ef6d162dffde9/tools.deps.alpha-0.7.535.jar!/clojure/tools/deps/alpha/reader.clj"]
( "clojure/tools/deps/alpha/util/io.clj")
=>
#object[java.net.URL
0x2b2beaaa
"jar:file:/Users/colin/.gradle/caches/modules-2/files-2.1/org.clojure/tools.deps.alpha/0.7.535/11de58e45ac07ea473f2322f335ef6d162dffde9/tools.deps.alpha-0.7.535.jar!/clojure/tools/deps/alpha/util/io.clj"]
#2019-08-2022:39hiredmanmaybe check (get (ns-publics ') 'read-edn)#2019-08-2022:39hiredmanthe namespace could have been created somehow and added to the lib-map, without loading the source file, stopping require from actually loading it#2019-08-2022:39cfleming(get (ns-publics ') 'read-edn)
=> nil
(ns-publics ')
=>
{printerrln #',
write-file #',
slurp-edn #'
#2019-08-2022:40cflemingAll the vars are there except that read-edn#2019-08-2022:42hiredmanno install-deps either (not sure if that is in 0.7.535, I am looking at master)#2019-08-2022:42cflemingNo, I don’t see that in the 0.7.535 rev#2019-08-2022:44cflemingI can’t see anything in the source doing anything funky like unmapping vars.#2019-08-2022:44hiredmanread the diff wrong, install-deps is a different namespace#2019-08-2022:45cflemingIf I just restart the REPL and then require http://c.t.d.a.u.io directly, then the publics are wrong right off the bat.#2019-08-2022:46cflemingIs there a good way to diagnose where the ns is actually being loaded from?#2019-08-2022:48hiredmanyou could change the require function to print debugging information(a stacktrace, maybe clojure.lang.Compiler/LOADER, but I guess that is just a dynamic classloader) every time it is called#2019-08-2022:51hiredmanyou could also see what (.getResource (RT/baseLoader) "clojure/tools/deps/alpha/util/io.clj") returns, maybe check for aot'ed classfiles too#2019-08-2022:51cfleming(.getResource (RT/baseLoader) "clojure/tools/deps/alpha/util/io.clj")
=>
#object[java.net.URL
0x37339a
"jar:file:/Users/colin/.gradle/caches/modules-2/files-2.1/org.clojure/tools.deps.alpha/0.7.535/11de58e45ac07ea473f2322f335ef6d162dffde9/tools.deps.alpha-0.7.535.jar!/clojure/tools/deps/alpha/util/io.clj"]
#2019-08-2022:51cflemingThe same as before, looks like it’s coming from the right jar.#2019-08-2022:52hiredmanyou could also look in the jar and see if has the right file contents 🙂#2019-08-2022:53cflemingIt does 🙂#2019-08-2022:58hiredmansomething like (alter-var-root #'require (fn [orig]
(fn [& args]
(println (.getResource (clojure.lang.RT/baseLoader) "clojure/core.clj"))
(apply orig args))))
with the funky namespace's resource name instead of clojure/core.clj and then requiring the namespace might show something#2019-08-2023:00cfleming(alter-var-root #'require (fn [orig]
(fn [& args]
(println (.getResource (clojure.lang.RT/baseLoader) "clojure/tools/deps/alpha/util/io.clj"))
(apply orig args))))
=> #object[user$eval1256$fn__1257$fn__1258 0x1a272414 "#2019-08-2023:01cflemingI’m not sure why require would be called twice there.#2019-08-2023:03hiredmanthat may be a clue#2019-08-2023:03cflemingAnd if I use:
(alter-var-root #'require (fn [orig]
(fn [& args]
(println (slurp (.getResource (clojure.lang.RT/baseLoader) "clojure/tools/deps/alpha/util/io.clj")))
(apply orig args))))
then the correct file contents are printed out.#2019-08-2023:03hiredmanI think I've seen that behavior when the namespace in question has already been loaded (why that results in two require calls I have no idea)#2019-08-2023:03cfleming(twice)#2019-08-2023:04hiredmanin a fresh repl before requiring anything maybe try the ns-publics call for the namespace again#2019-08-2023:05cflemingIn a plain clj -Adeps REPL, where the publics are correct, it’s also printed twice.#2019-08-2023:05cflemingYeah, (find-ns ') returns nil in a fresh REPL.#2019-08-2023:09cflemingWeirdly though, requiring something like clojure.zip only prints once.#2019-08-2023:09Alex Miller (Clojure team)-Adeps loads an uberjar#2019-08-2023:09Alex Miller (Clojure team)That won’t play nice with the lib jar#2019-08-2023:09cflemingNo, I was just using -Adeps without a deps.edn to test in a clean REPL.#2019-08-2023:10cflemingi.e. to compare to what I was seeing in the problematic one.#2019-08-2023:11Alex Miller (Clojure team)Nothing unusual about how this ns is defined or loaded afaik #2019-08-2023:11cflemingNo, I can’t see anything.#2019-08-2023:13hiredmancould you have some aot classfiles somewhere on the classpath, if they have a newer last modified they will be loaded instead of the clojure source#2019-08-2023:14cflemingI’m wondering that. I have the debugger set up, and Compiler.load() is never called.#2019-08-2023:14Alex Miller (Clojure team)I don’t think that fn was added till 541#2019-08-2023:16cflemingUgh, that was the problem.#2019-08-2023:17cflemingRogue AOT classes sitting around.#2019-08-2023:17cflemingSorry for the noise.#2019-08-2023:17cflemingThanks for the help!#2019-08-2023:18cflemingI’m not sure why there would have been AOT classes for the .io namespace but not the one calling it, though. I guess it must have been a previous transitive compilation.#2019-08-2120:44Alex Miller (Clojure team)clj 1.10.1.469 and tools.deps.alpha 0.7.541 released...#2019-08-2120:44Alex Miller (Clojure team)- FIX: exclusions were not canonicalized and could fail to match and exclude
- PERF: cache Maven session and use Maven session cache
- FIX: Remove slf4j-nop as dependency of tools.deps.alpha
#2019-08-2120:47Alex Miller (Clojure team)if you're a user of tools.deps.alpha, you can now define Maven session boundaries around your use of tools.deps.alpha by wrapping clojure.tools.deps.alpha.util.session/with-session around one or more calls into tda (see https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/script/make_classpath.clj#L58 for example)#2019-08-2120:48Alex Miller (Clojure team)you may also need to add slf4j-nop as a dep if you were previously relying on getting it via tools.deps.alpha#2019-08-2123:36seancorfield@alexmiller Under what circumstances might I expect with-session to make a difference?#2019-08-2123:37seancorfieldI just updated clj-new to use the latest t.d.a and wrapped the pair of t.d.a calls in that, but saw pretty much no difference in time spent generating a template (from an empty M2 repository each time). (session/with-session
(-> (deps/resolve-deps deps resolve-args)
(deps/make-classpath (:paths deps) {})
(str/split (re-pattern java.io.File/pathSeparator))
(->> (run! pom/add-classpath)))))
#2019-08-2123:49Alex Miller (Clojure team)It will only help if you’re talking to repos of course#2019-08-2123:50Alex Miller (Clojure team)But in particular it makes a difference taking to s3 as it’s slow#2019-08-2123:51Alex Miller (Clojure team)For building a large classpath (without cp cache), when talking to s3 it was as much as 50% better#2019-08-2123:51Alex Miller (Clojure team)And in other cases, no observable difference#2019-08-2123:54seancorfieldAh, OK. So it's harmless at worst, and might be helpful for some small class of clj-new users...?#2019-08-2203:21Alex Miller (Clojure team)yes, should be no worse, and occasionally much better#2019-08-2221:42danielcomptonhttps://ask.clojure.org/index.php/8440/equivalent-of-leiningens-managed-dependencies-in-deps-edn#2019-08-2221:46dominicmAfk, but default deps come to mind#2019-08-2221:54danielcomptonThanks! :override-deps also look promising#2019-08-2221:57dominicmThat sounds right too#2019-08-2221:57dominicmI think the downside is the alias requirement#2019-08-2221:59danielcomptonYeah, was just going to say that#2019-08-2222:00danielcomptonDoesn't communicate it in the same way that managed dependencies does#2019-08-2222:02Alex Miller (Clojure team)default-deps is addressing the same use case as dependency management but there is still work to do to make it actually useful#2019-08-2222:29danielcomptonCan you use it at the root of a deps.edn or does it require an alias?#2019-08-2223:05Alex Miller (Clojure team)Alias #2019-08-2223:42seancorfield@danielcompton I added an answer explaining how we use :override-deps at World Singles Networks -- because we use it extensively!#2019-08-2223:43danielcomptonThanks to both of you! This feels like a great use-case for Ask Clojure to me#2019-08-2223:45seancorfieldhttp://ask.clojure.org is my home page in my browser 🙂#2019-08-2316:40aaelonyI shared this link at work and the feedback was that it is a PHP site...#2019-08-2316:50seancorfieldAlex has addressed that already. It is an off-the-shelf solution that is easy to maintain/administer.#2019-08-2316:50seancorfieldAnd provides all the functionality needed for the task. There is no Clojure/Script equivalent to use (but folks are welcome to write such things and offer them up to Cognitect/the public).#2019-08-2317:07aaelonymakes sense to me#2019-08-2314:53astrasheIs there an example anywhere that shows how to use deps.edn with a local java library? I need a local java library to deal with a protobuf#2019-08-2314:59andy.fingerhutIf the Java library is compiled to a JAR file, then you should be able to add it as a local dependency using something like this: {:deps {somejavalib {:local/root "/path/to/the/jar/file.jar"}}} If you search for ":local" on the reference doc page for deps, you should find that, too: https://clojure.org/reference/deps_and_cli#2019-08-2315:00astrashe@andy.fingerhut Thanks! The JAR file seems like the way to go#2019-08-2315:03vlaaadI wish future clojure releases focused more on java interop, particularly around proxies and class creation. tools-deps is a great value proposition, with the downside being that using .java files in the same project is clumsy. I really don't want to go back to lein...#2019-08-2315:04bronsahttps://clojure.atlassian.net/browse/CLJ-1255#2019-08-2315:21Alex Miller (Clojure team)I think that's unlikely given that Clojure is in philosophical opposition to concrete derivation#2019-08-2315:23Alex Miller (Clojure team)other than https://dev.clojure.org/jira/browse/CLJ-1255 which is in consideration for 1.11#2019-08-2315:33vlaaadI think that's what I need actually, having troubles with proxy not having access to some protected stuff and using reflection without type hinting this..#2019-08-2407:44dharriganIs there any chance that "add-libs" branch on tools deps can be rebased from master so that the latest changes on master for the just released version can be brought in? The "add-lib" functionality is really really useful to have (along with the removal of the slf4j-nop library....)#2019-08-2417:00just.sultanovAlso waiting for these changes. I had to select a separate library with small edits for my needs. If interested, https://github.com/just-sultanov/clj-deps#2019-08-2417:03seancorfield@U064X3EF3 is usually pretty good about merging master back to the add-lib branch -- esp. once we add-libbers ask for it 🙂#2019-08-2421:18Alex Miller (Clojure team)I merged it#2019-08-2421:18Alex Miller (Clojure team)merged master to it that is#2019-08-2500:09seancorfieldYou rock! Thank you!#2019-08-2500:10seancorfield@U11EL3P9U @U1EQNSHL4 SHA 148cab8f154711f8495bd632ebdc53a14dff7f09 to save you both going to look it up. I'm updating my dot-clojure repo right now...#2019-08-2506:44dharriganw00t w00t!#2019-08-2510:40just.sultanovthank you @U04V70XH6 @U064X3EF3#2019-08-2510:53just.sultanovIn my current project, I need to add libraries to the classpath not only in the REPL at development time. Therefore, I made small changes to the function add-loader-url.#2019-08-2510:56just.sultanov(defn- add-loader-url
"Add URL string or URL to the highest level DynamicClassLoader or URLClassLoader URL set."
[url]
(let [url (if (string? url) (URL. url) url)
thread (Thread/currentThread)
cl (loop [loader (.getContextClassLoader thread)]
(let [parent (.getParent loader)]
(if (instance? DynamicClassLoader parent)
(recur parent)
loader)))]
(if (instance? DynamicClassLoader cl)
(.addURL ^DynamicClassLoader cl url)
(let [loader (.getContextClassLoader thread)
urls (into-array URL [url])
cl (URLClassLoader. urls loader)]
(.setContextClassLoader thread cl)))))#2019-08-2407:44dharrigan🙂#2019-08-2407:45dharriganThank you 🙂#2019-08-2421:04seancorfieldseancorfield/clj-new 0.7.8 -- minor maintenance release: fixes -v / --verbose option (betcha didn't know it was broke, eh?), updates dependencies and pins some new dependencies to avoid weird conflicts in some (Leiningen) templates -- I think that's due to the difference between tools.deps and lein/`boot` in terms of resolving conflicts (gotta love those pesky Jackson libraries!).#2019-08-2618:34seancorfieldseancorfield/depstar 0.3.2 -- encountering an unknown file type in a dependency no longer prevents building a JAR (you'll either get a warning or not, depending on whether the name of the file is on the "excluded" list -- you can see more of what is going on either via the -v option or by enabling debugging).#2019-08-2716:04nathanmarztrying to use a git coordinate on a repo that has .clj and .java source files, but trying to use a namespace from the dependency in a repl fails because the java source isn't being compiled#2019-08-2716:04nathanmarzis there a way to handle this?#2019-08-2716:10Alex Miller (Clojure team)With deps, no#2019-08-2716:10Alex Miller (Clojure team)For projects that require a build step, you’ll need to use an artifact#2019-08-2716:10hiredmantools.deps doesn't provide any built in way to run a build step (like calling javac), I think when it has come up in the past people have talked about calling out to javac, but I am not sure if anyone has wrapped that up into something re-usable#2019-08-2716:13hiredmanand I wouldn't hold my breath for something that solves that in the general case, because you would really need something that knows how to build arbitrary java projects, much easier to rely on built artifacts#2019-08-2818:28onetomI'm wondering how can I test running clj for the 1st time in a deps.edn project, without rm -r ~/.m2
I remember in the past I could just do something like env M2_HOME=$(realpath .)/m2 boot repl or something like that, but now this is the only way I found to affect that path:
mvn -Dmaven.repo.local=$(realpath .)/m2 help:evaluate -Dexpression=settings.localRepository
but if I try something similar with clj, it still seem to use the default location (since it's not downloading anything):
rm -r .cpcache; time clj -J-Dmaven.repo.local=$(realpath .)/m2 -e 1
#2019-08-2818:37onetomAnd, btw, I'm still seeing abysmal performance when accessing Maven Central from Hong Kong.
After rm -r ~/.m2, it took 9minutes to download 25MB of jars (just clojure, netty, aleph, compojure).
While my connection is shit and hardly ever goes over 2MByte/s and it's below 1MB/s mostly, 9minutes is unreasonable.
It seems to me that the main reason is the number of requests, which are also made serially (and might not even use keep-alive?)
So I don't think only S3 is slow.#2019-08-2818:40onetomI've also checked the related DNS chain:
. 1735 IN CNAME .
. 21599 IN CNAME .
. 29 IN A 151.101.196.215
and that IP points to San Jose.#2019-08-2818:41onetomI had similar issues with the nixos binary caches. When they switched from AWS CloudFront to Fastly (which is significantly cheaper, plus they actually sponsor them), then the performance from Hong Kong dropped drastically.#2019-08-2818:42Alex Miller (Clojure team)you can set :mvn/local-repo in your deps.edn#2019-08-2818:42Alex Miller (Clojure team)to use a different (presumably empty) local repo dir#2019-08-2818:42onetomso not from the command line or environment?#2019-08-2818:42Alex Miller (Clojure team)you can clj -Sdeps '{:mvn/local-repo "foo"}'#2019-08-2818:42Alex Miller (Clojure team)so that's command line#2019-08-2818:43onetombut im not talking about specifying dependencies#2019-08-2818:43onetomim talking about specifying the maven cache directory#2019-08-2818:43Alex Miller (Clojure team)the local repo is the maven cache directory#2019-08-2818:43Alex Miller (Clojure team)usually it's ~/.m2/repository#2019-08-2818:44onetomindeed, it works#2019-08-2818:44Alex Miller (Clojure team)are you using latest clj?#2019-08-2818:44Alex Miller (Clojure team)1.10.1.469#2019-08-2818:44onetomwhere is this documented?#2019-08-2818:44Alex Miller (Clojure team)clj -Sverbose will tell you#2019-08-2818:46Alex Miller (Clojure team)we are making use of maven session caches as of that version, so would be interested in whether that makes a difference for you#2019-08-2818:46onetomversion = 1.10.1.466#2019-08-2818:46Alex Miller (Clojure team)it should mostly cut down on repeated download of metadata files though, which are pretty tiny#2019-08-2818:47onetomit would make sense to use a more common option for this, like -v; i keep forgetting it 😕#2019-08-2818:47onetomthanks, i will give it a go#2019-08-2818:47Alex Miller (Clojure team)jars are serially downloaded#2019-08-2818:49Alex Miller (Clojure team)there are ways to change that, but, it's somewhat involved#2019-08-2818:50onetomis there an off the shelf solution for setting up a maven repo manager specifically for clojure usage?
the https://maven.apache.org/repository-management.html page mentions a lot of options.
a few years ago i've tried a few of them but i was struggling to make them work.
it should be practically a caching http proxy...
i was really hoping to find some small, turn-key solution, which i would run in the cloud, so i can share it between the office an home#2019-08-2818:51Alex Miller (Clojure team)there are several turn-key products for this, not sure if any qualify as "small"#2019-08-2818:52Alex Miller (Clojure team)I've used Nexus a lot#2019-08-2818:52Alex Miller (Clojure team)but there is also JFrog and Artifactory#2019-08-2818:53Alex Miller (Clojure team)sorry, those are the same#2019-08-2818:53Alex Miller (Clojure team)Artifactory is by JFrog#2019-08-2818:53hiredmanyou can just use a caching http proxy too, you just don't get the nice management interfaces, and the ability to merge sources, etc#2019-08-2818:54onetomlatest nixpkgs contains 1.10.1.469 on its master branch. downloading it now:
uhu:multiboxx onetom$ nix-shell -I nixpkgs=~/nixpkgs/ -p clojure
these paths will be fetched (307.02 MiB download, 990.58 MiB unpacked):
/nix/store/0d8mpzq2dah05xqd6i1c9g02blvsvcnj-bash-interactive-4.4-p23-man
looks like it will take about 10minutes.
i will try a clean download of the same 25MB dependencies afterwards#2019-08-2818:54Alex Miller (Clojure team)there are some benefits to using a maven-aware product I think#2019-08-2818:55Alex Miller (Clojure team)I guess one question I have is why you care what the perf is if you do it just once and then cache it forever?#2019-08-2818:55onetom@hiredman and what would you recommend as a caching http proxy? would it work with https? so for example datomic can be cached too?#2019-08-2818:56onetom@alexmiller these things never happen only once.
i have multiple personal computers, multiple office computers, more and more servers and hoping to add more colleagues too.
these caches should be primed on all those filesystems and having a bad 1st experience doesn't help...#2019-08-2818:56hiredmanI think it could be done, but it is a trade off, if you really want something a lean caching solution a caching proxy would be that, but you will have to invest time getting it working#2019-08-2818:57onetomalso on CI systems it's good to be able to build stuff afresh from time to time at least#2019-08-2818:57Alex Miller (Clojure team)I'm not trying to be flip about it, obviously perf matters, just trying to probe a little closer#2019-08-2818:58Alex Miller (Clojure team)the structure of maven is designed that you don't need to refresh - these are immutable, uniquely versioned artifacts#2019-08-2818:58Alex Miller (Clojure team)CI is one use case where we see this come up#2019-08-2818:58onetommore specifically, i have 2 clojure projects im working on at the moment and i just wanted to know how much are their dependencies#2019-08-2818:58hiredmanat my last job we had a nexus setup that aggregated a number of different repos, and our builds were setup to only check our nexus, and that worked pretty well#2019-08-2818:58Alex Miller (Clojure team)but most CI allow you to keep a maven cache alive#2019-08-2818:59onetomi could have built an uber jar, but my knowledge is outdated on how to do that with t.d
im just getting back to using clojure after a ~2yr break (when i was working with ethereum and js...)#2019-08-2818:59seancorfieldhttps://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging for making uberjars with t.d.a.#2019-08-2818:59Alex Miller (Clojure team)there might also be better maven central mirrors you could use from HK, not sure#2019-08-2819:00Alex Miller (Clojure team)clj (latest) supports Maven mirrors#2019-08-2819:01onetomi looked into the mirrors but most of them doesn't work.
there is a UK one which is not offline but to do a performance test, i wanted to know how can i relocate the m2 cache, since it's not fun to delete my ~/.m2 directory 🙂#2019-08-2819:01Alex Miller (Clojure team)there's one on google storage I know#2019-08-2819:02Alex Miller (Clojure team)<settings>
<mirrors>
<mirror>
<id>google-maven-central</id>
<name>Google Maven Central</name>
<url>https://maven-central.storage.googleapis.com</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
</settings>#2019-08-2819:02onetomnever seen that being mentioned anywhere#2019-08-2819:03Alex Miller (Clojure team)https://cloudplatform.googleblog.com/2015/11/faster-builds-for-Java-developers-with-Maven-Central-mirror.html#2019-08-2819:03Alex Miller (Clojure team)looks like they have an asia-pacific one there too#2019-08-2819:06onetomi was googling for hong kong maven central proxy and came across these:
• https://repo.maven.apache.org/maven2/.meta/repository-metadata.xml
• https://maven.apache.org/guides/mini/guide-mirror-settings.html
• and one more which was also listing ibiblio.{net,org}, which are down#2019-08-2819:07onetomthanks a lot, @alexmiller !
if these work, it will really help with years of frustration 🙂#2019-08-2819:08Alex Miller (Clojure team)please report back, would be very interested to hear if those are any better#2019-08-2819:12onetomhmm... it took one minute to download 18MB, but only the 1st dependency was printed
[nix-shell:/Volumes/Data/lab/multiboxx]$ clj -Sverbose
version = 1.10.1.469
install_dir = /nix/store/0k1f3llrx4aggbgx7rhh70mrardqvrgz-clojure-1.10.1.469-prefix
config_dir = /Users/onetom/.clojure
config_paths = /nix/store/0k1f3llrx4aggbgx7rhh70mrardqvrgz-clojure-1.10.1.469-prefix/deps.edn /Users/onetom/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/1871601414.cp
Refreshing classpath
[nix-shell:/Volumes/Data/lab/multiboxx]$ time clj -e 1
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.pom from
1
real 1m18.496s
user 0m13.404s
sys 0m0.742s
[nix-shell:/Volumes/Data/lab/multiboxx]$ ls m2/
cheshire/ clj-tuple/ commons-io/ me/ riddley/ tigris/
clj-http/ com/ commons-logging/ org/ seancorfield/
clj-soup/ commons-codec/ etaoin/ potemkin/ slingshot/
#2019-08-2819:13onetomah, nvm, i forgot to rm -r m2 .cpcache beforehand#2019-08-2819:17onetom2nd run is printing all the downloaded deps:
[nix-shell:/Volumes/Data/lab/multiboxx]$ rm -r m2 .cpcache/; time clj -e 1; du -hsc m2
Downloading: org/clojure/clojure/1.10.1/clojure-1.10.1.pom from
...
Downloading: org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar from
Downloading: com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.7.5/jackson-dataformat-smile-2.7.5.jar from
Downloading: org/clojure/data.codec/0.1.0/data.codec-0.1.0.jar from
1
real 1m30.756s
user 0m16.289s
sys 0m1.102s
18M m2
18M total
this is magnitudes better!
i guess i should make a counter test with the old version just to confirm im not just experiencing some change in network conditions#2019-08-2819:18Alex Miller (Clojure team)using -Sforce will let you force a classpath recompute (don't need to nuke your .cpcache then)#2019-08-2819:19Alex Miller (Clojure team)is that difference with just the new clj version or the new clj version + mirror?#2019-08-2819:21onetomonly the clj version is different#2019-08-2819:21onetomthis is my deps.edn:
{:paths ["src" "rsc"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/data.csv {:mvn/version "0.1.4"}
clj-soup/clojure-soup {:mvn/version "0.1.3"}
me.raynes/fs {:mvn/version "1.4.6"}
org.clojure/java.jdbc {:mvn/version "0.7.9"}
org.xerial/sqlite-jdbc {:mvn/version "3.28.0"}
com.microsoft.sqlserver/mssql-jdbc {:mvn/version "7.2.2.jre8"}
seancorfield/next.jdbc {:mvn/version "1.0.5"}
etaoin {:mvn/version "0.3.5"}}
:aliases {:test {:extra-paths ["test"]}}
:mvn/local-repo "m2"
:mvn/repos {
"central" {:url ""}
;"uk" {:url ""}
"clojars" {:url ""}
}}
#2019-08-2819:22onetomnext thing after the old version finishes is to try the google mirror#2019-08-2819:23onetomwhich i guess, i can simply do by replacing the :url for the "central" entry to the google mirror's url, right?#2019-08-2819:23onetombut currently im seeing 7-30kbyte/s download rates with the old version, so it will take awhile...#2019-08-2819:24onetomnope, it took only ~3mins:
uhu:multiboxx onetom$ clojure -Sverbose
version = 1.10.1.466
install_dir = /nix/store/m2v1xd7cybi544jyl77w3mzjw1xklw41-clojure-1.10.1.466-prefix
config_dir = /Users/onetom/.clojure
config_paths = /nix/store/m2v1xd7cybi544jyl77w3mzjw1xklw41-clojure-1.10.1.466-prefix/deps.edn /Users/onetom/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/1871601414.cp
Refreshing classpath
^Cuhu:multiboxx onetom$ rm -r m2 .cpcache/; time clj -e 1; du -hsc m2
rm: cannot remove '.cpcache/': No such file or directory
...
Downloading: org/clojure/data.codec/0.1.0/data.codec-0.1.0.jar from
1
real 3m21.764s
user 0m18.844s
sys 0m1.207s
18M m2
18M total
#2019-08-2819:25onetomstill, it's quite significant difference, so thanks a lot for telling me about this enhancement and for implementing these improvements!#2019-08-2819:29onetomclj 1.10.1.466
mirror: https://maven-central-asia.storage-download.googleapis.com/repos/central/data/
uhu:multiboxx onetom$ rm -r m2; time clj -Sforce -e 1; du -hsc m2
...
real 1m40.781s
user 0m21.119s
sys 0m1.290s
18M m2
18M total
#2019-08-2819:32onetom2/3 of the dependencies come from maven central and the rest is clojars:
uhu:multiboxx onetom$ rg -c maven-central-asia 466-deps.txt
71
uhu:multiboxx onetom$ rg -c 466-deps.txt
22
uhu:multiboxx onetom$ wc -l 466-deps.txt
92 466-deps.txt
(it almost adds up :)#2019-08-2819:35onetomclj 1.10.1.469
mirror: https://maven-central-asia.storage-download.googleapis.com/repos/central/data/
uhu:multiboxx onetom$ rm -r m2; time clj -Sforce -e 1; du -hsc m2
...
real 0m44.863s
user 0m13.960s
sys 0m1.059s
18M m2
18M total
#2019-08-2819:35onetomthat consistently seems to be a more than 2x difference!#2019-08-2819:36Alex Miller (Clojure team)pretty big combined change! :)#2019-08-2819:36onetomalthough it's possible that im warming up some CDN caches...#2019-08-2819:37onetomdo you happen to know any asia-pacific clojars mirror by any chance? 🙂#2019-08-2819:37onetomi will try this mirror on my other project too#2019-08-2819:39Alex Miller (Clojure team)I don't believe there is anything region specific. they do have a cdn mirror though#2019-08-2819:39Alex Miller (Clojure team)I have no idea of difference in perf though#2019-08-2819:48onetomthis google mirror understands HTTP2:
uhu:reap onetom$ curl -vso /dev/null --http2 2>&1 | rg -i http2
* Using HTTP2, server supports multi-use
is the HTTP client in clj talking HTTP2?#2019-08-2819:49Alex Miller (Clojure team)that's several layers below the code I'm in, so don't know#2019-08-2819:49Alex Miller (Clojure team)it would not surprise me if the answer was no#2019-08-2819:53Alex Miller (Clojure team)the maven resolver uses org.apache.httpcomponents/httpclient and the version it's pinned on does not look like it supports http 2 (the newest series does though)#2019-08-2819:54Alex Miller (Clojure team)so my strong guess would be: no, but it potentially could#2019-08-2819:59onetomin my other project; 24MB of deps:
clj 1.10.1.469
mirror: https://maven-central-asia.storage-download.googleapis.com/repos/central/data/
[nix-shell:~/lab/reap]$ rm -r m2; time clj -Sforce -e 1; du -hsc m2
...
real 2m53.788s
user 0m24.007s
sys 0m1.858s
24M m2
24M total
#2019-08-2820:06onetomclj 1.10.1.466
mirror: https://maven-central-asia.storage-download.googleapis.com/repos/central/data/
[nix-shell:~/lab/reap]$ rm -r m2; time clj -Sforce -e 1; du -hsc m2
...
real 3m33.490s
user 0m26.609s
sys 0m1.819s
24M m2
24M total
that's less of a difference, but still quite noticable (~20%)
anyway, enough testing for tonight; it's 4am in HK, so i shall sleep.
thanks again for your attention and advice!#2019-08-2904:33onetomit happened again.
when i've added the :mvn/local-repo "m2" to my deps.edn and ran time clj -e 1 for the 1st time, i've only seen some of the dependencies being printed, while it was actually downloading all of them:
Downloading: ring/ring-jetty-adapter/1.6.3/ring-jetty-adapter-1.6.3.pom from
Downloading: crypto-equality/crypto-equality/1.0.0/crypto-equality-1.0.0.jar from
Downloading: potemkin/potemkin/0.4.5/potemkin-0.4.5.jar from
1
real 1m42.094s
user 0m16.879s
sys 0m0.933s
raven:reap onetom$ du -hsc m2
22M m2
22M total
#2019-08-2904:37onetomafter an rm -r m2, on subsequent run, i could see all of the deps printed as they were being downloaded and the size of the maven cache ended up being the same:
raven:reap onetom$ rm -r m2
raven:reap onetom$ time clj -Sforce -e 1
Downloading: org/clojure/data.json/0.2.6/data.json-0.2.6.pom from
...
Downloading: com/google/code/gson/gson/2.7/gson-2.7.jar from
1
real 0m38.062s
user 0m16.855s
sys 0m1.167s
raven:reap onetom$ du -hsc m2
22M m2
22M total
something, somewhere might be caching these artefacts, because the 2nd time was so much faster...
but MAYBE the first time is slower because of the same bug, which causes only a few of the downloaded dependencies being printed!#2019-08-2904:38onetomfor the record, my deps.edn looks like this:
{:paths ["src" "rsc" "java" "classes"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
clj-http {:mvn/version "3.10.0"}
cheshire {:mvn/version "5.1.1"}
org.clojure/data.json {:mvn/version "0.2.6"}
com.squareup.wire/wire-runtime {:mvn/version "3.0.0-rc01"}
com.google.protobuf/protobuf-java {:mvn/version "3.9.1"}
com.google.protobuf/protobuf-java-util {:mvn/version "3.9.1"}
zprint {:mvn/version "0.4.16"}
ring/ring-core {:mvn/version "1.7.1"}
ring/ring-devel {:mvn/version "1.7.1"}
ring/ring-jetty-adapter {:mvn/version "1.6.3"}
aleph {:mvn/version "0.4.6"}
compojure {:mvn/version "1.6.1"}
hiccup {:mvn/version "1.0.5"}
;org.clojure/clojurescript {:mvn/version "1.10.520"}
}
:aliases {:test {:extra-paths ["test"]
:extra-deps {
;clj-http-fake {:mvn/version "1.0.3"}
}}}
:mvn/local-repo "m2"
;
:mvn/repos {"central" {:url
""
#_""}
"clojars" {:url ""}}}
#2019-08-2904:39onetomand my clj version info looks like this:
raven:reap onetom$ time clj -Sverbose -e 1
version = 1.10.1.469
install_dir = /nix/store/bqs5l2mc2jlj5z1a21ix0sz0ak8cvn0l-clojure-1.10.1.469-prefix
config_dir = /Users/onetom/.clojure
config_paths = /nix/store/bqs5l2mc2jlj5z1a21ix0sz0ak8cvn0l-clojure-1.10.1.469-prefix/deps.edn /Users/onetom/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/1871601414.cp
1
real 0m0.858s
user 0m2.045s
sys 0m0.131s
#2019-08-2904:39onetom/Users/onetom/.clojure/deps.edn is all commented out#2019-08-2913:35hoynkIs it possible to just download the dependencies of my project, without running anything? Like with lein deps.#2019-08-2913:36dominicmDoing -Spath will print the classpath, and won't execute anything.#2019-08-2913:37hoynkNice trick, thanks!#2019-08-3021:26andy.fingerhutFYI, I have used deps.edn to start both Clojure/Java and ClojureScript/Node.js REPLs, and run tests for each, for the core.rrb-vector that I have been experimenting with some proposed bug fixes for, here: https://github.com/jafingerhut/core.rrb-vector/blob/proposed-fixes-for-4-issues/deps.edn#2019-08-3021:27andy.fingerhutIt basically uses a bunch of aliases, some prefixed with ":clj-" and others with ":cljs-", for similar functionality in each Clojure flavor. The best alternative I could imagine to that approach would be to have two different directories with different deps.edn files, one for Clojure/Java, the other for ClojureScript.#2019-08-3021:29andy.fingerhutCertainly open to other ideas if anyone thinks of improvements to that approach.#2019-08-3021:33Felipe Cortezshould add-lib work for every lib or are there limitations?
(require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
(add-lib 'cheshire {:mvn/version "5.9.0"})
(require '[cheshire.core :as json])
fails with
Syntax error (ClassNotFoundException) compiling new at (cheshire/factory.clj:57:11).
com.fasterxml.jackson.core.async.ByteArrayFeeder
Syntax error compiling at (cheshire/core.clj:1:1).
namespace 'cheshire.factory' not found
#2019-08-3021:36dominicmDoes it work if you include it in your deps.edn?#2019-08-3021:36Felipe Cortezyep#2019-08-3021:47seancorfieldI ran into something like this with clj-new and the Luminus template... as I recall it's a peculiarity of the Jackson library as far as its dependencies go...#2019-08-3021:47seancorfield(or maybe it's a bug in how Cheshire is packaged?)#2019-08-3021:47seancorfieldI ended up needing to explicitly depend on various Jackson libraries in order to get that to succeed...#2019-08-3021:50Alex Miller (Clojure team)add-lib will interpret new libraries in the context of your existing libs#2019-08-3021:50Alex Miller (Clojure team)so if you already have an existing version of jackson on the classpath, and you add cheshire, it (can't) override the versions you already have#2019-08-3021:51Alex Miller (Clojure team)if cheshire needs a newer version than the one you have, it may not work#2019-08-3021:51Alex Miller (Clojure team)whereas if you had run a deps resolution on the full set of deps, it might have been able to resolve to a better set of deps#2019-08-3021:52Alex Miller (Clojure team)jackson tends to be involved in 75% of dependency conflicts, so it's always a likely culprit#2019-08-3021:52seancorfieldYeah, that sounds like the problem -- Cheshire/Jackson version mismatches.#2019-08-3021:54Alex Miller (Clojure team)this is a totally normal thing that can happen with add-libs#2019-08-3021:54seancorfieldS3 wagon lib pulls in Jackson 2.5.5#2019-08-3021:56Felipe Cortezmakes sense, thanks!#2019-08-3021:57seancorfield@clojurians884 Here's a combination that works: (! 508)-> clj -A:deps -Sdeps '{:deps {com.fasterxml.jackson.core/jackson-core {:mvn/version "2.7.5"}}}'
Clojure 1.10.1
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'cheshire {:mvn/version "5.6.3"})
true
user=> (require 'cheshire.core)
nil
user=>
#2019-08-3021:57seancorfieldNote the Jackson override on the command line. Without that, it won't work.#2019-08-3021:58Felipe Cortezawesome! thanks for the help 🙂#2019-08-3021:58seancorfieldYou don't need to add-lib on the smile sub-lib either (edited to remove that)#2019-08-3021:59seancorfieldIf you want to use the latest Cheshire, you need Jackson 2.9.9: (! 510)-> clj -A:deps -Sdeps '{:deps {com.fasterxml.jackson.core/jackson-core {:mvn/version "2.9.9"}}}'
Clojure 1.10.1
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'cheshire {:mvn/version "RELEASE"})
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
true
user=> (require 'cheshire.core)
nil
user=>
#2019-08-3022:00seancorfield(I would love to excise Cheshire and Jackson from our code base for exactly this reason!)#2019-08-3022:01seancorfield(and the problem is because tools.deps brings in s3-wagon-private which in turn brings in Jackson 2.5.5 so you need an explicit override to make it all work)#2019-08-3022:01Felipe Cortezdo you recommend using data.json instead?#2019-08-3022:04Alex Miller (Clojure team)I've got a spike of some work to use the cognitect s3 lib completely in place of the s3 wagon for tools.deps, would love to shift things over, but needs some work still (and prob streaming support in the s3 lib)#2019-08-3022:05Alex Miller (Clojure team)I think the cognitect stuff uses data.json, which is definitely not as fast as cheshire/jackson (but way less problematic)#2019-08-3022:07seancorfieldIf data.json does what you need -- and does it fast enough -- then, yes, use it instead of Cheshire to avoid the pit of dependency hell that is Jackson 🙂 But, in reality, you'll find it hard to avoid Cheshire as other libraries bring it in too...#2019-08-3022:10seancorfield(having said that, I'm now having a hard time finding a single library that depends on Cheshire! oh how I wish that http://cross-clj.info was still alive!)#2019-08-3022:11seancorfield(it was without - I think but that brings up an online casino that is squatting on the domain!)#2019-08-3022:13Felipe Cortezagain, thanks for the help 😄#2019-08-3103:59sogaiuis the purpose of -Dclojure.libfile=... in the invocation at the end of the clojure bash (powershell) script diagnostic / for debugging?#2019-08-3104:36seancorfieldI'm not seeing anything in the code at first glance that uses it so... I don't know... but it's a good question to ask Alex when he's around (Tuesday, probably, given that it's Labor Day weekend here) @sogaiu#2019-08-3104:40sogaiu@seancorfield tnx!#2019-08-3110:53Alex Miller (Clojure team)It’s to communicate for add-lib or tools like it#2019-08-3111:20sogaiu@alexmiller thanks for the explanation -- i presume it's not considered an api one can rely on (i.e. the ability to access the info via the clojure.libfile system property), is that right?#2019-08-3111:25Alex Miller (Clojure team)It has to be considered optional. The impl in add-lib knows how to use several means to detect the lib set, including classpath introspection#2019-08-3111:26Alex Miller (Clojure team)That whole thing is potentially something we may add to Clojure itself, tbd (which is why add-lib is not final)#2019-08-3111:35cflemingI’d really like to use Deps as a lightweight way to manage deps for projects which aren’t always Clojure, or all Clojure. Currently there’s no way to globally exclude Clojure itself from the dependencies, correct?#2019-08-3111:36cflemingAs I understand it, in current Deps versions the system deps.edn is always read from the bundled copy in t.d.a, and there’s also no way to override that - is that correct?#2019-08-3111:38cflemingI did try this in my deps.edn, but it didn’t work: {org.clojure/clojure {:mvn/version "1.10.1" :exclusions [org.clojure/clojure]}}#2019-08-3111:40cflemingWould there be interest in a patch adding global exclusions support to deps, say using a top-level :exclusions key?#2019-08-3111:44cflemingI also tried: {org.clojure/clojure {:local/root "empty"}} where empty/deps.edn is just {}, also no dice.#2019-08-3114:50kennyGlobal exclusions would be very useful for us too. #2019-08-3115:29Alex Miller (Clojure team)there is currently no way to "remove" a dependency from the top level, and clojure is included as a default dependency, so the practical effect is that clojure is always included#2019-08-3115:29Alex Miller (Clojure team)I don't consider this to be a major problem as the in-scope use case here is running Clojure programs (I don't consider clj to be a tool for generic deps mgmt)#2019-08-3115:31Alex Miller (Clojure team)that said, we do have a ticket for global exclusions and there certainly are times when that would be useful. similarly, it's pretty hard to use a different dep type (like local or git) to override the clojure version (due to issues in overrides of different dep type and version comparison), and that's something that I sometimes need to do#2019-08-3115:33Alex Miller (Clojure team)anyhow, I'm still thinking about the best solution to these#2019-08-3120:46cflemingAnother use case I have is running a Clojure fork.#2019-08-3121:08Alex Miller (Clojure team)Yeah, that’s unintentionally difficult as I discovered recently when I was trying to do something similar#2019-09-0119:37andy.fingerhutIf I run clj -Sdep '{:deps {org.clojure/clojure {:git/url "" :sha "...."}}}' I get Error: Could not find or load main class clojure.main. Is there a way to run arbitrary git commits of Clojure using clj command?#2019-09-0119:38andy.fingerhutI had an actual SHA hex string instead of ..... there, of cousre.#2019-09-0119:43andy.fingerhutThis is with version 1.10.1.469 of clj on Linux#2019-09-0119:44andy.fingerhutWait, I guess that cannot work, since it needs to compile Java code. Never mind.#2019-09-0215:48Drew Verleewhats a short summery of what the .cpcache folder contains?#2019-09-0216:28Alex Miller (Clojure team).cp files (final classpaths) and .lib files (intermediate dep files)#2019-09-0216:29Alex Miller (Clojure team)clj either uses the cached cp file or has to spin a separate jvm with tools.deps to compute the classpath#2019-09-0216:30Alex Miller (Clojure team)It is truly a cache (key is a hash of a bunch of stuff) so it’s safe to rm it#2019-09-0216:49Drew Verleemakes sense, thanks for the explanation Alex.#2019-09-0315:46kennyHas anyone tried compiling the tools-deps with graal to increase performance?#2019-09-0315:48Alex Miller (Clojure team)there's really two stages in clj and the first one (cp computation) should be pretty amenable to it#2019-09-0315:48Alex Miller (Clojure team)it makes the install story (which is already complicated) much more complicated#2019-09-0315:49Alex Miller (Clojure team)and in most cases the result of cp computation is a) IO-bound and b) cached and skipped#2019-09-0315:49Alex Miller (Clojure team)given all that, exploring this is not a high priority for me#2019-09-0316:05kennyPrimarily interested in the first one#2019-09-0317:44Alex WhittHi! I think I asked a while back about using self-signed certs for accessing a self-hosted maven repo over https with tools.deps. (The Leiningen equivalent is adding the path to the cert in :certificates [].) IIRC, at the time that I asked, this was not supported with tools.deps. I can't seem to find anything about this by googling or looking on JIRA. Could anyone give me an update on whether this works these days, or point me in the right direction for how to handle this scenario?
Relevant config in deps.edn:
:mvn/repos {"central" {:url "https://<my-nexus-server>/repository/maven-public/"}}
Everything works fine over http with tools.deps, and even with https with lein. But with https on tools.deps I get "unable to find valid certification path to requested target."#2019-09-0318:07Alex Miller (Clojure team)Nothing has been added in this area#2019-09-0318:08Alex Miller (Clojure team)Could add it to Java’s certs?#2019-09-0318:17Alex WhittThat's a good solution on most OSs, but I'm on NixOS which aims to be "immutable" so the keystore is read-only. I should probably look into adding the ability to add certs to the java keystore declaratively in NixOS.
Anyway, thanks for the update!#2019-09-0317:59Adrian SmithI'm struggling to import a library that I've forked on my own repo details: https://ask.clojure.org/index.php/8507/failure-on-github-import-via-deps#2019-09-0318:00Alex Whitt@sfyire I'm pretty sure that implies that the repo in question does not have a deps.edn (which it doesn't). To my knowledge, if you want to use :git/url, the target must have a deps.edn.#2019-09-0318:01Adrian Smith@alex.joseph.whitt ah thank you#2019-09-0318:03Alex Miller (Clojure team)You can tell it to do that#2019-09-0318:05Adrian Smith@alexmiller how would I do that?#2019-09-0318:05Alex Miller (Clojure team)If you add :deps/manifest :deps to the coord that overrides the autodetection (although you won’t get the deps in project.clj)#2019-09-0318:06Alex Miller (Clojure team)You could add those to the caller though if you don’t have access to modify the project#2019-09-0318:06Alex Miller (Clojure team)If you’ve forked it, then I’d just add a deps.edn#2019-09-0318:06Adrian Smithyeah, that's cool to know though#2019-09-0318:07Alex WhittTIL#2019-09-0318:17seancorfield(I added the above as an answer on the ask site)#2019-09-0318:25Adrian Smiththank you#2019-09-0522:30QuestWhat's the equivalent of lein install for a clj deps.edn library?#2019-09-0522:32Quest(I don't have any packaging library right now -- just looking through the options at https://github.com/clojure/tools.deps.alpha/wiki/Tools for something easy to get running)#2019-09-0522:37andy.fingerhutIf you mean creating a JAR file of Clojure source code, I believe depstar is a good choice.#2019-09-0522:50QuestAppreciate the recommendation, easy enough to get a jar generated with clojure -A:depstar -m hf.depstar.jar northstar-model-0.0.1.jar.
The consume the library artifact from other local projects, I take it I manually install to my ~/.m2/ from here -- i.e. copying pom.xml & northstar-model-0.0.1.jar to ~/.m2/northstar/model/0.0.1/#2019-09-0523:05seancorfield@quest If you have mvn installed, that's probably the easiest way to do it "right".#2019-09-0523:06seancorfield(`mvn` is what I use to deploy JARs built with depstar to http://clojars.org -- I've never tried to to install locally but I'm pretty sure it's a similar command)#2019-09-0523:09seancorfieldmvn install:install-file -Dfile=northstar-model-0.0.1.jar -DpomFile=pom.xml
I believe#2019-09-0523:10seancorfield#2019-09-0523:14QuestHmm, I'll keep looking through maven install & see if I can find something that works. Faking the .m2 autogen files isn't working out#2019-09-0523:15seancorfield@quest The above command seemed to work for me, based on limited testing#2019-09-0523:17QuestYes! I can confirm that it's working. Thanks @seancorfield
clojure -A:depstar -m hf.depstar.jar northstar-model-0.0.1.jar
mvn install:install-file -Dfile=northstar-model-0.0.1.jar -DpomFile=pom.xml
;; include dependency [northstar/model "0.0.1"]
#2019-09-0523:17QuestIt seems like it'd be good to document this somewhere, but I'm not exactly sure where :thinking_face:#2019-09-0523:18seancorfieldI should add it to the readme...#2019-09-0523:18QuestI'll open a PR with my best attempt to capture it, edit it as needed from there? 🙂#2019-09-0523:19seancorfieldSure! Thank you!#2019-09-0523:19seancorfieldDeploy to Clojars is mvn deploy:deploy-file -Dfile=${1}.jar -DpomFile=pom.xml -DrepositoryId=clojars -Durl=#2019-09-0523:20seancorfield(where ${1} is the jar file name -- and this assumes your credentials are in ~/.m2/settings.xml like <settings>
<servers>
<server>
<id>clojars</id>
<username>someperson</username>
<password>topsecret</password>
</server>
</servers>
</settings>
#2019-09-0523:45Questopened https://github.com/seancorfield/depstar/pull/17#2019-09-0615:43TravisI may have over looked this in the docs but is there something kind of the equivalent of lein deps
. Basically trying to make sure all maven deps are downloaded for caching#2019-09-0615:55seancorfield@camechis use clojure -Spath or clojure -Stree (and just ignore the output).#2019-09-0615:56Travissweet, thnx @seancorfield#2019-09-0616:49jjttjj@quest this might not fit your use case but if you're just looking to use code from one of your local projects it's dead easy to just refer to local source directories with clj: :deps {
...
{:local/root "/home/jjttjjj/code/goodlib"}
...
}#2019-09-0616:50Alex Miller (Clojure team)you should use :paths for that#2019-09-0617:50Quest@jjttjj Unfortunately my other projects aren't deps.edn yet -- adapting the re-frame 10x (shadow-cljs) template isn't something I'm ready for yet. Will keep that in mind though#2019-09-0702:15seancorfieldseancorfield/depstar "0.3.3" is available -- no longer uses clojure.xml so you won't get "Illegal Reflective Access" warnings on JDK11 any more!#2019-09-0718:32tianshuI'm trying to create my own clj project template using clj-new. when using it, there's an error:
Failed with: Failed to resolve version for :home/tianshu/develop/ministack/lein-template:jar:RELEASE: Could not find metadata :home/tianshu/develop/ministack/lein-template/maven-metadata.xml in local (/home/tianshu/.m2/repository)
Execution error (ExceptionInfo) at clj-new.helpers/resolve-remote-template (helpers.clj:148).
Could not load artifact for template: /home/tianshu/develop/ministack
Tried coordinates:
[/home/tianshu/develop/ministack/boot-template "RELEASE"]
[/home/tianshu/develop/ministack/lein-template "RELEASE"]
Full report at:
/tmp/clojure-14768781517611096608.edn
what does this mean, why a lein-template and boot-template is needed?#2019-09-0718:36hiredmanyou are somehow declaring a maven dependency with a group-id of /home/tianshu/develop/ministack/ and an artifact-id of lein-template#2019-09-0718:36hiredmanwhich is obviously bogus#2019-09-0718:38tianshuBut it is said I should use qualified symbol or multi-segment clojure symbols.
Project names must be valid qualified or multi-segment Clojure symbols.#2019-09-0718:38tianshuOhh#2019-09-0718:38tianshuI know#2019-09-0718:38tianshusorry#2019-09-0718:40tianshuI give it the wrong template path#2019-09-0718:41seancorfieldclj-new looks for a clj template, then a boot template, then a lein template -- so you can use it to create projects with other tools' templates.#2019-09-0718:42seancorfield(it's based on boot-new which looks for a boot template, then a lein template -- and I based that library in turn on the innards of lein new, and in fact both boot-new and clj-new include some Leiningen code, with permission)#2019-09-0718:47tianshuthanks for the explain!#2019-09-0718:48seancorfieldThe idea behind "Project names must be valid qualified or multi-segment Clojure symbols." is to avoid the everything.core default that Leiningen (and Boot) fall back on if you provide a single segment name.#2019-09-0718:52seancorfieldFor qualified names, it sort of expects you to use something relatively unique like your GitHub username when creating a project: clj -A:new some-template tianshu/example which will create a folder called example but the main namespace in that project would be tianshu.example#2019-09-0718:53seancorfieldOf course you can just go ahead and produce the same thing as Leiningen: clj -A:new some-template example.core 🙂 The folder will be example.core but the main ns will be example.core from src/example/core.clj ... ugh!#2019-09-0718:55seancorfieldAnd finally if you combine them: clj -A:new app tianshu/example.core you'll get a top-level example.core project folder with src/tianshu/example/core.clj for a main ns of tianshu.example.core#2019-09-0807:45tianshuyeah, make sense. thanks very much! @seancorfield#2019-09-0822:10cflemingI’m adding code to Cursive to cache project dependency resolution, and also classpaths for things like running REPLs. Here’s what I’m hashing to create a cache key:
1. Global config (whether using CLI or t.d.a directly, and identifying properties like install path & version in each case), and user and system (if applicable) deps.edn files.
2. Hash of all non-whitespace non-comment content from project deps.edn, and all transitive deps.edns reachable via :local/root.
3. Aliases in use, depending on operation.#2019-09-0822:11cflemingIn the case of the execution classpath, I’ll also add in details from the execution, e.g. main namespace, env vars etc, basically all the info from the run configuration.#2019-09-0822:11cflemingIs there anything else that might affect that I’ve forgotten?#2019-09-0822:20seancorfieldWhat about the user-level deps.edn which may include aliases/extra-deps/override-deps?#2019-09-0822:22seancorfieldAt World Singles Networks, we use CLJ_CONFIG to set the "user-level" to be our repo-wide deps.edn and then each subproject has a deps.edn file. Those two combined then form the set of deps and aliases etc for each given subproject. We use the repo-wide ("user-level") deps.edn file to "pin" versions by having a :defaults alias with all our :override-deps in.#2019-09-0900:16cfleming@seancorfield Ah, yes, thanks - I actually include that already in option #1#2019-09-0900:16cflemingAlthough Cursive doesn’t yet support CLJ_CONFIG - that’s on my TODO list#2019-09-0900:18cflemingI’ve edited the list above to make that clear.#2019-09-0900:19cflemingThose files include the same non-whitespace non-comment content logic.#2019-09-0900:23Alex Miller (Clojure team)did you look at the clojure script? you can see everything included in the hash now#2019-09-0900:24Alex Miller (Clojure team)seems like matching that is best for matching behavior#2019-09-0900:25seancorfield@alexmiller Did you say CLJ_CONFIG support might go away? Is there an alternative in mind where a (large) project/monorepo may want a "base" deps.edn file and then to have a separate deps.edn for each subproject?#2019-09-0900:25Alex Miller (Clojure team)no, I didn't say that#2019-09-0900:25seancorfieldAh, then I perhaps misunderstood you 🙂#2019-09-0900:26Alex Miller (Clojure team)more likely to add something to insert more#2019-09-0900:26seancorfieldOK. Having four deps.edn files being considered would be fine too. System, user, (project/repo -- new!), (sub)project/current dir.#2019-09-0900:27seancorfield(I mostly don't care about the first two in this case -- I really care about the "third" and fourth 🙂 )#2019-09-0900:29cflemingI did look at the script, but a while ago. IIRC it doesn’t include deps file contents, only the config paths, and definitely not the contents of transitive local roots.#2019-09-0900:29cflemingAt least, I couldn’t see where the deps file contents were taken into account.#2019-09-0900:33Alex Miller (Clojure team)yeah it's just the paths and aliases and the -Sdeps contents#2019-09-0900:34cflemingActually, does the -nt test take the file timestamps into account too? My google-bash-fu is failing me.#2019-09-0900:34Alex Miller (Clojure team)"newer than"#2019-09-0900:35cflemingRight, that was what I suspected, but Google wasn’t very helpful.#2019-09-0900:35Alex Miller (Clojure team)so the "stale" calculation is separate from the hash#2019-09-0900:35cflemingRight.#2019-09-0900:36cflemingIn my case I don’t need a location (i.e. what you’re using the hash for), I just need the staleness test.#2019-09-0900:36Alex Miller (Clojure team)so yeah, that's largely timestamp based#2019-09-0900:37cflemingOk, thanks. In my case it’s relatively easy to do the full transitive contents check, so I’ll stick with that.#2019-09-0900:37Alex Miller (Clojure team)well, and assuming the hash finds there is a cp to use#2019-09-0900:38cflemingRight.#2019-09-0900:43cflemingActually, one other question I had: since the system deps file is now read from a resource, am I correct in thinking that for recent versions I only need to pass the user config, and not both?#2019-09-0900:44cflemingI looked at the code a while back, and it looked like the resource will always be put first in the list, so if I also pass the system deps file then that will override the one from the resource, but will presumably be the same?#2019-09-0900:45cflemingIs there any way to tell from e.g. -Sdescribe whether I need to pass the system file or not? The path still appears in the config-files list.#2019-09-0900:48cflemingIf not I can just check the version and choose based on that.#2019-09-0900:52seancorfield@cfleming Looks like you could check :install-dir against the first path in :config-files -- but I will note that t.d.a. has an API for getting the deps.edn files that returns just the user + project level ones (because it has the system one baked in).#2019-09-0901:07cflemingInteresting. It’s potentially tricky to call that API ahead of time when delegating to the CLI.#2019-09-0901:07cflemingI’ll take a look at that though, thanks.#2019-09-0902:06Alex Miller (Clojure team)it wouldn't hurt to include both#2019-09-0903:04cflemingIs there a reason that t.d.a and tools.gitlibs print progress information (“Downloading: xxx” and “Checking out: xxx”) to stderr?#2019-09-0903:20Alex Miller (Clojure team)so you can shunt it off if you don't want to see it#2019-09-0903:22Alex Miller (Clojure team)normally it's silent on stdout unless you do something that expects a printed result, like -Stree or -Spath etc#2019-09-0904:07cflemingAh, ok, thanks.#2019-09-1016:37jjttjjIs there any obvious reason I'd get the message
Skipping paths: resources
after running clj -Spom when resources is in my :paths?#2019-09-1016:38Alex Miller (Clojure team)it means it saw you had that in paths, but there is no canonical way to do that in pom.xml so, it just didn't do anything with it#2019-09-1016:39Alex Miller (Clojure team)in Maven, you typically need to pull in a plugin and do several steps to work with a resources dir#2019-09-1016:40jjttjjstrange, I somehow managed to deploy to clojars on a different computer (with depstar+maven), and don't remember any issues getting resources in the jar#2019-09-1315:48plexusI'm having some trouble using a jar from a private Maven repo (we're using Gitlab's built-in maven repos)#2019-09-1315:49plexusI can install it fine with Maven, and see it in ~/.m2, but when I do clj it seems it still tries to fetch it, Error building classpath. Could not find artifact group:artifact:jar:version in central ()#2019-09-1315:49plexusis this to be expected, or am I holding it wrong?#2019-09-1315:56dominicmI think it will try there, but it should also try from your repo that's configured#2019-09-1320:21vlaaadIf anyone is curious: I've setup a simple testing workflow using github actions and tools-deps: https://github.com/cljfx/cljfx/blob/master/.github/workflows/test.yml#2019-09-1320:25vlaaadCouple of notes:
- there is java 8 by default, so if you don't need a newer one you can omit actions/ action
- use clojure instead of clj, duh#2019-09-1320:27vlaaadI also want to say again how much I love tools-deps. Aliases and their simplicity is amazing: adding workflow-specific aliases is such a breeze, and mixing them in different contexts is very straightforward. Thank you very much cognitect and @alexmiller#2019-09-1320:30vlaaadI have :test alias to add test directory and test-specific deps, I have :runner alias to run tests, I have :headless alias to run cljfx without display. During dev I have :test alias on, I can run tests locally with -A:test:runner, and I can make CI run tests in headless mode with -A:test:runner:headless. So simple clj#2019-09-1320:31gerredhey! as someone doing a lot with github actions too, this is great. I might recommend using some more Docker-based actions rather than shell-based ones, which might make things a little bit faster.#2019-09-1320:38vlaaadBut then I'd need to figure out docker in addition to github actions, I'm too lazy 😛#2019-09-1715:23aviTotally understandable Vlad, but if and when you do want to give it a try, I think GitHub has done a good job making this super simple. Basically you just add a container key to your job definition and Actions takes care of getting the image, starting a container, and then running your job’s steps within that container.
So for example you could add container: clojure:tools-deps to your job definition and then in your steps you wouldn’t need to set up Java nor install Clojure. Presumably this would become faster over time as GitHub does a better job of caching docker image layers. And there are a bunch of variants of the clojure image, e.g. with different OS’s, different JREs, etc.
HTH but feel free to ignore if it’s not the right time!#2019-09-1715:27gerred👍 also happy to help 🙂#2019-09-1715:29gerredit'll even build a repo from a Dockerfile, which adds time but is great for development#2019-09-1715:30aviOoo really? I didn’t know that — that’s excellent! I gotta look up the docs on that. Thanks!#2019-09-1715:30gerredof course. if you use actions/checkout and actions/docker, it's all Dockerfile based so there's pre-steps to build those images.#2019-09-1715:31gerredi'd like to optimize that, but here's an example workflow: https://github.com/kudobuilder/kudo/blob/master/.github/workflows/release-on-master.yml#2019-09-1715:31gerredactions/checkout and actions/docker will get build at the start of the workflow, which you can see here: https://github.com/kudobuilder/kudo/commit/7ee65861db6b426e87618bd44b94b469a0e99802/checks#2019-09-1715:31gerred"Build x..." aren't steps in our workflow#2019-09-1715:31gerredthey just happen#2019-09-1715:32aviAha, that makes sense! It’s kinda naturally intuitive once I unlearn the constraints I learned from CircleCI, wherein for a given job I have to choose between a VM and a container, so there’s no concise way to build an image and then use it.#2019-09-1715:32aviThanks!#2019-09-1715:33gerredof course.#2019-09-1715:34gerredi'm also well integrated with the actions team and the implementation since we're doing a lot kubernetes-wise with this, so if you need any help or support, let me know and I can either help or point you at the right people within github.#2019-09-1715:41aviWill do — thanks!#2019-09-1715:42vlaaadHey, many thanks for the info! I have a question: what problem does docker solve that actions don't? You mentioned that you can have different docker images with different OSs, JREs — isn't it something already solved by actions matrix?#2019-09-1715:43gerredactions in-and-of-themselves are either based on Docker or javascript.#2019-09-1715:43gerredworkflows are made up of a series of actions, which build up matrices, etc.#2019-09-1715:44aviI’m not sure about actions matrix; I haven’t looked closely at it yet. As for using docker in general, it can sometimes make jobs more concise and sometimes, theoretically, faster, because the images might be cached.#2019-09-1715:44gerredyou can also technically just run shell commands on the host runs-on I guess.#2019-09-1715:44gerredbrb, doing a CNCF TOC presentation, I'll be back in a bit.#2019-09-1715:44aviIt can also make it easier to debug/troubleshoot issues in a CI job because you can then use the same exact environment locally to reproduce the situation#2019-09-1715:46vlaaadAh, you mean if I have docker setup at local machine#2019-09-1715:46aviYup#2019-09-1715:46vlaaadGot it, thanks!#2019-09-1715:46gerredand you can think of an action as an individual step (but reproducible as a lego block) in a whole workflow#2019-09-1715:46gerredwith the v2 actions API#2019-09-1320:31gerred(I have a clj-kondo action currently in progress as well)#2019-09-1320:40vlaaadWhat I'd really like to have is a tools-deps action, with .m2 and .gitlibs caching etc. Configurable by aliases and deps.edn in a project.#2019-09-1320:32gerredthis is great though!#2019-09-1719:29avi👋 are there any known gotchyas with using gitlibs via t.d.a in CircleCI? I just switched a lib from Maven coords to git coords; it worked fine locally for me but over in CircleCI the deps were downloaded but then the files couldn’t be found on the classpath during the test run.#2019-09-1719:29aviFailed job is here if it’s helpful to see it: https://app.circleci.com/jobs/github/FundingCircle/fc4-framework/2711#2019-09-1719:37ghadi@aviflax hard to say without seeing what bin/tests-with-coverage is doing#2019-09-1719:37ghadiin general if you have private ssh repos, you'll need to use an ssh-agent#2019-09-1719:37aviclojure -J-Xmx2g -A:test:test/coverage
#2019-09-1719:37aviIt’s not a private repo#2019-09-1719:38aviI was able to reproduce the problem just now locally via Docker so I should 😅 be able to figure it out#2019-09-1719:46aviIt looks like the problem is that when the classpath is generated, there’s an extra segment main at the end; that dir doesn’t exist in the repo hosting this particular library#2019-09-1719:46aviHere’s the cp entry: /home/circleci/.gitlibs/libs/clj-chrome-devtools/clj-chrome-devtools/58a396f54f469e98b5aae92b9357d65434914228/src/main#2019-09-1719:48aviAnyone have any suggestions on why that might be the case? I’ve been looking through https://clojure.org/reference/deps_and_cli and so far haven’t found any clues#2019-09-1719:48ghadiSeems like that lib has an invalid paths inside its deps.edn?#2019-09-1719:48aviah, maybe, I’ll take a look#2019-09-1719:49aviNope#2019-09-1719:49avihttps://github.com/tatut/clj-chrome-devtools/blob/master/deps.edn#2019-09-1719:53ghadiIt needs to specify paths explicitly#2019-09-1719:54aviOhh wait I remember this#2019-09-1719:54ghadi:paths ["src"]. I think there is a bug when it isn't set#2019-09-1719:54avithe dep is inheriting the source path that was specified in my project root#2019-09-1719:54ghadiYup#2019-09-1719:54aviI mean, my project’s deps.edn#2019-09-1719:54aviIIRC this is a known bug#2019-09-1719:55aviI’ll see if I can find the JIRA issue#2019-09-1719:55dominicmMake sure you vote!#2019-09-1719:56aviwill do, if/when I find it#2019-09-1719:56aviso far no dice#2019-09-1719:56vlaaadwasn't it fixed?#2019-09-1719:57aviaha! yes, it was fixed: https://clojure.atlassian.net/browse/TDEPS-52#2019-09-1719:58aviIt must be that the Docker image I’m using hasn’t picked up the fix yet#2019-09-1719:58aviI’m using circleci/clojure:openjdk-11-tools-deps-browsers#2019-09-1719:58avi@ghadi you reported that bug!#2019-09-1719:59ghadiIt's the circle of life #2019-09-1720:01avi😂#2019-09-1720:01aviThanks for the help!#2019-09-1720:07aviAlex commented on the issue:
> Released in clj 1.10.1.458
And I’ve confirmed that the CircleCI image I’m using has 1.10.0.442 installed.
Interestingly, even the official Docker images for Clojure don’t appear to have any variant with a version of Clojure more recent than 1.10.0.442. This is surprising…#2019-09-1720:10Alex Miller (Clojure team)there are no official Docker images for Clojure#2019-09-1720:11gerredi don't even think there should be official Docker images for Clojure in a world of uberjars, to get edgy#2019-09-1720:11aviah, well that explains it#2019-09-1720:11aviwho maintains the images here? https://hub.docker.com/_/clojure#2019-09-1720:11gerredthat's Docker owned, like the company.#2019-09-1720:12aviSo if I think they should be updated, I guess I pester Docker, the company 😬#2019-09-1720:12aviOK thanks all!#2019-09-1720:12gerredyup! if you're interested in custom images and interested in sponsoring some sort of support model, I'd love to chat though.#2019-09-1720:13gerredwe made a #cloud-native channel lately.#2019-09-1720:13gerredso I'm in a process of taking in problems/opinions and trying to distill something#2019-09-1721:11aviNext question:
• I’ve been using Cambada to AOT my code and build an uberjar containing my code and all my dependencies.
• Now that I’ve switched one of my dependencies to a gitlib, Cambada is having trouble building the uberjar.
So: Does anyone happen to know whether there’s an existing solution to this somewhere?
(I mean either some way to continue using Cambada, maybe a fork, etc; or some alternative to Cambada (whether a single tool or a combination of tools) that I might want to consider switching to.)#2019-09-1813:20mike_ananev@aviflax try this fork of Cambada. There are several issues have been fixed. may be your problem is already fixed too.
https://github.com/mikeananev/cambada#2019-09-1813:23aviWill do — thanks!#2019-09-1815:58avi@mike1452 it doesn’t seem to work; it seems to fail with the same error as the root cambada.
I see you have GitHub’s issues feature disabled in your fork but the feature is enabled in the root repo — would you be interested in a bug report opened as an issue in that repo, which we could use to discuss and debug the problem, and to potentially craft a fix to be merged into your fork?#2019-09-1816:24avi@mike1452 never mind, I spoke too soon — I took a closer look at the error output and it was not the same error — it was a different error that I’ve seen before and already have a way to work around. And once I applied that workaround, the uberjar process completed successfully. So your fork does in fact resolve my issue with gitlibs. This is great — thank you!#2019-09-1818:33aviFor anyone who might have a similar question: @mike1452 suggested I try this fork of Cambada: https://github.com/mikeananev/cambada — and it does indeed seem to have fixed the problem I was encountering with gitlibs, AOT, and uberjarring. Thanks Mike!#2019-09-1721:43seancorfield@aviflax Is AOT a requirement?#2019-09-1721:44seancorfieldHere's all the packaging tools available: https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging#2019-09-1721:44aviI guess it’s not a 100% hard requirement but I’d prefer to preserve it. The project is a CLI tool so it needs to start quickly.#2019-09-1721:45aviThanks, I was just reviewing that list. I was just wondering if anyone had an answer off the top of their head. The descriptions of those tools don’t always include lots of details.#2019-09-1721:45seancorfieldI don't know how many of those do AOT although, in theory, you could always do that step separately and then package stuff: -e "(compile 'my.main.ns)"#2019-09-1721:46seancorfieldI am fairly certain that depstar works with git deps and local deps but it does not do AOT.#2019-09-1721:46aviI believe Cambada is AOT compiling all the Clojure code in my project, including in dependencies. I like that.#2019-09-1721:47aviWould calling compile do the same? I’d guess not...#2019-09-1721:47seancorfieldSince AOT is transitive, I suspect just compiling the main ns will get you much the same result...?#2019-09-1721:47aviIt’s worth trying out for sure!#2019-09-1721:47aviGotta get on my bike now, will check back in shortly. Thanks!#2019-09-1721:48seancorfieldYou'll need to ensure classes exists as a top-level folder in your repo and add it to the :paths vector so it's on the classpath.#2019-09-1818:33aviFor anyone who might have a similar question: @mike1452 suggested I try this fork of Cambada: https://github.com/mikeananev/cambada — and it does indeed seem to have fixed the problem I was encountering with gitlibs, AOT, and uberjarring. Thanks Mike!#2019-09-1800:25cflemingHas anyone made a spec for deps files that I could use to check validity?#2019-09-1800:26cflemingIt’s probably not hard to put together, but I wasn’t sure if anything already existed.#2019-09-1800:26Alex Miller (Clojure team)I think there are some in tools.deps#2019-09-1800:26Alex Miller (Clojure team)but I don't actually use them atm, so they might not be totally up to date (but I'd be happy to fix that)#2019-09-1800:26cflemingOk, I’ll check them out, thanks.#2019-09-1800:26Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/specs.clj#2019-09-1800:27cflemingI’d like to make at least a basic check that the file looks good before automatically trying to resolve it on file modification.#2019-09-1800:27Alex Miller (Clojure team)::deps-map in particular#2019-09-1800:28cflemingYou think ::deps-map might not be up to date?#2019-09-1800:28Alex Miller (Clojure team)just eyeballing it, it can't be too far off#2019-09-1800:29cflemingI’m mostly interested validating the input, i.e. the deps file itself, not what t.d.a returns.#2019-09-1800:29Alex Miller (Clojure team)yeah that ::deps-map is a deps.edn file contents#2019-09-1800:29cflemingAh, ok.#2019-09-1800:31Alex Miller (Clojure team)I'm pretty sure I'm not using those specs in the mainline code right now so it's probably as good as any code you aren't using#2019-09-1800:31cflemingOk. I might have a go at converting this to my internal validator, which means I could show errors in the editor too.#2019-09-1800:32cflemingAs well as checking before trying to resolve automatically.#2019-09-1800:33Alex Miller (Clojure team)I don't see anything wrong in those specs. it's missing :sha and maybe one or two other things#2019-09-1800:33cflemingOk, if I see any obvious errors I’ll make a PR#2019-09-1800:33cflemingOr a patch, rather.#2019-09-1800:34Alex Miller (Clojure team)or just tell me and I'll fix it :)#2019-09-1800:34cflemingThese days, when supplying a patch, does it have to be attached to a JIRA? Or can it be attached to http://ask.clojure.org?#2019-09-1800:34cflemingOr that, yes 🙂#2019-09-1800:35Alex Miller (Clojure team)as someone that's supposed to be writing a talk right now, I'm in the market for sources of procrastination :)#2019-09-1800:35cflemingMeh, you still have two weeks, what’s the issue?#2019-09-1801:11Alex Miller (Clojure team)one week!!#2019-09-1802:08cflemingWhat are plane flights for?#2019-09-1800:39cflemingThis is probably more of a question for #spec, but I’m curious about this:
(s/def :local/coord (s/keys :req-un [:local/root] :opt-un [::path]))
#2019-09-1800:39cflemingDoes that mean that :req-un can be used for namespaced keywords as long as the ns is explicitly specified?#2019-09-1800:41cflemingAh, no, ignore me - never mind.#2019-09-1800:42cflemingActually, no - the doc says:
> These variants specify namespaced keys used to find their specification, but the map only checks for the unqualified version of the keys.#2019-09-1800:42cflemingDoes that mean that the spec would accept :root as well as :local/root?#2019-09-1801:02Alex Miller (Clojure team)ha, I was off checking many things and independently discovered this - it's a bug#2019-09-1801:02Alex Miller (Clojure team)I ran it on all the deps.edn in my system and that barfed at me in a few places#2019-09-1801:09Alex Miller (Clojure team)fixed and pushed#2019-09-1801:09Alex Miller (Clojure team)also added the required :sha attr for git coords#2019-09-1802:00Alex Miller (Clojure team)released as tda 0.7.549 if you need it#2019-09-1802:08cflemingGreat, thank you!#2019-09-1803:17Ahmed HassanWhat is :resource-paths (leiningen) counterpart in Tools Deps?#2019-09-1803:17Ahmed HassanIs it :paths?#2019-09-1803:48cfleming@ahmed1hsn Yes - Deps doesn’t make a distinction.#2019-09-1812:47Alex Miller (Clojure team)to be more accurate, the jvm doesn't make a distinction, and thus neither does deps#2019-09-1813:19dominicmI stopped making the distinction in my programs too. I realised it was rather arbitrary.#2019-09-1815:41seancorfield@ahmed1hsn You may find this blog post helpful https://corfield.org/blog/2018/04/18/all-the-paths/ since it talks about the terminology used by lein (and boot) and how that maps onto t.d.a.'s world.#2019-09-2022:05shaun-mahoodFor clj-on-windows, how would I start a socket repl? I'm trying powershell -command clj '-J"-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}""' but not sure if I have the quotes wrong or if there is something else going on. The REPL starts up, but I can't get to the socket repl at all.#2019-09-2022:08seancorfield@shaun-mahood Did you try with comma instead of space in that?#2019-09-2022:09shaun-mahoodOf course not - I never think of the Corfield comma 🙂#2019-09-2022:09seancorfieldAlternatively, just use an alias in your deps.edn (which is how I do it on Powershell).#2019-09-2022:10seancorfield(and it has to have commas in the deps.edn version because of the round-tripping through the .cpcache files and shell etc)#2019-09-2022:15seancorfield@shaun-mahood Did it work with the comma?#2019-09-2022:16shaun-mahoodNo luck yet - I might be putting the commas in the wrong place though#2019-09-2022:18shaun-mahoodI just realized I have to put lein on that machine, though, so I might use that instead - saves me from figuring out how to get Cursive to connect to a socket repl#2019-09-2022:19seancorfieldCursive requires nREPL doesn't it? So you wouldn't want to start a Socket REPL anyway.#2019-09-2022:20shaun-mahoodYeah, I think there's a way around that (using https://github.com/mfikes/tubular), but nREPL is the way to go for this one I think.#2019-09-2022:21seancorfieldclj -A:nrepl 🙂#2019-09-2022:21seancorfield:nrepl {:extra-deps {nrepl {:mvn/version "RELEASE"}}
:main-opts ["-m" "nrepl.cmdline"]}
#2019-09-2022:22shaun-mahoodYeah that looks nice and easy 🙂#2019-09-2022:22seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L47-L57#2019-09-2022:23seancorfieldThis is why I have so much stuff in my .clojure/deps.edn file -- and I have it on every machine and under every shell (macOS, Windows Powershell, WSL, etc)#2019-09-2022:54shaun-mahood@seancorfield It looks like it was a firewall issue, not a Clojure issue - thanks for the help, using your aliases I at least found where the problem was. Now to figure out why Windows Server is ignoring me when I tell it to open a port...#2019-09-2514:38val_waeselynckIs it possible to deps-consume a JAR that's versioned in a Git repo?
Context: I have a little private java library that I want to consume in a deps.edn project. Putting the generated JAR under version control is weird, but this seems to be a minor inconvenience compared to setting up a private Maven distribution infrastructure.#2019-09-2514:50dominicmIf something was going to work, it would be :deps/root#2019-09-2514:54val_waeselynckLike this?
{:git/url "#2019-09-2514:54val_waeselynck(Assuming the JAR was generated at the root of the project).#2019-09-2520:09Alex Miller (Clojure team)That won’t work#2019-09-2605:36val_waeselynckThanks @U064X3EF3. Do you see any other way?#2019-09-2607:27Alex Miller (Clojure team)you could add the jar file to your :paths in the deps.edn of the git dep#2019-09-2611:04dharriganHello! Could master of tools.deps.alpha be merged into add-libs please? The add-libs is of tools.deps.alpha-0.7.541 (August 9th), not tools.deps.alpha-0.7.549 (September 17th)?#2019-09-2611:04dharriganThank you very much 🙂 I guess, any reason why add-lib can't be brought in to master? It's super useful 🙂#2019-09-2611:11Alex Miller (Clojure team)I don’t think you’re missing anything from master - just some tweaks in the specs (which aren’t used)#2019-09-2611:12dharriganfantastic! 🙂 Thank you (for clarifying!)#2019-09-2611:13Alex Miller (Clojure team)There are issues with add-lib in some environments and it’s unlikely to end up included in its current form (some of it may actually be moved into Clojure itself)#2019-09-2611:14dharrigankk, that's good to know. Havin the ability to dynamically pull in a dep - for experimentation on the repl, is amazingly useful to have 🙂#2019-09-2611:14dharrigan(and given clojure's great interop, doing that playing around is a very happy process!)#2019-09-2612:14dominicmWhy move it into clojure @alexmiller? Which bits?#2019-09-2613:01Alex Miller (Clojure team)the part that tracks the current “libs” currently in use could be built in. It’s already designed to be initialized in a couple ways so even non tools.deps tools could convey the lib set via those means. add-lib itself relies on tools.deps so that probably cannot move, but the parts that update the lib set could be genetically useful#2019-09-2613:02Alex Miller (Clojure team)Generically :)#2019-09-2613:06dominicmWhat's the goal of making the concept of a library first class? I can imagine using it to affect requires in different ways (e.g. Namespace versioning). Is there a set of goals in mind yet?#2019-09-2613:08Alex Miller (Clojure team)No, none of this is baked which is why it’s on a branch#2019-09-2613:56Alex Miller (Clojure team)There are a lot of potentially interesting things that could be done though#2019-09-2715:27plexusseems Clojure CLI uses a jar that contains Clojure + tools.deps. How would I build such a jar myself? or put in a different way, how can I make clojure use my local changes to tools.deps to try them out?#2019-09-2715:39Alex Miller (Clojure team)The easiest way (what I do) is to just hack my clojure script to put my local source directory at the front of the tools classpath#2019-09-2715:39Alex Miller (Clojure team)Then you can pick up changes with no build at all#2019-09-3004:08taylorI have a deps.edn project/repo that I’d like to also keep example projects, but ideally those wouldn’t get pulled when another deps.edn project depends on mine e.g. via git. Would tools.deps pull those “extra” files from my repo, and the README and everything? Should I stop worrying about a few kilobytes?#2019-09-3004:14seancorfield@taylor If someone depends on your project via git deps, the whole repo will be checked out (regardless of which parts are actually used).#2019-09-3004:15seancorfieldIf you publish an artifact (on Clojars or wherever) then you can choose what goes into that.#2019-09-3006:43sogaiuyou get a bare clone of the repository + one or more checkouts corresponding to whatever commits are used, right? at least, that's how it looks under ~/.gitlibs/#2019-09-3012:49Alex Miller (Clojure team)Yes, that’s correct#2019-10-0100:36plexusIf there are gitlab users here that would be interested in using the private Maven repositories provided by gitlab, then please check out this patch and provide feedback. https://clojure.atlassian.net/browse/TDEPS-140#2019-10-0100:38plexusGitlab uses a token in a http header for authentication instead of the normal username/password method, this patch adds support for that.#2019-10-0104:17mattyulrichis it possible to resolve a pom type dep (rather than a jar) in my deps.edn file?#2019-10-0104:21mattyulrichI think in lein, one could specify :type "pom" and it would resolve the pom alone without requiring a jar at the end...#2019-10-0104:29mattyulrichoh.. nevermind - I just found the :extension "pom" attribute - did the trick perfectly!#2019-10-0112:17kirill.salykinHi
I am playing with Github actions and tools.deps, seems there is an issue with installing deps from private github repositories
Did anyone also experienced it? Do I need to install my ssh key?#2019-10-0112:23Alex Miller (Clojure team)Yes, you would need to do that#2019-10-0112:23kirill.salykink, thanks#2019-10-0112:23Alex Miller (Clojure team)I believe there is a special env var for it automatically in the action#2019-10-0112:24Alex Miller (Clojure team)But you’d need to add it to the ssh-agent#2019-10-0112:24Alex Miller (Clojure team)I haven’t done this yet myself so don’t have a recipe for you #2019-10-0112:24kirill.salykinthanks, will try to install ssh key and add it to agent#2019-10-0112:59kirill.salykinadding ssh key didnt help much
name: CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/#2019-10-0112:59kirill.salykinError building classpath. connector is not available:
com.jcraft.jsch.agentproxy.AgentProxyException: connector is not available:
at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.java:120)
#2019-10-0112:59kirill.salykinplease advice what can be wrong?#2019-10-0114:34ghadi@kirill.salykin in Github Actions, do processes persist between steps?#2019-10-0114:42kirill.salykinI don’t know, to be honest
testing now#2019-10-0114:55kirill.salykinyes, you are correct#2019-10-0114:55kirill.salykinrun: eval $(ssh-agent -s) && ssh-add -k ~/.ssh/id_rsa && ./bin/run-tests-ci.sh
this works#2019-10-0115:02kirill.salykin> My guess is eval $(ssh-agent -s) dies at the end of that step?
this is my guess as well#2019-10-0115:03kirill.salykinhttps://medium.com/tokenmill/how-to-build-a-clojure-project-that-depends-on-private-git-repositories-in-docker-635c7a576726#2019-10-0114:35ghadiYou can find out by running
- run: ssh-add -l
in a subsequent step#2019-10-0114:35ghadiI have github actions building private git deps, but I'm doing it all within a container#2019-10-0114:39kirill.salykini am able to clone that repo via just git#2019-10-0114:40ghadibut tools.deps isn't using git#2019-10-0114:40kirill.salykinyeah, just to ensure that I can clone it#2019-10-0115:02ghadiMy guess is eval $(ssh-agent -s) dies at the end of that step?
@kirill.salykin#2019-10-0115:02ghadioh just saw the thread#2019-10-0115:04ghadiyou should probably pin the https://github.com host key statically in source code#2019-10-0115:04ghadiotherwise it will accept any known host key -- MITM attack#2019-10-0115:04kirill.salykintrue, thanks#2019-10-0316:58dominicm@alexmiller bit of a weird one https://github.com/juxt/pack.alpha/issues/64 no repro yet exactly, but it looks like RepositorySystem is coming back as nil. I don't suppose you're clued up on this and can think of an obvious cause?#2019-10-0317:28Alex Miller (Clojure team)I looked at that when it was filed as it was mentioned here too. It's weird, don't know what to make of it. I think from the stack this pre-dates the session cache additions?#2019-10-0317:30dominicmIt's still happening after updating too.#2019-10-0317:30dominicmDigging in, the impl of the service locator does attempt to add the repository system, so that should be there.#2019-10-0317:31dominicmBumping from 1.1.1 of the maven dependencies to 1.4.1 doesn't fix it either.#2019-10-0317:37dominicmthis is the problem with service locators. All the damn coupling.#2019-10-0317:45dominicmhttps://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/tools-deps/near/159950271#2019-10-0317:57Joe LaneHey friends, I'm having an issue cloning my own private github repo with tools deps. Even though I'm able to clone the repo with the ssh url into a local tmp directory, JGit is still telling me that Caused by: org.eclipse.jgit.errors.TransportException: .
I'm on a mac with these version numbers:
Joes-MacBook-Pro:snapestry-ion magemasher$ clj -Sverbose
version = 1.10.1.469
install_dir = /usr/local/Cellar/clojure/1.10.1.469
config_dir = /Users/magemasher/.clojure
config_paths = /usr/local/Cellar/clojure/1.10.1.469/deps.edn /Users/magemasher/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/479583893.cp
The ssh key I have on github matches this key below:
ssh-add -l -E md5
4096 MD5:10:ef:06:6b:9b:94:86:fc:06:83:9f:17:0a:8c:bb:10 /Users/magemasher/.ssh/id_rsa_jplane (RSA)
Does anybody have any ideas? I'm really stumped here.#2019-10-0317:57Joe Lane#2019-10-0317:58Alex Miller (Clojure team)is ssh-agent running?#2019-10-0317:59Joe Laneyes#2019-10-0317:59Joe Lane(I just confirmed by starting it, same failure)#2019-10-0318:00Alex Miller (Clojure team)there are some known failure modes with host key algorithms etc, but this doesn't really look like that#2019-10-0318:00Alex Miller (Clojure team)I assume you can git clone with that key#2019-10-0318:00Joe Laneyep#2019-10-0318:01ghadiDo you have entries in .ssh/config ?#2019-10-0318:01ghadiSometime jsch tries to read those instead of the ssh agent#2019-10-0318:01Alex Miller (Clojure team)some tips at https://clojure.atlassian.net/browse/TDEPS-91#2019-10-0318:02Joe LaneI do, are they safe to paste?#2019-10-0318:02ghadiYes#2019-10-0318:02ghadiIt’s usually just file names#2019-10-0318:02Alex Miller (Clojure team)there's one like IdentityKey or something that might need to be removed#2019-10-0318:02Joe LaneHost *
AddKeysToAgent yes
IdentityFile ~/.ssh/id_rsa_jplane
IdentitiesOnly yes
Host
HostName
User admin
IdentityFile ~/.ssh/id_rsa_jplane
#2019-10-0318:03Alex Miller (Clojure team)yeah, remove the IdentityFile line in first block#2019-10-0318:03ghadiComment out the first block#2019-10-0318:03Joe LaneHey, it works!#2019-10-0318:04Joe Lane(commenting out just the IdentityFile) So, the reason I had that line in there in the first place was because of these instructions. https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#2019-10-0318:05Joe LaneSo, while I'm glad this was fixed for me, are the github instructions conflicting with what Jsch needs?#2019-10-0318:08Joe LaneEither way, thank you both for your assistance! I see similar advice was given by Alex on the jira issue.#2019-10-0318:09dominicm@alexmiller I managed to wrangle a stacktrace https://github.com/juxt/pack.alpha/issues/64#issuecomment-538061024#2019-10-0318:10dominicmThe problem is, I can't figure out what would set a version resolver! There's nothing in the default service locator code.#2019-10-0318:11ghadi@lanejo01 JSch is going to be a constant source of headaches#2019-10-0318:38dominicmOkay. Figured out what sets the version resolver, but I don't understand how I'm getting java.lang.ClassCastException: Cannot cast org.apache.maven.repository.internal.DefaultVersionResolver to org.eclipse.aether.impl.VersionResolver when defaultversionresolver is like this:
public class DefaultVersionResolver
implements VersionResolver, Service
#2019-10-0318:42dominicmDoesn't that explicitly state that it's a VersionResolver?!#2019-10-0318:44seancorfieldIs it crossing a class loader boundary perhaps?#2019-10-0318:56Alex Miller (Clojure team)yeah, must be same class loaded from different loader#2019-10-0318:57Alex Miller (Clojure team)looks like you're buried in some capsule stuff - doesn't that do weird things with classloaders?#2019-10-0318:59Alex Miller (Clojure team)or is this just setting up capsule stuff#2019-10-0319:04Alex Miller (Clojure team)I'm not sure what the setup is, but there must be same class available via different loaders somehow here#2019-10-0319:05dominicmThis is setting up capsule stuff, so no classloaders involved here yet.#2019-10-0319:05dominicmcapsule is very good on the classloader front fwiw btw. It just extracts to a directory and runs it, much like leiningen. OneJar is the classloader one.#2019-10-0319:10dominicmI'll see if I can make this a minimal repro at least. Remove pack from the equation.#2019-10-0319:16dominicmrlwrap: Oops, crashed (caught SIGABRT) - this should not have happened!If you need a core dump, re-configure with --enable-debug and rebuildResetting terminal and cleaning up...
Today is not my day apparently! 🙂#2019-10-0319:22Alex Miller (Clojure team)wow#2019-10-0319:22ghadithat happens a lot with long lines#2019-10-0319:23ghadiprobably a method to constrain it. haven't looked#2019-10-0319:23ghadilong like 10MB lines, in my experience#2019-10-0319:29andy.fingerhutI sometimes idly wonder how many different behaviors exist among line-based utility programs for very very long lines.#2019-10-0320:51dominicmWhat I don't understand is how simply commenting out oz would change things. How does that cause a classloader?!#2019-10-0320:52Alex Miller (Clojure team)I have no idea#2019-10-0320:59Alex Miller (Clojure team)I don't have a theory for why this is happening, just trying to work backward from the evidence#2019-10-0321:00dominicmI can't create a minimal repro either. Taking the exact pack code and placing it in another project stops it from happening... :thinking_face:#2019-10-0321:00Alex Miller (Clojure team)can you get pack out of it? Just call tools.deps with oz stuff?#2019-10-0321:00Alex Miller (Clojure team)I can't remember if I did that when it was logged or not#2019-10-0321:01dominicmDoing that doesn't throw. There's something contextual/environmental that I am not accounting for.#2019-10-0321:02Alex Miller (Clojure team)are you running via clj? if so, what's the classpath? Are there any uberjars such that same class has two possible sources?#2019-10-0321:04dominicmRunning via clj. The classpath is pretty big because of oz.#2019-10-0321:22dominicmduplicates looking likely#2019-10-0321:23dominicmclojure -Spath -A:pack:packit | (while IFS= read -rd: line; do jar tf "${line}"; done) | sort | uniq -d I did this and got results#2019-10-0321:38dominicmI know why my repro isn't working, it's because I'm not co-locating oz and t.d.a on the classpath in my repro, but I am when I load pack into an existing project. That's also why this happens when used with add-lib.#2019-10-0321:40dominicmIt's nothing to do with dependency resolution (not really anyway). It's to do with the classpath tda is running under.
I saw maven duplicates with the above classpath, so I'm guessing that's the root cause. Although I've never seen issues like this where one wins out over the other in different places.#2019-10-0322:07hiredmanthe classpath isn't canonical, classloaders are, the classpath is interpreted by some classloader, so if some wants a classloader that behaves sort of like the system classloader (or whatever the root classloader is called these days) it can make a classloader that interprets the classpath but has a slightly different behavior#2019-10-0322:21Alex Miller (Clojure team)Maven has all kinds of magic dependency injection and it’s entirely possible there is some service config somewhere that is giving weird behavior. Usually I just avoid using all of that stuff when I call the maven APIs but it’s surprisingly invasive.#2019-10-0407:35dominicmThe alternative in this case is creating impl classes, and I'm not sure if that's better or worse. It's unclear whether that's a stable api to me, because the artefact is named impl too#2019-10-0407:57dominicmSo the repro is this:
;; deps.edn
{:paths ["src"]
:deps
{org.clojure/tools.deps.alpha {:mvn/version "0.7.541"}
metasoarous/oz {:mvn/version "1.6.0-alpha5"}}}
;; src/io/dominic/tda_npe/core.clj
(ns io.dominic.tda-npe.core
(:require
[clojure.tools.deps.alpha :as tools.deps]))
(def deps-map
'{:paths ["src"]
:deps {org.clojure/clojure #:mvn{:version "1.9.0"} }
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}})
(defn -main
[& _]
(tools.deps/resolve-deps deps-map {}))
#2019-10-0408:23dominicmclojure -Spath | while IFS= read -rd: line; do
if [ -f "${line}" ]; then
jar tf "${line}" | while read -r file; do echo "${line}: ${file}"; done
fi
done | sort -k2 | uniq -f1 --all-repeated=separate
Here's a script for finding duplicates on the classpath 🙂#2019-10-0408:33dominicmmetasoarous/oz {:mvn/version "1.6.0-alpha5"
:exclusions [org.apache.maven/maven-aether-provider]}
This fixes it (haven't tested oz). Looks like there's 2 sources for the default impl of VersionResolver!#2019-10-0408:34dominicmIt's coming transitively from pomegranate, so I think you can't have pomegranate and t.d.a on the classpath together without this exception.#2019-10-0411:06delaguardohttps://github.com/marketplace/actions/setup-clojure
I wrote github action to setup hosted runner with clojure cli.#2019-10-0412:02vlaaadThis is amazing, thank you very much!#2019-10-0412:42delaguardoyou are welcome to open an issue if something isn’t working as expected)#2019-10-0412:49vlaaadIntegrated it into cljfx, works great! https://github.com/cljfx/cljfx/blob/master/.github/workflows/test.yml#2019-10-0416:52delaguardoalso published action for provision graalvm instead of java - https://github.com/marketplace/actions/setup-graalvm-environment#2019-10-0418:34seancorfieldAdding this to next.jdbc to matrix test on Java 8 and Java 11. Thank you!#2019-10-0418:52seancorfieldWow, it takes a long time for those jobs to get started! 21 minutes for the first one. Still waiting for the second one in the matrix.#2019-10-0419:04seancorfield29 minutes end-to-end. It'll be interesting to see how they tune that during the beta. CircleCI started almost immediately (and completed in 30 seconds, compared to an actual run time of about a minute on GitHub).#2019-10-0419:36vlaaadFor me it usually starts immediately...#2019-10-0419:37seancorfieldMaybe it was just because this was the first time I've used it? I'll report back on subsequent runs.#2019-10-0420:54seancorfieldJust pushed another commit and, yeah, it started up nearly immediately.#2019-10-0510:56ikitommihi. I have a transitive deps issue. metosin/malli depends on both borkdude/sci and borkdude/edamame via sha. sci uses edamame via real mvn version. When I try to use malli, the versions conflict:
➜ ~ clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "28b4ee63bd571ce612e8cd0e9dab6ef86722597e"}}}'
Checking out: at 28b4ee63bd571ce612e8cd0e9dab6ef86722597e
Downloading: borkdude/edamame/0.0.6/edamame-0.0.6.pom from
Error building classpath. Unable to compare versions for borkdude/edamame: {:mvn/version "0.0.6", :deps/manifest :mvn} and {:git/url "", :sha "9b3e825b65ec6fe0dc536988fd036e6e681813de", :deps/manifest :deps, :deps/root "/Users/tommi/.gitlibs/libs/borkdude/edamame/9b3e825b65ec6fe0dc536988fd036e6e681813de"}
#2019-10-0510:58ikitommiIf I add explicit sha to edamame from command like, it works, but is not what I would like to be needed.#2019-10-0511:00ikitommie.g. this works:
➜ ~ clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "28b4ee63bd571ce612e8cd0e9dab6ef86722597e"}, borkdude/edamame {:git/url "", :sha "9b3e825b65ec6fe0dc536988fd036e6e681813de"}}}'
#2019-10-0511:21ikitommiquick-hacked this so that malli depends on edamame mvn version. Still, curious could it be made to work with just Sha dep to it#2019-10-0511:27dominicmWhat do you think should happen here? The problem is that maven and git don't have a comparison mechanism that makes sense.#2019-10-0511:34borkdude@ikitommi I think it's best to depend on a mvn version of edamame then from now on. I'll try to do that with sci as well.#2019-10-0512:01Alex Miller (Clojure team)This can be made to work, but I haven’t done so yet. Maven artifacts generally record their sha at build time so it is possible to version compare them.#2019-10-0512:02borkdudewhat if you have two branches that are diverged from the same maven version?#2019-10-0513:58Alex Miller (Clojure team)This is just back to the same rules we use for git comparisons - if one version is a descendant of another, its preferred. If there is no ancestor relationship, they aren’t comparable#2019-10-0513:59Alex Miller (Clojure team)In that case, it will throw and you need to be explicit at the top level about which dep to use (top dep version always used)#2019-10-0715:52Alex Miller (Clojure team)if anyone here is an add-lib user, can I ask the scenario where you're using it? is it
1) to definitely add a lib you know you need from prior experience
2) to add a lib you might want to use but are not sure if you need
3) to do a task at the repl that you definitely don't want in your deps.edn#2019-10-0717:16seancorfieldIs 3) meant to include situations where you have a "scratch" REPL running just to try stuff out (e.g., helping beginners on Slack) and it would be less convenient to restart that REPL with difference deps rather than just adding a lib to the REPL? For me, such situations are usually not accompanied by deps.edn at all.#2019-10-0717:18Alex Miller (Clojure team)you always implicitly have a deps.edn, even if it's just including clojure from the system deps.edn#2019-10-0717:19Alex Miller (Clojure team)sounds like a mixture of 2 and 3#2019-10-0717:19seancorfieldWell, yeah, I meant the "project" (current directory) one. Libraries that I use add-lib for wouldn't generally go in either of the other files.#2019-10-0717:20seancorfieldI have a scratch directory that has deps.edn with {} in it and I usually start a plain REPL with -Sdeps on the command-line, but if add-lib was easier to use, I'd just start that plain REPL and then add-lib in anything I wanted to experiment with and just leave it running (instead of restarting with different deps).#2019-10-0807:47plexusFor me it would be mostly 1), especially in the early stages of a project I want to be able to add dependencies without having to restart the process. It could also be 2) if it's a lib I haven't used before, to be able to quickly evaluate it before committing to it.#2019-10-0715:56dominicmI have played around with the idea of 3, maybe even for use in dev tooling libraries. But I haven't actually done it yet.#2019-10-0715:58Alex Miller (Clojure team)by which you mean you would have a lib that called add-lib?#2019-10-0716:00dominicmYeah. Although essentially it's a library of things you'd do at the repl.
I can't remember the details of why I was doing this now. Possibly to keep things "light" at someone's request, or maybe because I wanted to load things for dev time based on the config.edn#2019-10-0715:57Alex Miller (Clojure team)and if you answered 1 or 2, how do ensure the dep gets added to deps.edn? do you do it at the same time you add-lib or rely on remembering later?#2019-10-0715:58delaguardotests#2019-10-0715:59Alex Miller (Clojure team)not sure what that means? you add tests and rely on that failing later to tell you to add the lib?#2019-10-0716:01delaguardoyes, not a perfect solution, but covers most of the problems
something like Could not locate chesire/core__init.class, chesire/core.clj or chesire/core.cljc on classpath. usually is enough to remember what went missing#2019-10-0716:11kennyYep, pretty much the same thing as @U04V4KLKC. If the lib ends up getting used in the project, it will get added to a namespace's :requires. This will cause the project to fail CI tests if someone forgot to add it to the deps.edn.#2019-10-0716:40dharriganremember later#2019-10-0717:14seancorfieldFor 1) I do it at the same time because it's a deliberate act. For 2) I do it later, once I've decided whether the lib is really what I need (so, yeah, I run the risk of forgetting but if I really need the lib, it'll end up required in the code base somewhere so tests will fail if I have forgotten to add it).#2019-10-0715:58dominicmLater. Although I've been working on something which would remember what is add-lib'd and allow me to commit it. And also an alternative entry point for loading it and committing simultaneously.#2019-10-0715:59Alex Miller (Clojure team)I'm currently exploring similar ideas#2019-10-0716:01dominicmAs an fyi, I currently don't find the git tag rewriting useful.
I put comments into my deps.edn, and losing those is annoying. I've got a few toys that use rewrite-clj to avoid that (including a implementation of the git tag stuff), but they're difficult to write.#2019-10-0716:03dominicmOr rather, it's useful, but that side effect makes it problematic.#2019-10-0716:03Alex Miller (Clojure team)yeah, I'm probably going to remove that#2019-10-0716:03dominicmI know :) but the external version would have the same problem, so would a commit-add-lib!#2019-10-0716:04Alex Miller (Clojure team)well, I'm more exploring a "refresh" type option#2019-10-0716:04Alex Miller (Clojure team)so you would be the one doing the add#2019-10-0716:04dominicmAh sync-deps, I think someone tried that here. They requested knowledge of the aliases in use so they could merge them together :)#2019-10-0716:08Alex Miller (Clojure team)"sync" is ambiguous about the direction of the sync. "refresh" is deps.edn -> repl#2019-10-0716:13dominicmThat's fair. Refresh made me think of tools namespace and I struggled to separate the two.#2019-10-0717:16Alex Miller (Clojure team)How do you know the version of the lib for add-lib? Do you:
1) Look it up on github or other source
2) Use LATEST or RELEASE#2019-10-0717:17seancorfieldI use "RELEASE" a lot when I'm just experimenting with a library. I only bake the version into deps.edn once I know I'm going to use it.#2019-10-0717:18seancorfieldThe "refresh" scenario wouldn't cover a "scratch" situation where you're just trying a lib out in the REPL and don't want deps.edn created/updated.#2019-10-0717:23Alex Miller (Clojure team)yes, I'm aware. investigating alternatives, making tables :)#2019-10-0717:20dharriganMostly 1, sometimes 2.#2019-10-0717:28dominicmI usually look it up, I need to look up the coord anyway. I don't memorize those.#2019-10-0806:12orestisJust a data point - sometimes I’m repl’ed in a remote environment and I would find it useful to add a library without having to touch the remote deps.edn file, which might be tucked in a jar in any case. It’s on me to remember to do it after the fact, and that would require a proper deploy step anyhow. #2019-10-0806:14orestisI’m sure the table Alex is building will be much more nuanced, but the obvious (to me) building block is a function that can add a fully specified dep to the runtime. Figuring out the latest version, refresh the env from a deps.edn could be userland concerns, no?#2019-10-0813:19dominicmThis is what we have, so I think these might be baked in as well#2019-10-0821:49Michaël SalihiHi ! I I would like to know if it's possible to populate deps.edn dependencies in command line or some externals tools ?
Right now, I use clj-refactor with cljr-add-project-dependency in Emacs but it only works when a REPL is already running.#2019-10-0822:23ghadi@admin055 it’s just data - you can programmatically generate it#2019-10-0901:43mpisankohi there, pardon if totally missed this (but google didn't give me any answers). my question is: is there an alternative to lein's pedantic? in tools.deps#2019-10-0902:14seancorfield@mpisanko No. It's also worth bearing in mind that the algorithm for selecting versions is different in t.d.a. so there should be fewer potential conflicts. Or at least, fewer counterintuitive version selections 🙂#2019-10-0902:16seancorfieldI think there's still an undocumented :verbose true option you can add to deps.edn so that t.d.a. explains all of its decisions...#2019-10-0902:17mpisankoright, that's the premise of improved dependency resolution i guess. thanks @seancorfield!#2019-10-0902:42seancorfieldLooks like :verbose needs to be under an alias, not at the top-level... {:aliases {:v {:verbose true}} ,,,} and then clojure -A:v will show the resolution output as t.d.a. works.#2019-10-0902:57Alex Miller (Clojure team)actually there are tools that can help you with this in clj#2019-10-1819:03Alex Miller (Clojure team)is this in response to request on @U06HHF230 - are you working together? or are there two things here?#2019-10-1913:10zetafishWrong me#2019-10-2110:02rickmoynihanNot working together no.
I tried running with :verbose and it looks like the deps are coming from the git dep after all so I suspect this is another instance of not being able to configure updatePolicy on :mvn/repos, i.e. it looks like clojars is somehow using an updatePolicy of daily rather than never.
i.e. I think it’s this issue: https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-101?filter=allissues&orderby=priority%20DESC&keyword=101#2019-10-2110:08rickmoynihanIt’d be nice to support this, as it prohibits caching the deps in CI — which slows down the builds#2019-10-2112:44Alex Miller (Clojure team)Daily is the default so that matches what I would expect. Are any of these Snapshots?#2019-10-1905:32seancorfieldseancorfield/depstar "0.3.4" -- now follows symlinks when copying directories (thanks to @mgrbyte for reporting the problem)#2019-10-2021:05andy.fingerhutThis is probably too far off the beaten path to matter to anyone, but I noticed recently that Homebrew is also supported on Linux and Windows with WSL, and so tried out 'brew install clojure' on a Linux system. It seems to set things up in a way that is problematic for clojure finding the system-wide deps.edn file. I know there is the clojure install script for Linux, which is what I have always used before today, and will continue to do so. Just wanted to point this out in case anyone else tries it.#2019-10-2022:37Alex Miller (Clojure team)What’s the actual problem you saw? clojure actually no longer needs that file#2019-10-2022:45andy.fingerhutAfter brew install clojure completed with no apparent problems, I tried clojure -Sdescribe and got this message, which at the time stopped me from further experimentation: $ clojure -Sdescribe
cp: cannot stat 'PREFIX/example-deps.edn': No such file or directory
but I can try other things if you think it would be useful to know.#2019-10-2022:47andy.fingerhutA kind of unusual thing I noted about the results after brew install clojure completed was that Homebrew creates a /home/linuxbrew/.linuxbrew directory, where linuxbrew is a new directory, not the name of any user account on the machine, and puts all (or at least the ones I have seen) brew-installed files inside of that dir.#2019-10-2022:49andy.fingerhutComparing to my works-better macOS install via brew install clojure earlier, it appears that the Linux clojure shell script has one occurrence of PREFIX that is not replaced with the install directory.#2019-10-2023:54andy.fingerhutI see what happened -- I tried this on an Ubuntu Linux system with minimal install, and it had no ruby executable installed. brew install clojure's install.sh script uses ruby to replace occurrences of the string PREFIX with the install dir. That step of install.sh script failed, leaving the clojure shell script in its original unmodified form.#2019-10-2023:54andy.fingerhutsudo apt-get install ruby, brew uninstall clojure, then brew install clojure made the install go better.#2019-10-2023:55andy.fingerhutStrange that the Homebrew install went well without installing ruby itself.#2019-10-2023:57seancorfieldI thought brew used Ruby to do pretty much everything?#2019-10-2023:58seancorfieldPerhaps the clojure recipe needs to be updated to explicitly state ruby is a dependency?#2019-10-2023:58seancorfield(I assume brew can be used to install ruby?)#2019-10-2100:02andy.fingerhutI think it can. Homebrew's first platform was macOS, I believe, where ruby has been installed by default since a long time ago (although I hear that might be changing in the latest macOS just released? Or perhaps the next one?)#2019-10-2100:04andy.fingerhutAnd probably if I had done a non-minimal install of Ubuntu 19.04 Linux, it might have included ruby as well. In any case, sudo apt-get install ruby definitely works if you do that before brew install clojure. It seems that the main Homebrew install instructions for Linux work without ruby being installed first, surprisingly enough.#2019-10-2100:04seancorfieldI just tried install Homebrew on WSL based on https://docs.brew.sh/Homebrew-on-Linux and the first thing it did was install Ruby...#2019-10-2100:05seancorfield(but it might be an "embedded" Ruby for its own use?)#2019-10-2100:05andy.fingerhutRuby got installed after you did only this command? sh -c "$(curl -fsSL )" ?#2019-10-2100:07andy.fingerhutI will test on a fresh Linux VM and look at what happens a little more carefully while installing Homebrew.#2019-10-2100:07seancorfieldYeah, just confirmed it's an internal version of Ruby, just to run Homebrew! 😐#2019-10-2100:09seancorfieldHmm, and to get Ruby you need patchelf which in turn requires Clang / gcc... OK, not going down that rabbit hole! 🙂#2019-10-2100:09seancorfieldSo, could Clojure's brew recipe use something other than Ruby to do that textual change in the script?#2019-10-2100:10andy.fingerhutEven a minimal Ubuntu 19.10 Linux install includes gcc, and I remember brew installation installed patchelf early on before.#2019-10-2100:11seancorfieldThis is an earlier Ubuntu from an early WSL1 install (the joys of Insider builds)#2019-10-2100:13seancorfield(This is on my Mac running Win10 via Parallels with WSL as just a testing ground... maybe later I'll try it on my Win10 laptop although that's also WSL1 but I use it a lot more there... on this machine, I use Mac's Terminal for all my work)#2019-10-2100:15seancorfieldAlthough on Windows now I'm using Scoop to install Clojure etc in PowerShell#2019-10-2100:17andy.fingerhutI believe I have followed the Homebrew Linux install instructions faithfully, and while it does install a ruby executable, it seems to be in a directory that does not end up on your command PATH after following the instructions, but some other directory.#2019-10-2100:17seancorfieldRight:
> Yeah, just confirmed it's an internal version of Ruby, just to run Homebrew!#2019-10-2100:20andy.fingerhutIt seems that perhaps there ought to be a way (or perhaps already is?) for a Homebrew install script to access that internal version.#2019-10-2100:20andy.fingerhutbut that's a minor issue of polish, I think.#2019-10-2100:22seancorfieldThis looks like the only line that is problematic: https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/install.sh#L14 -- if that used any low-level, guaranteed-to-be-installed command to change the text, it wouldn't need Ruby and it would work just fine.#2019-10-2100:22seancorfieldsed perhaps?#2019-10-2100:37andy.fingerhutPerhaps. It does appear that sed has a -i option on the couple of systems where I have checked, for easier overwrite-the-input-file-in-one-shell-command convenience.#2019-10-2101:01seancorfieldGiven that there's a cp just below it, why not use this directly sed "s;PREFIX;$prefix;" < clojure > $prefix/bin/clojure#2019-10-2101:02seancorfieldSo it would be mkdir ...
sed ...
cp ...
#2019-10-2101:04seancorfieldI don't see much reason to modify the installed source files in place.#2019-10-2101:05seancorfield(I'll be interested to hear Alex's thoughts on this tomorrow)#2019-10-2103:21Alex Miller (Clojure team)sed has incompatible syntax between Mac and Linux #2019-10-2108:04dominicmIt's the -i flag that's incompatible. You can easily avoid it.#2019-10-2103:22Alex Miller (Clojure team)If you can make it work, patch wanted#2019-10-2104:59andy.fingerhutAny doc page you would recommend on how to test modifications to a Homebrew install script?#2019-10-2105:10andy.fingerhutI have tested on a Linux machine using brew install --interactive clojure where I edited the install.sh script to replace ruby with ${HOMEBREW_RUBY_PATH}, and that worked. It used the Homebrew-internal installation of ruby. I will try that on a macOS system next to see if it works there, too.#2019-10-2105:14andy.fingerhutYep, worked on a macOS system, too. That shell variable ${HOMEBREW_RUBY_PATH} was the full path name to a Homebrew-internal ruby executable on both a Linux system that had no other ruby installed, as well as on a Mac that had /usr/bin/ruby preinstalled before Homebrew was installed.#2019-10-2105:21andy.fingerhutI created a PR, even though they are likely not accepted for this project, just one convenient way to show the suggested change: https://github.com/clojure/brew-install/pull/5 . Feel free to close it.#2019-10-2105:25hiredmanI think "brew-install" in the project name is a misnomer, and install.sh is expected to work on Linux sans homebrew#2019-10-2105:26andy.fingerhutThe install.sh script that my PR suggests changing appears to be identical to a file with the same name that I found in both a Mac and Linux Homebrew installation when I installed the clojure Flask.#2019-10-2105:28andy.fingerhutThe line my PR suggests changing is the one that was failing on a Linux system with no ruby executable.#2019-10-2105:29andy.fingerhutI mean, I am fairly new to this Homebrew stuff, so could be making any number of mistaken assumptions here, but giving it a shot.#2019-10-2105:29hiredmanSure, and it is also, unless I am mistaken, identical to install.sh which is used for installing on Linux systems which do not use homebrew#2019-10-2105:30andy.fingerhutThere is another file named linux-install.sh in the same directory of that repo with install.sh, which uses install commands, that I suspect might be the non-Homebrew Linux install script.#2019-10-2105:30hiredmanAh, my mistake#2019-10-2105:32andy.fingerhutAlways glad for another mind on the problem, though. Thanks for looking into it.#2019-10-2105:39hiredmanI use Linux as my desktop os, and the idea of using homebrew to manage installed packages is so alien to me. I would consider just making the homebrew script exit with an error suggesting using the Linux install script on Linux. In which case you could depend on the bsd sed options and wouldn't need Ruby. But that is maybe a terribly unfriendly idea.#2019-10-2105:41andy.fingerhutThe script works fine on macOS already today, since /usr/bin/ruby comes preinstalled (until either Catalina, I think, or maybe the next macOS release, when I have read it will no longer come preinstalled). With this change, it should continue to work on macOS, and start working on Linux. That doesn't mean anyone must use Homebrew on Linux -- just another option.#2019-10-2108:05dominicmI would just not sed in place, and use it to write elsewhere.#2019-10-2108:05dominicm-i is just a convenience flag with sed#2019-10-2109:11tianshuhow to add a repository in deps.edn?#2019-10-2109:25dominicmThere's a :mvn/repos key#2019-10-2109:56sogaiuhttps://clojure.org/reference/deps_and_cli#_procurers ?#2019-10-2110:01tianshu@dominicm @sogaiu thanks!#2019-10-2110:22kommensince upgrading to clojure 1.10.1.478 (from https://github.com/Homebrew/homebrew-core/pull/45565) clojure fails to run#2019-10-2110:23kommen$ clojure
Error: Could not find or load main class clojure.main
$ clojure -Spath
Error: Could not find or load main class clojure.main
#2019-10-2110:23kommen$ brew info clojure
clojure: stable 1.10.1.478
The Clojure Programming Language
/usr/local/Cellar/clojure/1.10.1.478 (9 files, 21.1MB) *
Built from source on 2019-10-19 at 20:39:17
From:
==> Dependencies
Required: rlwrap ✔
==> Requirements
Required: java >= 1.8 ✔
==> Analytics
install: 3,096 (30 days), 11,637 (90 days), 49,842 (365 days)
install_on_request: 3,070 (30 days), 11,511 (90 days), 49,114 (365 days)
build_error: 0 (30 days)
#2019-10-2110:33kommenany body else having issues like this?#2019-10-2110:34dominicmHave you wiped your .m2?#2019-10-2110:34dominicm(I'm not suggesting you should)#2019-10-2110:38kommenno, I didn’t#2019-10-2110:39kommenI just tried to move .clojure/deps.edn somewhere else, suspecting a problem with it. but then I get:#2019-10-2110:39kommen$ clj
cp: PREFIX/example-deps.edn: No such file or directory
#2019-10-2110:39kommenwith an ~/.clojure/deps.edn file only containing {} I guet the same Error: Could not find or load main class clojure.main error#2019-10-2110:39dominicmAh.#2019-10-2110:48kommensince the -m flag of the java call was changed in this release, I suspect that this would be the problem:#2019-10-2110:48kommenhttps://github.com/clojure/brew-install/commit/0accd10be691101e6288e1bf614abe7371b43cc2#2019-10-2111:01dominicmI don't think it is. I think your install failed.#2019-10-2111:02dominicmGiven that you have issues with PREFIX#2019-10-2111:03kommenyes, you’re right#2019-10-2111:03kommentried to reinstall already, but no luck#2019-10-2111:04dominicmRuby powers the rewrite on Mac, that's all I know I'm afraid. Maybe there's a log somewhere?#2019-10-2111:04kommenwoudn’t know where to look for a log#2019-10-2111:05dominicmIt'll be a brew install log I'd guess.#2019-10-2111:05kommenso this is supposed to be replaced at time of brew install? https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/clojure#L175#2019-10-2111:05dominicmYes#2019-10-2111:07kommenI see: https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/install.sh#L14#2019-10-2111:13kommenok, so I manually fixed the install_dir= in /usr/local/bin/clojure and it works now, let’s see if this is only a problem with my machine or if some body else runs into it#2019-10-2111:25kommenfor future reference: macOS 10.15 (19A583), $ ruby --version
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]
brew config
HOMEBREW_VERSION: 2.1.15
#2019-10-2111:36kommenok, found the logs and this also explains it:#2019-10-2111:36kommen$ cat ~/Library/Logs/Homebrew/clojure/01.install.sh
2019-10-21 12:50:08 +0200
./install.sh
/usr/local/Cellar/clojure/1.10.1.478
truffleruby: the -p option is not implemented
#2019-10-2111:48dominicmHaha. Yes it does :)#2019-10-2111:57kommen@dominicm thanks for keeping company and pointing me in the right direction#2019-10-2112:49Alex Miller (Clojure team)Ok, so is there something to fix here or is this an issue mainly on your system?#2019-10-2113:01dominicmThe problem is that truffle ruby is installed, and that didn't support -p for doing the substitution. Anyone who's playing with graal may run into this.#2019-10-2113:01Alex Miller (Clojure team)so the prior discussion here about using the brew install ruby impl would presumably also address that#2019-10-2113:02dominicmYes. So would switching to sed.#2019-10-2113:02Alex Miller (Clojure team)-i has different incompatible syntax on mac and linux there#2019-10-2113:02Alex Miller (Clojure team)I used to use sed, then perl for a bit#2019-10-2113:03dominicmI know. But you can avoid using -i#2019-10-2113:03Alex Miller (Clojure team)I seem to remember there was some reason I didn't do that either, but it was a long time ago now#2019-10-2113:08dominicmProbably not a problem. Using brew means ruby anyway. The only impetus would be to unify the install scripts.#2019-10-2113:11plexusunfortunately some distros have stopped including ed... echo -e 'w clojure.bak\n1,$s^PREFIX^'$PREFIX'^\nwq' | ed clojure#2019-10-2113:41kommenI remember I once proposed a sed command which worked on linux and macos, but either my mind or jira search fails me#2019-10-2113:43andy.fingerhutThis is exactly the same problem I found on Linux with Homebrew with no Ruby preinstalled. This will be an issue for all macOS Catalina systems, too, which also have no ruby preinstalled for the first time with macOS since it was introduced almost 20 years ago.#2019-10-2113:43kommenbut Dominic is right, using brew means using ruby, so I guess it makes sense to be able to rely on ruby#2019-10-2113:43andy.fingerhutThere is an issue open for Homebrew for this since June 2019 I found last night.#2019-10-2113:44andy.fingerhutI believe that the patch I submitted for the Clojure Homebrew installer will work for Catalina as well, because it will force the Clojure install.sh to use a ruby installed inside of Homebrew, rather than a system-wide one.#2019-10-2113:44andy.fingerhutBut I do not have a macOS Catalina system to test on to be certain.#2019-10-2113:46Alex Miller (Clojure team)well, that certainly bumps it up my priority list#2019-10-2114:55markbastianDoes anyone know if the immutant folks intended to let their domain expire? When I go to http://immutant.org/ there's just a domain expiration notification.#2019-10-2115:05dominicmThis probably isn't the channel for that. @U06SGCEHJ you're associated I think?#2019-10-2115:37Alex Miller (Clojure team)definitely not the channel, but I'm not sure anyone is still working on immutant#2019-10-2115:40markbastianOk, thanks for the info. I'll see if there's a better place.#2019-10-2115:43markbastianOh, I just noticed I posted this in tools-deps, not clojure. Sorry!#2019-10-2115:55Alex Miller (Clojure team)tools.deps.alpha 0.8.578, clj 1.10.1.478 are now available - see #announcements for details#2019-10-2115:58Alex Miller (Clojure team)as a side note, there were changes made to the internal cli scripts between clj and tools.deps (clojure.tools.deps.alpha.script.make-classpath and clojure.tools.deps.alpha.script.generate-manifest). I doubt many people use any of that directly, but changes were made in clojure.tools.deps.alpha.script.make-classpath2 and clojure.tools.deps.alpha.script.generate-manifest2 to avoid any breakage.#2019-10-2115:58Alex Miller (Clojure team)no changes to tda apis#2019-10-2122:13ro6Is anyone successfully using (add-lib ...) in a Cursive repl? I get a true back, but then "could not locate ... on classpath" when requiring the lib.#2019-10-2122:17cfleming@alexmiller The new :deps and :paths support in aliases is interesting. Am I correct in thinking that this would allow overriding the system deps.edn, e.g. to use a fork of Clojure?#2019-10-2122:18Alex Miller (Clojure team)It still has install deps on the chain#2019-10-2122:18Alex Miller (Clojure team)So, no difference there#2019-10-2122:19cflemingI see. Yes, re-reading the announcement I see I misread it - it’s more for tools to not require the project classpath instead.#2019-10-2122:19cflemingThanks.#2019-10-2206:33dominicmOoh, that would be very useful for pack#2019-10-2207:11andy.fingerhutCorrection to something I said earlier -- macOS Catalina (the recently released version, earlier this month) does still include /usr/bin/ruby, perl, and python executables installed by default. I checked by installing it on one of my Macs in the house. Apple has announced that the next major release will not include them in the base macOS installation.#2019-10-2207:15andy.fingerhutI will soon be installing Homebrew on it, and brew install clojure, to see if it works for me.#2019-10-2208:40andy.fingerhutAfter installing a Java executable (I chose AdoptOpenJDK 11) and Xcode, and Homebrew, brew install clojure produced a working clojure command install on my macOS Catalina system. There is no need that I can see to change the install.sh script to make it work on that OS.#2019-10-2208:41andy.fingerhutIt isn't clear to me why someone earlier had a problem there, unless perhaps /usr/bin wasn't in their path so that /usr/bin/ruby was not found, but that seems unlikely to be the reason.#2019-10-2213:53kommen@andy.fingerhut if you mean me by someone earlier with macOS catalina: this was due to truffleruby (from graalvm) being used to run the install.sh script. and truffle ruby doesn’t support the -p flag yet: https://clojurians.slack.com/archives/C6QH853H8/p1571657798259600#2019-10-2213:57andy.fingerhutSo at the time you tried upgrading the clojure package using brew upgrade clojure, you had a ruby executable in your command PATH that was earlier than /usr/bin/ruby? If so, the results you got makes sense to me.#2019-10-2306:50kommen@andy.fingerhut no, that is the part I didn’t understand but also didn’t investigate yet: how brew install clojure came to use truffleruby. truffleruby is not on my PATH…#2019-10-2312:11delaguardoAre you using graalvm with ruby component installed?#2019-10-2312:13kommenyes#2019-10-2313:02delaguardoprobably you have graalvm bin directory in your path that overrides /usr/bin#2019-10-2313:03kommenno, that isn’t the case. also running ruby --version in the same shell as the brew commands shows that it uses a ruby version managed by rbenv#2019-10-2313:39delaguardothat is right, but during formula installation it loads new shell session with limited number of tools installed. The best way to check it - brew install --interactive clojure#2019-10-2313:40delaguardoAs an example:
Install to this prefix: /usr/local/Cellar/clojure/1.10.1.478
/Users/delaguardo/.zshrc:29: command not found: starship
/Users/delaguardo/.zshrc:32: command not found: jenv
/Users/delaguardo/.zshrc:49: command not found: rbenv
clojure-tools % which ruby
/usr/bin/ruby
clojure-tools % ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19]
#2019-10-2313:42andy.fingerhutIn any case, I saw that Alex Miller has committed a change that I suggested where brew install clojure will run a shell script named install.sh such that it will run ruby using ${HOMEBREW_RUBY_PATH}, which Homebrew should consistently set to the full path name of its own internal ruby executable. This should make brew install clojure much more robust in the face of arbitrary user environments.#2019-10-2313:42andy.fingerhuthttps://github.com/clojure/brew-install/commit/f238dcebaa0b7bce51305c322dfb89fd3e152304#2019-10-2313:42andy.fingerhutI don't think that commit is deployed "to the field" yet, but on its way in that direction.#2019-10-2313:44kommenthanks @U04V4KLKC, that would explain it. I will check later#2019-10-2313:46kommen@andy.fingerhut thanks for the info, I’ll make sure to test brew install clojure with the next release#2019-10-2313:46delaguardo> In any case, I saw that Alex Miller has committed a change that I suggested where brew install clojure will run a shell script named install.sh such that it will run ruby using ${HOMEBREW_RUBY_PATH}, which Homebrew should consistently set to the full path name of its own internal ruby executable. This should make brew install clojure much more robust in the face of arbitrary user environments.
I’m not sure that this is the best solution. Now brew-install repo has two almost identical scripts that are doing the same thing but differently (using different tools)#2019-10-2313:47delaguardorecently I did some changes - https://github.com/clojure/brew-install/compare/1.10.1...DeLaGuardo:1.10.1
to create one cross-platform installation script#2019-10-2313:51delaguardoNot in PR yet because I was hoping to find some time to build packages for other distribution platforms like debian’s ppa and rpm for centos#2019-10-2405:47amarjeetHi, is there a way to provide credentials while using private git repo as a dependency in deps.edn? I want to use my private git repos in my project, but I am getting credentials related error.#2019-10-2412:07Alex Miller (Clojure team)What kind of credentials?#2019-10-2412:07Alex Miller (Clojure team)ssh is supported, user/pw is not currently#2019-10-2414:51amarjeet@U064X3EF3 For ssh, should I supply an additional key as :ssh in the deps map, in addition to :git/url and :sha?#2019-10-2415:03Alex Miller (Clojure team)no#2019-10-2415:03Alex Miller (Clojure team)we use the ssh-agent as a source for identity checking#2019-10-2415:04amarjeetoh okay#2019-10-2415:04Alex Miller (Clojure team)see https://clojure.org/reference/deps_and_cli and search for "Git configuration" for more info#2019-10-2415:05Alex Miller (Clojure team)as a general principle, you will never put any kind of creds in deps.edn#2019-10-2415:05amarjeetGot it, makes sense. I will do it and try. Thanks much 🙂#2019-10-2510:13borkdudecan I specify a src folder for a local/root with a pom.xml (it's a boot project)#2019-10-2510:14borkdudesomehow the source path doesn't show up in the pom.xml and when I run clojure -Spath it returns src/main/clojure#2019-10-2510:14borkdudethe library is called dre.common and this is on my classpath: /Users/borkdude/dre/DocSearch/common/src/main/clojure#2019-10-2510:40borkdudeAdding "../common/src" to the extra-paths works, but not sure if that's the right way#2019-10-2512:30delaguardoI have some problem trying to reposition library coordinates:
Library A’s deps.edn file
{:deps {B {:mvn/version "1"}}}
Library B’s deps.edn file
{:deps {C {:mvn/version "1"}}}
Library C’s deps.edn file
{:deps {}}
All version exists and can be resolved
Then I want to use git version of C library in A
Updated deps.edn for lib A:
{:deps {B {:mvn/version "1"}
C {:git/url ""
:sha "commit-sha"}}}
I’m expecting git version to take precedence in resolved dependency tree.
Instead I got an error:
Error building classpath. Unable to compare versions for C: {:mvn/version "1", :deps/manifest :mvn} and {:git/url "", :sha "commit-sha", :deps/manifest :deps, :deps/root "/home/user/.gitlibs/libs/C/commit-sha"}
Looks like a bug in clojure-cli for me. Happy to file a ticket if anybody can confirm (or agree) that this is a bug#2019-10-2512:44Alex Miller (Clojure team)there is a ticket already I think for being unable to compare versions for the same library across different coordinate types#2019-10-2512:44Alex Miller (Clojure team)still a todo#2019-10-2512:45Alex Miller (Clojure team)I have not read the above carefully, just going by the error#2019-10-2512:49delaguardoTo summarize all above:
main problem that I’m trying to override dependency coordinate by adding it in top-level dep.edn. That is working fine within the same coordinate type (maven and git) but fails when I need git dependency to overtake maven. So yes, I think adding ability to compare different coordinate types should solve it#2019-10-2512:54Alex Miller (Clojure team)yes, this has not yet been implemented#2019-10-2512:54Alex Miller (Clojure team)although I've thought about it a lot and know what to do#2019-10-2512:59delaguardonice! thanks#2019-10-2512:59delaguardoalso I’m happy to help if you don’t mind to share you ideas around it#2019-10-2513:55gphilippNot sure if it’s of any interest to anybody, but I’ve this small one-liner which pretty-print an EDN form in the clipboard:
clojure -Sdeps '{:deps {mvxcvi/puget {:mvn/version "1.2.0"}}}' -e "(require '[puget.printer :as puget]) (puget/cprint (read-string (first *command-line-args*)))" $(mktemp) "$(pbpaste)"
I’m not sure why the first arg needs to be a file since I’m using the -e option (maybe https://clojure.atlassian.net/browse/TDEPS-56) but well, it works.#2019-10-2514:46borkdude@U050S183X if you're interested in a tool with milliseconds startup time:
$ jet --pretty <<< "$(pbpaste)"
{:a 1, :b 2, :c 3}
#2019-10-2514:55gphilippNice ! Is there an option for color highlighting, as this is why I’m using the command above ? For speed, I use https://github.com/kkinnear/zprint/blob/master/doc/filter.md#appcds-version with echo pbpaste | zprint, it’s also compiled with GraalVM.#2019-10-2514:56borkdudejet doesn't have colors yet.#2019-10-2518:38andy.fingerhutIs there some kind of option to clj/clojure commands to cause them to show the java commands they execute?#2019-10-2519:06seancorfieldbash -x clojure <args>#2019-10-2519:07seancorfield(that shows "everything" but it does show the java command as well 🙂 )#2019-10-2519:07seancorfieldbash -x clojure
...
+ exec /usr/bin/java -Dclojure.libfile=.cpcache/1562564545.libs -classpath src:/Users/sean/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/sean/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar clojure.main
Clojure 1.10.1
user=>
#2019-10-2519:08seancorfield(you can't do bash -x clj ... because it doesn't pass -x into clojure when it runs it via rlwrap)#2019-10-2519:08andy.fingerhutHuh, didn't know about -x option to bash. Looks generally useful for all bash scripts.#2019-10-2519:09seancorfieldYeah, I use it a lot for debugging scripts.#2019-10-2519:54Alex Miller (Clojure team)ditto :)#2019-10-2520:22andy.fingerhutI mean, I knew about the set -x command by editing my own bash scripts to debug them, and sometimes did that on things owned by root if I had the permission, but nice that it can also be done without editing the script.#2019-10-2523:02sogaiuis there something comparable for powershell?#2019-10-2523:13sogaiuah, may be this: https://ss64.com/ps/set-psdebug.html#2019-10-2905:07JohanHello, did anyone successfully use nRrpl and repl-y together to be launched via a clojure -A:repl ?#2019-10-2905:33seancorfield@johan178 Something like this? clj -Sdeps '{:deps {reply {:mvn/version "RELEASE"}}}' -e "((requiring-resolve,'nrepl.server/start-server))" -m reply.main
That will start an nREPL server on a random port (and report it) as well as starting REPL-y.#2019-10-2905:35seancorfieldThen you can connect into that server from other clients. I just tried connecting in with Leiningen to check it worked.#2019-10-2905:35seancorfieldYou could provide a specific port number if you wanted in the call to start-server.#2019-10-2906:36Johan:nrepl {:extra-deps {nrepl {:mvn/version "RELEASE"}
reply {:mvn/version "RELEASE"}
cider/cider-nrepl {:mvn/version "RELEASE"}}
:main-opts ["-e" "((requiring-resolve,'nrepl.server/start-server))"
"-m" "reply.main"] }
I got an alias working.
Trying to get nrepl midlerware (cider-nrepl) working#2019-10-2911:18borkdude@johan178 this is what my cider-nrepl alias looks like: https://github.com/borkdude/clj-kondo/blob/master/doc/dev.md#toolsdeps#2019-10-2911:19borkdude(fwiw I don't start a REPL in terminal, I just connect to it from my editor)#2019-10-3105:23maxpIs usage of "RELEASE" / "LATEST" in :mvn/version documented somewhere?#2019-10-3105:36seancorfield@maxp They are not really recommended.#2019-10-3105:38seancorfieldThey are not supported in Maven 3, I believe. They are only available in Maven 2 based tools.#2019-10-3105:39seancorfieldRELEASE means the most recent non-snapshot release, so it can be an alpha, for example, or a "gold" version. LATEST means the most recent -SNAPSHOT release which can change frequently.#2019-10-3105:41seancorfieldThat said, I use RELEASE for dev-only tooling, as a "convenience" for it always being up-to-date but with the caveat that it could of course break any day. I never use LATEST.#2019-10-3105:41seancorfieldDoes that help @maxp?#2019-10-3106:36maxp@seancorfield - yes. Thank you/#2019-10-3116:31Drew VerleeIts possible to specify multiple deps files and have later ones override earlier ones right?#2019-10-3116:35Alex Miller (Clojure team)there are multiple deps files that override, but that is not an open set#2019-10-3116:35Alex Miller (Clojure team)the deps chain is: install deps.edn (now a resource in tools.deps.alpha), user deps.edn (~/.clojure/deps.edn), project deps.edn, -Sdeps#2019-10-3116:36Alex Miller (Clojure team)if you're using tools.deps directly, you can of course do whatever you like#2019-10-3116:36Alex Miller (Clojure team)it has tools for loading and merging deps.edn files#2019-10-3116:42Drew VerleeThanks alex.#2019-10-3116:58Drew Verleeso if you want to override a project dep it would be something like "-Sdeps {:aliases {:foo {:override-deps ...}}} -A:foo"#2019-10-3117:07Alex Miller (Clojure team)you don't need an alias for that#2019-10-3117:07Alex Miller (Clojure team)can just -Sdeps '{:deps {the/lib {:mvn/version "whatever"}}}'#2019-10-3117:08Alex Miller (Clojure team):deps merge in#2019-10-3117:08Alex Miller (Clojure team)you can do what you said too though#2019-10-3117:12Drew Verleemakes sense.#2019-11-0314:19athomasoriginalI love cofield’s dot-clojure repo: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn. I would like to make an alias in my own global deps.edn file that will use the latest version of ClojureScript e.g.
{:aliases
{:cljs
:extra-deps {org.clojure/clojurescript {:git/url ""
:sha "6431e6435ad9fc85e766752f4d6939ff978db5a5"}}
:main-opts ["--main" "cljs.main" "--repl-env" "node" "--repl"]}}}
But when I call the above, I get Error building classpath. Destination path "clojurescript" already exists and is not an empty directory. What am I missing here?#2019-11-0316:29athomasoriginalI did a little digging and was able to resolve my issue by executing the following steps:
1. Delete ~/.gitlibs/_repos/github.com/clojure/clojurescript (which was empty)
2. Run clj -A:cljs again
Now everything seems to be working again. Is there a command in the tools.deps API that can do this automatically?#2019-11-0316:30Alex Miller (Clojure team)I’ve seen a few people hit this but I have not been able to reproduce it and don’t have a ready fix for it#2019-11-0316:42athomasoriginalFair. I tried to setup a minimal repro based on what my previous dir structure looked like, but can’t get it right. The best I found is this other interesting scenario
1. Successfully run clj -A:cljs as I have in the code snippet above (then stop the repl)
2. delete content of ~/.gitlibs/libs/org.clojure/clojurescript + ~/.gitlibs/_repos/github.com/clojure/clojurescript (both clojurescript still exist, but they are empty)
Now when I run clj -A:cljs I get the following error
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate cljs/main__init.class, cljs/main.clj or cljs/main.cljc on classpath.
Full report at:
/var/folders/qc/mv_jl6857850zkr0qv6gj89m0000gn/T/clojure-5520844173070466309.edn
The above is resolved by deleting .cpcache#2019-11-0316:44athomasoriginalThe above seems odd, but that seem to be how my local looked at the time.#2019-11-0318:36borkdudefwiw I had a similar problem recently (went away after deleting .gitlibs and/or .cpcache)#2019-11-0320:55seancorfield@tkjone Or perhaps putting -Sforce on that clj command, to force it to rebuild the cache?#2019-11-0320:59sogaiui've been doing -Sforce more often than deleting .cpcache with some success recently -- iiuc this will leave older bits in .cpcache if you want to investigate issues later, helpful when reporting issues perhaps?#2019-11-0321:03seancorfieldYeah, you get a separate set of cached files for each alias combination you use, so it's nice to keep the ones that work fine and just override the ones that don't, via -Sforce.#2019-11-0411:05vlaaadAnother use-case for https://clojure.atlassian.net/browse/TDEPS-116 — I want to create a library that has different code for different jdk versions. I do it using 2 aliases:
{:jdk8 {:extra-paths ["src/jdk8"]}
:jdk11 {:extra-paths ["src/jdk11"]}}
It would be much easier to depend on such library when you can specify extra alias when declaring a dependency to it#2019-11-0411:45vlaaadI think I found a bug: :paths in aliases are not picked up when there are also :deps present. Repro:
deps.edn:
{:paths ["src" "jdk11"]
:aliases {:jdk8 {:paths ["src" "jdk8"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}}}}}
clj -Srepro -Sforce -Spath output:
src:jdk11:/home/vlaaad/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/vlaaad/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/vlaaad/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
clj -A:jdk8 -Srepro -Sforce -Spath output:
src:/home/vlaaad/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/vlaaad/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/vlaaad/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
notice: second output does not have jdk8 folder on a classpath#2019-11-0411:50vlaaad#2019-11-0413:16stijn@vlaaad you have a typo in the deps.edn, :deps is not inside the :jdk8 alias.#2019-11-0413:17vlaaadwhat, where? it is!#2019-11-0413:18vlaaadoh, you are right#2019-11-0412:11dominicmIn maven that's a classifier I think. Although jdk11 had a system for having jdk-specific files added.#2019-11-0412:22dominicm@vlaaad jep238: multi release jars#2019-11-0412:23dominicmIt would be cool if deps.edn supported MRJAR for fetching dependencies.#2019-11-0413:01Alex Miller (Clojure team)I don’t think mrjars have any effect on deps? That’s really a JVM thing?#2019-11-0413:01Alex Miller (Clojure team)On the problem above, what version of clj are you on?#2019-11-0413:06vlaaadlatest, 1.10.1.148#2019-11-0413:10vlaaad@dominicm will jep 238 work with clj files as well? is it okay to make library muti-release?..#2019-11-0413:10dominicmI see no reason why not. Try it and report back!#2019-11-0413:12vlaaadI think jep238 might not be a fitting solution for me, because I also need different dependencies for different java versions...#2019-11-0413:29Alex Miller (Clojure team)Yeah, won’t do that, but should work for clj files as they are loaded as resources, just like class files#2019-11-0413:32vlaaadyeah, makes sense#2019-11-0413:34Alex Miller (Clojure team)This is really where maven uses classifiers and jdk properties#2019-11-0413:10dominicmI was thinking of fetching the appropriate pom.xml from the jar based on jdk version#2019-11-0413:32Alex Miller (Clojure team)I don’t think that’s a thing#2019-11-0413:32Alex Miller (Clojure team)Although I have not looked at all at the maven support for mrjars#2019-11-0413:35dominicmIt doesn't look like it is. It could be for local/root on a jar, but for maven a separate fetch is made for the pom, completely unaware of the jdk#2019-11-0413:39vlaaadis there a way to exclude dependency using alias?#2019-11-0413:41Alex Miller (Clojure team)Not right now#2019-11-0413:54vlaaadsad, I'm running out of ideas how to add jdk8 support without breaking git dependencies to cljfx 😞#2019-11-0413:56vlaaadmaven is easy, I can build poms with different deps and source paths and publish them either as different artifacts, or with different classifiers/activations#2019-11-0413:57vlaaadbut I already have jdk11-specific deps in root deps.edn, and it seems there is no way to exclude them when using git dependency?#2019-11-0413:59Alex Miller (Clojure team)you could try using a lib with nil coordinate - that will merge over, but I don't remember how tools.deps handles it (could easily not work, I just don't remember)#2019-11-0414:01vlaaadusing :override-deps?#2019-11-0414:01Alex Miller (Clojure team)yeah#2019-11-0414:04vlaaaddoesn't seem to work:
given this deps.edn:
{:paths ["src" "jdk11"]
:deps {org.openjfx/javafx-base {:mvn/version "13"}}
:aliases {:jdk8 {:paths ["src" "jdk8"]
:override-deps {org.openjfx/javafx-base nil}}}}
running clj -A:jdk8 -Srepro -Sforce -Spath still includes javafx-base:
src:jdk8:/home/vlaaad/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/vlaaad/.m2/repository/org/openjfx/javafx-base/13/javafx-base-13.jar:/home/vlaaad/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/vlaaad/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/home/vlaaad/.m2/repository/org/openjfx/javafx-base/13/javafx-base-13-linux.jar
#2019-11-0414:05vlaaadand using :deps in alias seems to interfere with :paths so jdk8 path is not seen, as reported earlier#2019-11-0414:05Alex Miller (Clojure team)yeah, I'm going to look at that in a bit once I'm past my morning calls#2019-11-0414:06Alex Miller (Clojure team)that's surprising to me#2019-11-0414:07Alex Miller (Clojure team)could you also post me your clj -Sdescribe and whether you have anything in ~/.clojure/deps.edn?#2019-11-0414:09Alex Miller (Clojure team)wait, you said latest but 1.10.1.148 is not a thing, should be 1.10.1.478#2019-11-0414:10vlaaadI'm using -Srepro, so ~/.clojure/deps.edn shouldn't be an issue?#2019-11-0414:10vlaaadI'm using 1.10.1.478, it was a typo#2019-11-0414:10Alex Miller (Clojure team)ah, gotcha#2019-11-0414:10Alex Miller (Clojure team)on both#2019-11-0414:11vlaaadclj -A:jdk8 -Srepro -Sforce -Sdescribe
{:version "1.10.1.478"
:config-files ["/usr/local/lib/clojure/deps.edn" "deps.edn" ]
:config-user ""
:config-project "deps.edn"
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/vlaaad/.clojure"
:cache-dir ".cpcache"
:force true
:repro true
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ":jdk8"}
#2019-11-0414:16Alex Miller (Clojure team)oh, this is the expected behavior#2019-11-0414:16Alex Miller (Clojure team)the :paths and :deps in an alias are new with the newest release and REPLACE the project :paths and :deps#2019-11-0414:17Alex Miller (Clojure team)if you want to add, you need :extra-paths or :extra-deps#2019-11-0414:17vlaaadI want to replace both#2019-11-0414:17Alex Miller (Clojure team)I thought your complaint above was that the project :paths was not included?#2019-11-0414:18Alex Miller (Clojure team)re-reading what you said above...#2019-11-0414:18vlaaadNo, replacement paths are not picked up#2019-11-0414:51Alex Miller (Clojure team)ah, it is a bug and I see the fix#2019-11-0415:14Alex Miller (Clojure team)and there is a test that should have caught this, but the test is missing the is, so it didn't fail :)#2019-11-0415:14Alex Miller (Clojure team)programming is hard, y'all#2019-11-0415:31andy.fingerhuteastwood will probably warn about such tests, if it works on your code base at all.#2019-11-0416:03kszaboclj-kondo as well#2019-11-0415:46Alex Miller (Clojure team)@vlaad logged as https://clojure.atlassian.net/browse/TDEPS-142, fixed, and released in tools.deps.alpha 0.8.584, the rest is working its way through to clj#2019-11-0416:09vlaaadthanks a lot!#2019-11-0416:11Alex Miller (Clojure team)thanks for a concise and useful report! :)#2019-11-0416:03kszaboclj-kondo as well#2019-11-0416:07Alex Miller (Clojure team)I didn't that see that with clj-kondo, but I'm not running latest version#2019-11-0416:08borkdudedepends on the example. what's the code?#2019-11-0416:10borkdudefound it. yes, that's a case I want to improve. currently it only checks the direct children of (deftest ...) like (deftest foo (= 1 2)), but these cases were inside a let: https://github.com/borkdude/clj-kondo/issues/462#2019-11-0416:30andy.fingerhutThere are also typically many is/are expressions inside of testing subexpressions inside of deftest, or let/testing/let/... etc.#2019-11-0416:31borkdudeyeah. does eastwood test for that btw?#2019-11-0416:34andy.fingerhutEastwood uses tools.analyzer to analyze everything, then looks for all expansions of is forms, I believe.#2019-11-0416:35andy.fingerhutIt is one way to do it, but can be painful to recognize the expansions of is, which is an odd macro.#2019-11-0416:35borkdudebut the point is to detect that is was forgotten. so it warns when you have a deftest without a single occurrence of is?#2019-11-0416:35borkdudeglobally? or does it also point to the location where the is was expected?#2019-11-0416:36andy.fingerhutI'd have to go back to the code to check, but I think it warns about all expressions that are not likely to be called only for their side effects, that do not have is or are around them.#2019-11-0416:37andy.fingerhutI'll follow up in #eastwood channel after checking the code for a little bit.#2019-11-0417:10kszabo@alexmiller If you run the tests through Kaocha, then this might have caught it as well: https://github.com/lambdaisland/kaocha/blob/53d06ab7c56e48cec19bc2ab41a98e0f7b2b860c/doc/clojure_test/assertions.md#clojuretest-assertion-extensions#2019-11-0417:12borkdudeI guess even clojure.test could have built-in warnings like: you're defining a deftest but you never called is.
Or: you called testing, but you never called is.#2019-11-0417:50Alex Miller (Clojure team)But it’s difficult to verify that beyond lexical scope#2019-11-0417:54borkdudeI'm not sure how clojure.test works internally, but is there not some kind of registry to which test an assertion belongs?#2019-11-0417:56borkdudeThere are all kinds of hooks. Maybe it's possible to verify during one of those hooks that nothing got actually checked when running a test.#2019-11-0417:57Alex Miller (Clojure team)Possibly #2019-11-0417:57Alex Miller (Clojure team)I’ve also written tests with negative assertions in code that shouldn’t run in the past. In those cases, even no assertions is correct#2019-11-0417:58Alex Miller (Clojure team)But those are prob rare#2019-11-0418:02borkdudethat's meta: assert on a hook that the number of assertions is zero#2019-11-0418:07Alex Miller (Clojure team)you could also write those kind of tests as a zero? check on a "reached" count in an atom I suppose so there would always be an assertion#2019-11-0418:08borkduderight#2019-11-0418:10Alex Miller (Clojure team)@U08E8UGF7 the build here is a Maven build (as with all contrib builds) so not trivial to integrate#2019-11-0418:10Alex Miller (Clojure team)ironically of course, given it's tools.deps#2019-11-0418:13borkdudeit seems the feature we discussed for clojure.test is something kaocha also does, checking at runtime when the test ends using a hook:
https://github.com/lambdaisland/kaocha/blob/c97a2cbfbb029604edd42e1340c66bd4704007cf/src/kaocha/type/var.clj#L13#2019-11-0418:18borkdudeI would like to see this in clojure.test, but I think the chances are fairly slim it would ever get in#2019-11-0418:18borkdudebut as a library it could work#2019-11-0418:21kszabothis is one of the many reasons why we use kaocha, of course a static analysis check would be better as well. The negative assert feature could be fixed with dedicating another name for it like (clojure.test/never-reached!) which macros could check for. If I got the negative assert idea correct#2019-11-0418:37borkdude@alexmiller https://gist.github.com/borkdude/c43d5b8548d9185f8454bcced3861064#2019-11-0419:01borkdudehttps://github.com/borkdude/naw#2019-11-0419:23kszabocool! so productive, @U04V15CAJ 🙂 Can you pretty please add a deps.edn as well for us tools.deps users.#2019-11-0419:24borkdudeon it 🙂#2019-11-0419:28borkdude@U08E8UGF7 it's in there now. I'm trying to make it work under CLJS as well, but registering the multimethod doesn't seem to work#2019-11-0420:50borkdudeNow also works in ClojureScript.#2019-11-0422:29borkdudeFound an error in one of my own projects!
https://twitter.com/borkdude/status/1191482400599158785#2019-11-0504:02Alex Miller (Clojure team)if anyone has clj on windows and can check https://ask.clojure.org/index.php/8820/why-does-not-merge-deps-across-multiple-aliases-in-tools-deps, that would be cool. I tried it with this deps.edn on mac and couldn't repro:
{:aliases
{:a {:extra-deps {org.clojure/data.xml {:mvn/version "0.2.0-alpha6"}}}
:b {:extra-deps {org.clojure/data.json {:mvn/version "0.2.6"}}}}}#2019-11-0504:05seancorfieldTaking a look...#2019-11-0504:09seancorfieldThere's definitely something odd going on: PS C:\var\www\async.http\sketch> clojure -R:a -R:b -Spath
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.xml\0.2.0-alpha6\data.xml-0.2.0-alpha6.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar;C:\Users\seanc\.m2\repository\org\clojure\data.codec\0.1.0\data.codec-0.1.0.jar
PS C:\var\www\async.http\sketch> clojure -A:a -A:b -Spath
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.xml\0.2.0-alpha6\data.xml-0.2.0-alpha6.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar;C:\Users\seanc\.m2\repository\org\clojure\data.codec\0.1.0\data.codec-0.1.0.jar
PS C:\var\www\async.http\sketch> clojure -Sforce -A:a -A:b -Spath
WARNING: Specified aliases are undeclared: [:a ]
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar
PS C:\var\www\async.http\sketch> clojure -Sforce -R:a -R:b -Spath
WARNING: Specified aliases are undeclared: [:a ]
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar
PS C:\var\www\async.http\sketch>
#2019-11-0504:10seancorfieldI got the same results from -R and -A tho' in all cases (including -A:a:b and -R:a:b, not shown) but with -Sforce, the first alias is somehow not found...#2019-11-0504:12seancorfieldAfter running that -Sforce variant, running the command without -Sforce is now behaving as if :a was not included.#2019-11-0504:12Alex Miller (Clojure team)that extra space in [:a ] is probably important - I presume that's causing the alias not to be found there#2019-11-0504:13seancorfieldPS C:\var\www\async.http\sketch> clojure -R:a:b -Spath
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar
PS C:\var\www\async.http\sketch> clojure -R:b:a -Spath
src;C:\Users\seanc\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\seanc\.m2\repository\org\clojure\data.json\0.2.6\data.json-0.2.6.jar;C:\Users\seanc\.m2\repository\org\clojure\data.xml\0.2.0-alpha6\data.xml-0.2.0-alpha6.jar;C:\Users\seanc\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\seanc\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar;C:\Users\seanc\.m2\repository\org\clojure\data.codec\0.1.0\data.codec-0.1.0.jar
Note that swapping the aliases produces the correct result (since it's treated as a new invocation)#2019-11-0504:15seancorfieldThen I ran it with -Sforce -R:a:b and got the correct behavior back. So I'm suspecting a problem with parsing options if -Sforce is used?#2019-11-0504:18seancorfieldYeah, almost certainly something about -Sforce messing up alias parsing or combining somehow based on further experiments.#2019-11-0504:20seancorfieldThat's with PS 1.10.1.469 BTW.#2019-11-0504:22seancorfieldAnd it's peculiar to the PS script. The exact same sequences of commands on Linux just work.#2019-11-0504:25seancorfield(how did he get .483 on Windows? I tried scoop update clojure and it thinks .469 is the latest... maybe the wiki page has not been updated?)#2019-11-0504:25Alex Miller (Clojure team)I updated the wiki page#2019-11-0504:25Alex Miller (Clojure team)assuming you mean https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#2019-11-0504:26Alex Miller (Clojure team)I don't anything about the scoop thing though#2019-11-0504:26seancorfieldYeah, the scoop script is supposed to use that link in that page to pick up a newer version...#2019-11-0504:26Alex Miller (Clojure team)I wouldn't expect these to differ in this though#2019-11-0504:27Alex Miller (Clojure team)so it's likely same in 483#2019-11-0504:27seancorfieldFair enough. Now I'm still puzzled about the scoop update process 🙂 I'll open an issue...#2019-11-0504:27Alex Miller (Clojure team)I assume these are bugs in the ps script#2019-11-0504:42seancorfieldInteresting. Now I've managed to get this in a clean folder without using -Sforce: PS C:\var\www\async.http\sketch> clojure -A:a -A:b
WARNING: Specified aliases are undeclared: [:a ]
Clojure 1.10.1
user=>
#2019-11-0504:43seancorfieldThat makes a bit more sense given the report on ask...#2019-11-0504:46Alex Miller (Clojure team)[:a ] has an extra space which I assume is the problem#2019-11-0504:47seancorfieldI think all of these need to be wrapped in $(,,, -join '') https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/ClojureTools.psm1#L259-L273#2019-11-0504:47seancorfieldIn the same way as is done here https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/ClojureTools.psm1#L224#2019-11-0504:48seancorfieldI suspect the default conversion for multi-element collections is to have spaces between the elements.#2019-11-0504:48Alex Miller (Clojure team)the tools.deps code splits on colons, so could easily end up with keywords with spaces in them#2019-11-0504:48seancorfieldSo $ToolsArgs += "-R$($ResolveAliases -join '')"
etc#2019-11-0504:51Alex Miller (Clojure team)both $ there?#2019-11-0504:51Alex Miller (Clojure team)guess so#2019-11-0504:53seancorfieldYup, that's the fix. I just updated my .psm1 script to have that and it works as expected now.#2019-11-0504:53seancorfieldAll of those $ToolsArgs += lines need that...#2019-11-0504:54Alex Miller (Clojure team)https://github.com/clojure/brew-install/commit/cc4fba82aee7e8a83269eed666dcfd1180e5069a ?#2019-11-0504:55seancorfieldLooks good!#2019-11-0504:56Alex Miller (Clojure team)ok, there's a windows build out there at .486 now#2019-11-0504:57Alex Miller (Clojure team)https://download.clojure.org/install/win-install-1.10.1.486.ps1#2019-11-0504:57seancorfieldWiki page updated?#2019-11-0505:00Alex Miller (Clojure team)yes#2019-11-0505:02Alex Miller (Clojure team)heading to bed, thx for the help#2019-11-0505:37andy.fingerhutI tried installing Homebrew then brew install clojure on a couple of Ubuntu Linux VMs with different versions, and they both installed version 1.10.1.478 still. brew update clojure on my Mac upgraded to 1.10.1.483, so perhaps Homebrew Linux is taking longer to update package info somewhere?#2019-11-0703:23seancorfield@U0CMVHBL2 FWIW, I installed brew on Ubuntu (actually WSL on Windows 10), and then brew install clojure installed 1.10.1.483.#2019-11-0703:32andy.fingerhutCool. I will check on an Ubuntu Linux system again soon and report back, to see if it gets the latest version now.#2019-11-0505:39andy.fingerhutStrangely enough, though, brew install clojure worked fine on those Linux machines, with no problem finding a working version of ruby executable to do what it needed to install the clojure thingy (replace "thingy" with correct Homebrew term, whatever that is)#2019-11-0512:45Alex Miller (Clojure team)Formula#2019-11-0512:45Alex Miller (Clojure team)Maybe Linux brew is downstream somehow#2019-11-0521:53seancorfieldAnother Windows glitch? Can someone else on Windows PowerShell confirm/deny this for me? PS C:\Users\sean> clojure -Sdescribe
{:version "1.10.1.486"
:config-files ["C:\Users\sean\scoop\modules\ClojureTools\deps.edn" "C:\Users\sean\.clojure\deps.edn" "deps.edn"]
:config-user "C:\Users\sean\.clojure\deps.edn"
:config-project "deps.edn" when run in a folder that does not have it still reports it in describe and as the :config-project path. The macOS/Linux version displays it as :config-project (which still seems wrong) but correctly omits it from the :config-files path.#2019-11-0521:57Alex Miller (Clojure team)The behavior in Mac is right - it’s telling you where it’s looking#2019-11-0521:57Alex Miller (Clojure team)config-files is not actually used now#2019-11-0521:58Alex Miller (Clojure team)The Clojure code will do the right thing if it’s missing#2019-11-0522:02seancorfieldOK, so a small glitch in the PS script but not an important one.#2019-11-0522:03seancorfieldmacOS (! 720)-> clojure -Sdescribe
{:version "1.10.1.483"
:config-files ["/usr/local/Cellar/clojure/1.10.1.483/deps.edn" "/Users/sean/.clojure/deps.edn" ]
:config-user "/Users/sean/.clojure/deps.edn"
:config-project "deps.edn"
#2019-11-0522:03Alex Miller (Clojure team)Seems ok to me#2019-11-0522:07seancorfieldGiven that tooling should no longer be running clojure -Sdescribe via (clojure-env), and such tooling was broken on Windows anyway(!), I agree that nothing really needs to be done. It's just slightly odd to look at.#2019-11-0522:09Alex Miller (Clojure team)config-files is not even passed to the script anymore#2019-11-0614:14Michaël SalihiHello everybody, I would to know if I can kick off a one command line simple http server with Clojure like Python does python3 -m http.server ?
With tools.deps for eg. ? clj -Sdeps '{:deps {some/clojars {:mvn/version "x.x.x"}}}' -m http.server#2019-11-0615:40souenzzohttps://github.com/kachayev/nasus#usage#2019-11-0617:30Michaël SalihiPerfect ! Thx @U2J4FRT2T#2019-11-0615:27dominicmI think there was a snippet for that kicking around#2019-11-0617:26rickmoynihan@dominicm: I’m currently using the capsule builder for pack.alpha, but running into annoyances it brings wrt the JVM trampoline, tunnelling args and also wanting to extend the classpath of the “uber.jar” from the outside via java -cp uber.jar:./my-file ,,,. Is the one-jar builder likely to cause me less friction?#2019-11-0617:27rickmoynihantheir docs describe the problem, but not really their solution other than as a custom classloader that redirects requests.#2019-11-0618:04dominicmHaven't used either in this way.#2019-11-0618:04dominicmOneJAR will likely work well for this case, based on my recollection of its function. It'll delegate.#2019-11-0703:24seancorfieldHow to confuse yourself: in Ubuntu bash on Windows (via WSL 1), type scoop update clojure, and wonder why, after completing successfully, when you type clojure -Sdescribe it shows the version you previously had installed...#2019-11-0703:25seancorfield...then switch to PowerShell and run clojure -Sdescribe and realize that it was updated: Windows figured out scoop was a Windows command, even tho' you typed it into Ubuntu, and ran it on Windows anyway... 👀#2019-11-0703:33andy.fingerhutI'm sticking with Linux in a VM, if I ever use a Windows machine regularly 🙂#2019-11-0703:34seancorfieldWSL 2 is a full VM-based approach. I'm stuck with WSL 1 for... reasons... on both of my Windows systems.#2019-11-0703:34seancorfield(I run Windows 10 on my Mac, BTW, and pretty much only use macOS for Terminal and Docker!)#2019-11-0704:09Drew Verleeso the command here: clj -A:pack mach.pack.alpha.jib --image-name --image-type registry -m my.main
from https://github.com/juxt/pack.alpha. seems to suggest that by passing -m its going to record that as the main namespace. but whats happening is that its "running" that namespace. which makes sense given that passing -m to clj does just that.
what i'm i missing?#2019-11-0704:10Drew Verleehmmm#2019-11-0704:12Drew Verleeactually that might not be the issue. is mach.pack.alpha.jib a script/function im passing clj and the rest are the args?#2019-11-0704:12Drew Verleeyea, thats probably right#2019-11-0704:13Drew Verleemy issue was that i told the pack alias to run the main function.#2019-11-0707:41andy.fingerhutI tried on an Ubuntu 16.04 Linux machine (freshly installed, latest updates from Canonical as of today), to install Homebrew for Linux, then brew install clojure. It installed the latest version 1.10.1.483 which brought up a REPL just fine. The Linux VM I tested on had no previous installation of ruby, clojure, nor Java. brew install clojure installed them all automatically as dependencies, choosing AdoptOpenJDK 8 for the JDK (plus some things that surprised me, but didn't hurt -- e.g. gcc seemed weird)#2019-11-0710:56robert-stuttafordProbably a very-FAQ, but googling didn't help me at all. how do teams deal with sharing deps.edn files that use :local/root? one option is to symlink the common code folder on everyone's machines, but i'm wondering if the lib itself supports something out of the box#2019-11-0710:58dominicmIt depends 🙂#2019-11-0710:58dominicmIf they're released separately, then I wouldn't use :local/root#2019-11-0710:59robert-stuttafordthis is just to support a short period of active new development#2019-11-0710:59robert-stuttafordrelease will use the usual s3 maven repo nonsense, until we can get everything transitioned to deps, and then we'll use the git sha thing#2019-11-0711:04robert-stuttafordis there inbuilt capability to resolve local relative paths, or is symlinks the way?#2019-11-0711:06dominicmjust as a workflow thing for what I do: as someone doing multi-repo development I personally setup aliases in my ~/.clojure/deps.edn file.#2019-11-0711:06dominicmSome of my colleagues don't do any cross-development, so they don't have to care. When I do the cross development, I just activate the extra profile(s) on launch.#2019-11-0711:06dominicmrelative paths do work though.#2019-11-0711:08robert-stuttafordah, i tried relative to home ~/Code/... but not relative to folder ../..., going to try that now#2019-11-0711:09robert-stuttafordthanks Dominic!#2019-11-0711:09dominicmYeah, no support for ~. I think it's been discussed & rejected. .. works though, used heavily in Edge.#2019-11-0711:16robert-stuttafordnice!#2019-11-0722:00Drew Verleeif your projects deps.edn uses git/url to point at another project, does it have to also have a deps.edn? or would a project.clj work? i dont think so, but its been a while.#2019-11-0722:01kszabothe former#2019-11-0722:01Drew Verleesorry, could you clarify? i asked my question awkwardly. It must have a deps?#2019-11-0722:03Alex Miller (Clojure team)it must have either a deps.edn or a pom.xml#2019-11-0722:03Alex Miller (Clojure team)or neither#2019-11-0722:04Alex Miller (Clojure team)deps does not currently support project.clj as a manifest as there is no good way to canonically load it and find the deps without basically running all of lein#2019-11-0722:04Drew Verleeright, thanks alex.#2019-11-0722:04Alex Miller (Clojure team)because project.clj is evaluated, can be modified by plugins, etc#2019-11-0808:47plexus@drewverlee maybe obvious but worth noting, if you control the repo you can just lein pom and check in the pom.xml next to the project.clj#2019-11-0811:07vlaaadI'm starting to feel that both lein pom and clj -Spom is a case of easy over simple. There is a dependency format — pom, that has many different stuff in it. It's much more than list of deps + artifact + version, but tools like lein pom and clj -Spom give only the most common baseline, which might cover some percentage of use cases, but whenever I want to publish a library, I realize that that's not enough. I need to specify developer/license information. I need to specify version as tag in scm information. Someone else will want to specify git sha instead of tag in scm. Dependencies I use during development might go to different profiles, with different activation rules, so what lein/`clj` provides to update dependency list might even be wrong.
I guess ideally I want to produce pom from code I write, aided with information from tools-deps/git and some helpers to generate xml from data, but ultimately be the one who decides what goes where. I guess where is no such tool yet because you can just check in pom.xml and update it with scripts using sed or whatever, and that's good enough.#2019-11-0811:29dominicmUsing sed for xml was hard :) Garamond works okay in this space.#2019-11-0811:30dominicmI think it's okay to say that pom.xml is a combination of data sources into one. I personally would not want to manage the tag/version stuff as I'd definitely get it wrong.#2019-11-0815:37Drew VerleeInteresting. So assuming no plugin modifies the deps. then its as easy as generating a pom for say a project like: https://github.com/weavejester/integrant?#2019-11-0815:38dominicmYup#2019-11-0815:38dominicmAlthough a deps.edn is probably easier#2019-11-0815:43Drew Verleewhat would the output of clj -Spath | grep integrant containing integrant twice imply? Im trying to undertsand a build error.
/home/drewv/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/src:/home/drewv/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/src/main/clojure#2019-11-0815:45Alex Miller (Clojure team)it doesn't contain integrant twice#2019-11-0815:45Alex Miller (Clojure team)it contains two source directories from within integrant (src and src/main/clojure)#2019-11-0815:47Alex Miller (Clojure team)the latter doesn't exist, so no worries#2019-11-0815:47Alex Miller (Clojure team)(that's the standard Maven source path for Clojure source files)#2019-11-0815:49Drew Verleethanks alex. So {:warning "could not find classpath entry", :path "/home/drewv/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/src/main/clojure"} isn't indicating something is wrong with my build? its because ____ produces that path and then a later tool can't find the entry on it?#2019-11-0815:52Alex Miller (Clojure team)I don't know what that is#2019-11-0815:52Alex Miller (Clojure team)where do you see it?#2019-11-0815:52Alex Miller (Clojure team)it's certainly not an issue to the jvm to have a classpath entry that doesn't exist#2019-11-0815:53Alex Miller (Clojure team)which I assume is why that's a warning#2019-11-0815:54Drew Verleethats the output from running depstar https://clojars.org/seancorfield/depstar.
Right, makes sense.
thanks#2019-11-0819:24seancorfield@drewverlee Integrant doesn't have a deps.edn file and that path indicates you are depending on it via a :git/url?#2019-11-0819:25Drew VerleeIt's a fork.#2019-11-0819:27seancorfieldAh, with a pom.xml file. Got it.#2019-11-0819:28seancorfield@alexmiller Does t.d.a. assume src/main/clojure when it sees a POM file in a Git dep?#2019-11-0819:29lreadhello! I am wondering, should :override-deps affect the pom.xml generated/updated by -Spom?#2019-11-0819:29seancorfield@lee If you specify an alias that brings in that override...#2019-11-0819:30seancorfield(at least, that's why I would expect -- I've never tried it)#2019-11-0819:32lreadYep, I am specifying the alias and it does not affect pom.xml in my little test. Could be I botched something.#2019-11-0819:32seancorfieldBased on a quick test, it doesn't seem to.#2019-11-0819:33seancorfield(! 815)-> cat deps.edn
{:paths ["resources" "src"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}}
:aliases
{:test {:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "0.10.0"}}}
:old {:override-deps {org.clojure/clojure {:mvn/version "1.9.0"}}}
:runner
{:extra-deps {com.cognitect/test-runner
{:git/url ""
:sha "76568540e7f40268ad2b646110f237a60295fa3c"}}
:main-opts ["-m" "cognitect.test-runner"
"-d" "test"]}}}
Fri Nov 08 11:33:00
(sean)-(jobs:0)-(~/clojure/pommy.example)
(! 816)-> clj -A:old
Clojure 1.9.0
user=> ^D
#2019-11-0819:33seancorfieldBut clj -A:old -Spom produces a pom.xml that uses 1.10.1 not 1.9.0#2019-11-0819:35lreadthanks sean! -Stree behaves as expected:
> clj -A:old -Stree
org.clojure/clojure 1.9.0
org.clojure/spec.alpha 0.1.143
org.clojure/core.specs.alpha 0.1.24
#2019-11-0819:38seancorfieldI'd say that's a bug. @alexmiller?#2019-11-0820:26Alex Miller (Clojure team)this is trickier than it looks. I may actually have broken it in my recent set of changes. the tricky part is that the aliases are about resolving transitive deps and building a classpath, but you're not actually building a classpath here, you're just making a different manifest file#2019-11-0820:28Alex Miller (Clojure team)so if you supplied override-deps for some lib N levels down, what does that mean? do you add it to the root deps to get the same effect? or just ignore it?#2019-11-0820:54seancorfieldI would certainly expect the dependencies in pom.xml to match the top-level deps in -Stree output but I agree that it's really hard to decide what the transitive dependencies should be. Hmm :thinking_face:#2019-11-0821:03Alex Miller (Clojure team)I used to do a bunch of fiddly logic to half apply stuff from the aliases during gen, but I stopped doing that in my last round of changes and just reused the same core from making classpaths, but did not properly take that all the way#2019-11-0821:38lreadForgive my naive question @alexmiller, I'm sure you've put much thought into this. Wouldn't adding the override of the transitive dep to the root deps give the intended result?#2019-11-0821:39lreadAlso do you want me to raise an issue for this? Or is it via ask clojure now?#2019-11-0821:50Alex Miller (Clojure team)I've actually fixed it locally, so no need to do anything. I'll probably log a jira just for tracking before I commit it#2019-11-0821:51Alex Miller (Clojure team)on the transitive deps, the decision I had made in the past was to only modify the root deps and not include overrides in transitive deps#2019-11-0821:52Alex Miller (Clojure team)and that's what I'm matching in the update. inevitably, all decisions here will be wrong for someone.#2019-11-0821:55lreadtrue enough! So long as behavior is consistent and easy to understand, it works for me! (for what that's worth!)#2019-11-0822:16Alex Miller (Clojure team)@lee https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-143, fix in master but not released#2019-11-0822:16lreadcool, thanks alex!#2019-11-0921:52dominicmhttps://github.com/stuarthalloway/lancet perhaps interesting to this group 🙂#2019-11-0922:05sogaiusomehow i am reminded of boot and badigeon#2019-11-0922:06sogaiu@dominicm thanks, had never seen that 🙂#2019-11-0923:26Alex Miller (Clojure team)Lancet was the long running example in the first edition of Programming Clojure#2019-11-1220:03Alex Miller (Clojure team)I'm going to do a talk about tools.deps / clj next week at the conj - any burning questions I should be sure to answer?#2019-11-1220:06dominicmSettle the debate on namespaces. Talk about java. Leak the plans for tools.ant#2019-11-1220:07seancorfield@alexmiller Talk about add-lib 🙂#2019-11-1220:10Alex Miller (Clojure team)@dominicm if those are serious, I'm not sure what any of them mean#2019-11-1220:12dominicmI'm just teasing :) I was hoping tools.ant would give it away.
Unless there really is a tools.ant...#2019-11-1220:13Alex Miller (Clojure team)well, Lancet was kind of like tools.ant :)#2019-11-1220:16dominicmIt would be interesting to know if there's ideas you have that you'd like others to execute. (E.g. Expanding on Lancet, building X, the community would gladly step up and at the least explore the domain for you)#2019-11-1220:19Alex Miller (Clojure team)yeah, I'll cover some future idea stuff for sure#2019-11-1220:11Alex Miller (Clojure team)@seancorfield maybe, we'll see#2019-11-1220:11dominicmIn earnest, the big questions I've had are:
- how do I do java
- how do I do thing I do with lein (usually uberjar)
- how do I load the tests of my dependencies (usually in the context of a monorepo)
- why would I use this rather than lein#2019-11-1220:12Alex Miller (Clojure team)"do java" == "compile java"?#2019-11-1220:13dominicmSorry, yes. How do I fold java into my program, or use java libraries from source. Stuff like that.#2019-11-1220:13Alex Miller (Clojure team)don't? :)#2019-11-1220:14dominicmGood slide, yep :) (Although the team I said that to then introduced lein and took away all my lovely deps.edn)#2019-11-1220:17seancorfieldNot so much "do java" but "do compile" in the presence of gen-class etc I think would be interesting to at least mention (`compile`, binding the output directory, making sure it exists).#2019-11-1220:18Alex Miller (Clojure team)you guys are too far on one end of the audience distribution :)#2019-11-1220:18dominicmThese are really clojure questions, but is interesting how lein did all this stuff that we didn't understand. Even though it's simple to do yourself.#2019-11-1220:19Alex Miller (Clojure team)but I will keep those in mind#2019-11-1220:20dominicmI guess the recurring theme is "lein did X, deps.edn doesn't, where do I go?"#2019-11-1220:21seancorfieldhttps://github.com/seancorfield/cfml-interop/blob/master/deps.edn -- does compile of a gen-class namespace (caveat about the compile target dir needs to exist). I don't think compile/AOT is that far too one end of the spectrum really -- lots of people seem to assume you need AOT because "everything" out there assumes :aot main.core and uberjar with a compiled -main 🙂 That's a big difference when you switch to CLI/`deps.edn` from the lein world (or even the boot world).#2019-11-1220:26seancorfieldIntro to lein tends to be lein new app myapp; lein run; lein test; lein uberjar; java -jar path/to/the.jar -- so I think the equivalent story with CLI/`deps.edn` needs to be out there "officially" with at least some explanation of the differences.#2019-11-1220:27seancorfield(the current docs expect you to create your need project completely manually, for example, right?)#2019-11-1220:36Alex Miller (Clojure team)where "build" == what?#2019-11-1220:44seancorfield@alexmiller edited. Sorry, meant "create".#2019-11-1220:36Alex Miller (Clojure team)create and deploy an artifact?#2019-11-1220:37Alex Miller (Clojure team)"build" by itself is not a very precise word in this context#2019-11-1220:37vlaaadhow to build your own lein with tools-deps 🙂#2019-11-1220:38vlaaadwhen TDEPS-116 will be solved 😄#2019-11-1220:39vlaaad(link if you are interested: https://clojure.atlassian.net/browse/TDEPS-116)#2019-11-1220:39Alex Miller (Clojure team)it'd be a lot better if that ticket talked about a problem, not a solution#2019-11-1220:39vlaaadit is#2019-11-1220:40Alex Miller (Clojure team)well, not by my reading#2019-11-1220:41vlaaad...just not very clearly: I want to author more fine-grained libraries that may have optional code/dependencies which won't bloat classpath when unneeded. With maven it can be done using classifiers. With gitlibs, it's all or nothing#2019-11-1220:42Alex Miller (Clojure team)I have like 3 questions about every part of that statement#2019-11-1220:43vlaaadI'm ready to answer!#2019-11-1220:43Alex Miller (Clojure team)but I'd suggest pulling the "optional code/dependencies" string as a place to start#2019-11-1220:45vlaaadI don't have an account there unfortunately, can't edit the issue.#2019-11-1220:45Alex Miller (Clojure team)I don't have time to have a long conversation about this right now but something like that is a far better title/thing to describe in a ticket.#2019-11-1220:46Alex Miller (Clojure team)"Allow specifying aliases in coordinates that point to deps.edn projects" is a specific change that might solve a problem, but unlikely to be the only possible change to solve those problems#2019-11-1220:48Alex Miller (Clojure team)Maven classifiers by themselves are just another dimension to your artifact coordinate (and tdeps already supports classifiers). They only get at this kind of a problem in combination with the Maven property system that allows you to conditionalize a dependency coordinate.#2019-11-1221:12sogaiumay be it's not worth saying to this audience, but does the following seem off?
my general impression was that clj / deps.edn tries to address some key issues (e.g. pomegranate-related, git deps, etc.) and wasn't out to replace lein, boot, etc. i thought a somewhat natural path would be for existing tooling to start using clj / deps.edn (or tools.deps, gitlibs, etc.) to take advantage of its capabilities. iiuc, there are also new efforts to build on or make use of tools.deps + related: https://github.com/juxt/pack.alpha, https://github.com/EwenG/badigeon, etc.
i've used at least one lein plugin (https://github.com/reifyhealth/lein-git-down) which, iiuc, makes use of gitlibs with varying degrees of success -- and iiuc there are other plugins (see the bottom of lein-git-down's README for some) that have made attempts. may be that these or similar things haven't "made it into the core of lein" is where some pain comes from?#2019-11-1221:23andy.fingerhutIf your future work part drops any hints about the more recent version of codeq tht you mentioned a few weeks ago in chat, you might be inundated with dozens of requests for more info, unless it is provided during your talk 🙂#2019-11-1221:39Alex Miller (Clojure team)that has nothing to do with tools.deps so dont see why it would come up#2019-11-1221:30seancorfield@sogaiu I went down that path with boot-tools-deps but ultimately it was too difficult to meld t.d.a. and Boot due to their different worldviews on paths and dependencies for anything but the very simplest use cases. I don't know how lein-tools-deps fares in that regard.#2019-11-1221:32seancorfieldI think if lein/`boot` adopted t.d.a. wholesale as their way to handle paths and dependencies at their core, then that "natural path" could happen -- but that's a lot of rework and potentially breaking as well.#2019-11-1221:35seancorfieldAnd it's turned out that the effort that might have gone into that has perhaps gone into building tools on top of t.d.a. that provide the Leiningen-like features as point solutions you can mix'n'match (which is, to some extent, Clojure's general mindset). Both Leiningen and Boot are "easy" rather than "simple" because they are both "batteries included" -- which have served the community well for a long time but the CLI and the t.d.a.-based tools we have now feel more "Clojure-y" to me.#2019-11-1221:41sogaiuthanks for the insights#2019-11-1221:45vlaaadall "burning questions" for Alex to answer seem to be answered immetiately 🙂#2019-11-1221:48vlaaadI agree wholeheartedly with Sean, tools-deps are simple and that empowers me to quickly hack together tools I personally need, because I don't have to deal with semantics imposed by build tool#2019-11-1221:50vlaaadI think there is a lot of space for libraries to emerge that help with that process of developing projects#2019-11-1221:54vlaaadI made a couple of libraries with tools-deps, and the thing that wasn't there is aid to generate pom.xml, for example. clj -Spom is not enough — it's a good starting point, but my build scripts usually have to update different fields in pom, and dependency list provided by tools-deps might end up in different place... I'd like to see a tool that just helps me generating pom from data, and then I will use tools-deps programmatic APIs to provide information I need#2019-11-1221:58seancorfieldYeah, having a minimal pom.xml that satisfies the needs of clojars/cljdoc would be a big improvement over what -Spom produces by default but that would be a fairly simple tool to write I guess. Also something to update the version/tag in the POM file.#2019-11-1221:59seancorfield(shouldn't be core t.d.a./CLI functionality -- the community can easily build this... I just haven't quite felt enough pain yet to do write it myself 🙂 )#2019-11-1221:59kszaboit seems metav just added POM generation with versioning just yesterday 🙂
https://github.com/jgrodziski/metav/commit/e0bfe47f3af0659ac2a0bc624db82f90e2e0a6fe#2019-11-1222:00seancorfieldI could probably even build that into clj-new to be honest so at least you start with a somewhat fleshed out pom.xml file...?#2019-11-1222:00kszabohttps://github.com/jgrodziski/metav/issues/11#2019-11-1222:01kszabo> Pom.xml generation
>I have added the option to generate a pom.xml file based on what is already available from clojure.tools.deps. It actually uses the sync-pom function from tools deps then goes to the generated/updated pom and adds the group-id, artefact-name and version based on what metav already establishes.#2019-11-1222:02kszabosource: https://github.com/jgrodziski/metav/pull/12#2019-11-1222:01kszabothis looks like it fits the bill#2019-11-1222:04kszabowe are very happy users of metav so this is a welcome addition, maybe it would be nice to add this dependency to clj-new? We are mainly using it’s sha-based versioning but of course folks could use it for semver (who haven’t watched Spec-ulation 🙂 )#2019-11-1222:06Alex Miller (Clojure team)I haven't looked at it in many years, but Maven actually has support for storing pom data in arbitrary formats and there was even a clojure format way back when this was conceived of#2019-11-1222:08Alex Miller (Clojure team)maybe that's just taking on a lot more problems :)#2019-11-1222:10dominicmIs there a dimension in which maven is not extensible?#2019-11-1222:10Alex Miller (Clojure team)sadly, no#2019-11-1222:13Alex Miller (Clojure team)https://stackoverflow.com/questions/11170057/what-happened-to-maven-polyglot#2019-11-1318:42rickmoynihanHow do people manage and maintain their aliases?
For example in one of our projects:
To start a basic dev repl (with all configured customers data on the classpath) you need the aliases: -A:dev:test:cljs:prj/all
For me to start a dev repl I need/want to add the extra aliases :cider:sc to those.
To run the tests in CI or via kaocha you need: -A:kaocha:test:prj/all:cljs
To build a specific release for a specific customer we need: -A:prj/customer-1:prod
Plus there are various other configurations of aliases. Right now I tend to “document” these by having scripts e.g. ./bin/cider ./bin/repl ./bin/kaocha apply them. However this is opaque to the tooling.
So cursive users have to manually select the specific aliases they want to translate the scripts to the aliases they want to use when they start a repl; and Emacs users need to run the right script with their own :cider alias appended and cider-connect to the repl.
Then a different project will have different combinations.
I find our alias definitions tend to become smaller over time, so they can be reused in other contexts — but then you end up with lots of small things that you have to remember to add back in when you start your REPL/tests/etc.
It’d be nice if tools deps aliases could also be sets of aliases… e.g. I’d really like to define the following:
:composite-aliases {:dev #{:prj/all :test :cljs :user/dev}
:prj/all #{:prj/customer-1 :prj/customer-2 :prj/customer-3 ,,,}
:prod/customer-1 #{:prod :prj/customer-1}
:prod/customer-2 #{:prod :prj/customer-2}}
#2019-11-1319:25seancorfieldAlias-aliases like that would be very handy sometimes.#2019-11-1319:34hiredmanAt work @seancorfield wrote a shell script that manages a lot of that stuff for us. Outside of work I rely pretty heavily on shell history for that kind of thing (type it in once, that search for it in history to run it again forever after), which has the added benefit of working for things outside of clojure too, but is not really sharable for a team.#2019-11-1319:46seancorfieldThat script knows about the high-level aliases and expands them into the set of low-level ones. It also knows how to run multiple clojure commands in sequence, to simulate lein/boot's ability to run multiple commands in one go#2019-11-1320:03hiredmanI think you might be better off creating coarser aliases. It seems to often be the case then when you add a meta language to a system (in this case aliases in deps are an aggregate built on top of clojure functions, but you can draw a similar analogy with things like di systems) the meta language starts out providing a coarse aggregate of base level functionality, and then overtime becomes finer overtime until it corresponds so closely with the base level that you lose the aggregate functionality and start building a new meta level (:composite-aliases) to try and regain the aggregation#2019-11-1409:18rickmoynihan> I think you might be better off creating coarser aliases
Yeah, I’ve tried that too. That’s essentially what the :prj/all alias is. It’s an alias that adds all (customer) project dependencies to the classpath; and it is essentially the union of all the customer aliases.
I find it becomes hard to then know what deps/paths/config belong to which alias though… e.g. I’m a believer that in the repl you should have both :dev-tooling and :test stuff available. But :test may also want to be included in the alias the CI uses to run the tests.
The problem then is that the groupings become lost, and you need to resort to comments to group things which is far from ideal.#2019-11-1320:05hiredmanlike, maybe get rid of the separate alias for adding cider, add it to the class path for all the dev aliases, but add some kind of runtime configuration option so it only launches nrepl on developers machines that want it#2019-11-1320:08dominicm@rickmoynihan 1. copy and paste 2. Not including things like cider in projects, that's userland stuff#2019-11-1409:21rickmoynihanI agree cider is userland stuff. We treat it as such.
This adds a further complication that each user might define cider in a different alias, e.g. I have :cider-22 :cider-21.#2019-11-1320:10dominicmYou can probably configure cider to call clojure always as -A:cider <prompted arguments>.#2019-11-1409:28rickmoynihanYeah I think you can do that but often we need to run the JVM in a particular environment, with env vars etc set…
So I tend to prefer to cider-connect to an instance started in a terminal.
I’ve solved some of the issues here by starting the dev repl and dependent services, along with shadow-cljs --watch etc with overmind; which uses a Procfile to start stuff; and allows you to share env vars across services.
I then use direnv to share project wide, not just repl wide, configuration… e.g. a port/location of a shared database across services etc.#2019-11-1320:11dominicmDev/test/cljs would probably be copy and pasted into each other as much as they were mutually dependent.#2019-11-1320:11dominicmI'm not sure what's different between customers for you, so it's hard to say what I'd do#2019-11-1409:30rickmoynihanMainly just assets on the resource path; and occasionally customer specific reference data.#2019-11-1409:34rickmoynihanThough some of those assets are bundled into webjars… e.g. css skins.#2019-11-1409:46plexusI quite like having wrapper scripts for this, this is also why I promote adding a bin/kaocha to projects. That also gives you an opportunity to further reduce friction by doing extra setup steps, doing precondition checks (e.g. minimum java version), setting environment variables, passing along JVM options, ...#2019-11-1410:00rickmoynihan@plexus: Yeah we use kaocha; it’s great, thanks for that btw 🙂
Though I think adding bin scripts is a bit of a hack/workaround, as it’s opaque to introspection and tool support. Which is another thing projects (particularly internal ones) then need to document; often for each editor/environment in use on the team.#2019-11-1410:57plexusI agree, but I think that's exactly why they are great, they are executable documentation. I'd rather see bin/cider_repl / bin/proto_repl / ... then having one liners in the README (not too bad) or having them shared ad-hoc through back channels (more common). It removes friction and cognitive overhead for the most common cases. You can still pass on extra args to clojure, and if that doesn't cut it then run it with bash -x, copy the final command into your shell and take it from there.#2019-11-1410:59plexusit also means most developers are doing the same thing. I've been on too many projects where it turns out every single person runs their dev process slightly different from everyone else, causing all kinds of reproducibility issues.#2019-11-1411:33rickmoynihanTIL. nice tip:
> run it with bash -x, copy the final command into your shell and take it from there.#2019-11-1411:36rickmoynihan> I’ve been on too many projects where it turns out every single person runs their dev process slightly different from everyone else, causing all kinds of reproducibility issues.
I think this is a side effect tools.deps has encouraged, by doing less.
Expectations and workflows on lein projects were typically more uniform.#2019-11-1417:51seancorfieldIt's why we have a standardized build script that runs clojure, and also why we have a standardized "baseline" deps.edn which we control via CLJ_CONFIG in the build script.#2019-11-1417:52seancorfieldNo user-level aliases interfering. No possibility of different versions across the team.#2019-11-1417:52seancorfieldThe baseline deps.edn contains all the aliases that anyone needs.#2019-11-1510:54rickmoynihanSo assuming I was on your team for a second…
Presumably I’d need any personal aliases to be committed into that file? What about if I wanted to use classpath overrides etc, to stub a dep with a localised version etc? I like to use scope-capture, assuming noone did already would that need to be a conversation with the team, or would I just add it in as a dep into everyones :dev alias?
I’m curious how the build script works… Does it start a repl as well as build stuff? Does it accept aliases as arguments? Or does it manages alias groups? i.e. ./build repl cider would start it with cider ./build repl cursive suitable for cursive etc?
What environments do people on your team use? Does everyone use chlorine and nrebl like you? Or do folk also user cider, intellij etc?#2019-11-1510:57rickmoynihanYou have me thinking though, that there’s a benefit to the consistency wrt to transitive deps of always at least having things like the cider deps on the classpath even if not everyone uses it; it wouldn’t need to be loaded after all :thinking_face:#2019-11-1511:13danierouxWe optimise for pairing. And therefore value consistency very high.
We have Makefiles. And a make repl creates an nRepl on a known port, and both Cider and Cursive is configured to connect to it. We also share configurations.#2019-11-1511:41dominicmCider doesn't really have deps#2019-11-1513:18rickmoynihanyeah they’re shaded by mr anderson — but not all tools/libs will do this.#2019-11-1514:27dominicmThen we must be careful in our selection#2019-11-1514:28dominicmPick development tooling that itself only has safe deps#2019-11-1514:29rickmoynihanif its always there yeah… if it’s something you can opt in to it doesn’t matter as much#2019-11-1514:32dominicmI would say it's the other way round 😊 if it is bundled with the project and is always present then it doesn't matter. If I load it as a user, then it ought to be stable so I know I'm not causing the reproducibility problems that were mentioned.#2019-11-1514:37rickmoynihanYou might be right but I was thinking more for stuff like:
https://github.com/clojure-goes-fast/clj-async-profiler
I tend to only include it one off when I need it; rather than always have it there… not sure what its deps are tbh.#2019-11-1516:45dominicmThat's a good example of something you only use sometimes#2019-11-1511:42dominicmTooling dependencies have to be carefully selected because of what you describe#2019-11-1511:43dominicmAs someone who likes to tinker and improve on my development process, I find the idea of not supporting personal aliases distasteful.#2019-11-1516:54seancorfieldNo one on our team uses CIDER or nREPL these days but there are still two aliases in our shared deps file for starting such things (old and new versions). #2019-11-1516:58seancorfieldYes, the build script takes a list of aliases to use. Yes, it can start a REPL, REBL, Socket REPL, and several other variants (including old and new nREPL servers).#2019-11-1815:26practicalli-johnWhat does the S in -S option for the Clojure CLI tools stand for? The other option names seem to be fairly obvious in meaning when reading clojure --help. For example -A is aliases, -C is classpath, -R resolve, etc.
Or is -S just a generic 'catch all' prefix for all other commands?#2019-11-1815:46dharriganIf I was to hazzard a guess, I think it means script - as examining the clojure script file that controls the process, S is used for script commands (internal to the script) and also a tree function, in the script package in tools deps alpha.#2019-11-1815:46dharriganI could be 100% waaaay of the mark here ... 🙂#2019-11-1816:09Alex Miller (Clojure team)It stands for "settings"#2019-11-1816:09dharriganaha!#2019-11-1816:09dharrigan🙂#2019-11-1816:09Alex Miller (Clojure team)so, yes a generic prefix to catch many settings#2019-11-1816:11Alex Miller (Clojure team)in particular, the command line has a mix of clj args, clojure.main args, and when used as a runner, your own clojure main args#2019-11-1816:31sogaiuat least now i have two mnemonics :)#2019-11-1818:26practicalli-johnmy brain can rest at ease now, thank you Alex.#2019-11-1823:32danielsz@alexmiller Hi Alex, the console-listener Var in clojure.tools.deps.alpha.util.maven appears to be an implementation of the TransferListener interface. However it isn't a complete implementation, and I am seeing errors to that effect.#2019-11-1823:32danielszReceiver class clojure.tools.deps.alpha.util.maven$reify__7307 does not define or inherit an implementation of the resolved method 'abstract void transferSucceeded(org.eclipse.aether.transfer.TransferEvent)' of interface org.eclipse.aether.transfer.TransferListener.#2019-11-1823:34danielszPlease note that the docs suggest to extend the abstract class AbstractTransferListener rather than implementing the interface. The abstract class ships with no-op callbacks so a partial extension is a-OK and no errors would occur.#2019-11-1823:41Alex Miller (Clojure team)Thx#2019-11-2020:00dadairWhat's the recommended way for evaluating the dependency tree for deps.edn based projects (e.g., lein deps :tree)? I've tried clj -A:dev:test -Spom && mvn dependency:tree -Dverbose=true, but I just get an exception that maven can't resolve the project deps#2019-11-2020:01dadairI'm trying to find out if a dep somewhere is loading an old tools.reader that ends up spitting out some warnings in my project#2019-11-2020:02dadairI've used clj -A:dev:test -Stree, but wasn't sure if there is a more verbose one, as the -Stree approach only shows one dependency on tools.reader and it's definitely of a version that includes a fix to not have the warnings..#2019-11-2020:05nwjsmith@dadair do you happen to have cloverage in your test dependencies?#2019-11-2020:05Alex Miller (Clojure team)can you share the exception?#2019-11-2020:07dadair@nwjsmith yes, through kaocha-cloverage#2019-11-2020:07nwjsmithit contains an AOT-ed version of tools.reader, which doesn’t show up in the dependency tree#2019-11-2020:08Alex Miller (Clojure team)evil#2019-11-2020:08nwjsmithI have a commit in master which brings the dependency up-to-date (https://github.com/cloverage/cloverage/commit/882c21041384c76d397d6429780e10531de9dc40)#2019-11-2020:37dadairCannot target the sha via deps as cloverage doesn't provide a deps.edn#2019-11-2020:09nwjsmithbut a release needs to be cut for it (only @lvh can do that)#2019-11-2020:09nwjsmithand we should probably stop AOT-ing that dependency#2019-11-2020:09nwjsmithI’ve had problems with it too#2019-11-2020:09dadair@alexmiller Could be of note, I am running through WSL2 (Ubuntu 18.04)#2019-11-2020:10Alex Miller (Clojure team)looks like missing clojars repo or something#2019-11-2020:14lvhHi: I think I mentioned this in a ticket somewhere but CI should be building snapshots#2019-11-2020:14lvhI’m on my way to the Conj right now so I can’t check#2019-11-2020:17nwjsmithOh! good call. @dadair you might get rid of those warnings by adding cloverage {:mvn/version "1.1.3-SNAPSHOT"} to your deps.edn.#2019-11-2020:20dadairDoesn't seem to be making a difference#2019-11-2020:21dadairConfirmed that the override is in place, just not working
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "2c484d5a45486c68697f6778637c0168611f6d647f6d"}, :content ("[email protected]")}#2019-11-2020:24Alex Miller (Clojure team)If cloverage has the other one and ends up first in the classpath it will win#2019-11-2020:25Alex Miller (Clojure team)Whether it’s first is pretty arbitrary#2019-11-2020:26Alex Miller (Clojure team)-Spath should be exactly what you’re running#2019-11-2020:29dadairhmm, using -Spath, cloverage is on the snapshot version (1.1.3-SNAPSHOT), and tools.reader is at the fixed version 1.3.2, yet the warnings are still printing..#2019-11-2020:37miikkaFWIW, cloverage-1.1.3-20191022.214104-8.jar still contains AOT'd tools.reader#2019-11-2020:39miikkaOh, right, as it should#2019-11-2022:12Alex Miller (Clojure team)you can take clj out of the loop by doing java -cp <the-path-from-Spath> clojure.main <your-main-args> and then feel free to reorder / edit the cp to try things#2019-11-2106:33miikkaThe Cloverage snapshot still seem to cause the problem. I get the warning with the following classpath by running java -cp ... clojure.main -e "(require 'clojure.tools.reader)":
/Users/miikka/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/miikka/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/miikka/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/Users/miikka/.m2/repository/cloverage/cloverage/1.1.3-SNAPSHOT/cloverage-1.1.3-SNAPSHOT.jar
#2019-11-2106:52miikkaI'm now wondering if :refer-clojure :exclude and AOT actually work correctly together.#2019-11-2106:53miikkaShould be easy to test, but I have to do something else now.#2019-11-2113:16dominicmDepending on coverage goals, one could avoid the monroe shebang: https://www.crystae.net/posts/2019/11/08/two-shebang-papercuts/#2019-11-2114:22Alex Miller (Clojure team)clj 1.10.1.489, tools.deps.alpha 0.8.599
* TDEPS-143 - Fix: use aliases when generating pom with -Spom
* TDEPS-127 - Add: include pom resource directories in dep paths in -Spom (thanks Miikka Koskinen)
* TDEPS-140 - Add: support for Maven server HTTP header properties (thanks Arne Brasseur)
* Fix: Use of multiple aliases in Windows clj script
* Add -Strace for debugging dep expansion#2019-11-2119:06mssgetting a weird error when attempting to use depstar to build an uberjar:
Execution error (FileAlreadyExistsException) at com.sun.nio.zipfs.ZipFileSystem/createDirectory (ZipFileSystem.java:422).
META-INF/versions/9/
#2019-11-2119:07mssI can’t tell if I’ve configured something incorrectly in intellij, if I’m missing an option in building the classpath, or something else entirely. not much is coming up when googling around about a META-INF related FileAlreadyExistsException#2019-11-2119:09mssif people have a recommendation for tooling to use to build standalone jars, I’m all ears as well#2019-11-2119:10mssinterestingly, using depstar to build a slim jar and attempting to run that jar results in a Error: Invalid or corrupt jarfile error#2019-11-2119:39seancorfield@mss I'm at Conj so I won't be able to look at that until next week but if you can put a repro case up on github and create an issue against depstar with a link to your repro case, that would really help #2019-11-2119:41seancorfieldYou can't run a "slim jar" - you can only use that as a library (since it won't contain any dependencies).#2019-11-2121:40jfntnIs there a supported approach for merging deps from a parent directory in a monorepo?#2019-11-2123:09seancorfield@jfntn At work we use the CLJ_CONFIG environment variable to point to the "master" deps file so that it gets combined with the subproject's deps file within our monorepo.#2019-11-2123:10seancorfieldcd subproject
CLJ_CONFIG=.. clojure
(our master deps file is in ../versions not .. but the same principle applies)#2019-11-2218:09jfntnInteresting but that would prevent using user defined aliases etc right?#2019-11-2218:10jfntnAlso how do you enforce the env var, do you have to use a makefile or some additional env tooling?#2019-11-2223:02seancorfield@jfntn Yes, a build shell script that runs clojure one or more times so we can have a single command run multiple tasks, all with CLJ_CONFIG set. And, yes, no user-defined aliases can interfere with our build process. Any developer tooling is available via aliases in that master deps.edn file.#2019-11-2312:20daveliepmannI borked a copy/paste in my deps.edn and briefly had a dependency pointing to a different git repo. In other words the key is library X but the git repo coordinate is library Y. What's the expected behavior in this situation?
org.clojure/tools.deps.alpha {:git/url ""
:sha "e5f3d9e6839191f1e37ddfa51cf442b2d5403ff3"}#2019-11-2312:33Alex Miller (Clojure team)There’s no way for tda to tell #2019-11-2314:14dominicmhttps://github.com/clojure/tools.deps.graph 👀#2019-11-2314:18kszaboI am getting:
Error building classpath. Could not find artifact org.clojure:tools.deps.graph:jar:0.1.5 in central ()
#2019-11-2314:23dominicmThe repo was empty a moment ago#2019-11-2314:25seancorfieldMaven Central can take several hours to list a new release. I suspect that this is something Alex will debut in his talk in a few hours. #2019-11-2314:27kszaboI wish I could be there 😢#2019-11-2314:29seancorfieldAll the talks are showing up on YouTube already!#2019-11-2314:36Alex Miller (Clojure team)that was a dummy version - it's up there now at 0.1.14#2019-11-2314:49dominicmThe conj is still on?! Yay, more to watch#2019-11-2314:49dominicmHaving the videos go up live is fantastic for those of us across the pond#2019-11-2315:48sogaiueven on the same lilypad, it's much appreciated 🙂#2019-11-2315:54seancorfieldEven if you are actually here it's great. I missed a talk yesterday but was able to watch it online later before going out for dinner! #2019-11-2316:07sogaiunice!#2019-11-2314:32Alex Miller (Clojure team)shhh#2019-11-2415:29andy.fingerhutAlex, you mentioned what I think you called logically inconsistent deps that one occasionally comes across in the deps of projects. Is there an existing option to clj/clojure, tools.deps, or some other tool that tells one if a project has such logically inconsistent deps?#2019-11-2415:36Alex Miller (Clojure team)No, it will just muddle through#2019-11-2415:49andy.fingerhutSure. Just wondering for yours (and others) debug purposes if something like that existed, as a machine-detectable property that can signify that here might be dragons.#2019-11-2419:49lreadEnjoyed your talk @alexmiller https://youtu.be/7CM7Ef-dPWQ thanks!#2019-11-2420:26lvhI want to run a tool specified in a deps.edn file as an alias (i.e. with a :main-opts), but with a multiple sets of command line arguments to debug an issue down to a particular combination of arguments. do I just use clojure.main repeatedly or is there a smarter way to do that?#2019-11-2613:41Lone RangerGood morning! Darn conj has me so full of ideas now 😮
So what does it take to hook into the tools-deps machinery? Is there a general primer-faq? Reason I'm asking is b/c I'm working with Chris Nuernberger on libpython-clj and I think it would be a very good idea to be able to specify Python deps declaratively. Is tools-deps designed to be extensible like that or should we use a seperate .edn file inspired by deps.edn?#2019-11-2613:42dominicmIt is extensible, but the clj tool currently doesn't define a way for extensions to be loaded#2019-11-2613:43vlaaadI thinks it's not extensive in that direction. it's a tool to resolve java dependency and build java classpath, it probably can't be extended for other ecosystems such as python or js. they are too different.#2019-11-2613:49Alex Miller (Clojure team)clj is about making classpaths so the question is whether there is any use in putting python stuff on a classpath#2019-11-2613:51Lone RangerSo the use case here is for library authors using the libpython-clj tool#2019-11-2613:51Alex Miller (Clojure team)You could use just the resolve-deps part with custom providers to do the transitive expansion#2019-11-2613:52Lone RangerThere's a chicken/egg problem -- basically the Python VM needs to be loaded before any libpython-clj libraries can be pulled in#2019-11-2613:52Alex Miller (Clojure team)Unclear if that’s worth doing via tolls deps#2019-11-2613:53Lone RangerOf course! I was't sure if it was designed for that or not#2019-11-2613:57Alex Miller (Clojure team)I don’t know enough about python stuff to tell but the provider extensions are pretty generic so it could probably be made to work, just not sure if it’s worth doing#2019-11-2614:40orestisPython has literally a PYTHONPATH which is pretty similar to the classpath. But in practice I think people don’t do that. The Python community changed the “de-facto” dependency manager 4 times in the last 5 years already (I haven’t looked deeply into python rencetly, might be wrong).#2019-11-2615:35dominicmI think they still use the same repo though don't they?#2019-11-2615:37dominicmThe real question is:
- How does libpython-clj search for python code? It could search the classpath, jruby can.
- How does graal python find python code?
Those are the two consumers I can think of.#2019-11-2817:55souenzzoHow do I import https://mvnrepository.com/artifact/org.apache.cxf/cxf/3.3.4 in deps.edn?
I'm trying org.apache.cxf/cxf {:mvn/version "3.3.4" :extension "pom"}
It add a .pom to my classpath, but I can't import anything
(filter
#(string/includes? % "cxf")
(string/split(System/getProperty "java.class.path") #":"))
=> ("/home/souenzzo/.m2/repository/org/apache/cxf/cxf/3.3.4/cxf-3.3.4.pom")
#2019-11-2817:58dominicmhttps://maven.apache.org/pom.html#Dependencies type is documented here for others who have never heard of this 🙂#2019-11-2818:00dominicmokay, https://maven.apache.org/ref/3.6.3/maven-core/artifact-handlers.html is way more interesting. tools.deps maybe doesn't support pom extensions because they're not added to the classpath.#2019-11-2818:06souenzzoOkay. So I will not be able to use this dep at this time?#2019-11-2818:06seancorfieldYou'll need to manually specify all the deps that the POM specifies I think.#2019-11-2818:07seancorfieldI thought there was a way for t.d.a. to pull in things via a pom dep but I can't think what it is right now...#2019-11-2818:14dominicmYou definitely don't want it on the classpath, that's for sure 🙂#2019-11-2818:16souenzzoNice to know#2019-11-2818:15souenzzoSolved by use https://mvnrepository.com/artifact/org.apache.cxf/cxf-core/3.3.4#2019-11-2822:36sogaiui am seeing what looks like extra strings "/main" and "/java" being tacked on to a couple of the non-jar items of a classpath stored in some files in .cpcache.
specifically, for: https://github.com/FundingCircle/fc4-framework/tree/master/tool#running-the-tests , running clj leads to .cp and .libs files containing:
• .gitlibs/libs/clj-chrome-devtools/clj-chrome-devtools/a1b1b0903ae5ba56eaada8496ed3785a53d8cc06/src/main
• .gitlibs/libs/clj-chrome-devtools/clj-chrome-devtools/a1b1b0903ae5ba56eaada8496ed3785a53d8cc06/src/java
looking at the repository for the library in question (https://github.com/tatut/clj-chrome-devtools), i see a src directory but no subdirectories named "main" nor "java".
there is a deps.edn file, but it has no :paths key
is this expected behavior?#2019-11-2822:43seancorfieldThe :paths in fc4-framework/tool/deps.edn has src/main so that will propagate I believe. Not sure where src/java is coming from. What's in your ~/.clojure/deps.edn file?#2019-11-2822:46seancorfieldHmm, perhaps not. I just cloned the repo and ran clj -A:dev:test in the tool folder and when I look in .cp I do not see src/main or src/java#2019-11-2822:46seancorfield(! 858)-> cat .cpcache/1648314215.cp | tr ':' '\n'|fgrep gitlib
/Users/sean/.gitlibs/libs/expound/expound/297e0812eb77ae3aa8f30e823f8bdbbb4650fa28/src
/Users/sean/.gitlibs/libs/clj-chrome-devtools/clj-chrome-devtools/a1b1b0903ae5ba56eaada8496ed3785a53d8cc06/src
#2019-11-2822:47seancorfield^ @sogaiu Maybe this is from your user-level deps.edn file?#2019-11-2822:53sogaiu@seancorfield thanks for checking
my ~/.clojure/deps.edn file is basically a slightly modified version of your public one 🙂 grepping for java in it doesn't turn up any culprits (just the javafx deps for rebl)
i see my clojure-tools is 1.10.1.447 though, may be i should update that and see if it makes any difference#2019-11-2822:57seancorfield(! 861)-> clj -Sdescribe
{:version "1.10.1.492"
:config-files ["/usr/local/Cellar/clojure/1.10.1.492/deps.edn" "/Users/sean/.clojure/deps.edn" "deps.edn" ]
#2019-11-2822:59seancorfield(after updating your CLI, remember to delete .cpcache to make sure it regenerates it)#2019-11-2823:02sogaiuthanks -- what you said about propagating seems to be happening. i failed to mention i was not on the master branch. the branch i'm on has "src/java" as one of its paths.#2019-11-2823:03seancorfieldAh, interesting. With the latest CLI I see src/main in .cp but only for the tool repo itself, not for any transitive deps.#2019-11-2823:03sogaiufwiw, the branch i'm trying is native-image#2019-11-2823:03seancorfieldSo if you update and re-generate and no longer see src/main/`src/java` then it sounds like a bug that got fixed since .447#2019-11-2823:03sogaiuright -- will work on that#2019-11-2823:05seancorfieldI switched to that branch, deleted .cpcache, and re-ran clj -A:dev:test and now I see src/java in .cp but only for the and it does not propagate into the git deps.#2019-11-2823:10sogaiuthanks for checking -- hopefully upgrading will fix it 🙂#2019-11-2823:16sogaiu@seancorfield upgrading seems to have fixed things -- thanks a lot for your help 🙂#2019-11-3020:40Nolantools.deps.graph is wonderful. the missing piece of rooting out any unexpectedness in deps projects. 🙏 x`Inf` @alexmiller#2019-12-0204:26mike_ananevHi! Is there any way to build WAR using tools deps? I know only leiningen plugin for building war, but it is not convenient to use both tools.#2019-12-0204:58seancorfield@mike1452 I don't know if any of these tools can build WAR files https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging#2019-12-0204:58seancorfieldBut a WAR is "only" a JAR/ZIP with some additional metadata files in it...#2019-12-0205:06sogaiuthis was once claims to: https://github.com/EwenG/badigeon#2019-12-0205:07sogaiuperhaps this part of the sample is relevant? https://github.com/EwenG/badigeon/blob/master/sample/badigeon/sample.clj#L200#2019-12-0208:15dominicmSomeone requested it for pack, but I never got round to it because I'm not very familiar with war. I think it should be straightforward using the internal API, because the layout is very similar to the other supported types.#2019-12-0221:26didibusHow does tools.deps resolve conflicts for git deps and local deps ?#2019-12-0221:36seancorfieldI believe it can't resolve conflicts across different types of deps? For two git deps it can resolve to the "most recent SHA" I think. Not sure how it would handle two conflicting local deps.#2019-12-0223:09Alex Miller (Clojure team)Git - newest wins, local - nothing implemented now but I think that is rare#2019-12-0221:41seancorfieldI can't find a JIRA issue for this. @alexmiller is there an issue for this that I'm just not seeing? Or is this one of those "known issues" that hasn't gotten as far as JIRA yet?#2019-12-0223:10Alex Miller (Clojure team)There is a ticket similar to this but I could not repro it last week, hoping to clarify soon#2019-12-0221:43seancorfield@didibus It feels like a local dep should always be preferred over a git dep which in turn should be preferred over a Maven/Clojars dep but maybe that's just my misguided intuition...#2019-12-0222:17didibusYa, it can be tricky. I wonder if introducing a deps concept of version could help, like if you declared the version in deps.edn#2019-12-0222:17didibusNow it could be used to compare across revolvers#2019-12-0222:18didibusAnother thing could be to allow to specify a version when the resolver has no concept of it. So I declare a local deps, maybe I could also tell tools.deps what version it is#2019-12-0222:19didibusBut that one would seem more confusing#2019-12-0223:06Alex Miller (Clojure team)Yeah, not planning to do that. Most Maven artifacts include their shas (central enforces, not sure if Clojars does) which gives you comparison basis w git deps (even across forks)#2019-12-0223:08Alex Miller (Clojure team)Not all cases will be resolvable - in that case it’s an error and you (human) must resolve by picking one via top dep or override#2019-12-0223:27seancorfield@alexmiller Does that mean that if you have a :git/url and a :mvn/version and the pom.xml for the latter includes a <tag> that t.d.a. currently resolves that to a SHA and figures out whether it's before/after the :git/url's :sha? Or is that just speculative "t.d.a. could maybe do this perhaps"?#2019-12-0223:28seancorfield(I'm guessing that it's speculative and in many cases it could not resolve that anyway because there isn't always valid tag/scm information in the POM anyway)#2019-12-0223:47Alex Miller (Clojure team)There usually is scm info#2019-12-0223:47Alex Miller (Clojure team)I have not yet implemented this but all the pieces are already there#2019-12-0300:01didibusI see, so local deps always wins, and in the case of conflicting local deps it would error.#2019-12-0300:01didibusBut maven and git can both use the commit sha to find out which one is newest#2019-12-0300:09didibusI'm not seeing anything scm related or git related in the manifest spec: https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Manifest_Specification there's a version-number field, but that assumes digits#2019-12-0300:10seancorfield@didibus it would be in pom.xml not the manifest#2019-12-0300:11seancorfieldfor example
<scm>
<url></url>
<connection>scm:git:</connection>
<developerConnection>scm:git:@github.com/seancorfield/next-jdbc.git</developerConnection>
<tag>05a6d8a47fcea7dc2928e5e2c176310fb7205e0c</tag>
</scm>
(from next.jdbc's pom.xml that is used for deployment to Clojars)#2019-12-0300:12didibusI thought it was common to put tag to HEAD ?#2019-12-0300:13didibusAlso, what does maven do with that? Is the pom.xml uploded to Clojars as well?#2019-12-0300:15seancorfieldYes, the pom.xml is what tells Clojars how to link to the commit that is tied to the release.#2019-12-0300:16seancorfieldIt can either be a specific tag or a SHA.#2019-12-0300:16didibusI see#2019-12-0300:16seancorfield(I happen to use SHAs, just because I have a deploy script that does that automatically)#2019-12-0300:17didibusSounds like days worth of fun debugging conflicts to be honest 😛 But hopefully it mostly works itself out in practice#2019-12-0300:17seancorfieldWhy would you think that?#2019-12-0300:17seancorfieldClojars shows which commit is referenced in pom.xml#2019-12-0300:19didibusWell, maybe I'm wrong. But I can see the tag not mapping to the correct version, and issues incurring. Though I guess there's nothing to do about that#2019-12-0300:19seancorfieldSure, you can "lie" in pom.xml if you don't have some automated release process...#2019-12-0300:20didibusAnd what does happen if you set HEAD as the tag?#2019-12-0300:20didibusIs that automatically replaced by the HEAD sha at time of deploy?#2019-12-0300:20seancorfieldDeployment is independent of Git (so, no).#2019-12-0300:21seancorfield(at least, that is my understanding)#2019-12-0300:22seancorfieldCertainly, you can mess up the <scm> part of pom.xml -- when I first started publishing to Clojars via clojure-generated POM files, I messed up several times. Hence, automation 🙂#2019-12-0300:22seancorfieldcljdoc also relies on pom.xml being accurate so it can pull the correct version of your repo from GitHub to generate docs that match the release artifact...#2019-12-0300:23didibusI mean, the example on Maven shows HEAD as tag: https://maven.apache.org/pom.html#SCM#2019-12-0300:23didibusIt says: tag: Specifies the tag that this project lives under. HEAD (meaning, the SCM root) should be the default.#2019-12-0300:27didibusAnyways, it does seem more standard then I thought. It should be a pretty nice way to handle cross resolver conflicts.#2019-12-0300:28didibusI guess a local deps could also use the pom.xml found in the directory for conflict resolution ?#2019-12-0300:31Alex Miller (Clojure team)it could if there is one#2019-12-0300:31Alex Miller (Clojure team)although I think the chances that value is meaningful+correct in a local dep is low#2019-12-0300:32Alex Miller (Clojure team)vs the rev in an artifact, which is almost always updated during deployment#2019-12-0300:32Alex Miller (Clojure team)open question what to do for a local dep that also happens to be a git repo#2019-12-0300:33Alex Miller (Clojure team)I would need to think about that for a while and make some tables#2019-12-0300:35didibushammock#2019-12-0300:35Alex Miller (Clojure team)there are requests for git deps on the file system too (which we don't support right now) so that intersects#2019-12-0300:37didibusThe world of dependency management is hard. I've lost so many hours of my life managing dependencies 😛 At least Clojure libs rarely break anything, so which version ends up chosen doesn't matter as much 😛#2019-12-0300:39Alex Miller (Clojure team)yeah, I've been doing this stuff for 22 years in the Java universe. those early years pre-Maven were kinda weird.#2019-12-0300:40didibusI do love how tools.deps resolves conflicts by picking latest though. That's a huge improvement for me with regards to Maven.#2019-12-0300:40seancorfieldSomehow I managed to completely skip Maven until I got to Clojure... 👀#2019-12-0300:42Alex Miller (Clojure team)managing your deps manually is a very instructive process (not very efficient in the long run, but doing it a little really gives you an appreciation)#2019-12-0300:42Alex Miller (Clojure team)I used (and wrote) a lot of bespoke tools pre-Maven#2019-12-0300:44didibusI never lived in the pre-maven world, for the better I guess#2019-12-0310:16niclasnilssonadding libs on the fly in the repl? If I’m not mistaking the commit pointed to in http://insideclojure.org/2018/05/04/add-lib/ is gone, and github says there is no add-lib function or a repl ns in the repo. Any ideas on how people do this nowadays?#2019-12-0310:18dharrigan@niclasnilsson If you look at this: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#2019-12-0310:18dharriganand in particular line 139#2019-12-0310:18dharriganyou'll see how it is used 🙂#2019-12-0310:36niclasnilsson@dharrigan, thanks (I wonder why github didn’t find add-lib in that very repo though?) . But, I don’t get it work though. Installing the deps worked, but I get a NullPointerException when trying to add a lib. Anything you recognize?
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'honeysql {:mvn/version "0.9.8"})
Execution error (NullPointerException) at clojure.tools.deps.alpha.util.maven/make-session (maven.clj:173).
null#2019-12-0310:39Alex Miller (Clojure team)add-lib exists only on a branch, has never been integrated on the main line#2019-12-0310:40niclasnilsson@alexmiller, yep, understood. I used the sha in Sean’s deps.edn right now though#2019-12-0310:44Alex Miller (Clojure team)That sha is on the add-lib branch which typically lags master#2019-12-0310:41niclasnilsson(when I get the NPE)#2019-12-0310:41Alex Miller (Clojure team)You should never see that error but I have seen people report this. I’ve never been able to repro#2019-12-0310:42niclasnilssonInteresting. Any special investigation I can undertake on my env?#2019-12-0310:43Alex Miller (Clojure team)It should be impossible by my understanding of the code so clearly some assumption I have is wrong#2019-12-0310:43niclasnilssonEven more interesting 🙂#2019-12-0310:44dharrigan@niclasnilsson seems to work for me!#2019-12-0310:44dharriganuser=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'honeysql {:mvn/version "0.9.8"})
true#2019-12-0310:44dharriganTry deleting your .cpcache?#2019-12-0310:45niclasnilssonGood idea. Will try. (But I’ll save it some place else)#2019-12-0310:45Alex Miller (Clojure team)Should t be using cpcache#2019-12-0310:45dharrigank#2019-12-0310:45Alex Miller (Clojure team)That’s for caching classpaths but you’re not making those here, you’re dynamically updating runtime#2019-12-0310:46Alex Miller (Clojure team)What repl env are you in?#2019-12-0310:46dharrigan:thumbsup:#2019-12-0310:46Alex Miller (Clojure team)Not all repls have the right classloader setup for this to work#2019-12-0310:49niclasnilssonClearing .cpcaches didn’t work#2019-12-0310:50Alex Miller (Clojure team)I’d be shocked if it did :)#2019-12-0310:50niclasnilssonrepl: Vanilla clj#2019-12-0310:51niclasnilsson(no socket, no nrepl)#2019-12-0310:51Alex Miller (Clojure team)Well, don’t know then#2019-12-0310:52niclasnilssonThe docs say “Note that for successful use, you must be in a REPL environment where a valid parent DynamicClassLoader can be found in which to add the new lib urls.“, but I suppose that’s the case when I just fire up clj?#2019-12-0310:52Alex Miller (Clojure team)Yes#2019-12-0310:52niclasnilssonOk#2019-12-0310:54niclasnilssonHmm… It works if I have a deps.edn only contaning
{:deps
{org.clojure/tools.deps.alpha {:git/url ""
:sha "148cab8f154711f8495bd632ebdc53a14dff7f09"}}}
#2019-12-0310:55niclasnilssonBut not in my project, so likely something in there messes it up. I’ll reduce my deps.edn and see if I can find it.#2019-12-0310:57dharriganyou can do clojure -Stree to get a list#2019-12-0310:57Alex Miller (Clojure team)If you have something that doesn’t work now, I’d take that#2019-12-0310:58dominicmI had a repro for this involving dependency conflicts#2019-12-0310:59dominicmhttps://github.com/juxt/pack.alpha/issues/64 oz transitively brings in a different maven package.#2019-12-0311:00dominicmThe session lookup fails because its the wrong class#2019-12-0311:00dominicmIt's one from a different dependency#2019-12-0311:01dominicmIt's not specific to the add lib branch, it's just the only time users interact with maven outside of the clj internal classpath#2019-12-0311:11niclasnilsson{:deps
{metasoarous/oz {:mvn/version "1.6.0-alpha5"}
org.clojure/tools.deps.alpha {:git/url ""
:sha "148cab8f154711f8495bd632ebdc53a14dff7f09"}}}
and
(require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
(add-lib 'honeysql {:mvn/version "0.9.8"})
causes the NullPointerException.
Unfortunately oz pulls in half the world…#2019-12-0311:13niclasnilssonAh, I see @dominicm already pointed out oz and the cause.#2019-12-0311:15dominicm@alexmiller let me know if there's anything I can do to help with explaining or documenting this. I spent a lot of effort debugging it, and I hate for that to be a waste.#2019-12-0311:15Alex Miller (Clojure team)Oh interesting, I had not seen all that debugging before#2019-12-0311:15Alex Miller (Clojure team)Do you have jira access?#2019-12-0311:15dominicmYou helped me with it 😜#2019-12-0311:15dominicmI do, yes.#2019-12-0311:16Alex Miller (Clojure team)If you could file a tdeps ticket, that would be great for tracking#2019-12-0311:16Alex Miller (Clojure team)I don’t think there is one#2019-12-0311:20Alex Miller (Clojure team)Certainly oz should not be publishing an uberjar, that’s evil#2019-12-0311:21Alex Miller (Clojure team)I’d like to be able to fail better in tdeps though#2019-12-0311:26dominicmoz shouldn't be doing an uberjar, but this is unrelated to that.#2019-12-0311:27dominicmthe problem is that maven-aether-resolver and maven-resolver-provider conflict. I think oz gets the former through pomegranate, but icr right now, will double check before I file the jira.#2019-12-0311:27dominicm@alexmiller I think I did mention the resolution in this channel, is there a better place to mention that kind of thing to make sure you don't miss it next time?#2019-12-0311:34Alex Miller (Clojure team)Jira#2019-12-0311:29dominicmAs a total aside, on my wishlist if for maven to support "provides", as in "juxt/tools.namespace" provides "org.clojure/tools.namespace", so there should only be one of them in the classpath (same for things where a -complete artifact is provided such as antlr. There's N artifacts and a "bundled" artifact too.)#2019-12-0321:55markbastianI'm having a deps issue with (drumroll, please), jackson. I have com.fasterxml.jackson.core/jackson-core {:mvn/version "2.10.1"} in my :deps map, but when I do an ion-dev push it reports a conflict of com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.8"}. This manifests itself when my ns gets loaded with the error "com.fasterxml.jackson.core.exc.InputCoercionException," which is a 2.10 addition. I tried adding the library to :override-deps instead, without success. Any tips on what I am doing wrong (besides needing jackson)?#2019-12-0322:00Alex Miller (Clojure team)can you share your ion-dev alias?#2019-12-0322:00Alex Miller (Clojure team)specifically, is it using :extra-deps or :deps?#2019-12-0322:00Alex Miller (Clojure team)and if it's using :extra-deps, can you switch it to :deps? (need newish version of clj for that to work)#2019-12-0322:01markbastianSure, latest attempt:
{:aliases
{:ion-dev
{:deps {com.datomic/ion-dev {:mvn/version "0.9.247"}}
:main-opts ["-m" "datomic.ion.dev"]}}
:paths ["src" "resources"]
:deps {com.cognitect/anomalies {:mvn/version "0.1.12"}
com.datomic/client-cloud {:mvn/version "0.8.78"}
com.datomic/ion {:mvn/version "0.9.35"}
org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/data.json {:mvn/version "0.2.6"}
metosin/ring-http-response {:mvn/version "0.9.1"}
ring/ring-defaults {:mvn/version "0.3.2"}
ring/ring-json {:mvn/version "0.5.0"}
metosin/jsonista {:mvn/version "0.2.5"}
metosin/reitit {:mvn/version "0.3.10"}
javax.xml.bind/jaxb-api {:mvn/version "2.4.0-b180830.0359"}
}
:override-deps {com.fasterxml.jackson.core/jackson-core {:mvn/version "2.10.1"}
com.fasterxml.jackson.core/jackson-databind {:mvn/version "2.10.1"}}
:mvn/repos {"datomic-cloud" {:url ""}}}#2019-12-0322:02Alex Miller (Clojure team)so the error is not during push, it's when your code is run in the ion#2019-12-0322:03Alex Miller (Clojure team)right?#2019-12-0322:03Alex Miller (Clojure team)that :override-deps in your deps.edn isn't doing anything - that only works in an alias#2019-12-0322:04Alex Miller (Clojure team)but you can just add those to your main :deps#2019-12-0322:04markbastianRight, I get a warning when I push stating that I have the dependency conflict on 2.9.8 and the error is at runtime. I am assuming this is because the ion is using the older version without the InputCoercionException class.#2019-12-0322:04markbastianYeah, that override-deps was just my attempt to get the override to work. It was in deps before.#2019-12-0322:04Alex Miller (Clojure team){:aliases
{:ion-dev
{:deps {com.datomic/ion-dev {:mvn/version "0.9.247"}}
:main-opts ["-m" "datomic.ion.dev"]}}
:paths ["src" "resources"]
:deps {com.cognitect/anomalies {:mvn/version "0.1.12"}
com.datomic/client-cloud {:mvn/version "0.8.78"}
com.datomic/ion {:mvn/version "0.9.35"}
org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/data.json {:mvn/version "0.2.6"}
metosin/ring-http-response {:mvn/version "0.9.1"}
ring/ring-defaults {:mvn/version "0.3.2"}
ring/ring-json {:mvn/version "0.5.0"}
metosin/jsonista {:mvn/version "0.2.5"}
metosin/reitit {:mvn/version "0.3.10"}
javax.xml.bind/jaxb-api {:mvn/version "2.4.0-b180830.0359"}
com.fasterxml.jackson.core/jackson-core {:mvn/version "2.10.1"}
com.fasterxml.jackson.core/jackson-databind {:mvn/version "2.10.1"}}
:mvn/repos {"datomic-cloud" {:url ""}}}
#2019-12-0322:06Alex Miller (Clojure team)ok, well then I would send you to #datomic - this is not the first time they've seen something like this, but I don't know the recommended path forward. I think you're right on with what the problem is.#2019-12-0322:06markbastianI'll give that shot and report the output. I think I had done that before. BTW, the jaxb addition was because of this warning when I do the ion push: [s3-transfer-manager-worker-6] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant#2019-12-0322:06markbastianCool, let me give this a shot and I'll let you know what happened then bounce over to datomic.#2019-12-0322:06Alex Miller (Clojure team)yeah, that's likely due to being on java 9+#2019-12-0322:07Alex Miller (Clojure team)it's included in java 8#2019-12-0322:07Alex Miller (Clojure team)but they've been shedding all those j2ee libs from the jdk#2019-12-0322:07markbastianIs the datomic runtime 8 or 9?#2019-12-0322:07Alex Miller (Clojure team)dunno#2019-12-0322:07Alex Miller (Clojure team)I mean, it must be 8 or 11#2019-12-0322:07Alex Miller (Clojure team)9 and 10 are dead and I can't imagine they're using those, so I'd guess 8#2019-12-0322:09markbastianFull output:
Downloading: com/datomic/java-io/0.1.11/java-io-0.1.11.pom from
(cognitect.s3-libs.s3/upload "datomic-code-d70d7189-fa46-4cd5-8004-423b714e486b" [{:local-zip ".datomic-ions/datomic/apps/ion-repl/stable/ff0ed784b940e81fa02f8a0808a3fc06719bfc1b.zip", :s3-zip "datomic/apps/ion-repl/stable/ff0ed784b940e81fa02f8a0808a3fc06719bfc1b.zip"}] {:op :push})
[s3-transfer-manager-worker-6] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant
{:rev "ff0ed784b940e81fa02f8a0808a3fc06719bfc1b",
:deploy-groups (ion-repl-Compute-1TPY2S14U7NT9),
:dependency-conflicts
{:deps
{com.cognitect/transit-java #:mvn{:version "0.8.311"},
org.clojure/clojure #:mvn{:version "1.10.0"},
commons-codec/commons-codec #:mvn{:version "1.12"},
com.datomic/client-api #:mvn{:version "0.8.37"},
com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.8"},
com.datomic/client #:mvn{:version "0.8.86"},
org.msgpack/msgpack #:mvn{:version "0.6.10"},
com.datomic/client-impl-shared #:mvn{:version "0.8.69"},
com.cognitect/transit-clj #:mvn{:version "0.8.285"},
com.cognitect/s3-creds #:mvn{:version "0.1.23"},
com.datomic/client-cloud #:mvn{:version "0.8.80"},
com.amazonaws/aws-java-sdk-kms #:mvn{:version "1.11.479"},
com.amazonaws/aws-java-sdk-s3 #:mvn{:version "1.11.479"}},
:doc
"The :push operation overrode these dependencies to match versions already running in Datomic Cloud. To test locally, add these explicit deps to your deps.edn."},
:deploy-command
"clojure -A:ion-dev '{:op :deploy, :group ion-repl-Compute-1TPY2S14U7NT9, :rev \"ff0ed784b940e81fa02f8a0808a3fc06719bfc1b\"}'",
:doc
"To deploy to ion-repl-Compute-1TPY2S14U7NT9, issue the :deploy-command"}#2019-12-0322:10markbastianI'll try the deploy, but I think it's going to be broken. I'll try #datomic. Thanks!#2019-12-0322:10Alex Miller (Clojure team)yeah, I mean it's telling you that in the result there#2019-12-0322:10markbastianRight. "We're using this library that hasn't got a class you're going to need."#2019-12-0322:52Alexis Vincent@alexmiller Hi there Alex, thanks for building/maintaining deps. And all the rest you do for the clj community. Much appreciation!
Had a couple of questions.
1. I’m trying to use tools.deps as a lib. But have an issue using resolve-deps . One of the deps.edn files uses a package from a mvn repo, where the repo is declared in the deps.edn file, and where the cli picks it ub correctly. But it blows up for me when using the lib with:
Could not find artifact io.confluent:kafka-schema-registry-client:jar:5.3.0
in central ()
#2019-12-0322:53Alexis Vincentim calling it using
(-> {:deps {'root/root {:local/root "<path to root project>"}}
:mvn/repos mvn/standard-repos}
(deps/resolve-deps nil
{:trace true})))#2019-12-0322:55Alexis Vincentthe required file includes the mvn repo as
:mvn/repos
{"confluent" {:url ""}}#2019-12-0322:58Alexis Vincent2. I have a case where that file adds a jar to its paths. But when using juxt’s pack, it thinks the jar lives at one of the transative local/root dependencies dirs. I dont know if this is a pack problem or something with tools.deps. But im guessing deps.#2019-12-0408:01dominicmI don't fully follow this, sorry. Can you expand?#2019-12-0323:00Alexis VincentHow do I best give you info to help fix the bug? I’ve managed to work around it for now using depstar#2019-12-0403:38Alex Miller (Clojure team)@mail024 so the mvn/repos declaration is in a transitive deps.edn? I suspect that's probably not handled right now#2019-12-0403:46Alex Miller (Clojure team)we don't have a ticket for that but same issue exists with pom deps (https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-46)#2019-12-0403:56Alex Miller (Clojure team)I added a note there#2019-12-0414:54FiVoT#2019-12-0414:54FiVoThe link https://clojure.github.io/tools.analyzer.jvm/spec/quickref.html in the tools.analyzer.jvm README is broken.#2019-12-0414:55FiVoA sorry wrong channel.#2019-12-0415:15Alex Miller (Clojure team)works for me?#2019-12-0415:16Alex Miller (Clojure team)I guess he fixed it#2019-12-0415:16Alex Miller (Clojure team)I must have accidentally nuked it yesterday#2019-12-0415:16FiVoYes it's up again.#2019-12-0620:04dominicmI just responded to this pack issue which I think is a tdeps bug, but I don't know much about maven/pom.xml, so it might be helpful to @drewverlee if someone who was more familiar took a look: https://github.com/juxt/pack.alpha/issues/68#issuecomment-562717952
Essentially it doesn't seem like the sourceDirectories in pom.xml are being read.#2019-12-0620:13Drew VerleeOn my phone atm, I'll be back home on Sunday and I'll look into this.#2019-12-0620:13Drew VerleeThanks for looking into the issue.#2019-12-0620:15dominicmLooks like I'm wrong, I'll take another look in a minute :)#2019-12-0620:09Alex Miller (Clojure team)I don't understand what the expected and actual behavior is?#2019-12-0620:09Alex Miller (Clojure team)I tried
clj -Spath -Sdeps '{:deps {integrant {:git/url "" :sha "c673b85130e553feec6d4c5d2d1ec773a49c929c"}}}'
#2019-12-0620:09Alex Miller (Clojure team)and got
src:/Users/alex/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/alex/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/src:/Users/alex/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/src/main/clojure:/Users/alex/.gitlibs/libs/integrant/integrant/c673b85130e553feec6d4c5d2d1ec773a49c929c/resources:/Users/alex/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/alex/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/Users/alex/.m2/repository/weavejester/dependency/0.2.1/dependency-0.2.1.jar
#2019-12-0620:10Alex Miller (Clojure team)what is expected that's different?#2019-12-0620:11Alex Miller (Clojure team)that includes the srcDirectory (src)#2019-12-0620:12dominicmOh. So it does. My mistake. Reading long strings is hard. I missed that.#2019-12-0620:23sogaiu(side note) ^ been feeding classpaths to sed to separate them darn things 🙂
also was tipped off to using grep's color option to pick out specific things: https://github.com/jpmonettas/clograms/issues/3#issuecomment-560017206#2019-12-0620:25ghadiI feed them to
clj -Spath | tr ':' '\n'#2019-12-0620:30sogaiunice#2019-12-0620:47dominicmAh, the user is using a version of pack running an older tdeps which predates this jira: https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-96#2019-12-0620:49Alex Miller (Clojure team):thumbsup:#2019-12-0620:52dominicmI need to upgrade to the 0.8.x series really, I'm just not keeping on top of this stuff very well at the moment. Life is busy.#2019-12-0620:56Alex Miller (Clojure team)well, should be no breaking changes :)#2019-12-0622:25seancorfield@alexmiller I noticed that recent versions of clojure/t.d.a. actually strip existing newlines from POM files -- was that intentional when you fixed TDEPS-29?#2019-12-0622:26seancorfieldIn my pom.xml files, I tend to have blank lines separating sections at the top-level. clojure -Spom now removes those blank lines. Which is really annoying -- it should not mess with the existing layout.#2019-12-0622:39Alex Miller (Clojure team)It was not intentional but makes sense why that would be#2019-12-0622:40Alex Miller (Clojure team)As you know, parsing, modifying, then emitting xml is tough to do w/o something like this happening#2019-12-0622:41Alex Miller (Clojure team)(Really, xslt is a better tool for this job but I didn’t have much luck going down that route)#2019-12-0622:41Alex Miller (Clojure team)Patches welcome :)#2019-12-0623:02dominicmIs xslt whitespace preserving?#2019-12-0623:41seancorfieldI don't consider it quite annoying enough to create a JIRA issue so I'll just learn to live with it 🙂#2019-12-0623:44Alex Miller (Clojure team)xslt has a lot of options, not sure. I did look seriously at it back when I wrote the pom sync, but that was a couple yrs ago, don't remember the tradeoffs now.#2019-12-0706:45pyrHi, I know it's not tools.deps per-se#2019-12-0706:46pyrbut related, I discovered pulling private repositories with gitlibs was not possible with HTTP auth#2019-12-0706:46pyrwhen using github tokens for automation tooling this can be an issue for finer grained repo access control#2019-12-0706:46pyrI have the following patch if you think this is worth considering: https://github.com/pyr/tools.gitlibs/commit/6b5c18d37450cd86afb70130a126c242f3fad4b9#2019-12-0709:16Alex Miller (Clojure team)We don't want any credentials to show up in deps.edn - with this you'd have plaintext passwords in git urls wouldn't you?#2019-12-0709:22Alex Miller (Clojure team)we've had a lot of discussion about https auth (https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-91, https://clojure.atlassian.net/browse/TDEPS-33) and I think the main question to resolve is whether we keep doing jgit or whether we start shelling out to git and I've been trying to get that decision up to the top of my stack (and even briefly did so a couple months ago before I was interrupted).#2019-12-0709:25Alex Miller (Clojure team)If we're using git auth, you have a few different choices - I don't think anyone wants to manually type in passwords every time (or any time) and that ruins pretty much any automation workflow, but you can use "store" mode (stored in cleartext file), or "oskeychain" mode on mac, or the "Git Credential Manager for Windows" mode on windows which uses Windows Credential Store, or I think there may be other more generic password oracle options for Git.#2019-12-0709:27Alex Miller (Clojure team)If we're using jgit, then it's a matter of where we can read pws from - again you talk to os-dependent things like the keychain, credentials store, etc but we're probably building that ourselves (or maybe something exists, don't know)#2019-12-0709:27Alex Miller (Clojure team)But my main constraint is, don't put it in deps.edn.#2019-12-0711:36pyr@alexmiller OK understood. A possible alternative is to go with the NetRCCredentialsProvider. Shelling out to git would solve these issues#2019-12-0711:36pyrI'm using tools.gitlibs here from outside of tdeps which is why having the tokens in the config makes sense to me in this case but I get that for tdeps not necessarily.#2019-12-0711:39pyrI'll drop my commit for now, thanks for the clarification#2019-12-0711:40pyrLooking at the issue, it's indeed shell-out to git or a custom CredentialsProvider implementation which looks for credentials.edn#2019-12-1113:11borkdudeit is possible to generate a classpath without clojure and spec itself on it?#2019-12-1113:12borkdudee.g. here:
$ clojure -Srepro -Sdeps '{:deps {myscript {:git/url "" :sha "89f66ba2101180a98acf88441693840875288ec9"}}}' -Spath
I want only the git dep to be on the classpath#2019-12-1113:12borkdudebut I'm getting:
src:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/borkdude/.gitlibs/libs/myscript/myscript/89f66ba2101180a98acf88441693840875288ec9:/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
#2019-12-1113:13borkdudeI want to use tools.deps to create a classpath and download deps for a clojure interpreter tool I'm writing#2019-12-1113:14Alex Miller (Clojure team)Via clj, no (due to the install deps.edn). Via tools.deps, yes.#2019-12-1113:15borkdudeso I'd have to write my own tool that uses tools.deps as a lib? ah#2019-12-1113:15Alex Miller (Clojure team)Yes#2019-12-1113:15Alex Miller (Clojure team)clj is a Clojure runner#2019-12-1113:15Alex Miller (Clojure team)It assumes you’re using Clojure#2019-12-1113:16borkdudeok#2019-12-1113:29borkdudebtw I could see this feature be useful for other environments like self-hosted Clojure / planck, which is already using the clojure like this#2019-12-1113:29borkdudeI'll ask in their channel how they solve this "problem" (more of an optimization really)#2019-12-1113:46Alex Miller (Clojure team)You can run tools.deps from clj btw - there is a built in alias for it -A:deps #2019-12-1113:48Alex Miller (Clojure team)It’s not exactly a 1 liner but the tools are there (really depends precisely on what you want to do)#2019-12-1114:00borkdudeI'll figure something out. Btw, dirty hack: {:deps {org.clojure/clojure {:local/root "."}}}#2019-12-1114:12Alex Miller (Clojure team)really ugly 1-liner:#2019-12-1114:12Alex Miller (Clojure team)clj -A:deps -e "(println ((requiring-resolve 'clojure.tools.deps.alpha/make-classpath) ((requiring-resolve 'clojure.tools.deps.alpha/resolve-deps) {:deps {'org.ow2.asm/asm {:mvn/version \"7.2\"}} :mvn/repos @(requiring-resolve 'clojure.tools.deps.alpha.util.maven/standard-repos)} nil) nil nil))"#2019-12-1114:12Alex Miller (Clojure team)prints to stdout /Users/alex/.m2/repository/org/ow2/asm/asm/7.2/asm-7.2.jar#2019-12-1114:13Alex Miller (Clojure team)asm just used as an example#2019-12-1114:15Alex Miller (Clojure team)you could also slurp from deps.edn or whatever (some tools for that stuff in clojure.tools.deps.alpha.reader)#2019-12-1115:10borkdudenice, thanks!#2019-12-1115:36borkdudeI think I'll just use the existing tools though, the entire tooling is already there, caching and all, it'd be a lot of work and error-prone to replicate this. I'll just skip over some known deps that aren't interesting for the interpreter#2019-12-1115:37borkdudeThe idea is that the interpreter is a drop-in replacement for a program you can otherwise run with normal Clojure (give or take some reader conditionals)#2019-12-1115:37borkdudeso re-using most of what's there makes also sense from that perspective#2019-12-1115:38Alex Miller (Clojure team)gotcha - to some degree the :classpath-overrides alias option is a good match for this#2019-12-1115:38Alex Miller (Clojure team)you could just override the classpath part for org.clojure/clojure to nothing#2019-12-1115:39Alex Miller (Clojure team)I guess you'd need to also do clojure's deps though#2019-12-1115:41Alex Miller (Clojure team)clj -Sdeps '{:aliases {:bye {:classpath-overrides {org.clojure/clojure "" org.clojure/spec.alpha "" org.clojure/core.specs.alpha ""}}}}' -A:bye -Spath#2019-12-1115:42borkdude{:aliases {:babashka {:classpath-overrides {org.clojure/clojure nil
org.clojure/spec.alpha nil
org.clojure/core.specs.alpha nil}}}}
$ clj -C:babashka -Spath
src:::
#2019-12-1115:43borkdudePerfect!#2019-12-1115:50Alex Miller (Clojure team)I should probably be killing the nils in there :)#2019-12-1115:50Alex Miller (Clojure team)but yeaeh#2019-12-1115:54Alex Miller (Clojure team)committed that fix for whenever I do next release#2019-12-1115:56borkdudemaybe remove empty?#2019-12-1115:56Alex Miller (Clojure team)?#2019-12-1115:57borkdudeinstead of remove nil? so empty strings are also "killed"#2019-12-1115:57Alex Miller (Clojure team)well, would use str/blank? for that case#2019-12-1115:58Alex Miller (Clojure team)imo it's bad form to seq-ize strings with empty? vs using a string-specific predicate#2019-12-1115:59borkdudeok#2019-12-1115:59Alex Miller (Clojure team)done#2019-12-1116:02borkdudeawesome 🙂#2019-12-1116:12borkdudeNot that I need it right now, but it is possible to override the tools.deps version for the clojure tool to a git dep on master?#2019-12-1116:50Alex Miller (Clojure team)no, not without hacking the script#2019-12-1116:51Alex Miller (Clojure team)hacking works fine though, I test stuff that way all the time :)#2019-12-1116:52Alex Miller (Clojure team)I guess maybe for this use of -A:deps, you might be able to use :override-deps or :classpath-overrides to make that work#2019-12-1116:58Alex Miller (Clojure team)that is, there is no hook to affect the classpath of the jvm run that uses tools.deps.alpha to make the classpath, but you could use the existing alias modifiers to change the classpath of the program you're running if it uses tools.deps.alpha#2019-12-1217:00scottloweHi there. I’m working on a project that has a dependency on a large repo of test data, which we don’t want comitted in the main project repo.
Once retrieved, the directory is 295 MB uncompressed on disk. We are seeing OOMEs when tools.deps is retrieving via tools.gitlibs:#2019-12-1217:00scottloweCloning:
#2019-12-1217:00scottloweThis error never occurs locally on macos dev machines, but when running on a Travis CI Linux VM. Ubuntu Bionic.
I note that tools.gitlibs “0.2.64” is using jgit “4.10.0.201712302008-r”.#2019-12-1217:00scottloweDigging a bit futher it looks like jGit is on v5.x series and tools.gitlibs is on the v4.x series. Notice that some recent releases of jGit have memory leak fixes. Hard to say if this relates directly to our issues, though.#2019-12-1217:01scottloweIs this a bug that I should create a ticket for? Or are we using tool.deps outside of its parameters of intended use?#2019-12-1217:02scottloweI’ve tried giving the JVM a good deal of memory, and when that didn’t work, I’ve tried restricting memory to a smaller amount (1.2 GB) so that the JVM can better plan its memory use… but neither approach has worked.#2019-12-1217:03ghadi@scottlowe how specifically did you give the JVM more memory?#2019-12-1217:04scottloweHi Ghadi. I used an env var e.g.: JAVA_OPTS=-Xmx1280m. Less than was available on the Linux instance. I wanted to avoid the Linux OOM-Killer… which may or may not have been related.#2019-12-1217:08ghadiclearing up a couple misconceptions:
1) clj launches two JVMs, one to clone the repos, then another for your project. To modify the JVM for the first one, you need to change this line https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/clojure#L314 in /usr/local/bin/clojure or wherever your script lives
2) JAVA_OPTS isn't a thing, but a convention in some scripts -- not honored by clj#2019-12-1217:09scottloweWow. I didn’t know that. Thanks for the education 🙂#2019-12-1217:09ghadinp#2019-12-1217:10ghaditry bumping up from 256m and your problem may (?) go away -- definitely worth a issue though, right @alexmiller?#2019-12-1217:10scottloweI will go and try some different memory allocations. Could you just confirm that tools.gitlibs should be able to handle this size of dependency, if conditions are right?#2019-12-1217:11ghadinot sure what you want me to confirm 🙂 your analysis of the issue seemed right#2019-12-1217:12ghadibumping JGit is on the radar, as well as maybe making that whole subsystem more resilient#2019-12-1217:13dominicmSeems like loading the code into memory would be a mistake somewhere in the pipeline.#2019-12-1217:13dominicmIt can be written straight to disk.#2019-12-1217:13ghadiyou're making an assumption that it's loading into memory#2019-12-1217:13scottloweYeah. I was wondering about that… should be able to stream any amount of data straight through. Thanks. I’ll go away and experiment, then. Will report back either way… but it may take a few days to be sure that the memory tweaks have worked because we don’t see it every time, because of CI dependency caching etc.#2019-12-1217:14ghadiit might be a large git pack file being decompressed#2019-12-1217:38Alex Miller (Clojure team)I have some tabs and editors open working on validating the latest versions of jgit, but it's a notch or two down the stack right now. Kind of an interim step towards comparing latest jgit vs shelling out to git for https/ssh access#2019-12-1217:39Alex Miller (Clojure team)generally, I'd expect the jvm heap on the cp-making jvm to grow quite a bit larger, but it depends what jvm and version you're using#2019-12-1218:21scottloweAhh. JVM versions... that’s another longer story. In our dev and CI environment we’re currently restricted to using OpenJDK 8 because of a DB dependency. Our Clojure code should be fine running under version 11 and we hope to move to OpenJDK 11 early next year. It’s possible we could use multiple JVM versions in the meantime (to start the DB under a seperate JVM), using a version switcher or something, but that might be asking for trouble as well as more complexity!#2019-12-1218:21scottloweThanks for the comments Alex. I had heard about the shelling out approach for future versions of tools.gitlibs, and I’m hopeful about that approach. I look forward to your findings. 🤞#2019-12-1304:48mattyulrichHi all - I’m sure this has been asked before but I just can’t find the answer. Is there a way to signal in my deps.edn that one of my deps is a bom and not a jar?#2019-12-1305:10mattyulrichOr, actually more specifically, I seem to have problem with a bom as transitive deps.#2019-12-1308:13dominicmWhat's a bom?#2019-12-1308:17andy.fingerhuthttps://howtodoinjava.com/maven/maven-bom-bill-of-materials-dependency/#2019-12-1308:56dominicmOops. I was totally expecting it to be a typo of pom.#2019-12-1309:06andy.fingerhutYou may be right, but bom does have a relevant meaning here.#2019-12-1309:17plexusnot to be confused with a byte order mark#2019-12-1314:43mattyulrichHey all - actually noticed some updated support for this. Switched to the latest tools.deps and the transitive dep was resolved.. yeah - I was talking about a “bill of materials” dep..#2019-12-1311:02codeasoneWhat route should I take to get the equivalent of lein uberjar with :aot :all using the Clojure CLI tools?#2019-12-1311:04codeasoneI'm presently using depstar but there doesn't appear to be an equivalent option#2019-12-1317:00seancorfieldWe use depstar at work to build uberjars for production deployment and we do not AOT anything. #2019-12-1400:23codeasoneThanks for replying @U04V70XH6. Along with deploying our uberjar to ECS within a docker image, we also invoke it multiple times within our CI pipeline. My principle motivation was to shave some time off the CI runs, and additionally plain curiosity around why the facility wasn't being supported yet.#2019-12-1400:25seancorfieldFair enough. If you (manually) create a classes folder in your project and add it as an :extra-paths entry when building your uberjar, you can run clojure -e "(compile,'my.entry.point)" as a way to compile the entry namespace (`my.entry.point` here) and its transitive deps.#2019-12-1400:27codeasoneGreat tip: thanks!#2019-12-1400:32seancorfielddepstar tries hard to avoid messing with your project or your classpath: it just packages up whatever you tell it to, using the classpath as constructed by clojure using whatever options you've provided. Because compile requires that classes exists -- or else you can use binding to tell the compiler to put class files somewhere else, that must still exist -- and that the output path of the compiler needs to be on the classpath for depstar to include those files, it just doesn't seem like something depstar itself should try to do.#2019-12-1400:37seancorfieldNote that the depstar README does address this case https://github.com/seancorfield/depstar in terms of Main-Class in the manifest but it stops short of explaining specifically how to compile code and get the classes onto your classpath.#2019-12-1418:18codeasoneThanks for clarifying.#2019-12-1311:54kszaboAOTing is not recommended anymore unless you deeply care about startup perf#2019-12-1311:55kszabobut if you really want to do it, I prefer https://github.com/EwenG/badigeon#2019-12-1312:00borkdudeif you have an :aliases alias in ~/.clojure/deps.edn and one in your project deps.edn, it seems they are not merged?#2019-12-1312:01borkdudee.g. putting:
:babashka {:classpath-overrides {org.clojure/clojure nil
org.clojure/spec.alpha nil
org.clojure/core.specs.alpha nil}}
in ~/.clojure/deps.edn
and in the project.deps::
:babashka {:classpath-overrides {my_gist_script nil}}
does not seem to merge#2019-12-1312:50Alex Miller (Clojure team)They merge-with merge at the aliases level iirc#2019-12-1314:53mattyulrichI’m sure this has been asked before, but can’t seem to find it. When within a running clj repl, is it possible to resolve new deps; kind of like boot-clj set-env! ?#2019-12-1315:13Alex Miller (Clojure team)there's a speculative feature called add-lib in a branch for this#2019-12-1315:13Alex Miller (Clojure team)http://insideclojure.org/2018/05/04/add-lib/#2019-12-1315:13Alex Miller (Clojure team)(sha in that article is out of date)#2019-12-1919:35vinnyataidewhere do I find the correct sha?#2019-12-1919:47Alex Miller (Clojure team)whatever the newest sha in the add-lib branch is generally https://github.com/clojure/tools.deps.alpha/tree/add-lib#2019-12-1919:50Alex Miller (Clojure team)it was pretty far out of date so I just merged it in 19d197ab221d37db750423eb970880cb87a91100#2019-12-1920:08vinnyataidethanks#2019-12-1315:21mattyulrichWow - this is great! Thanks Alex!#2019-12-1315:24Alex Miller (Clojure team)I think we will probably add this, but some of the details are likely to change#2019-12-1317:00seancorfieldWe use depstar at work to build uberjars for production deployment and we do not AOT anything. #2019-12-1317:01seancorfield@codeasone why do you want to AOT?#2019-12-1317:02Alex Miller (Clojure team)it is perfectly reasonable to AOT a final uberjar for deployment#2019-12-1318:48hlolliI asked this question few months ago, but since that time, did anyone write a lein-ancient equivalent for tools.deps?#2019-12-1318:55Laverne SchrockThere's depot and deps-ancient (I've only used depot). https://github.com/clojure/tools.deps.alpha/wiki/Tools#deps-management#2019-12-1318:58seancorfieldWe've been using Depot very happily (although I've recently developed a bash script that does crazy stuff with clojure -Stree and a synthesized deps.edn file to get outdated deps in a way that suits us better)#2019-12-1319:16hlolliwow nice, essential tool in any toolkit. I knew someone had made it by now 🙂#2019-12-1319:16hlollithanks#2019-12-1319:19borkdude❤️ depot#2019-12-1319:20borkdude@U04V70XH6 care to share the bash script? I'd be curious if I can port it to babashka and make a nice example out of it 🙂#2019-12-1319:22seancorfieldTrust me when I say it's "crazy stuff" -- and it relies very heavily on our custom build shell script that drives our use of clojure...#2019-12-1319:22borkdudeok#2019-12-1319:25seancorfieldThe TL;DR is that it runs clojure -Stree to a file, runs that through sed etc to produce a temporary deps.edn with "RELEASE" for all of the dependency versions (that were :mvn/version explicit versions), then runs clojure -Stree on that, and then it compares the output of both trees, with some fudging to account for only the top two levels of dependencies...#2019-12-1319:26borkdudesounds like a good hack to me 🙂#2019-12-1319:33seancorfieldRunning Depot reliably on our huge monorepo and getting all the outdated deps that matter and avoiding the noise from transitive deps that I don't care about was very tricky, especially since we use CLJ_CONFIG to setup a baseline deps.edn with :override-deps for a lot of stuff...#2019-12-1319:36seancorfield...any t.d.a. tooling that tries to read and process deps.edn files in isolation can run into problems with our setup -- tooling that uses the same process to read/merge the environment as clojure itself does works better for us.#2019-12-1320:00borkdude@U04V70XH6 https://github.com/borkdude/babashka#list-outdated-maven-dependencies
Thanks for the idea 🙂#2019-12-1320:04seancorfieldWow, hahaha!! Important to note that your code won't work with any deps.edn that relies on ~/.clojure/deps.edn (or CLJ_CONFIG) to override/default any deps... 🙂#2019-12-1320:05borkdudeyeah, this is just a quick sketch which should be tuned to the needs of the scripter#2019-12-1320:06seancorfieldIt just goes to show what can easily be done -- and it would work for "most people" I suspect.#2019-12-1320:06borkdudeprobably one could come up with a way to use git to inspect newer shas as well#2019-12-1320:07borkdudebut that's an exercise for the reader 🙂#2019-12-1320:11borkdudeOne could even use this for project.clj by just parsing the deps from it, generate a deps.edn and then run this script#2019-12-1321:16scottlowe@ghadi I’m reporting back on yesterday’s tools.gitlibs OOME issue, which was seen when downloading large repo dependencies: bumping JVM Xms from 256m to 512m seems to have worked. I’ve not seen a failure yet, anyway. Thanks again for your instantaneous help.#2019-12-1321:16ghadiNo problem. What jvm version and vendor do you use btw?#2019-12-1321:18scottloweFor the CI environment it’s OpenJDK 1.8.0_222#2019-12-1321:28Alex Miller (Clojure team)seems pretty weird that jvm wouldn't grow to 512m#2019-12-1321:28Alex Miller (Clojure team)I wonder if you're getting client jvm by default instead of server#2019-12-1321:31scottloweIndeed - I had wondered myself why this would be so; I shouldn’t need to babysit the jvm (?); but it does seem to be the server jvm:#2019-12-1321:31scottlowe$ java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
#2019-12-1321:32Alex Miller (Clojure team)using same version locally, I get an initial of 256m and max of 4g with no settings#2019-12-1321:33Alex Miller (Clojure team)with -Xms256m as set in the script, I see the same#2019-12-1321:35Alex Miller (Clojure team)however, the max is dependent on the available ram - I think it's 1/4 total ram by default#2019-12-1321:35scottloweI agree that it is odd. This is on a Travis CI instance: Ubuntu 16.04.6 LTS (xenial). We hope to move to more recent versions of the JVM and Ubuntu soon. We’ve never seen this issue on developer machines (macos). The CI environment is more memory constrained than our dev machines. 7.5 GB, I think; but some of that memory is taken up by docker processes, as well as a JVM database process.#2019-12-1321:38andy.fingerhutMaybe the Travis CI instance has 1G RAM default?#2019-12-1321:40Alex Miller (Clojure team)certainly could be something like that#2019-12-1321:47scottloweHave checked the Travis CI instance: System Memory: 7680MiB, 64 bits#2019-12-1823:29caiois there an equivalent of lein's :java-source-paths for deps.edn?#2019-12-1823:32Alex Miller (Clojure team)no, clj doesn't support compilation or java source#2019-12-1823:33Alex Miller (Clojure team)it's intended for Clojure source only projects that don't require "building"#2019-12-1908:45vlaaadand yet there are tools already on top of tools-deps that do building and stuff#2019-12-1908:47vlaaadalthough they are mostly about making jars from clojure source, haven't seen any that compile java#2019-12-1909:04stijnis there a limit on the length of paths for tools.deps? when running clj (it's datomic ions push, but I don't think that matters) in Codebuild with local cache enabled for gitlibs it will fail with the following exeption
{:command-failed
"{:op :push :region us-east-1}",
:causes
({:message "Unable to transform path",
:class ExceptionInfo,
:data {:home "/root",
:prefix "/root/.gitlibs/libs",
:resolved-coord
{:git/url "<redacted>",
:sha "e7cc7c48ea0af52f9380c61d2d946cac5d7068e0",
:deps/manifest :deps,
:deps/root "/codebuild/local-cache/custom/b11f68d510f24e13d8132571e19ad44f6bd772898be1fc7b98e76106d7e1ab50/root/.gitlibs/libs/<redacted>/<repo>/e7cc7c48ea0af52f9380c61d2d946cac5d7068e0",
:paths ["/codebuild/local-cache/custom/b11f68d510f24e13d8132571e19ad44f6bd772898be1fc7b98e76106d7e1ab50/root/.gitlibs/libs/<redacted>/<repo>/e7cc7c48ea0af52f9380c61d2d946cac5d7068e0/src"]}}})}
If I run locally I don't see this problem and also when turning off cache for gitlibs, the problem disappears. Codebuild cache will create a symlink of /root/.gitlibs to /codebuild/local-cache/custom/b11f68d510f24e13d8132571e19ad44f6bd772898be1fc7b98e76106d7e1ab50/root/.gitlibs making the directory maybe too long?#2019-12-1913:39Alex Miller (Clojure team)I’m not aware of a length issue, should probably ask in #datomic#2019-12-1913:14sogaiu@vlaaad i think badigeon compiles java: https://github.com/EwenG/badigeon#2019-12-1913:20vlaaadoh, right!#2019-12-2118:56practicalli-johnIs it possible to create a project structure as follows with clj-new or any similar tools. Is it just a matter of creating my own template? Or just hacking the project files once its created
advent-of-clojure
|_ src
|_ practicalli
|_ advent_of_code_2019
|_ 01_trinity_of_rocket_equasion.clj
It would have the matching path for test branch too.
/cc @seancorfield#2019-12-2119:06seancorfield@jr0cket This will get you pretty close to what you want:
clj -A:new app practicalli/advent-of-code-2019.01-trinity-of-rocket-equation
#2019-12-2119:06seancorfield(! 951)-> tree advent-of-code-2019.01-trinity-of-rocket-equation/
|____
| |____.gitignore
| |____.hgignore
| |____CHANGELOG.md
| |____deps.edn
| |____doc
| | |____intro.md
| |____LICENSE
| |____README.md
| |____resources
| | |____.keep
| |____src
| | |____practicalli
| | | |____advent_of_code_2019
| | | | |____01_trinity_of_rocket_equation.clj
| |____test
| | |____practicalli
| | | |____advent_of_code_2019
| | | | |____01_trinity_of_rocket_equation_test.clj
#2019-12-2119:17practicalli-johnThat great thanks. All I would do is just edit the top level directory name and its perfect.
I was looking at the clj-new documentation and thinking I could also do this with generators. adding new namespaces with clj -A:new -m clj-new.generate ns=practicalli.advent_of_code_2019.01_,,,.clj . This would be handy to script up the adding of a namespace for each day, especially if I wanted a separate -data.clj file for my unique data (keeping the solution namespace cleaner).#2019-12-2119:30seancorfieldYup, once you have the project folder, you can add new namespaces via the generator logic.#2019-12-2512:59borkdudemade a thing: https://github.com/borkdude/deps.clj#2019-12-2518:44jApolgies if this is posted in the wrong group! I created a uberjar using clj -A:new app and clj -A:uberjar myproject.jar alias from @seancorfield’s dot-clojure aliases. I'm getting the following error message though: Error: Invalid or corrupt jarfile myproject.jar
I copied the aliases from the deps.edn from dot-clojure over to the deps.edn in the project to run the :uberjar alias. How do I make a working uberjar?#2019-12-2519:13seancorfield@watchtheblur The README on depstar explains all that...#2019-12-2519:13seancorfieldhttps://github.com/seancorfield/depstar#2019-12-2519:14seancorfieldAt work we run all our uberjars like this: java -cp /path/to/the.jar clojure.main -m entry.point -- which will run entry.point/-main as the "main" function. But the readme explains several other approaches.#2019-12-2519:16jIt worked! Thanks for the assist again!#2019-12-2607:09seancorfieldseancorfield/clj-new {:mvn/version "0.8.0"} -- https://github.com/seancorfield/clj-new/releases/tag/v0.8.0 -- adds new options and pom.xml generation (for library and template projects) -- Happy Holidays! 🙂#2019-12-2614:49rafaelHi. I'm wondering if there are best practices for locally referencing (or referencing via git) a project that relies on gen-class ? If I explicitly compile, everything works out fine, and it was easy to build a task to do that. I'm not sure how to fit it into the development workflow, though. Should I require every contributor to compile locally before working on dependent projects or should I just commit the class files to git and be done with in? Are there alternatives? Can I add a hook somewhere that will compile the gen-class project before starting a repl on a project that depends on it?#2020-12-3013:49mynomotoWe use make to ensure that the necessary classes are compiled when working on dependent projects.#2019-12-2810:09sveriHi. I am using cursive which can start an nrepl with deps. Now, when the repl is started I want to switch to a specific namespace and run a function inside.
Using the clojure.main repl I was able to do that by providing a main opts entry point like this:
:aliases {:repl {:main-opts ["-e" "((requiring-resolve 'de.sveri.sg.user/startup))"]}}
Is there something similar for the nrepl?#2020-12-3000:16seancorfieldseancorfield/clj-new {:mvn/version "0.8.1"} -- https://github.com/seancorfield/clj-new/releases/tag/v0.8.1 -- adds pom.xml to app template as well; adds jar/uberjar building aliases to all three built-in templates; expands documentation on the built-in templates how to modify pom.xml generation via the -e / --env options.#2020-12-3000:19seancorfieldIn particular, note that projects generated from the app template now let you run clojure -A:uberjar to produce a JAR file that can be run with java -jar -- the :uberjar alias includes a compile call on the main namespace and the classes folder is added to the paths when building the JAR.#2020-12-3009:49felipebarrosI would like to mimic the effect of npm install or lein deps with tools-deps. That is, to fetch the dependencies and exit. Running clj -R:alias always leaves me in a running REPL and I could not find an option in the documentation to do what I want.#2020-12-3009:54delaguardoclj -Stree will fetch all dependencies, print out complete deps tree and exit#2020-12-3009:55felipebarrosThanks, that will work :)#2020-12-3010:24borkdudeanything with -Sforce works#2020-12-3010:25borkdudee.g. -Sforce -Spath also works#2020-12-3014:32Alex Miller (Clojure team)These are good answers. you can also clj -e nil#2020-12-3014:32Alex Miller (Clojure team)if you want to run something and immediately exit#2020-12-3016:46felipebarrosThank you :)#2020-12-3101:57fabraoHello all, I´m confuse about using boot or deps.edn. What is the advantage of using one beside other? I switched from lein to boot because seancorfield told me that with boot we can load deps "on-the-fly" and I was using a lib in lein to do this. I saw this https://twitter.com/seancorfield/status/1171275237805060096?s=20 and https://www.youtube.com/watch?v=CWjUccpFvrg&t=46s and in terms of dev workflow it seems other way to do things. What is the most benefit of using deps.edn?#2020-12-3102:06seancorfield@fabrao Boot provides a whole bunch of tools "out of the box" -- although it does not incorporate a test runner (last I checked you needed to add Adzerk's boot-test for that, nor is "boot new" technically built-in but it's an easy command-line dependency to use). As noted, the boot repl allows you to add new dependencies dynamically, and it has some file management tooling built in (such as building jar and uberjar files).#2020-12-3102:07seancorfieldThe CLI/`deps.edn` is simple and lightweight, with a range of easy to add tooling for running tests, building jar files, etc. There's an add-lib branch of tools.deps.alpha which you can use -- illustrated in my dot-clojure file -- to add new dependencies dynamically.#2020-12-3102:08seancorfieldI would classify lein and boot as "easy" -- lein more so than boot -- whereas I would classify clj as "simple" (in the general Clojure sense of simple vs easy).#2020-12-3102:09seancorfieldclj is about small, composable tools -- very much in Clojure's "idiomatic world view".#2020-12-3102:12seancorfieldWe switched from Leiningen to Boot back in 2015 because we needed more/easier programmability in our build chain. We developed a lot of custom tasks once we switched to Boot -- thousands of lines of dev/test/build code. We kind of lost sight of simplicity there somewhere 🙂#2020-12-3102:14seancorfieldOur Boot setup already had dependencies in EDN files -- albeit a different format to deps.edn -- so when we were getting ready to switch, I wrote boot-tools-deps (since archived, as a bad idea), as migration step, but in the end we didn't use it.#2020-12-3102:16seancorfieldAs part of our switch to deps.edn, we took a long, hard view at all our Boot tasks and decided to streamline/simplify things. We're very happy with the decision to change over. We have just a simple shell script that wraps clojure so we can run multiple CLI commands from a single invocation.#2020-12-3102:17seancorfield(I will note that we were running into a couple of issues with Boot due to the size of our codebase and our heavy use of "pods", Boot's isolation feature, so we were "pushed" a bit -- it wasn't entirely a "jump")#2020-12-3102:22seancorfieldHappy to chat more via DM any time about tooling, if you want @fabrao (it looks late for you right now -- you're about six hours ahead of me).#2020-12-3102:31fabraowell, I don´t want to bother you at this time, but any time for me#2020-12-3102:34fabraoThe subject about deps.edn started in Clojure Brazil Telegram Group and your twit shows up and I research about it#2020-12-3102:41seancorfieldIt's early evening for me -- feel free 🙂 I'm online about 9am to 11pm Pacific most days 🙂#2020-12-3102:44fabraoHello Sean, how are you? I saw you using REBL instead REPL#2020-12-3117:23kennyDoes anyone recall the solution to "UnknownHostKey: https://github.com. RSA key fingerprint is ...."? We've started getting this on CircleCI. I'm fairly certain I've seen it before and can't remember the solution. The CI has a checkout SSH key.#2020-12-3117:41kennyTurns out our CI image didn't have ssh or git installed. Interestingly, CircleCI still had some way of getting the repo onto the machine without either of those. Adding them fixed this issue.#2020-12-3120:23ghadiThe CI image doesn’t need git or ssh#2020-12-3120:23ghadiBut it does need the github identity key in the .ssh/known_hosts#2020-12-3120:23kennyWell in this case it does 🙂 CircleCI adds the checkout ssh key only if ssh is installed. Because ssh wasn't installed, the key was not added. The CLI therefore could not clone the repo.#2020-12-3120:24ghadiI’d “pin” these in your circleCI yml file#2020-12-3120:24ghadik#2020-12-3119:39seancorfieldI finally caved and added an AOT option for uberjars to seancorfield/depstar {:mvn/version "0.4.1"} -- https://github.com/seancorfield/depstar -- there are some caveats: you need to create the classes folder yourself (as a one-off) and you need to ensure that folder is on the classpath when you actually run depstar (the README suggests a :uberjar alias that takes care of that). It only AOTs the -m / --main namespace (but that transitively compiles all namespaces that are directly referenced from it -- it is not "compile all", like Leiningen).#2020-12-3119:42seancorfieldI also updated seancorfield/clj-new {:mvn/version "0.8.2"} -- https://github.com/seancorfield/clj-new -- to take advantage of that for the app template; this also fixes the group/artifact in the pom.xml for generated template projects.#2020-12-3119:53seancorfield> clojure -A:new app myname/myapp
Generating a project called myapp based on the 'app' template.
> cd myapp/
> clojure -A:uberjar
Compiling myname.myapp ...
Building uber jar: myapp.jar
Processing pom.xml for {myname/myapp {:mvn/version "0.1.0-SNAPSHOT"}}
> java -jar myapp.jar
Hello, World!
> #2020-12-3120:01seancorfield(for lib and template types you can clojure -A:jar to build a JAR that should be deployable to http://clojars.org pretty much as-is, with a near-ready-to-roll pom.xml file)#2020-01-0208:56robert-stuttafordprobably an FAQ but my google-fu is lacking ; is there an order of precedence among :mvn/version and :local/root? could i somehow use both, where local is used for dev, but mvn is used for ci? has anything been written about producing this sort of environment specific variance?#2020-01-0209:27delaguardoWe are using special alias for CI builds with :override-deps key inside#2020-01-0211:07dominicmIt'll be an array map, so whichever namespace is found first most likely. As mentioned you should use override deps.#2020-01-0214:09Alex Miller (Clojure team)the way the code is written (https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions.clj#L32-L37) it depends on the first namespace key found in a map, so you should not assume any arbitrary order. Really, I'd say this is an error.#2020-01-0214:19Alex Miller (Clojure team)the whole point of stuff like :override-deps is to explicitly support the use case of changing a dep source#2020-01-0306:35robert-stuttafordthanks folks!#2020-01-0306:39robert-stuttafordWorks beautifully.#2020-01-0211:00dharriganDoes this -Sdeps DEPS Deps data to use as the final deps file take a map?#2020-01-0211:01dharrigani.e., if I want to pass in mvn/repo configuration on the command line, thus ignoring whatever is set in any deps.edn file, is it this switch I require?#2020-01-0211:08dharriganyup is the answer 🙂#2020-01-0211:16dharriganWhere does deps store downloaded deps, i.e., jars? In the .m2 directory?#2020-01-0213:20borkdude@dharrigan you can check that with -Spath#2020-01-0213:20dharriganah, thanks#2020-01-0213:21borkdudeThis is an example for mvn + git:
src:/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/borkdude/.gitlibs/libs/borkdude/medley/91adfb5da33f8d23f75f0894da1defe567a625c0/src:/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
#2020-01-0213:21dharriganyup#2020-01-0213:21dharriganthanks#2020-01-0217:49seancorfieldseancorfield/depstar {:mvn/version "0.5.0"} now manages the AOT compilation folder automatically -- no need to manually create classes or add it to the classpath; seancorfield/clj-new {:mvn/version "0.8.3"} updated to use latest depstar (so app template no longer needs classes folder).#2020-01-0221:06vlaaadAnd I am still on 0.2.1 because it does what it needs to do 🙂#2020-01-0221:28seancorfieldThat's good to hear! :thumbsup::skin-tone-2:#2020-01-0221:29seancorfield0.5.1 and 0.8.4 just released to add support for projects where the main namespace has - in its path and/or its name.#2020-01-0221:29seancorfield0.5.1 and 0.8.4 just released to add support for projects where the main namespace has - in its path and/or its name.#2020-01-0704:23andy.fingerhutHere is an alias I added to the deps.edn of the core.rrb-vector library: https://github.com/clojure/core.rrb-vector/blob/master/deps.edn#L30#2020-01-0704:24andy.fingerhutI wouldn't be surprised if there are better ways, but that is effective and straightforward#2020-01-0704:31seancorfield@deleted-user Have you looked at https://github.com/seancorfield/dot-clojure ? That has a lot of stuff that is very helpful when you're getting started with CLI/`deps.edn`#2020-01-0704:31potetmgonna guess the commas are because both ' and " are special in the shell#2020-01-0704:31potetmso , is a “guaranteed neutral space”#2020-01-0704:32seancorfieldThe main opts are round-tripped from shell clojure into files on disk and then back again into the shell and then to java.#2020-01-0704:33potetmIt’s usually safe to assume these kind of shenanigans when storing shell commands. Nobody does it right.#2020-01-0704:32seancorfieldSo the quoting/space combinations break in that round-tripping.#2020-01-0704:33seancorfieldCognitect's test-runner is in there.#2020-01-0704:33seancorfieldHence the :test alias and the :runner alias.#2020-01-0704:33potetmIt’s usually safe to assume these kind of shenanigans when storing shell commands. Nobody does it right.#2020-01-0704:34seancorfieldWell, you can run tests without the test-runner -- clojure.test does have some basic test running logic -- but you won't have a very nice experience 🙂#2020-01-0704:35seancorfieldclojure.test/run-all-tests will run all tests that have been loaded (in require'd code).#2020-01-0704:36deletedI wish there was a note about tests on the deps + cli page on http://clojure.org#2020-01-0704:46Alex Miller (Clojure team)if I could trouble you to file an issue so I remember, I'd be happy to do so https://github.com/clojure/clojure-site/issues#2020-01-0704:37seancorfieldAll the test runners out there look for "test" files (by various means), load the code, and then run all the tests, under whatever filters you provide.#2020-01-0704:39seancorfieldFWIW, Boot doesn't have a test runner built-in either. Only Leiningen does.#2020-01-0704:40seancorfieldMost of the test runners out there are additional dependencies that offer more features than Leiningen's built-in stuff...#2020-01-0704:42seancorfield@deleted-user This will probably be helpful for you https://cljdoc.org/d/expectations/clojure-test/1.2.1/doc/getting-started#running-tests#2020-01-0704:43seancorfieldIt's under consideration to spin off clojure.test into a Contrib lib for Clojure 1.11. I've volunteered to maintain it and one of the things I'll do is add a bunch of documentation to the project, that covers how to run tests with a variety of tools, and how to filter tests etc.#2020-01-0704:44seancorfieldYes, it's for the clojure.test-compatible version of Expectations that I maintain -- but a lot of what's at that link would become "canon" documentation in clojure.test, if I get to maintain it.#2020-01-0704:59seancorfieldwith-test is tricky to use because the default setup for all the test runners will not see those tests -- hence my notes about what needs to be done 🙂#2020-01-0808:55plexus@deleted-user you might want to look into Kaocha. It was conceived as a test runner for deps.edn that's less spartan than the cognitect runner#2020-01-0816:36practicalli-johnIf your editor has a test runner, eg. CIDER for Emacs, then you only need to ensure the 'test' directory is included in the path. You can add it to :path or as an alias.#2020-01-0816:37practicalli-johnI tend to switch betweeen cognitect-labs test runner and eftest (weavjester). I also am trying out Kaocha as its very comprehensive.
I have test-runner aliases in my $HOME/.clojure/deps.edn file
https://github.com/practicalli/deps-edn-examples/blob/master/deps.edn#L114-L137#2020-01-0817:21plexusCan you elaborate a bit on how you run tests in babashka? What would you expect from a test runner in that environment?#2020-01-0817:28borkdudeWe can take this to #babashka#2020-01-0817:28borkdudeBit of context: this is an intermediate lib for running tests until bb gets clojure.test:
https://github.com/borkdude/spartan.test#2020-01-1010:15Mehdi H.Hi all! I have some troubles removing SLF4J multiple bindings in my project. I know it's been debated here before (https://clojurians-log.clojureverse.org/beginners/2016-10-21 or even in this channel : https://clojurians-log.clojureverse.org/tools-deps/2018-10-12), but at the time it seems that you could find it easily only if you were using lein (I am on CLI tools). Has anything changed since then? That would help me a lot! Thanks!#2020-01-1010:50Mehdi H.Ok never mind, clj -Stree does the trick 🙂#2020-01-1307:34slipsetI ran into a subtle problem over the weekend with clj -Spom on https://github.com/clj-commons/pomegranate. The original pom.xml has the dep for wagon-provider-api here https://github.com/clj-commons/pomegranate/blob/master/pom.xml#L101 whereas the pom.xml generated by clj -Spom with this deps.edn https://github.com/clj-commons/pomegranate/blob/master/deps.edn#L12 put the dependency declaration for wagon-provider-api earlier in the pom.xml.#2020-01-1307:34slipsetOne reaction to this would of course be 🤷 , but from https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html we have that#2020-01-1307:34slipset> Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.#2020-01-1307:36slipsetAnd the kicker is that wagon-provider-api depends on an older version of plexus-util than the other deps of pomegranate, so by changing the order of the deps in the pom.xml, pomegranate broke.#2020-01-1307:38slipsetThinking of it, I’m not sure why I’m writing this, I guess it’s just as an amusing story.#2020-01-1310:48slipsetI guess there might be a question in here: Is the order in which the deps are listed in the generated pom.xml predictable in any way?#2020-01-1310:55dominicmI wouldn't think so. It operates on a hashmap.#2020-01-1314:01Alex Miller (Clojure team)no, it's not predictable#2020-01-1314:02Alex Miller (Clojure team)you could exclude the plexus-util in whatever branch and then it wouldn't matter, right?#2020-01-1314:35slipsetThat's correct, and what I ended up doing.#2020-01-1317:03aviDoes tools.deps.graph support gitlibs?#2020-01-1317:03Alex Miller (Clojure team)Yes#2020-01-1317:06aviHmm, I see, it looks like maybe not when --size is specified?#2020-01-1317:06aviwhich makes sense, but perhaps it would be good to display an error message rather than a stacktrace…#2020-01-1317:06avi/Users/avi.flax/.gitlibs/libs/expound/expound/9b5778a1a4ed91e2090308a6648ee9072076925a is not a relative path
java.lang.IllegalArgumentException: /Users/avi.flax/.gitlibs/libs/expound/expound/9b5778a1a4ed91e2090308a6648ee9072076925a is not a relative path
at $as_relative_path.invokeStatic(io.clj:414)
at $file.invokeStatic(io.clj:426)
at $file.invoke(io.clj:418)
at clojure.tools.deps.alpha.extensions.git$eval1027$fn__1029.invoke(git.clj:43)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.graph$get_size.invokeStatic(graph.clj:95)
at clojure.tools.deps.graph$get_size.invoke(graph.clj:93)
at clojure.tools.deps.graph$make_dep_node.invokeStatic(graph.clj:107)
at clojure.tools.deps.graph$make_dep_node.invoke(graph.clj:99)
at clojure.tools.deps.graph$make_graph$fn__1522.invoke(graph.clj:125)
#2020-01-1317:09Alex Miller (Clojure team)That’s possible, I thought I tested that, but maybe not. It’s supposed to fall back to 0 in that case#2020-01-1317:10aviWould you like a bug report somewhere?#2020-01-1317:10Alex Miller (Clojure team)I’ll take a look when I get a chance#2020-01-1317:42aviBTW when you do take a look at this, I’m thinking maybe using 0 for gitlibs is a little misleading? It’s more of an unknown… perhaps something like -1 or “unknown” or :man-shrugging: or just leaving it blank…?#2020-01-1317:10Alex Miller (Clojure team)Nah#2020-01-1317:10avi🆗 thanks2#2020-01-1317:10aviBTW reviewing the deps graph for my project is fascinating, and very informative — thank you!#2020-01-1317:10avi#2020-01-1317:11Alex Miller (Clojure team)Sorry size isn’t working!#2020-01-1317:13aviNo big deal!#2020-01-1317:14aviIt’s already immediately obvious that I have a single dep bloating my project. It’s a dep I’ve been planning to drop eventually, so now I’m even more motivated to so do!#2020-01-1317:15aviHere I’ve manually highlighted that dep and all its deps — it’s pretty dramatic!#2020-01-1317:51Alex Miller (Clojure team)nice - I'd like to also have an output mode to export the dot file itself instead of the image so you can further mess with it#2020-01-1317:51avi👍#2020-01-1317:51aviInterested in PRs?#2020-01-1317:52Alex Miller (Clojure team)nah, it's part of other changes I want to make#2020-01-1317:53avi:thinking_face: it’d be cool if there was a feature to assign a unique background color to each branch from the root — might make it easier to visualize the impact of each root-level dep#2020-01-1318:00Alex Miller (Clojure team)you could drop that in a question at https://ask.clojure.org if you want, category of "Contrib libs / tools.deps.graph" and label "request"#2020-01-1318:01Alex Miller (Clojure team)worth playing with. I experimented with a variety of color things, never found something that I thought helped, but I didn't try that!#2020-01-1318:03aviCool, will do!#2020-01-1320:21avihttps://ask.clojure.org/index.php/9028/suggestion-color-coding-dependency-subtrees#2020-01-1321:27seancorfield@alexmiller A question about tools.deps.graph: it seems that it reads just the current deps file so if you have overrides provided in another deps file, it won't see them? At work we have a "user-level" deps file in the monorepo that "pins" versions of everything via a :defaults alias and then we have {} for deps in each project deps.edn file.#2020-01-1321:27seancorfieldCan I specify multiple deps.edn files to read/merge on the command-line?#2020-01-1321:29seancorfield(so far everything I've tried in the context of our work repo has either failed outright or produces a blank canvas claiming the size is -1 x -1)#2020-01-1321:32seancorfieldIf I generate to a file, I seem to get a graph as expected, so it's perhaps only the default render to a graphics window that fails...#2020-01-1321:34Alex Miller (Clojure team)it's the same as clj#2020-01-1321:34Alex Miller (Clojure team)in that it uses mostly the identical code#2020-01-1321:34Alex Miller (Clojure team)so, no you can't use multiple deps.edn files#2020-01-1321:35seancorfieldAh, I was confused by the results I was seeing... it does take the user-level deps into account after all?#2020-01-1321:35Alex Miller (Clojure team)it should#2020-01-1321:35Alex Miller (Clojure team)unless I'm misremembering that#2020-01-1321:36Alex Miller (Clojure team)yeah, it does#2020-01-1321:38seancorfieldbuild [ graph api -a :defaults -o /tmp/graph.png ]
which ends up being
cd api; CLJ_CONFIG=../versions clojure -A:graph:defaults -a :defaults -o /tmp/graph.png
does indeed produce the correct graph for our api subproject. I think getting the blank popup was throwing me off there.#2020-01-1416:17rschmuklerHey all! I'm consuming a library that seems to have packaged itself as an uberjar including all of its dependencies. I also depend on one of the libraries it uses and it looks like that it's overwriting the requires to an old version (which requires a java class that isn't packaged in the new one). Is there any way for me to force my project to use the new version instead of what it's using?#2020-01-1416:18rschmuklerI'm not super familiar with the internals of how this all works, so if the above diagnosis is complete nonsense, please excuse me!#2020-01-1416:21Alex Miller (Clojure team)makes total sense#2020-01-1416:21Alex Miller (Clojure team)and no, not easily#2020-01-1416:21Alex Miller (Clojure team)you should tell that lib to stop doing that :)#2020-01-1416:23Alex Miller (Clojure team)the effect you need is to force your lib version earlier in the classpath ordering. tools.deps mostly does not provide support for explicit ordering#2020-01-1416:25Alex Miller (Clojure team)I possibly could come up with a hacky workaround, but it would be a hack. the real solution is the lib#2020-01-1703:30Alex Miller (Clojure team)tools.deps.alpha 0.8.264 and clj 1.10.1.496 are now available.#2020-01-1703:30Alex Miller (Clojure team)The main change in this is that support for s3 maven repos has been completely replaced and is now using a new impl of the Maven resolver transporter API based on the cognitect aws-api. The old version used the transporter-wagon adapter, the unmaintained spring s3 wagon, and the AWS Java SDK. Those deps have been happily dropped on the floor, and all other deps were bumped as well. Support for some newer AWS credential sources may be better, particularly when running on AWS. Also, you can explicitly set the s3 bucket region in your deps.edn like {:mvn/repos {"my-bucket-repo" {:url ""}}} - this is a feature of this transporter, not a general AWS thing.#2020-01-1703:31Alex Miller (Clojure team)The intention here is that any existing use of an s3 maven repo should continue to work as it did before. If you find that not to be the case, please let me know! It has been tested in a number of configurations by several people, but you know how it goes.#2020-01-1703:33Alex Miller (Clojure team)tools.deps tool makers may want to do a quick check as this version has some significant transitive dependency differences (hopefully mostly for the better!)#2020-01-1703:38Alex Miller (Clojure team)docs have not yet been updated, but I'll do that tomorrow#2020-01-1704:45tianshuHow can I get my clojure cli version?
I usually install with system package manager, It only shows the version of Clojure.#2020-01-1707:58Michaël SalihiTo get the clojure cli verison without launching the REPL as clj -Sverbose do, you can also use clj -Sdescribe.
@alexmillerRight ?#2020-01-1705:31Alex Miller (Clojure team)clj -Sverbose#2020-01-1705:33tianshu@alexmiller thanks!#2020-01-1707:07slipsetI was wondering about the version thing the other day as well, and tried to invoke clj --version which printed the help-text, which I failed to parse (ie I didn’t understand that Print important path info to console meant “and also print version”#2020-01-1707:07slipsetWould it be a nice addition to print the version string in the help-text as well?#2020-01-1707:21slipsetlooking through clojure/brew-install this should be simple enough, but is there a Jira project for this project where one might submit a patch?#2020-01-1714:33Alex Miller (Clojure team)you can use TDEPS for tickets, but I just added it to the repo#2020-01-1715:38slipsetThanks 🙂#2020-01-1715:38slipsetHave a great weekend!#2020-01-1715:54Alex Miller (Clojure team)thx, you too#2020-01-1709:32borkdudedeps.clj updated to 1.10.496: https://github.com/borkdude/deps.clj/releases/tag/v0.0.6#2020-01-1718:28rschmuklerHey all! I'm looking to publish a library on clojars with a deps.edn file that specifies a :git/url based dependency. I realize that this won't work with a pom.xml - does tools-deps find transitive dependencies using a deps.edn files in addition to pom.xml files?#2020-01-1719:37Alex Miller (Clojure team)Yes, both, prefers deps.edn#2020-01-1719:38Alex Miller (Clojure team)Or rather will use exactly one but can be either#2020-01-1718:48borkdude@slipset there's also:
$ clojure -Sdescribe
{:version "1.10.1.496"
,,,}#2020-01-1719:26seancorfield@borkdude Are any of these differences between deps.clj and the regular clojure script of any concern?
PS C:\Users\sean\clojure\cool> deps -Sdescribe
{:deps-clj-version "0.0.6"
...
:config-files ["deps.edn" "C:\\Users\\sean\\.clojure\\deps.edn" "deps.edn"]
:config-user "C:\\Users\\sean\\.clojure\\deps.edn"
...
:install-dir nil
...
:force ""
:repro ""
...}
PS C:\Users\sean\clojure\cool> clojure -Sdescribe
{...
:config-files ["C:\Users\sean\scoop\modules\ClojureTools\deps.edn" "C:\Users\sean\.clojure\deps.edn" "deps.edn"]
:config-user "C:\Users\sean\.clojure\deps.edn"
...
:install-dir "C:\Users\sean\scoop\modules\ClojureTools"
...
:force False
:repro False
...}#2020-01-1719:27seancorfield:config-files seems odd with deps.edn in there twice (and :force/`:repro` seem to be "" but that's truthy?)#2020-01-1719:28seancorfield(I can understand :install-dir being nil since deps.clj is self-contained)#2020-01-1719:29borkdudeyes, that's the reason why install-dir is nil. the rest can be fixed if it's bothering anyone. also the escaping seems different?#2020-01-1719:30borkdudeI think False may be coming from Powershell? it doesn't seem like something cmd.exe would produce#2020-01-1719:32borkdude@seancorfield I may have put a str too many here: https://github.com/borkdude/deps.clj/blob/4c374238ceec510a097749df4b2f4a5a17a3e355/src/borkdude/deps.clj#L426#2020-01-1719:32borkdudeshould probably read boolean#2020-01-1719:33seancorfieldThe macOS/Linux version shows:
:force false
:repro false
#2020-01-1719:33seancorfieldFalse is just PS's version of false 🙂#2020-01-1719:33borkdudeI'll make an issue for this. It doesn't seem something that's very urgent, but I'll fix it#2020-01-1719:34seancorfieldI assumed the \\ was due to being produced by Clojure/Java, whereas \ is native PS (or cmd).#2020-01-1719:34borkdudeyeah, this is produced using pr instead of bash#2020-01-1719:35seancorfieldI guess :config-files is no longer used by any tooling so it doesn't matter -- it just looked odd with deps.edn in there twice.#2020-01-1719:35borkdudeyeah, might be indicative of a small bug somewhere.#2020-01-1719:36borkdudeI basically did the naive thing and ported bash straight into clojure without really think hard 🙂#2020-01-1719:38seancorfielddeps is a great option on Windows -- much appreciated!#2020-01-1719:39borkdudehttps://github.com/borkdude/deps.clj/issues/7
if you think this is fun, I also take PRs. there are build / dev instructions in the other open issue.#2020-01-1719:39Alex Miller (Clojure team)Yeah Sean, get off the bench and help out for a change! ;)#2020-01-1719:40Alex Miller (Clojure team)I kid, I kid#2020-01-1719:40borkdude😄#2020-01-1719:40borkdudebtw bb.exe is now a thing on Windows so you can execute the deps.clj script as well (so you can tweak it in Notepad++)#2020-01-1719:41borkdude(I'll now stop the shameless promotion of my tools for the rest of the day)#2020-01-1719:43rschmuklerLooking through the source it doesn't look like transitive deps.edn files are supported in jars... Would a PR be welcome that adds this to the local resolver? similar to how find-pom works currently?#2020-01-1720:13Alex Miller (Clojure team)deps.edn in jars is not a thing#2020-01-1720:14Alex Miller (Clojure team)and if I wanted to make it a thing, I'd try to design that first#2020-01-1720:26dominicmWhat would designing that look like, or do you mean that you would design it?#2020-01-1720:33Alex Miller (Clojure team)like designing anything else - starting with goals, thinking about alternatives, examining their tradeoffs, making decisions, implementing#2020-01-1720:34Alex Miller (Clojure team)I don't have time to go down that path atm#2020-01-1805:15seancorfieldseancorfield/clj-new {:mvn/version "0.8.5"} -- generated lib and template projects now include :install and :deploy aliases, using @slipset’s excellent deps-deploy, so you can create a JAR clojure -A:jar, and either install locally clojure -A:install or deploy to Clojars clojure -A:deploy; also bumps depstar to 0.5.2 for the NPE bug fix in uberjars with no main class specified.#2020-01-1805:26seancorfieldMy dot-clojure/deps.edn file also has these aliases added (although they require a JAR filename as an argument, whereas they're baked into the generated lib and template projects since those already have a name).#2020-01-2002:52tapGot an error trying to start clj repl with rebl
Error building classpath. org.apache.maven.repository.internal.DefaultModelResolver.<init>(org.eclipse.aether.RepositorySystemSession, org.eclipse.aether.RequestTrace, java.lang.String, org.eclipse.aether.impl.ArtifactResolver, org.eclipse.aether.impl.VersionRangeResolver, org.eclipse.aether.impl.RemoteRepositoryManager, java.util.List)
java.lang.NoSuchMethodException: org.apache.maven.repository.internal.DefaultModelResolver.<init>(org.eclipse.aether.RepositorySystemSession, org.eclipse.aether.RequestTrace, java.lang.String, org.eclipse.aether.impl.ArtifactResolver, org.eclipse.aether.impl.VersionRangeResolver, org.eclipse.aether.impl.RemoteRepositoryManager, java.util.List)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(pom.clj:49)
at clojure.tools.deps.alpha.extensions.pom$model_resolver.invoke(pom.clj:39)
at clojure.tools.deps.alpha.extensions.pom$read_model.invokeStatic(pom.clj:60)
at clojure.tools.deps.alpha.extensions.pom$read_model.invoke(pom.clj:54)
at clojure.tools.deps.alpha.extensions.local$eval925$fn__927.invoke(local.clj:68)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$fn__1191.invoke(alpha.clj:191)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:190)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:168)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:236)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:216)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invokeStatic(make_classpath2.clj:56)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invoke(make_classpath2.clj:48)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:82)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:73)
at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:102)
at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:96)
at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:147)
at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:119)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
deps.edn
{:deps {}
:aliases
{:rebl {:extra-deps {org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/core.async {:mvn/version "0.4.490"}
com.cognitect/rebl {:local/root "/path/to/REBL.jar"}}
:main-opts ["-m" "cognitect.rebl"]}}}
The command I run
clj -R:rebl
Has anyone had the similar issue?#2020-01-2003:15seancorfieldWhat version does clojure -Sdescribe report?#2020-01-2003:17seancorfield@tap And you're aware that -R:rebl will add those dependencies but won't run REBL -- it will run the default behavior, which is to start a regular clojure.main REPL.#2020-01-2003:18tap{:version "1.10.1.496"
:config-files ["/usr/local/Cellar/clojure/1.10.1.496/deps.edn" "/Users/xxx/.clojure/deps.edn" "deps.edn" ]
:config-user "/Users/xxx/.clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/usr/local/Cellar/clojure/1.10.1.496"
:config-dir "/Users/xxx/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
#2020-01-2003:18seancorfieldYour deps.edn file doesn't look valid -- you have five } on that line with com.cognitect/rebl#2020-01-2003:19tapSorry#2020-01-2003:19seancorfieldThat would put :main-opts outside the whole { } of the deps.#2020-01-2003:20tapYeah. Fixed up there. I removed :main-opts just to test it out. Miss that when I readded it to ask question here.#2020-01-2003:21tapYes. I'm aware that -R won't run REBL. Just want to make it clearer that it's not related to :main-opts.#2020-01-2003:22seancorfieldWhich version of REBL is that local JAR file?#2020-01-2003:23tapREBL-0.9.220.jar#2020-01-2003:25seancorfieldOk, I have that setup locally... just a sec...#2020-01-2003:25seancorfieldYeah, with -R:rebl I get that same error. I've just never run it that way -- that's odd.#2020-01-2003:26tapI got the same error with -A as well#2020-01-2003:29seancorfieldLet me switch computers and try something...#2020-01-2003:29tapI saw a quick fix related to pom for tools.deps.alpha recently. Not sure if it's related.
https://github.com/clojure/tools.deps.alpha/commit/fbe57d942af2016229e0aabba9121e62de848729
I'm also not sure if I'm using the latest tools.deps.alpha at the moment.#2020-01-2003:32seancorfieldCould be that or https://github.com/clojure/tools.deps.alpha/commit/70917267ca38fa971b1b0ebcc73084516f9f8af3#2020-01-2003:32seancorfieldI tried it with the add-lib branch SHA that I normally use and got the same exception.#2020-01-2003:33tapThanks for your help looking into this#2020-01-2003:35seancorfieldYeah, I can't start my regular dev environment with the latest clojure version with REBL (I've been on vacation for a several days so this is the first time I've tried since the last clojure/`tools.deps.alpha` update) -- /cc @alexmiller looks like the latest release breaks when used with REBL#2020-01-2003:43seancorfieldLooks like clojure isn't set up in a way that earlier releases can be installed with brew 😞#2020-01-2003:43seancorfieldbrew install fails#2020-01-2003:44sogaiu@seancorfield @tap it looks like i am able to reproduce the problem as well (this is a linux box)#2020-01-2003:45sogaiui have an install script for 1.10.1.492 i can try#2020-01-2003:46seancorfieldYeah, I think the latest changes in clojure/`tools.deps.alpha` are incompatible with the current release of REBL -- because it is AOT'd#2020-01-2003:46seancorfieldLooks like Cognitect will need to release a new version of REBL to fix this (which in turn won't be compatible with earlier clojure/`tools.deps.alpha` builds, I suspect).#2020-01-2003:47sogaiui just tried with 1.10.1.492 and it works with the REBL-0.9.220 jar#2020-01-2003:48sogaiuswitching to 1.10.1.496 gives the exception#2020-01-2003:49seancorfieldWeirdly, there's nothing I can see in the REBL JAR or its pom.xml that suggests it wouldn't be compatible... so I guess we have to wait for @alexmiller and/or some other Cognitect folks to chime in tomorrow. Very frustrating we can't downgrade clojure via Homebrew tho'...#2020-01-2003:52sogaiuindeed#2020-01-2003:54sogaiui guess the bit about "installing from a specific commit" here: http://effectif.com/mac-os-x/installing-specific-version-of-homebrew-formula doesn't apply?#2020-01-2003:56sogaiumay be the info is dated...#2020-01-2003:59seancorfieldWorkaround:
$ curl -O
$ curl -O
$ tar xvfz clojure-tools-1.10.1.492.tar.gz
and then edit your current clojure shell script to point to the clojure-tools/clojure-tools-1.10.1.492.jar from that expansion instead of the installed 496 version.#2020-01-2004:00seancorfield(at least I'll be back up and running at work tomorrow morning now!)#2020-01-2004:01Alex Miller (Clojure team)the exception above from @tap is a regression that is fixed in master and I'll release it tomorrow#2020-01-2004:02Alex Miller (Clojure team)you can actually get the older brew formulas if you want to download#2020-01-2004:02Alex Miller (Clojure team)at https://download.clojure.org/install/clojure-1.10.1.492.rb#2020-01-2004:04Alex Miller (Clojure team)rename that to clojure.rb, then brew reinstall --force clojure.rb#2020-01-2004:04Alex Miller (Clojure team)brew can only reinstall older versions that you have previously downloaded and installed that you still have in the brew cache#2020-01-2004:05Alex Miller (Clojure team)there is no repository of versions like there is with Maven or something, just "the current version"#2020-01-2004:11seancorfieldIs there an option on upgrade or in a specific formula to not delete the old version?#2020-01-2004:11seancorfield(and good to know on the force install from a downloaded older version)#2020-01-2004:16Alex Miller (Clojure team)by default, I think it leaves it#2020-01-2004:18Alex Miller (Clojure team)ls ~/Library/_Caches_/Homebrew#2020-01-2004:19Alex Miller (Clojure team)and then Formula/clojure under that#2020-01-2004:19tapIs it appropriate to write about this how to downgrade brew clj package on https://clojure.org under Guides section? I'll do that if y'all think so.#2020-01-2004:20Alex Miller (Clojure team)nah, I'd prefer to bury it in reference and link#2020-01-2004:21tapok#2020-01-2004:43seancorfieldUnder ~/Library/Caches/Homebrew is just the current version of each installed package.#2020-01-2019:15Alex Miller (Clojure team)Latest (1.10.1.502) with regression fixes is now available in brew etc#2020-01-2019:15Alex Miller (Clojure team)Fixed for pom deps and pom gen#2020-01-2019:18seancorfieldWorks with REBL now -- thank you for the swift update!#2020-01-2019:21Alex Miller (Clojure team)I also added some tests to catch it earlier next time!#2020-01-2020:48sogaiuseems to work on a linux here :thumbsup:#2020-01-2021:25Alex Miller (Clojure team)oh also, this release has the version as the first line of clj -h#2020-01-2021:43andy.fingerhutYou have been worn down explaining the -Sdescribe and -Sverbose options 🙂 Soon you will succumb to the pull to the dark side of adding a --version option.#2020-01-2021:58Alex Miller (Clojure team)even that won't be the end - there will be confusion between the clojure tools version, the clojure tools default clojure version, and the version of clojure specified by the current project :)#2020-01-2022:03andy.fingerhutLong ago I gave myself slightly more sanity by including a line like the following at the beginning of Eastwood's output, by default: == Eastwood 0.3.6-SNAPSHOT Clojure 1.10.0 JVM 1.8.0_192 ==#2020-01-2022:30seancorfieldHeh, that's why all our services report their git tag/SHA, along with the version of Clojure and the JVM at startup! 🙂#2020-01-2021:29littleliseems to work on Windows too#2020-01-2021:58borkdudehttps://github.com/borkdude/deps.clj/releases/tag/v0.0.7#2020-01-2115:54aviHas anyone here ever managed to get Snyk to scan a Clojure project using tools.deps for vulnerabilities? I’m giving it a try and Snyk is helping out but I’m running into Maven problems.#2020-01-2115:54avi[ERROR] Failed to execute goal on project project: Could not resolve dependencies for project project:project:jar:0.1.0: The following artifacts could not be resolved: com.gfredericks:test.chuck:jar:0.2.10, clj-commons:clj-yaml:jar:0.7.0, hawk:hawk:jar:0.2.11: Could not find artifact com.gfredericks:test.chuck:jar:0.2.10 in central () -> [Help 1]
#2020-01-2115:55aviI thought maybe I was running into this: https://github.com/snyk/snyk/issues/207#2020-01-2115:55aviBut I checked and the pom.xml generated by clojure -Spom already includes both Central and Clojars.#2020-01-2115:55aviSo I’m stuck…#2020-01-2115:59Alex Miller (Clojure team)could you post the pom?/#2020-01-2115:59avisure#2020-01-2116:00aviJust re-running the CI job now to regenerate it.#2020-01-2116:03avi#2020-01-2116:04aviDoh! picard-facepalm repositories is empty#2020-01-2116:04aviNever mind!#2020-01-2116:05aviOr… is there a way to get clojure to populate repositories?#2020-01-2116:05aviI’m using 1.10.1.483 but could switch to something newer#2020-01-2116:07Alex Miller (Clojure team)I'm trying to remember if it's supposed to do that now or not#2020-01-2116:07aviI’m going to try it with 1.10.1.502 and report back#2020-01-2116:07Alex Miller (Clojure team)looks like it is, so might be broken#2020-01-2116:10avi😅#2020-01-2116:11Alex Miller (Clojure team)I guess it will only right now add repos explicitly listed in the deps.edn#2020-01-2116:11aviI’ve got both repos explicitly in deps.edn#2020-01-2116:12aviI’m testing locally now with 1.10.1.502#2020-01-2116:12avirepositories is empty 😅#2020-01-2116:12avimy deps.edn has this at the root:
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}#2020-01-2116:15Alex Miller (Clojure team)I'm looking at it#2020-01-2116:16avithanks2#2020-01-2116:20Alex Miller (Clojure team)ah, it's subtle but I see the problem#2020-01-2116:20avibananadance#2020-01-2116:25aviany possible workaround? Or maybe I should brush up on my sed skills and try to just patch the pom for now?#2020-01-2116:25Alex Miller (Clojure team)for the moment you'll need to manually modify your pom.xml after gen#2020-01-2116:26aviah ok#2020-01-2116:26aviwill do — thanks!#2020-01-2116:26aviWould you like a bug report somewhere?#2020-01-2116:26Alex Miller (Clojure team)nah, I've got it#2020-01-2116:26avi💪 thanks3#2020-01-2117:27Alex Miller (Clojure team)fixed for next release, but not sure when that will be (and added a test for future)#2020-01-2117:28aviFantastic — thanks so much!#2020-01-2118:59danielcomptonIs there a way to get tools.deps to print out all of the :paths and :extra-paths that a given set of aliases resolve to?#2020-01-2119:00danielcompton-Sdescribe prints out some information, but nothing about the composed config#2020-01-2119:01Alex Miller (Clojure team)not currently, although I've been making my way towards making something like that possible#2020-01-2119:01Alex Miller (Clojure team)I've wanted something like mvn help:effective-pom so that's been a back burner project of mine#2020-01-2119:01Alex Miller (Clojure team)do you care about :paths vs :extra-paths?#2020-01-2119:02Alex Miller (Clojure team)or just "all local paths"#2020-01-2121:45danielcomptonAll local paths for my usage (send them to clj-kondo)#2020-01-2121:48Alex Miller (Clojure team)ok, that's good#2020-01-2119:37weavejesterDoes anyone know if there’s an uberjar library for tools.deps that supports merging of files like data_readers.clj?#2020-01-2119:38ghadidepstar does it#2020-01-2119:38ghadimaybe?#2020-01-2119:38ghadiIt can be taught to, easily#2020-01-2119:38weavejesterI looked into depstar, but it looks like it has hardcoded that part. I could submit a PR, but I was hoping there was something around already.#2020-01-2119:39ghadihttps://github.com/seancorfield/depstar/blob/master/src/hf/depstar/uberjar.clj#L57-L61#2020-01-2119:39ghadihave you tested it @weavejester ?#2020-01-2119:40weavejesterYeah, that’s what I mean by it being hard-coded, @ghadi.#2020-01-2119:40weavejesterNotice data_readers.clj is explicitly checked for.#2020-01-2119:41weavejesterAnd clash-strategy doesn’t look like it can be overridden.#2020-01-2119:41ghadioh you meant other files, not that one in particular#2020-01-2119:41weavejesterYeah#2020-01-2119:42ghadinot extensible as written#2020-01-2119:55dominicm@weavejester pack avoids merging altogether.#2020-01-2119:55dominicmCapsule is the one that works with most of your libraries :)#2020-01-2120:14seancorfield@weavejester Happy to entertain PRs on depstar to extend the clash handling logic. I inherited that as-is and haven't enhanced that part yet...#2020-01-2123:12weavejesterAre there any disadvantages to Capsule jars? I guess I’m inherently wary of anything that builds on known deployment mechanisms.#2020-01-2207:28dominicmThey use a little bit more memory because they start trampoline jvm, much like lein does#2020-01-2207:28dominicmOnejar is a little more special, that uses a class loader to do its work#2020-01-2207:28dominicmYou might also want to consider the docker mode, which will directly create a docker image.#2020-01-2306:26tianshu@U09LZR36F can I package a clojure project into a docker image?#2020-01-2307:43dominicmYeah#2020-01-2308:34tianshu@U09LZR36F can you share your workflow?#2020-01-2308:38dominicm@U0NBGRGD6 My understanding is that you provide a docker registry and it will upload it there. People are using it with Google.#2020-01-2311:55rickmoynihan@weavejester: Yes there are definite disadvantages to capsule jars…
Also they extra the jars contents into a cache to build the classpath.
You also get a jvm dance at startup; and you then need to care about proxying various JVM options to the real app vm; i.e. if you want to set the heap etc.
I’ve been meaning to look at onejar as an alternative.
FWIW in spite of the occaisional frustration we currently avoid the issue you’re on about with capsule jars. I think there’s an argument that merging the files has problems too; it would be better to resolve all of them at runtime; i.e. uberjars are really just a hack.
I just wish the JVM platform itself supported this.#2020-01-2311:57rickmoynihanIt might just be better to zip up the classpath and write it in a shell script that starts the jvm tbh.#2020-01-2311:58rickmoynihanIIRC I wrote this stuff up on a duct issue somewhere… 👀#2020-01-2311:58rickmoynihanhttps://github.com/duct-framework/duct/issues/92#issue-449273982#2020-01-2312:02rickmoynihanAlso lein-tools-deps and using lein for just the packaging and deployment may also be a solution for you. (Full disclosure: I wrote it)#2020-01-2314:00dominicmIf I had control over how people deployed, I'd tell them to use the skinny mode, aka, dump my project into a folder#2020-01-2314:01dominicmThere's an idea on what you just said though#2020-01-2314:01dominicmZip will ignore the prefix leading up to finding the header. So you can have zips which are scripts.#2020-01-2314:02dominicmUnzip itself to a cache directory, and pass all arguments to the jvm#2020-01-2213:00Alex Miller (Clojure team)The object dirs and working trees are split there#2020-01-2310:24sveriHi, is there a way to define an alias that extends an existing alias?#2020-01-2310:30vlaaadno alias composition, as far as I know#2020-01-2310:34sveriOk, thank you 🙂#2020-01-2320:49samAre private git repos supported in deps.edn? The documentation says public and private are supported, but I'm not sure how to provide credentials for private and I can't find any instructions.#2020-01-2320:53samNvm, I think I found documentation here: https://clojure.org/reference/deps_and_cli#_procurers.#2020-01-2320:55Alex Miller (Clojure team)yes and yes#2020-01-2320:55Alex Miller (Clojure team)it can be a bit fiddly though, so when it doesn't work, come back here :)#2020-01-2320:58samThanks!#2020-01-2321:09samI got the private repo part to work, but got this error:
Error building classpath. Manifest type not detected when finding deps for provider-data-pipeline/provider-data-pipeline
which is apparently due to the fact that that the target repo is using leiningen for dependency management instead of deps.edn. Is that right?#2020-01-2321:14seancorfield@samwagg0583 Correct. t.d.a can only process deps.edn and pom.xml based git dependencies.#2020-01-2321:16borkdudeif it's a dependency that only has sources and no other deps in its project.clj you can declare a manifest type:
Optional key :deps/manifest
Specifies the project manifest type
Default is to auto-detect the project type (currently either :deps or :pom)
This works for e.g. libs like medley#2020-01-2402:25aviI like this tip! Thanks!#2020-01-2321:19seancorfieldYes, true, you can "force" :deps and it treats it as if deps.edn exists with {} as its contents.#2020-01-2321:30samThanks everyone.#2020-01-2321:54samWhat about using lein pom to generate a pom and committing that to the target git repo? I do have control over the target, but I might not be able to convert it to deps.edn, at least in the short term. Curious if you guys have heard of anyone using that pattern and how it's worked out.#2020-01-2321:56samI mean, I'd rather not have 2 different listings of deps like that. But it seems like an option.#2020-01-2322:09seancorfield@samwagg0583 If the repo has dependencies, then adding a pom.xml file seems like a good compromise.#2020-01-2322:09seancorfieldJust bear in mind that if someone updates project.clj and forgets to update pom.xml you'll have inconsistent dependencies downstream.#2020-01-2322:10seancorfieldAnother alternative would be to put the basic dependencies from project.clj into deps.edn in that repo and use a plugin to pull them in from that file...#2020-01-2322:10seancorfieldhttps://github.com/RickMoynihan/lein-tools-deps lets you do that#2020-01-2322:11seancorfieldSo then deps.edn would be the "system of record" for the primary dependencies and both the Lein project and downstream consumers via git will see the same file, same deps.#2020-01-2413:49rickmoynihan@samwagg0583 author of lein-tools-deps here. It does let you do what @seancorfield says; though I wouldn’t recommend ltd for projects that have complex transitive downstream dependencies.
The reasoning is covered in the README here: https://github.com/RickMoynihan/lein-tools-deps#do-you-really-want-to-use-this
However there is an open issue to fix this, and a known solution for fixing it: https://github.com/RickMoynihan/lein-tools-deps/issues/91
(As an aside I’m unlikely to find the time to fix this myself anytime soon; but am open to PRs). I think this is the most important issue to fix in that project.#2020-01-2822:09sam@U06HHF230 Thanks for the info!#2020-01-2822:12samFor the moment, I went the route of publishing a pom.xml in the target repo. But I may explore your lib more in the future.#2020-01-2414:22Jakub Holý (HolyJak)Hello! Does anyone have an example how to create a simple clojure + java based project with deps.edn? (Clojure will use some Java classes). Quick googling revealed nothing relevant...#2020-01-2414:32Alex Miller (Clojure team)deps does not explicitly support Java compilation#2020-01-2414:33Alex Miller (Clojure team)so you either need something that can combine deps with javac, or split the project into java part and clojure part#2020-01-2415:04sogaiu@holyjak if you haven't checked out @mike1452's recent 2 announcements for the clj-new templates slib and sapp, may be it's worth a look?#2020-01-2416:00mike_ananev@holyjak try this
https://github.com/mikeananev/sapp#2020-01-2416:04Jakub Holý (HolyJak)Thank you!#2020-01-2416:02Alex Miller (Clojure team)moving here from announcements sub-thread. @seancorfield I'm seeing an error trying to use clj-new with that sapp template:#2020-01-2416:02Alex Miller (Clojure team)clj -A:new qwerty/asd
Failed with: org.apache.maven.repository.internal.DefaultModelResolver.<init>(org.eclipse.aether.RepositorySystemSession,org.eclipse.aether.RequestTrace,java.lang.String,org.eclipse.aether.impl.ArtifactResolver,org.eclipse.aether.impl.VersionRangeResolver,org.eclipse.aether.impl.RemoteRepositoryManager,java.util.List)
Execution error (ExceptionInfo) at clj-new.helpers/resolve-remote-template (helpers.clj:149).
Could not load artifact for template: slib
Tried coordinates:
[slib/boot-template "RELEASE"]
[slib/lein-template "RELEASE"]
Full report at:
/var/folders/2k/2l7ch1xn7lb5gj86tgp1ygww0000gn/T/clojure-13105343161273067757.edn
#2020-01-2416:03Alex Miller (Clojure team)The DefaultModelResolver thing is either the problem I just fixed in latest tools.deps.alpha or a version conflict due to pomegranate (both kind of looked similar errors). I notice that clj-new is using a pretty old version of tools.deps?#2020-01-2416:06mike_ananevhmm, interesting. i just checked out template on clear centos 7 with latest clj-new 0.8.5 and tools deps. everything works fine. can't reproduce DefaultModelResolver error.#2020-01-2416:06Alex Miller (Clojure team)My clj-new alias is
:new {:extra-deps {seancorfield/clj-new {:mvn/version "0.8.5"}}
:main-opts ["-m" "clj-new.create"]}#2020-01-2416:07mike_ananevso, my alias looks the same:
:new {:extra-deps {seancorfield/clj-new {:mvn/version "0.8.5"}}
:main-opts ["-m" "clj-new.create"]}#2020-01-2416:08Alex Miller (Clojure team)if it's the conflict with transitive pomegranate maven deps, that may be the kind of thing that is arbitrary and resolves differently on different machines. can you dump me clj -Stree -A:new ?#2020-01-2416:08Alex Miller (Clojure team)so I can compare#2020-01-2416:09Alex Miller (Clojure team)and also, what version of clj are you on (`clj -Sdescribe`)#2020-01-2416:10mike_ananev{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "e5888c8e80a5888795d5d7"}, :content ("[email protected]")}#2020-01-2416:16Alex Miller (Clojure team)that was same for me#2020-01-2416:16Alex Miller (Clojure team)bumping to latest tools.deps with clj-new worked#2020-01-2416:16Alex Miller (Clojure team):new {:deps {seancorfield/clj-new {:mvn/version "0.8.5"}
org.clojure/tools.deps.alpha {:mvn/version "0.8.640"}}
:main-opts ["-m" "clj-new.create"]}`#2020-01-2416:17Alex Miller (Clojure team)probably because I'm not calling the DefaultModelResolver at all in latest#2020-01-2416:17Alex Miller (Clojure team)I guess @seancorfield that I would suggest maybe bumping tda in clj-new#2020-01-2416:20mike_ananevI will add org.clojure/tools.deps.alpha {:mvn/version "0.8.640"} as recommendation to README in case of errors#2020-01-2416:22Alex Miller (Clojure team)well, I'd hold off on that, I hate to cargo cult workarounds like that#2020-01-2416:22Alex Miller (Clojure team)I could file an issue?#2020-01-2416:23Alex Miller (Clojure team)I work with a hacked up clj environment a lot of the time and this may truly be just something on my machine (although I think I have undone all of that)#2020-01-2416:27seancorfield@alexmiller Can you open an issue on clj-new to bump the deps? That way I won't forget when I get to work, and I'll cut 0.8.6 today.#2020-01-2416:27Alex Miller (Clojure team)sure#2020-01-2416:32Alex Miller (Clojure team)done#2020-01-2417:05seancorfield@mike1452 seancorfield/clj-new {:mvn/version "0.8.6"} is up on Clojars -- can you try that out (without adding t.d.a.) and see if it resolves your issues?#2020-01-2417:06seancorfieldAlso @alexmiller 🙂#2020-01-2417:07seancorfieldI also bumped Pomegranate to the latest version, which is on clj-commons now, which may (or may not) cause new problems since the group ID has changed and code will possibly get two versions of it...#2020-01-2417:09seancorfield(the Jackson dependency hell conflicts with various templates are why 2.7.5 of that is also pinned in clj-new since Luminus' template was breaking before -- due to it relying on Leiningen's version resolution avoiding a conflict there)#2020-01-2417:24Alex Miller (Clojure team)btw, as of the new s3 transporter in tda, it no longer has a jackson dep#2020-01-2417:33seancorfieldOK, I may remove that and see if Luminus or other project templates still break.#2020-01-2417:34Alex Miller (Clojure team)you may still be pulling it in from multiple places, but tda is not one of them :)#2020-01-2417:36seancorfieldLuminus templates seem to generate OK without clj-new's deps on Jackson so in 0.8.7 I'll take them out.#2020-01-2417:36Alex Miller (Clojure team)also, you probably want the clj-new alias to be :deps, not :extra-deps so that you override, instead of add to, the deps of the current project deps.edn if there is one#2020-01-2417:36seancorfieldGood point. That wasn't available before.#2020-01-2417:36Alex Miller (Clojure team)usually it probably doesn't matter as you're making a new project#2020-01-2417:36Alex Miller (Clojure team)(but note that this requires a sufficiently new clj version, and you'll get an error if using older)#2020-01-2417:37Alex Miller (Clojure team)I'd recommend at least clj 1.10.1.483 (Nov 2019 vintage)#2020-01-2417:38Alex Miller (Clojure team)so maybe that's something to hold off on#2020-01-2417:41seancorfieldYeah... I'll give it a month or two and a few more releases of clj before I make that change. I just saw someone still using clj-new 0.5.5 😐#2020-01-2417:16mike_ananev@seancorfield I'll check. just a moment.#2020-01-2417:22Alex Miller (Clojure team)@seancorfield worked#2020-01-2417:23defaI have a tools-deps based project with a Clojure backend (lacinia-pedestal) und a re-frame frontend (ClojureScript). So it is a mixed Clojure/ClojureScript project. I’m using :aliases to run the server and the frontend (via shadow-cljs). Am I supposed to define the deps as :extra-deps in the related aliases? I just debugged an issue where jetty stopped working with very strange exceptions and that was related to having pedestal-lacinia and re-graph in :deps which caused incompatible versions of jetty libraries loaded. Moving the re-graph dep into the frontend alias solved the problem.#2020-01-2417:25Alex Miller (Clojure team)if you have deps that are only needed when the alias is enabled, then they should be :extra-deps in that alias#2020-01-2417:26Alex Miller (Clojure team)sorry if I misinterpreted the question#2020-01-2417:28defaI guess that’s the answer, since it is working. Is it in general a good/bad idea to have client and server in one deps.edn?#2020-01-2417:30Alex Miller (Clojure team)I don't think it's either good or bad#2020-01-2418:27dominicmTrue neutral?#2020-01-2418:28Alex Miller (Clojure team)💯#2020-01-2418:28Alex Miller (Clojure team)or maybe lawful neutral#2020-01-2418:51defaAt least it is very convenient when working with cursive inde IntelliJ IDEA. Guess I really have to give emacs a try one day…#2020-01-2421:12dominicmFwiw, I find it equally useful with vim, and my colleagues do with emacs.#2020-01-2421:13dominicmI try to defer splitting projects as long as possible. Development is faster when you don't have to fabricate arbitrary boundaries between things where you haven't found the boundary yet#2020-01-2517:35sveriWhile I agree with this sentiment in general, dealing with dependencies will cause troubles sooner or later (not only in clojure / java, but development in general).
If I have learned something over the decades it's these rules.
1. Uses as few dependencies as possible.
2. Separate these as much as possible.
3. Use the same version of a dependency everywhere, if you can.
4. Avoid gradle and eclipse, as eclipse still cannot separate classpathes like gradle does.#2020-01-2518:28dominicmTools deps has made my life significantly better in this space, so has the clojure culture of not breaking the api#2020-01-2417:40mike_ananev@seancorfield @alexmiller I have updated templates on GitHub with clj-new 0.8.6+#2020-01-2722:56salami hope this is the right channel to ask questions about deps.edn and the clojure cli tools.
i was wondering whether deps.edn can be spec'ed so that if one makes a mistake (e.g., putting a string literal where a symbol is expected), the cli tools can give some kind of feedback on the said mistake as spec errors. has the core team considered this?#2020-01-2723:30andy.fingerhutThis is a reasonable place to ask such questions about those tools. I'm not part of the core team, and do not know their views on stricter checking of deps.edn file contents.#2020-01-2723:49salamthanks for the response, @U0CMVHBL2 🙂#2020-01-2723:39Alex Miller (Clojure team)There are specs in the tools.deps repo#2020-01-2723:40Alex Miller (Clojure team)It’s not currently checked as we were trying to support older Clojure versions although we could probably bump that req now#2020-01-2800:00salamaha, indeed:
https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/specs.clj
yeah, being able to leverage this in clojure 1.10.x (or later versions) would be great.
i'll settle for the comment form at the bottom of that spec file for now. 🙂 thanks!#2020-01-2800:01Alex Miller (Clojure team)Yeah, I used that to validate all the deps.edn I had lying around :)#2020-01-2822:55Alex Miller (Clojure team)so I kind of got distracted into implementing parallel downloads for tools.deps today#2020-01-2822:56Alex Miller (Clojure team)might need some hearty volunteers to try it before I release it though#2020-01-2822:57Alex Miller (Clojure team)it was one of those things where I got it working in about 20 minutes, then spent 3 hrs properly rewriting it to account for error handling and all the other real work#2020-01-2822:57Alex Miller (Clojure team)but it's definitely faster when you're downloading a bunch of junk#2020-01-2823:00Alex Miller (Clojure team)it all of a sudden got a lot slower a while ago and I thought I had broken something until I realized my son was downloading a giant game 😄#2020-01-2823:32seancorfieldHow should we test it for you?#2020-01-2823:44Alex Miller (Clojure team)well, I'm still hacking on it a bit, but I'll try to make something available#2020-01-2904:10dharriganHappy to volunteer too!#2020-01-2904:48Alex Miller (Clojure team)the interesting bit is that parallelizing the jar downloads only got me so far because with Clojure artifacts, the pom is of significant size compared to the jar files and we end up downloading a lot more metadata during version selection#2020-01-2904:50Alex Miller (Clojure team)so, working through parallelizing the metadata download too, which is a lot harder to work into the algorithm#2020-01-2904:50Alex Miller (Clojure team)but, still seems doable#2020-01-2907:50dominicmVery cool. Does maven do parallel at all?#2020-01-2907:50Alex Miller (Clojure team)yeah#2020-01-2907:51Alex Miller (Clojure team)I have parallel metadata during expansion working now#2020-01-2907:51Alex Miller (Clojure team)the benefits vary a lot depending on how much metadata/how much data/how much parallelism/how bushy the dep tree is, etc#2020-01-2907:52Alex Miller (Clojure team)for something like the luminus template, for lein to download the full dep tree is ~43 sec#2020-01-2907:52Alex Miller (Clojure team)clj before was 73 sec and is now 10 sec#2020-01-2907:52Alex Miller (Clojure team)that's kind of an extreme (lots of jars, huge fanout, lots of java, so big jars) etc#2020-01-2907:53Alex Miller (Clojure team)but 20-60% improvement is a common range#2020-01-2907:54Alex Miller (Clojure team)most of the time of course you're not really seeing those full numbers because you already have some or most of the libs downloaded in the cache#2020-01-2909:15p-himikIt seems that "RELEASE" and "LATEST" are not available in Maven 3, but tools.deps seem to support them.
However, I cannot find any mention of these aliases in the documentation.
What's the deal with them? Can people rely on them? Can developers safely recommend using them for e.g. such things as tools in the user's deps.edn?
Also, I know that "RELEASE" is susceptible to cache issues - if no new dependency is needed to be retrieved during dependency resolution, the latest release available locally will be used (i.e., no check for a new release will be made) or something like that.
Is "LATEST" also susceptible to such issues? Or does it reliably make a check for a new version each time the dependency resolution is done?#2020-01-2913:26Alex Miller (Clojure team)They are not supported and break the cache checking#2020-01-2913:27Alex Miller (Clojure team)That said, they do actually work and can be useful for one off calls in something like -Sdeps#2020-01-2913:28Alex Miller (Clojure team)I recommend using real versions instead#2020-01-2913:30p-himik@thheller ^ FYI
Yeah, I use "RELEASE" only for clj-kondo, tools.deps.graph, etc. Something that runs only to facilitate the development.#2020-01-2915:01Alex Miller (Clojure team)note that if you use RELEASE for tools, you generally won't get new versions of the tools after you first run them, unless you -Sforce (because the hashed key is just seeing "RELEASE" every time)#2020-01-2917:18seancorfieldGood to know. Explains some of the slightly odd behavior I thought I was seeing from time to time since my dot-clojure deps.edn uses RELEASE a lot.#2020-01-2917:19seancorfieldclj-new has moved away from that and tracks explicit versions now.#2020-01-2917:20seancorfieldIt would be nice if there was some way in deps.edn to flag a dependency that should be rechecked periodically rather than cached indefinitely.#2020-01-2917:24Alex Miller (Clojure team)The issue is that the cache check is done in bash to avoid jvm load so the hash is purely text based#2020-01-2917:25Alex Miller (Clojure team)So there is no easy way to do what you’re suggesting without a lot of trade offs#2020-01-2917:27seancorfieldYeah, I think it would need an additional file in .cpcache for "dependencies that should be rechecked" which is probably more complexity than anyone would want for the occasional use of this feature.#2020-01-2917:31seancorfieldIs it worth me spending time thinking about this problem or would such a proposal not be considered anyway?#2020-01-2918:43Alex Miller (Clojure team)Way down my list#2020-01-2918:46seancorfieldI'll take that as a "no" then 🙂#2020-01-2915:02Alex Miller (Clojure team)probably you're changing your local deps.edn or clj install frequently enough that you are periodically re-caching and don't notice this#2020-01-2915:21borkdudeit would be cool if tools.deps could resolve RELEASE in addition to tags maybe?#2020-01-2915:22borkdudeas in replace RELEASE with the most recent version in your deps.edn file#2020-01-2915:35Alex Miller (Clojure team)I'm not planning on adding any more mucking with deps.edn#2020-01-2915:36Alex Miller (Clojure team)but does not need to be in tools.deps - this itself could be a tool#2020-01-2915:36Alex Miller (Clojure team)I'm not sure that's actually the best solution though#2020-01-2915:38dharriganhttps://github.com/Olical/depot does something similar, although I do not know if it processes "RELEASE" versions, but it updates the deps.edn in place definitely for deps that have versions.#2020-01-2915:44dharriganMaybe it does Besides SNAPSHOT versions --resolve-virtual will also handle the special version strings "RELEASE" and "LATEST"#2020-01-2915:51borkdudeThis script uses RELEASE to print the most recent versions of libraries that are newer than you current ones:
https://github.com/borkdude/babashka/#list-outdated-maven-dependencies
A bit of a hacky light version of depot (thanks seancorfield)#2020-01-2916:23robert-stuttafordcan i remove a dependency in an override?#2020-02-0820:50borkdude@mss fwiw, deps.clj does support this with the non-standard -Sdeps-file option#2020-02-0821:13seancorfield@mss If you do CLJ_CONFIG=/path/to/other/folder cllj it will look there for the user-level deps.edn (instead of you home directory) which might give you part of the solution?#2020-02-0821:19seancorfieldI like the -Sdeps-file idea tho'. I would use that quite often I suspect if clojure supported it. @alexmiller would that even be considered as an enhancement?#2020-02-0821:22borkdudeI often use it when I need the classpath from some other project in some other directory to lint with clj-kondo#2020-02-0821:23borkdudealthough $ ( cd /tmp/project ; clojure -Spath ) works too#2020-02-0821:24seancorfieldI suppose you can always use clj -Sdeps "$(cat /path/to/deps.edn)"#2020-02-0821:26borkdudethat works, but your local deps.edn will still be merged in, which is not always what I want#2020-02-0821:26borkdude-Sdeps-file will regard that file instead of the local deps.edn#2020-02-0821:28seancorfieldYeah, there are a lot of possibilities around which combination of files you might want to work from.#2020-02-0821:55Alex Miller (Clojure team)I've walked down the path of adding this a couple times for different reasons but there are a lot of unobvious consequences#2020-02-0821:55Alex Miller (Clojure team)like do you interpret relative paths in deps.edn from the deps.edn location or current location?#2020-02-0821:56Alex Miller (Clojure team)there are also now a lot of code locations that assume the deps.edn name, not just in tools.deps.alpha but in external tools around it#2020-02-0821:58Alex Miller (Clojure team)at the moment the tradeoffs don't seem worth it to me but I am interested to hear about cases where people need stuff like this though so I can continue to evaluate that#2020-02-0901:02deletedit's alpha, thankfully, so changes like this would be ok?#2020-02-0901:23andy.fingerhutI suspect Alex is in a mind set where even though it is alpha, adding things that he doesn't see value in supporting long term is like intentionally planning to pull the rug out later.#2020-02-0903:22Alex Miller (Clojure team)it's not alpha in my mind#2020-02-0903:22Alex Miller (Clojure team);)#2020-02-0903:23Alex Miller (Clojure team)Rich and I just haven't had the time to think through what changing it to be non-alpha means yet#2020-02-0907:51andy.fingerhutIf I interpret what Alex said correctly, it is currently alpha, but he and Rich want it to become non-alpha soon-ish.#2020-02-0907:56Alex Miller (Clojure team)it has alpha in the name, but I have been maintaining it like it's not alpha for a long time, and believe the api is stable#2020-02-0908:08borkdudedeps.clj non-standard features are certainly alpha though, I think it can be a place of experimentation as far as the bash part goes#2020-02-0910:07Rachel WestmacottMorning! I've just got an error from my IDE saying: "The following errors were found during project resolve: /home/peter/personal/mandel-zero/deps.edn: Destination path "see" already exists and is not an empty directory" - it doesn't tell me where this non-empty directory is though! Is there a place where tools-deps caches dependencies from github that I can go and delete things from?#2020-02-0910:14dharriganHave you tried to delete .cpcache?#2020-02-0910:16Rachel WestmacottI think its ~/.gitlibs#2020-02-0910:28Rachel Westmacottoh, dear, I seem to have broken it. it now just says 'Auth fail' instead#2020-02-0914:59Alex Miller (Clojure team)what happens if you are just at the command line?#2020-02-0915:00Alex Miller (Clojure team)these sound like ide specific issues, not tools.deps#2020-02-0915:20Rachel Westmacottstrangely not. The ide appears to be reporting an underlying error#2020-02-0915:21Rachel Westmacotton the plus side I do get a stack trace though#2020-02-0915:21Rachel WestmacottCloning:
#2020-02-0915:24Rachel Westmacotttbf I did go deleting directories in ~/.gitlibs to try and fix an earlier problem, which was probably a little reckless - I was sort of expecting them to get recreated as needed#2020-02-0915:48Alex Miller (Clojure team)what git url are you using?#2020-02-0915:52Alex Miller (Clojure team)I guess something like /cdn-cgi/l/email-protection:peterwestmacott/draw.git#2020-02-0915:52Alex Miller (Clojure team)what os are you on?#2020-02-0915:54Alex Miller (Clojure team)is your ssh-agent running? does it have your ssh identity?#2020-02-0915:54Alex Miller (Clojure team)ssh-add -l should list your key#2020-02-0916:26Rachel WestmacottThe agent has no identities.#2020-02-0916:26Rachel WestmacottI'm guessing that's bad#2020-02-0916:26Rachel WestmacottI'm running xubuntu#2020-02-0916:28Rachel Westmacottoh, that seemed to sort it out. THANKS ALEX!!!!#2020-02-1010:11andy.fingerhutIs there a way in one's $HOME/.clojure/deps.edn file to have a :local/root dependency with a relative path name starting from your home directory? I tried {:local/root "$HOME/path/to/my.jar"} and {:local/root "~/path/to/my.jar"} , but neither seemed to work#2020-02-1010:11andy.fingerhutI ask because it would be helpful to have such a thing for sharing a ~/.clojure/deps.edn file contents across multiple systems, where my home directory absolute path name differs between them.#2020-02-1010:14borkdudetildes are not expanded in .edn files#2020-02-1010:15borkdudeneither is $HOME#2020-02-1010:16borkdudebut I get the problem, would be nice to see a solution for#2020-02-1010:56sogaiumy current work-around is to put all source under ~/src/ (or some other directory right under my home directory) and make paths relative: e.g. "../REBL/REBL.jar"
ofc, that means on each system source code needs to live under a directory right under ~.
on a related note, to get the effect of "variables" in deps.edn, i've been thinking whether to generate deps.edn (possibly from some template).#2020-02-1010:57borkdudewith some lightweight scripting / templating you can get around it though#2020-02-1010:58borkdudeah, you also suggested that sogaiu 😉#2020-02-1013:45Alex Miller (Clojure team)I'm going to continue to resist variables in deps.edn for as long as possible :)#2020-02-1017:52andy.fingerhutIs tilde as alias for home directory in same category?#2020-02-1017:52andy.fingerhutNot an urgent need by any means, just asking while we are in the neighborhood.#2020-02-1017:52Alex Miller (Clojure team)yes#2020-02-1017:52Alex Miller (Clojure team)it's in the category of "processing step beyond reading as data"#2020-02-1017:53Alex Miller (Clojure team)having none of those is a huge advantage for downstream work#2020-02-1020:47mike_ananevJust hacking my own build tool right now, and I created new file build.edn and code for processing with comment in code
"This sections dropped to build.edn cause Alex Miller continue to resist variables in deps.edn"
😎#2020-02-1020:50andy.fingerhutI mean, you can give that as the reason if you want, but as an engineer, I'd say addressing the reasons for that design decision, rather than focusing on the person that made them, is a good practice. Sure, include the name of one or more people responsible for making the decisions if you want, but without the reasons, it can sound like you believe the decisions are arbitrary, or not grounded in any kinds of reasons. Not saying you have done that -- just pointing out the possibilities of misinterpretation.#2020-02-1014:03dominicmI think generating your own deps.edn is a good idea#2020-02-1016:08sparkofreasonAre there tools/methodologies around adapting lein projects containing java to be amenable to clj/`deps.edn` ? I'm looking at a potential dependency that is fairly new and fast-moving, would like to be able to reliably pin my reference to a known SHA, so I can reference the code at that point rather than depending on potentially out-of-date docs, etc. I could manually run lein and deploy it locally, but something more automated where I can just update the SHA and have clj do the right thing would be nice-to-have.#2020-02-1016:15borkdude@dave.dixon The most reliable way I know is fork it and add a deps.edn to it.#2020-02-1016:17sparkofreasonYes. But then how do I get the downstream project to compile the java and put the classes on the classpath? I think I could do it for the specific case of that project using an alias and a tool like badigeon, but curious if anybody had come up with anything more general.#2020-02-1016:18borkdudeif it contains java, it's a bit of a different story. I would just deploy it as a leiningen artifact to your own org on clojars in this case#2020-02-1016:19borkdude(my personal take on it, for compiling java I always use lein, since it just works)#2020-02-1018:32sparkofreasonSort of hacked it by building the java classes, and adding that directory to the :paths . Pushed to github, and that works. Curious if there's any reason to not do this.#2020-02-1109:44rickmoynihan> Curious if there’s any reason to not do this.
It’s not good to commit complied (and binary) artefacts into source control.
It would be better to compile the java classes and deploy it as a library. Either a shell script with javac and some mvn calls, or just move the java code into its own lein project and use lein & project.clj to compile and deploy the java bit, and depend on it from the tools.deps lib.
Whether to break out the java as a separate lib largely depends on how stable it is, and how often you expect to change it etc.#2020-02-1114:36sparkofreason> It’s not good to commit complied (and binary) artefacts into source control.
I've heard this, and my understanding (maybe wrong) for the rationale is that git doesn't handle large binaries well, as might be the case for a jar or fully linked executable. Class files, however, are pretty small, and map closely to the corresponding java source.#2020-02-1114:43sparkofreasonThe key risk is having the compiled class files be out of sync with the source. But that's kind of the problem I'm facing anyway, source is changing, and the clojars artifacts may not match, which makes it hard to figure out which features are there etc. More ideal would be a tool that built the java classes after clj pulled from the indicated :sha.#2020-02-1114:56rickmoynihanWell there are a whole bunch of reasons:
1. git doesn’t handle large files in a repo very well (operations on git repos slow to a crawl on large repo’s, e.g. changing branches etc)
2. you can’t diff a binary very well, so merging branches etc becomes painful. Should you accept a change or not?
3. you have a dependency between source and output files in the repo
4. related to 3, but for non-binary output files; e.g. compiling and minifying css/js you get huge diffs in your repo that can cause some tools to crash/struggle to render. You can set attributes on the files in git, but not all tools honor them properly.
5. different versions of javac generate different byte code for the same source code so you may get diffs and deltas just by upgrading java versions
6. tools.deps can’t handle files in a repo over ~100mb (due to limitations in jgit)
7. adopting this practice on repo early on, normalises the behaviour, and all sorts of binaries get commited; git performance tanks, and then you have to rewrite the whole history with filter-branch, or worse because of lack of patience around rewriting the history correctly the team decide to drop all the git history and start fresh in a new repo.
I’ve worked in many repo’s where this has been done, and it’s almost always caused problems. Not saying you won’t get away it; just be aware of the problems.#2020-02-1114:57rickmoynihanI’ve also personally experienced all of those at some point in the last 15+ years or so using git.#2020-02-1116:29sparkofreasonAgreed in general. And most of this sounds like something that should be fixed in git, making the abstraction less leaky, but that's a whole other thread. Clearly it would be better to have a way to trigger the java build after clj pulls and update the classpath accordingly.#2020-02-1116:39rickmoynihanThere are fixes in/around git for things like large-files, and git handles binaries ok… However you can’t really solve the issue of how to merge a random binary; it’s essentially pick one or the other or commit a new one. I’m not sure git can really do much more to help there.
For me philosophically it’s just the wrong thing to do; git is for source control and generated files aren’t source. I’m saying this as someone who has committed binaries into repos though; and may at some point do it again… I’d just always very much prefer to avoid it.#2020-02-1017:56andy.fingerhutI have not been working on this full time by any means, but from a programmatically generated file from http://Clojars.org, I have a list of 18,493 projects and their URLs they list in the artifact there, which appear to be on Github, downloading now. This is partial progress towards answering the question "what fraction of Leiningen project.clj files are readable using clojure.edn/read".#2020-02-1017:57andy.fingerhutAs an aside on the kinds of weird things I now expect more often in such large-ish data sets: 672 projects with the incorrect URL ssh-keygen and adding it to the openssh agent via ssh-add.
I think the relevant part of the stacktrace is:
Caused by: org.eclipse.jgit.errors.TransportException: : Authentication is required but no CredentialsProvider has been registered
#2020-02-1020:44samIs there something I’m missing? I was able to get it working fine on my mac.#2020-02-1020:44dominicmThat's https, you need to use ssh @samwagg0583#2020-02-1020:45dominicmChange the url to be the ssh one on github#2020-02-1020:45samoh wow, I can’t believer I did that and have been spinning my wheels for hours!#2020-02-1020:51samThat did it, @dominicm. I’m embarrased, but thanks so much.#2020-02-1022:12hlollione nix package (Lumo) suddenly started to fail with
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.clojure:clojure:pom:1.10.1 from/to central (): : Name or service not known
I'm using tools-deps to resolve the dependencies, but I highly doubt this is related to tools-deps. But could be some mirror setting I guess? This at least isn't network error since Hydra (the CI of nixpkgs) also reported this exact same error. Maybe this error sounds familiar to someone?
Full stack here 4wiw https://pastebin.com/sC6d0Cdn#2020-02-1022:15andy.fingerhutIn a web browser, I can go to
successfully. Maybe some transient failure, or networking problem between that host and public Internet?#2020-02-1022:17ghadiyou need to paste your settings for fetching these deps#2020-02-1022:18ghadior whatever nixpkgs is doing#2020-02-1022:18hlollithis was also reported to me here
https://gist.github.com/Mic92/e7ac6eb8652b72ddc9e5a349b8e7bc23#2020-02-1022:18ghadiseems pretty clear:
Caused by: https://java.net.UnknownHostException: https://repo1.maven.org: Name or service not known
at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)#2020-02-1022:19ghadieither transient name resolution failure, or JVM is mistakenly resolving maven through IPv6 instead of IPv4#2020-02-1022:19hlollilet repos = [
""
""
""
""
""
];
....
{
name = "org.clojure/clojure";
path = pkgs.fetchMavenArtifact {
inherit repos;
artifactId = "clojure";
groupId = "org.clojure";
sha512 = "f28178179483531862afae13e246386f8fda081afa523d3c4ea3a083ab607d23575d38ecb9ec0ee7f4d65cbe39a119f680e6de4669bc9cf593aa92be0c61562b";
version = "1.10.1";
};
}
...
so I download the jars then call clojure basically#2020-02-1022:20ghadialso the error you're pasting is a different version than the one in nix#2020-02-1022:21ghadi1.10.1 vs 1.10.0-beta5#2020-02-1022:21hlollithe latter one uses beta, yes, same error still. I tried bumping.#2020-02-1022:21ghadibut still, the error cause is accurate#2020-02-1022:21ghadiit has nothing to do with the version#2020-02-1022:21ghadithere's a name resolution error#2020-02-1022:21hlolliyes, something in that direction was my instinct#2020-02-1022:22hlollijust dns lookup?#2020-02-1022:22ghadiinstinct / gut feeling unnecessary 🙂
you just have to read the stacktrace#2020-02-1022:22ghadiit appears like it's resolving through IPv6#2020-02-1022:22andy.fingerhutWell, or it is using IPv6 when IPv6 is disabled somewhere in the network between you and DNS resolver, or between your host and where JAR file is.#2020-02-1022:22ghadinot sure if the JVM always appears like it's resolving through IPv6#2020-02-1022:22hlolliwell, I was only explaining why the first paste was in diferent version, but yeh, nevermind#2020-02-1022:23ghadiyou can try to set -.preferIPv4Stack=true on the JVM to see if it resolves differently#2020-02-1022:23ghadior try to resolve something that you know is invalid, and see if you get the same trace#2020-02-1022:24hlollinice, that sounds like something I can try. I'm using 8.8.8.8 like always. Maybe maven isn't discoverable there (for me dns lookups are black boxes)#2020-02-1022:24hlollithanks for the tip fellas 🙂#2020-02-1022:25ghadinp. you should test your hypothesis#2020-02-1022:25ghadialso check if nix changed their global ipv6 settings or something#2020-02-1022:26hlollithat's also plausible, tough it could only be the package manager that could have such power, all else is configured by the user#2020-02-1022:27hlolliit's a beginning of a bug hunt I guess, again, thanks#2020-02-1022:28andy.fingerhutSounds like it. We can throw the hot potato into what looks like the most likely hands for the problem, which at least in this case strongly appears not to be deps.edn code, nor any other particular package manager type of thing.#2020-02-1022:31Alex Miller (Clojure team)I was on a thread last week about some Maven deploy errors (different problem) and it sounded like they've been doing some server updates. could be something on their side. They have a jira system for reporting issues if you wanted to ask there. https://issues.sonatype.org/secure/Dashboard.jspa#2020-02-1022:34ghadias a first step you should try to figure out what DNS settings you have in /etc/resolve* and what address types you have set with ip addr show#2020-02-1023:04hlollireddest herring this week, I actually had stable and good connection to maven, the error was arriving at installation time and not build time, which in nix mean no unwanted side-effect (calling http(s)). So to take my quesiton to a new direction, why would the clojure executeable try to download clojure-1.10.1 when I specify its jar location on the classpath?
My classpath is (stored as classp)
/nix/store/58c2jsc0gq4mi73ma6pvphgnvnpwlnn8-com_cognitect_transit-java-0.8.332.jar:/nix/store/bayaxp12xw9kpbxn35pwab2s4ya8gpcn-org_clojure_data_json-0.2.6.jar:/nix/store/z26fq48r5xs7syp1j2df5pjv1qk37pjf-org_clojure_clojure-1.10.1.jar
...elided...
and command is
${clojure}/bin/clojure -Scp ${classp} -e "${compileClojurescript true}"#2020-02-1023:08hlolliwell, I'll open a ticket if I find a clear regression between previous versions. No need to solve it here now.#2020-02-1023:08andy.fingerhutAs a guess (I do not know implementation details), but there might be cases where clojure finds a different version in local file system, and time stamps or I-don't-yet-know-what cause it to make network connections to see if a newer version isa available?#2020-02-1023:09hlollithat could be, for security resons, all file timstamps are set to 0 (1jan1970).#2020-02-1023:10andy.fingerhutIf you want to absolutely prevent this possibility, you could use clojure command to create a classpath as its output, and save that in a one-line bash script#2020-02-1023:10andy.fingerhutUsing -Spath option.#2020-02-1023:11andy.fingerhutbefore you reach installation time, or whatever step doesn't like you to make network connections.#2020-02-1023:11hlollihmm, good point#2020-02-1023:12hlollithat should be easy thing to test. The maven endpoints are initially made this way. I make tools-deps create classpath and maven endpoints which I have nix download for me, and then provide it again to clojure. Has worked fine last 2 years.#2020-02-1023:13hlollithen the question is if the generator and the build environment are mismatching, tough I updated 2 months ago, there's a chance that nixpkgs has even newer clojure data.#2020-02-1023:13andy.fingerhutSomething changed, but I don't know what. Depending upon how much time you like tracking down such root causes, and how important it is, you can try, or just change steps to avoid it happening.#2020-02-1100:03Alex Miller (Clojure team)looking at the clojure script, I think it will still do staleness checking (whether one of the deps.edn files is newer than the cached classpath file), and potentially download and create an unused classpath file even if you -Scp#2020-02-1100:06Alex Miller (Clojure team)that seems like a bug, I don't think there's any reason to do that#2020-02-1100:14andy.fingerhutI was thinking one could use clojure -Spath ... > mypath.txt when you had network connections allowed, and "java -cp java command where you were not supposed to make network connections, but sure, making network connections unnecessarily in clojure / clj programs seems like a bad idea.#2020-02-1100:19Alex Miller (Clojure team)using clojure -Scp for the latter should have that effect#2020-02-1105:21richiardiandreaMaybe it is completely unrelated, but here we discovered that our Java mvn builds were throwing warnings cause we were still using user> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user> (add-lib 'clj-python/libpython-clj {:mvn/version "1.36"})
user> (require '[libpython-clj.require :refer [require-python]])
...
Error: failed to access class com.sun.jna.CallbackReference#2020-02-1104:24cartesian-theatricsWorked fine after restarting the repl with the dep.#2020-02-1104:24cartesian-theatricsAre there plans to ever merge add-lib into master?#2020-02-1104:28Alex Miller (Clojure team)it will probably be added in some form, but may change in api/impl#2020-02-1104:30cartesian-theatrics:thumbsup:#2020-02-1104:30cartesian-theatricsIt's still super useful in it's current form, so thanks for the tool!#2020-02-1104:30Alex Miller (Clojure team)add-lib will not always be able to safely add libs as it's trying to resolve new libs in terms of the already loaded library set#2020-02-1104:33Alex Miller (Clojure team)I'm not sure how the native stuff interacts with any of that either#2020-02-1104:34cartesian-theatricsI see. I'm not well versed on how deps are resolved, but I can imagine there's all sorts of edge cases. Thankfully it does very often work.#2020-02-1104:39seancorfieldAgree. I do all my dev work with Atom/Chlorine + REBL and the add-lib branch of t.d.a. and I love being able to add new dependencies on the fly! Every now and then I add one and get false back (indicating it didn't add it for some reason) but mostly it works exactly as expected.#2020-02-1104:40seancorfieldIt would be nice if, instead of just false, add-lib returned more information about why it didn't add a dependency.#2020-02-1104:41cartesian-theatricsFor sure, at some point think it would be great to give it (the feature) some signal boosting, especially if we can squash more of the edge-case issues. It really helps with flow.#2020-02-1104:41Alex Miller (Clojure team)well, it doesn't actually know why#2020-02-1104:43Alex Miller (Clojure team)it takes all currently loaded libs + the new lib and then does a full resolution, then compares the result with the prior set#2020-02-1104:43cartesian-theatricsIs there a conceivable "brute force" / descructive approach?#2020-02-1104:44seancorfieldFair enough @alexmiller -- it can't report what it does know! 🙂#2020-02-1104:45Alex Miller (Clojure team)there are multiple possible reasons why you wouldn't end up with something new#2020-02-1104:46Alex Miller (Clojure team)because everything already loaded is "pinned", adding a different version of something already loaded isn't possible (starting from a real initial dep set doesn't have those constraints, so you can use the version selection to resolve)#2020-02-1104:49cartesian-theatricsI see. Sounds like the current form is about as good as we can do then.#2020-02-1104:50seancorfield@alexmiller Here's a repeatable case of failure, from an empty REPL with the add-lib branch:
user=> (require '[clojure.tools.deps.alpha.repl :refer [add-lib]])
nil
user=> (add-lib 'jedi-time {:git/url "" :sha "8e756ecb71bbfa0b081e00d71a21c47037f1eae4" :deps/manifest :deps})
Cloning:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
Checking out: at 8e756ecb71bbfa0b081e00d71a21c47037f1eae4
false#2020-02-1104:51seancorfieldThis works fine as a regular dep:
$ clj -Sdeps '{:deps {jedi-time {:git/url "" :sha "8e756ecb71bbfa0b081e00d71a21c47037f1eae4" :deps/manifest :deps}}}'
Clojure 1.10.1
user=> (require '[jedi-time.core :as jdt])
nil#2020-02-1104:52seancorfieldIs it due to needing the :deps/manifest override?#2020-02-1104:55seancorfield(I see the add-lib branch is about two months behind master at this point)#2020-02-1105:09seancorfield(`add-lib` works if I use {:mvn/version "0.1.4"} instead of the git coordinates)#2020-02-1105:15andy.fingerhutOK, 13K GitHub repositories cloned, with URLs from Clojars projects, of which 12K have project.clj files in their home directories, ~500 deps.edn files in their home directories. More stats later.#2020-02-1105:16seancorfield@andy.fingerhut And the other ~500 are... boot? Or a mixture?#2020-02-1105:18andy.fingerhutDon't know yet. What file name should I look for? Is it boot.edn?#2020-02-1105:18seancorfield(are you counting deps.edn in projects that also have project.clj?)#2020-02-1105:18seancorfieldbuild.boot#2020-02-1105:18andy.fingerhutNo combinations checked for yet, but will do so.#2020-02-1105:18andy.fingerhutThat was just raw count of those files in root dirs.#2020-02-1105:18seancorfieldSweet! You're a rock star!#2020-02-1105:19andy.fingerhut635 build.boot files. Again no stats on projects with different combinations of those. As I said, bigger more detailed report I will send a link later, after I find out how many of these project.clj files can be read with clojure.edn/read#2020-02-1105:20andy.fingerhutAnd then I may want to get fancy and report not only the fraction, but also the weighted fraction, weighted by something like Clojars download counts.#2020-02-1105:20seancorfieldHmm, so CLI/`deps.edn` is already almost as popular as Boot. Interesting.#2020-02-1105:21andy.fingerhutAnd of course many of these repos may not have been updated for years...#2020-02-1105:22seancorfieldGood point. Year-by-year stats based on last update would be interesting if you're thinking of going that way 🙂#2020-02-1105:24andy.fingerhutThat may be interesting enough to spend the extra time to get it.#2020-02-1105:26Alex Miller (Clojure team)deps is way more than boot in the 2020 survey#2020-02-1105:27seancorfieldGood to know @alexmiller -- did you see my add-lib issue above (now scrolled off)#2020-02-1105:32Alex Miller (Clojure team)new deps-mgmt question was lein 84%, deps 49%, boot 6%#2020-02-1105:35seancorfieldWow! That's an impressive uptake!#2020-02-1105:36Alex Miller (Clojure team)and (obviously) that's multi-select, so presumably a lot of people using both lein and deps#2020-02-1105:28Alex Miller (Clojure team)I saw it, but not a priority for me right now - please file a question at https://ask.clojure.org#2020-02-1118:34seancorfield@alexmiller I went ahead and filed a JIRA on TDEPS: https://clojure.atlassian.net/browse/TDEPS-149#2020-02-1118:50Alex Miller (Clojure team)Thx#2020-02-1105:29seancorfieldGotcha. I'll see if I can narrow it down a bit first. Should git deps work, in general with add-lib?#2020-02-1105:30Alex Miller (Clojure team)off the top of my head, sure?#2020-02-1105:33Alex Miller (Clojure team)it's using the same resolve-deps as normal deps, so can't think why it wouldn't#2020-02-1105:34seancorfieldThanks.#2020-02-1107:49robert-stuttafordif i had an app project and a library project, and the library project had deps.edn aliases, could i invoke those from the app project?#2020-02-1108:17dominicmNope.#2020-02-1108:34robert-stuttafordthanks for confirming!#2020-02-1109:25robert-stuttafordis there a canonical reference on how to use deps.edn git coordinates with private github repos?#2020-02-1109:27dominicmI think it's just as simple as using the ssh url#2020-02-1109:38robert-stuttafordthanks!#2020-02-1113:19souenzzoHello
$ clj -Srepro -Sdeps '{:paths ["a" "b"] :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
a
b
/home/souenzzo/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/a
/home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/b
/home/souenzzo/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/home/souenzzo/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
a b paths from "root project" are "chained" to clj-kondo/lsp-server . I think that is a issue, maybe related to TDEPS-52
https://clojure.atlassian.net/browse/TDEPS-52#2020-02-1113:23souenzzoI expect /home/souenzzo/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src as the default :paths is ["src"]
May also be nice to control :deps/paths 👀#2020-02-1113:35Alex Miller (Clojure team)what version of clj are you using?#2020-02-1113:36Alex Miller (Clojure team)clj -Sdescribe will tell you#2020-02-1113:37Alex Miller (Clojure team)if you're not using latest (1.10.1.507) could you update and try again?#2020-02-1114:11souenzzoI'm on 1.10.1.507
I posted a "onliner" so it's easy to rest in many versions#2020-02-1113:38Alex Miller (Clojure team)TDEPS-52 is long fixed, so I'm wondering if you're on an older version#2020-02-1113:52borkdudethis is my output with the newest (507):
$ clj -Srepro -Sdeps '{:paths ["a" "b"] :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
Cloning:
Checking out: at 4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5
a
b
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/a
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/b
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar#2020-02-1113:53borkdudeso it might be an issue with :paths + :deps/root ?#2020-02-1113:55Alex Miller (Clojure team)dunno, must be a bug, please file either a jira or a question on https://ask.clojure.org#2020-02-1113:55Alex Miller (Clojure team)or I guess I'll just make a jira, nvm#2020-02-1113:57Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-148#2020-02-1114:15souenzzoatlassian login system is a pure mess
I have 3 accounts (clojure, work1, work2)
When I'm authed as ẃork1 and try to login to vote on clojure, it blows a auth error
then I need to go to work1 workspace, logout, login in clojure workspace, then(.................) 😓#2020-02-1114:22souenzzoafter create a new browser to login in jira, there is no vote option to vote on TDEPS issues?#2020-02-1114:28Alex Miller (Clojure team)the jira auth is for sure, not good, particularly cross accounts. identity is tied strongly to email but the gsuite tie-ins make everything pretty bad. I don't have a good solution to that other than private browser window#2020-02-1114:28Alex Miller (Clojure team)if you have a clojure jira logon, you should be able to vote. that said, they have been actively destroying the user experience with their "new ui" issue view. it could just be that it's buried a menu level or two down#2020-02-1114:29Alex Miller (Clojure team)I'm using the old ui, so not sure I can point to how to do it#2020-02-1114:38borkdudeI have an old account, but I'm not able to vote for a long time anymore, at least, can't find that option#2020-02-1114:44souenzzookay after spawn 5 firefox isolated instances and login with all my credentials, I'm sure that voting is disabled.
My account is {:tag :mailto:souenzzohotmail.comsouenzzohotmail.com, :attrs nil, :content nil} with "raw password" (without microsoft auth).#2020-02-1115:17Alex Miller (Clojure team)there is no way in jira to disable voting for a user#2020-02-1115:17Alex Miller (Clojure team)voting is enabled in our jira product instance#2020-02-1115:18Alex Miller (Clojure team)whether you can vote is dependent on whether you can edit issues and I have it configured in all projects that if you are a logged in user, you can edit issues#2020-02-1115:18Alex Miller (Clojure team)your acct looks normal - you have jira product access and good default roles like other users#2020-02-1115:19Alex Miller (Clojure team)if you go to something like https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-132 - you don't see a link "Vote for this issue" by the votes field on the right?#2020-02-1115:20borkdudeif voting is so hard to find, maybe we can just regard it as non-existing#2020-02-1115:22Alex Miller (Clojure team)it's a link at the top with the words "Vote for this issue" which seems pretty easy to find to me#2020-02-1115:23Alex Miller (Clojure team)(note that you can't vote for issues you create)#2020-02-1115:23borkdudeI have tried to find it multiple times, and apparently I'm not the only one. I gave up on it.#2020-02-1115:24Alex Miller (Clojure team)well there must be something I'm missing then#2020-02-1115:24Alex Miller (Clojure team)there is no intent for it to not be there#2020-02-1115:25borkdude#2020-02-1115:26andy.fingerhutI see how to vote using my JIRA account when looking at a ticket using the old view, but not the new one#2020-02-1115:27andy.fingerhutThere is a link "See the old view" near the top center of the frame.#2020-02-1115:27Alex Miller (Clojure team)yeah#2020-02-1115:27Alex Miller (Clojure team)if you go to https://clojure.atlassian.net/secure/ViewPersonalSettings.jspa you can configure this#2020-02-1115:28Alex Miller (Clojure team)turn OFF "New JIRA Issue view" - when you do, it asks you why. tell them you can't vote. :)#2020-02-1115:28andy.fingerhutBut agreed without explicit instructions somewhere, most people won't bother hard trying to find it.#2020-02-1115:31Alex Miller (Clojure team)here's a jira issue in jira about this https://jira.atlassian.com/browse/JRACLOUD-71251#2020-02-1115:31Alex Miller (Clojure team)sorry, I turned this off so long ago I forgot about it#2020-02-1115:31Alex Miller (Clojure team)278 votes to enable voting#2020-02-1115:32borkdudeok, I reverted to the old view of which I was unaware it existed and now I see the votes again#2020-02-1115:32andy.fingerhutThose were some determined voters, who figured out how to vote on it 🙂#2020-02-1115:32borkdudelol#2020-02-1115:35Alex Miller (Clojure team)also, I as an admin have no ability to change the default - only users can change their own setting. there is another jira about that.#2020-02-1115:35borkdudemaybe we can migrate to clubhouse, I've heard they built it in clojure, so it must be good#2020-02-1115:38Alex Miller (Clojure team)I'm done migrating issue data for a while :)#2020-02-1115:38borkdudejust start over. if it's important, the issues will pop up again 😛#2020-02-1115:38Alex Miller (Clojure team)it was nice that they added the new issue, with fewer features, and made it the default, and prevented admins from changing that default#2020-02-1115:39Alex Miller (Clojure team)many years ago, I had a boss that literally deleted the issue database with that theory#2020-02-1115:39borkdudehow did it play out?#2020-02-1115:40Alex Miller (Clojure team)not great#2020-02-1115:40borkdudeif users have a stake in the issue that was posted, you could also trust on them to migrate the issue, if they found it important#2020-02-1115:40borkdude(I'm speaking as devil's advocate)#2020-02-1115:40Alex Miller (Clojure team)mostly, they just felt like we didn't care about their problems#2020-02-1115:41Alex Miller (Clojure team)and we had no way to notify them when we did happen to fix stuff they'd logged earlier#2020-02-1115:41borkdudehmm, right, if it's customer issues, it might be different#2020-02-1115:42Alex Miller (Clojure team)there is some happy medium where you mark things as "won't fix" more aggressively than feels comfortable I think#2020-02-1114:01borkdude@souenzzo As a workaround, this seems to work:
$ clj -A:lsp:dev -Srepro -Sdeps '{:aliases {:dev {:extra-paths ["a" "b"]} :lsp {:extra-deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}}}' -Spath | tr ':' '\n'
a
b
src
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar#2020-02-1114:04borkdudeThis one also works:
$ clj -A:dev -Srepro -Sdeps '{:aliases {:dev {:extra-paths ["a" "b"]}} :deps {clj-kondo/lsp-server {:git/url "" :deps/manifest :deps :deps/root "server" :sha "4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5"}}}' -Spath | tr ':' '\n'
a
b
src
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.gitlibs/libs/clj-kondo/lsp-server/4ba4b40dc47c4e11e76f7a5ecebf7e2d40130fe5/server/src
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar#2020-02-1120:13hlolli(at)alexmiller, as a continuation of the bugs I mentioned a bit last night related to nixpkgs. It seems as if this isn't related to lumo, but clojure-tools itself. Since I've been tagged a lot on various tickets on nixpkgs due to lumo, I#2020-02-1120:13hlolliI'd love to find a workaround. (sorry pressed enter too early)
https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/clojure/default.nix#2020-02-1120:14ghadiIf I understand this correctly, it's a name resolution issue#2020-02-1120:17hlolliso the problem is that the current clojure, and newer ones for that matter, will by default call maven when called. While a clojure-tools does provide all the classpath needed to run the clojure executeable. So it's basically bootstrapped as it is released. The way to workaround this, would be to force clojure to not check for updates, and just use the bootstrapped jar on runtime for its own environment, and if the user then later wants to resolve dependencies with another deps.edn, it will in those cases ofc continue it's normal behaviour. The issue here is only the runtime environment of clojure and not for another project.#2020-02-1120:18andy.fingerhutIf your deployed application uses a command like java -classpath your:class:path:here and whatever other options you want, then it should not call Maven stuff, unless your application does.#2020-02-1120:19andy.fingerhutAlex noticed last night that clojure -Scp your:class:path:here might still call Maven and make network connections for that, which he considers a bug, I think.#2020-02-1120:19hlollito the default.nix I posted above, I added
doInstallCheck = true;
installCheckPhase = ''
CLJ_CONFIG=$out CLJ_CACHE=$out/libexec $out/bin/clojure -Spath
'';
this fails the same way, maven not found.#2020-02-1120:20hlolliit can't run itself without internet it seems#2020-02-1120:20andy.fingerhutHave you tried java command instead?#2020-02-1120:20andy.fingerhutmight require changes to other command line options, too, so not necessarily a one-line change.#2020-02-1120:20hlollinot sure it matters because clojure is just a shell for calling java anyway#2020-02-1120:20hlolli#! /nix/store/b85h3rw3y20hyj6s2aqgvlm26a8v61gy-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/rz4aba9s8895b69l3kmx8qb6ir8di2xv-clojure-1.10.1.507/bin:/nix/store/6xxk9hynq5q5nky6wh8l7k4y95y3aksv-rlwrap-0.43/bin:/nix/store/kqr55dy5zjmhklzkkwc85ghf7yxk11qq-openjdk-11.0.6-ga/bin'${PATH:+':'}$PATH
exec -a "$0" "/nix/store/rz4aba9s8895b69l3kmx8qb6ir8di2xv-clojure-1.10.1.507/bin/.clojure-wrapped" "#2020-02-1120:20andy.fingerhutdid you read my message above, completely?#2020-02-1120:21andy.fingerhutthe one that mentioned Alex by name#2020-02-1120:21hlolliahh yes ^#2020-02-1120:22ghadidoes the nixpkg build system intentionally prevent network calls?#2020-02-1120:22ghadi(I know very little about nix)#2020-02-1120:22hlolliyes, it allowes network calls, you just need to specify the endpoints by hand. This is by design ("purely functional" buzzwording)#2020-02-1120:23ghadimakes sense#2020-02-1120:24andy.fingerhutSo it seems you can wait for a bug fix to clojure command, if one is forthcoming, or try java command.#2020-02-1120:24hlollijust thinking outside the box until a fix arrives, perhaps making some sort of .cpcache file to prevent clojure from attempting to call maven#2020-02-1120:25andy.fingerhutSure, if you want to run experiments like that, go for it. If you do not know clojure internals, you are doing black box testing.#2020-02-1120:26hlollihence me asking, I'm also ok with solving this on a slow tempo, I'd just remove lumo temporarily in that case#2020-02-1120:27andy.fingerhutYou could also add https://repo1.maven.org to explicit whitelist for nixpkg, if you have such access, and then rinse and repeat as other unexpected host accesses arose.#2020-02-1120:29hlollithat I can't do, but what I can do, is getting all the jars clojure is attempting to download from nix and put them anywhere visible to clojure. But I just worry if adding -Spath to the clojure wrapper will interfere with normal useage (like two -Spath would be then plausable input parameter to clojure if user provides one and nix one)#2020-02-1120:30hlollithat is an alternative, but then again, all the files clojure needs to run are already downloaded, so I was hoping to do --offline-mode or something similar until going into more unknown terretory#2020-02-1120:31andy.fingerhutIt might work to run an experiment where you run a command like ps axguwww on the deploy box while your failing clojure command is running to see the full command line and options of all processes, and see what command line options are actually being used.#2020-02-1120:32hlollinice, if I find the root of the bug, there's always a possibility of patching (or using forked repo etc..). I'll try this! thanks!#2020-02-1120:32andy.fingerhutI don't know your time tolerance level for experiments 🙂#2020-02-1120:33hlolliI'll write a react app in assembler for fun 😄#2020-02-1120:33ghadihaha#2020-02-1120:38Alex Miller (Clojure team)the fix in clojure is just extending the flag checks around the tool invocation#2020-02-1120:38Alex Miller (Clojure team)(and the equivalent in the windows script)#2020-02-1120:38Alex Miller (Clojure team)if someone wants to make a patch, I will likely be doing a release this week and would be happy to include#2020-02-1120:44Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-150#2020-02-1120:48hlolliperfect, I think this describes it perfectly#2020-02-1121:35hlollijust reading trough the shell script quickly, everything makes total sense, and I like how the checksum is based on arguments and environment. Because it's hard to know if the current environment is "stale", so I'd be reluctant to want to make it less strict. Just a simple
if [[ "$stale" = true && "$describe" = false ]]; then
to
if [[ "$stale" = true && "$describe" = false && "$offline" == false ]]; then
or "$dont_recalculate_classpath" instead of $offline#2020-02-1121:36hlolli> > ...would be enough for my usecase#2020-02-1121:46Alex Miller (Clojure team)don't worry about the staleness check, just add it as a flag that affects the running of the tool#2020-02-1121:47Alex Miller (Clojure team)this line: if [[ "$stale" = true && "$describe" = false ]]; then#2020-02-1121:47Alex Miller (Clojure team)and not -Scp ...#2020-02-1121:48Alex Miller (Clojure team)whatever the bash is for that#2020-02-1121:50hlolliI see, are there any use-cases of people partially wanting to add classpath but otherwise rely on cached (or yet to be cached) classpath?#2020-02-1121:51hlollithere's extra-classpaths for that I guess, so -Scp would be total opt-out of make-classpaths mechanism#2020-02-1121:52Alex Miller (Clojure team)yes, that's it's purpose imo#2020-02-1121:53Alex Miller (Clojure team)so should shut off making a cp#2020-02-1121:53hlollimakes sense to me, with extra-classpaths well documented, shouldn't come as a surprise#2020-02-1122:31hlollibug fixed https://github.com/NixOS/nixpkgs/pull/79868 easy 😛#2020-02-1122:38Alex Miller (Clojure team)are you going to make a patch?#2020-02-1122:39hlolliyes! be happy to#2020-02-1122:46andy.fingerhutWow, just the raw numbers of issues, PRs, and commits on that project are mind-boggling. Sounds like a buzzing hive of activity: https://github.com/NixOS/nixpkgs#2020-02-1122:55Alex Miller (Clojure team)looks just like the homebrew core tap :)#2020-02-1121:48andy.fingerhutWhen investigating which project.clj files are readable via clojure.edn/read, I have noticed a couple of projects that have both deps.edn and project.clj, and their project.clj is not EDN because it has non-EDN executable Clojure that reads the deps.edn file. I am guessing that if there is both a project.clj and deps.edn file in the root directory of a project's source code, then it doesn't matter whether project.clj is EDN or not?#2020-02-1121:51andy.fingerhutI've also noticed @U04V15CAJ that you have joined "`#=` club" for several of your project.clj files. I have seen plenty that have an eval-able Clojure expression in place of a version string, that evaluates to a version string, that do not prefix that expression with #= , but simply have the expression. I haven't verified myself whether that actually works, but might be worth testing to see if it does. Either that or plenty of projects left their project.clj files broken in that way, or it only worked in some versions of Leiningen, etc.#2020-02-1122:05borkdude@andy.fingerhut I have joined that club for some projects but for e.g. this one I changed that to templating later: https://github.com/borkdude/clj-kondo.lsp/blob/master/script/update-project.clj
which is I think the better thing to do. I haven't heard any problems about the read-eval approach tooling-wise though#2020-02-1122:05borkdudeI think I only used the read-eval because I didn't know any better#2020-02-1122:15andy.fingerhutThere is no problem tooling-wise that I am aware of. This is just me noticing things while checking out ~12,000 project.clj files in projects, to see which are clojure.edn/read'able, which not, and an a few cases they are not, looking at why. Alex Miller mentioned the possibility of having deps.edn use clojure.edn/read for project.clj files if a project had no deps.edn, and I'm still doing the data collection....#2020-02-1123:03seancorfieldI've never seen #= in a project.clj but I've seen plenty of ~ to evaluate inline expressions... which wouldn't read as EDN, right?#2020-02-1121:53Alex Miller (Clojure team)deps.edn is preferred#2020-02-1121:53Alex Miller (Clojure team)so if it has that, project.clj won't be used#2020-02-1122:23andy.fingerhutThanks for confirmation. Will use that in my summary of findings, and mention it.#2020-02-1123:07seancorfieldSome projects will have both even if project.clj isn't using machinery that reads deps.edn since some people add the latter in response to folks complaining that their projects can't be used by the CLI as git deps 🙂 So they'll start off compatible in terms of general dependencies but may drift apart later if the maintainer forgets to update deps.edn when they update project.clj (in the same way that maintainers of Contrib projects added project.clj files alongside pom.xml files and then the POM was updated but not the project.clj file!). So it's worth mentioning that they could differ but, hopefully, such projects would be a tiny proportion of the whole...#2020-02-1123:14andy.fingerhutUnderstood they can differ, in the absence of such automated methods. Right now I'm partly documenting/understanding, and also wanting to make the results as accurate as I know how, by counting a project with both deps.edn and project.clj as "good for tools.deps to read project.clj file, in the last resort"#2020-02-1123:18andy.fingerhutSo I am planning to soon calculate which subset of these projects have each of the 16 possible combinations of files in their repository home directory: project.clj deps.edn build.boot pom.xml. Any others of interest to throw into that mix?#2020-02-1123:18andy.fingerhutThen I plan to report stats for, within each of those 2^4=16 groups, which ones have EDN-readable project.clj files.#2020-02-1123:29seancorfieldThose are the only four files I can think of that are interesting. It'll be interesting to see which notionally Clojure projects have none of those four (Gradle-based, maybe?).#2020-02-1200:21Alex Miller (Clojure team)I'm really concerned most about project that have only a project.clj (and no deps.edn or pom.xml)#2020-02-1200:24penryuFirst, wanted to thank @seancorfield for his dot-clojure. 😄
Also wondering if tools.deps has anything similar to leiningen's :injections functionality?#2020-02-1200:25Alex Miller (Clojure team)no#2020-02-1200:26Alex Miller (Clojure team)instead of running a repl, you could launch your own program that started its own repl that did something similar#2020-02-1200:27penryuack. are there any workarounds? something as simple as (require '[clojure.pprint :refer [pp]]) could save a lot of typing over time.#2020-02-1200:28penryuTrue. I'm in the process of converting from emacs/cider to vim/conjure. I'll pursue that route.#2020-02-1200:28penryuThank you!#2020-02-1200:29Alex Miller (Clojure team)like clojure.main effectively just does this:
(repl :init (fn []
(initialize args inits)
(apply require repl-requires)))
#2020-02-1200:30Alex Miller (Clojure team)the initialize part will already be handled by the main wrapper, so don't need that line#2020-02-1200:31Alex Miller (Clojure team)you'd want to supply the :eval hook: #(do (injections) (eval %))#2020-02-1200:32Alex Miller (Clojure team)something like that, the default is just eval#2020-02-1200:33penryu... I'm afraid that's a little beyond my level of knowledge of the repl/clojure.main at the moment, but I'll take some time this week to dig in to how that would work in practice.#2020-02-1200:33Alex Miller (Clojure team)that's all the code#2020-02-1200:33penryuAppreciated, nonetheless!#2020-02-1200:35penryuthe eval hook is defined in deps.edn?#2020-02-1200:40Alex Miller (Clojure team)as a command line:#2020-02-1200:40Alex Miller (Clojure team)clj -e "(require '[clojure.main :as m]) (m/repl :init #(apply require m/repl-requires) :eval (fn [form] (do (require '[clojure.pprint :refer [pp]]) (eval form))))"
#2020-02-1200:40Alex Miller (Clojure team)which you can then fold into an alias if you like#2020-02-1200:43Alex Miller (Clojure team);; deps.edn:
{:aliases {:repl {:main-opts ["-e" "(require,'[clojure.main,:as,m])(m/repl,:init,#(apply,require,m/repl-requires),:eval,(fn,[form],(do,(require,'[clojure.pprint,:refer,[pp]])(eval,form))))"]}}}#2020-02-1200:44Alex Miller (Clojure team)thanks to the Corfield Comma for serving as bash-safe whitespace... :)#2020-02-1200:43Alex Miller (Clojure team)then clj -A:repl#2020-02-1200:44Alex Miller (Clojure team)you could fold that tiny program into a tool that abstracted out the injections rather than hard-coding them and clean this up a bit, but that's the core of it#2020-02-1201:07seancorfieldWait... pp is available directly in the REPL started by clj already...#2020-02-1201:08seancorfield> clj
Clojure 1.10.1
user=> *clojure-version*
{:major 1, :minor 10, :incremental 1, :qualifier nil}
user=> (pp)
{:major 1, :minor 10, :incremental 1, :qualifier nil}
nil
user=>
#2020-02-1201:09seancorfield@penryu What injections do you want in the Clojure REPL, specifically? (given that pp already seems to be referred in by default)#2020-02-1201:14Alex Miller (Clojure team)it's only available in user#2020-02-1201:14Alex Miller (Clojure team)injections makes it available in every ns#2020-02-1201:15Alex Miller (Clojure team)I dropped the above in a blog in case it's helpful for posterity https://insideclojure.org/2020/02/11/custom-repl/#2020-02-1201:20kennyfyi#2020-02-1201:44Alex Miller (Clojure team)yeah, I know :)#2020-02-1201:15Alex Miller (Clojure team)and so I can just point someone to it next time :)#2020-02-1201:17penryuThanks for the explanation, @alexmiller!#2020-02-1201:18penryu@seancorfield might also be something like tools.logging or spyscope (as recommended in a post), though I’m having problems getting either of them to work right now.#2020-02-1201:20seancorfield(I never used :injections in all the years I used Leiningen and I'm not sure not sure I'd want a bunch of symbols auto-inserted into every namespace while I'm working so...)#2020-02-1201:44Alex Miller (Clojure team)well, me neither :)#2020-02-1201:58penryuI just found out about :injections today, but I'm getting a feel for the [mis]feature parity of deps vs lein.#2020-02-1202:02Alex Miller (Clojure team)the philosophy of clj is much more to give you the tools to solve your own problems, not to exhaustively provide every possible feature#2020-02-1203:14penryuThanks again for your response. I know leiningen has accumulated a lot of feature-cruft, and I appreciate how clj factors that out. Especially, thank you for the workaround in your blog post!#2020-02-1201:26penryuYes, it would need to be sparse in the extreme. But certain things (like pp or #spy) might be reasonable. And I intended to bury it behind an alias (`-A:spy:rebel` or just fold it into rebel) anyway, never in a clean clj session.#2020-02-1202:05ghadiWhat is the reason the Corfield comma is necessary in the first place?#2020-02-1202:24seancorfieldRound-tripping to text files and back into bash (and then into Clojure as command-line arguments).#2020-02-1202:24seancorfield(shell escaping is... facepalm )#2020-02-1202:25ghadiis there a technical reason it can not be done with full fidelity, or has no one dived into the GNU Bash Info page?#2020-02-1202:27ghadior whatever Windows uses for bash 😂#2020-02-1202:37seancorfield@ghadi The "official" clj-on-Windows uses Powershell -- which has different quoting rules from bash (you end up triple-quoting some things -- ugh!)#2020-02-1202:27ghadiwhich I think is now... bash#2020-02-1202:27andy.fingerhutI suspect it could be done with full fidelity, but the prevalence of multiple different versions of bash on Mac OS versus different flavors of Linux adds a level of difficulty to the problem. Also not just spaces, but single and double quotes, dollar signs, braces, ... It can make a grown person weep.#2020-02-1202:28andy.fingerhutNot sure it is a hill I want to even try to climb, nor die on.#2020-02-1202:29seancorfield@ghadi I'm sure Alex would welcome patches for it 🙂#2020-02-1202:47Alex Miller (Clojure team)There are patches for it. They are all scary #2020-02-1202:30seancorfield"It's hard!" is the technical reason I've heard from several people.#2020-02-1202:30andy.fingerhutI suspect it would be difficult to maintain the shell scripts, too, and lead to copious notes on what works and what doesn't across the platforms.#2020-02-1202:30ghadiI'm thinking of tools that accept main arguments (like Docker) usually accept a canonical form: an array of strings (which is sort of punting)#2020-02-1202:30seancorfieldThe complexity comes from clojure storing all that stuff in text files in .cpcache and then reading it back on subsequent runs.#2020-02-1202:31andy.fingerhutI know reasonable people might disagree with this statement, but bash is just barely a programming language.#2020-02-1202:33andy.fingerhutI will end my rant-ish statements on that topic for the evening 🙂#2020-02-1202:38andy.fingerhutI lied. As a starting point for consideration, bash --version on the penultimate macOS version (Mojave 10.14) gives this output:
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.
#2020-02-1202:39andy.fingerhutOn Ubuntu 18.04 Linux:
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
#2020-02-1202:39andy.fingerhutThere's 9 years of variation possible right there.#2020-02-1218:44slipsetAlso, Catalina wants you over on zsh because the more modern bash have a license that doesn’t please Apple#2020-02-1202:49Alex Miller (Clojure team)I actually have a recently raised alternative to the spaces issue that avoids all the escaping that I’m exploring. #2020-02-1202:50Alex Miller (Clojure team)Raised by rich and Stu for other purposes but happens to also cover this#2020-02-1206:35seancorfield@deleted-user Then you should write up a patch for clojure and/or tools.deps.alpha that fixes these problems 🙂#2020-02-1207:37sogaiucuz this is ridiculous: https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#escaping-quotes#2020-02-1207:38sogaiubut actually, may be that's a diff issue?#2020-02-1207:40andy.fingerhutThat shows differences in command line parsing for different command shells on Windows.#2020-02-1207:41andy.fingerhutThere are differences between command line parsing for different command shells on Linux/Unix/macOS, too, but bash is a pretty commonly used one, and what clj/clojure scripts are currently written in.#2020-02-1207:42andy.fingerhutThere are things involving single quotes, double quotes, white space, etc. that can be tricky to get correct even in bash, without considering the flavors of Windows command shells described at the link you gave.#2020-02-1207:44sogaiui guess if there ever is a cross-platform shell that gets adopted may be this issue will be less of a problem#2020-02-1207:45andy.fingerhutbash exists on Windows in fairly easily obtainable form. That makes things more common with other platforms, if you use bash on Windows, but it doesn't eliminate the existing difficulties with bash#2020-02-1207:47sogaiui use bash on windows via the git for windows installation -- but i hesitate to use it as a shell all of the time.
i think there's a chance that powershell core might at some point be a nicer alternative.#2020-02-1208:02andy.fingerhutOn Linux as well? And macOS? That seems unlikely to me, but stranger things have happened.#2020-02-1208:04sogaiufirst paragraph here seems to suggest some level of support for the 3: https://github.com/PowerShell/PowerShell#-powershell
i've not tried it on macos, but have on linux#2020-02-1208:06sogaiui'm not a power bash user but have dealt on and off with shell scripts since the 90s, and though i've historically not been an ms fan, the powershell stuff (at least in core) seemed way more sane to me (though i've used it far less than bash, sh, zsh, etc.)#2020-02-1208:07sogaiuit's easier to do so when you start over ofc#2020-02-1208:50penryuAt what point is it worth reading code from a file instead of a command line argument? It seems tools.deps has drawn that line in the sand.#2020-02-1208:55andy.fingerhutI am not sure what you mean by the line in the send. The clojure command lets you give Clojure expressions on the command line after the --eval option (in the init-opts part of the command line args), and/or a file name to evaluate code from named path in the help output by clojure -h#2020-02-1208:56andy.fingerhutBy the line in the sand, do you mean it lets you pick what you want?#2020-02-1218:09seancorfieldCaching. clojure does two passes -- and invokes the JVM twice -- if the classpath/main/JVM options are not cached (in a file). It does just one pass with the cache in place. That's a big deal.#2020-02-1214:46sparkofreasonExpanded version of my earlier question about javac: suppose I want to get in between the resolve-deps and make-classpath steps before running clj, say to do some other processing on files fetched from git, add resulting artifacts to the classpath. Are there any examples of tools (or whatever) doing such?#2020-02-1215:11delaguardoYou can try https://github.com/EwenG/badigeon
I’m not sure this is exact solution for your specific task, but could provide some missing tool’s parts#2020-02-1215:13sparkofreasonYes, assuming nothing else existed, that was going to be my starting point. Very useful stuff.#2020-02-1214:55dominicmNot that I'm aware of, that's a new, separate tool.#2020-02-1214:55dominicmHowever, I think that would be really useful for the javac case.#2020-02-1214:56dominicmOther tools are generally shell scripts which you run first, but that reduces the smooth integration with cider, etc.#2020-02-1218:40andy.fingerhutOK, I do not have stats weighted by number of http://Clojars.org project downloads right now, but I do have some stats showing that out of 11,736 Github projects I successfully cloned, and that is the subset of them that have a project.clj file but no deps.edn file in their root directory, 91.8% of those project.clj files were readable using clojure.edn/read , and had their defproject form first, and a reasonable-looking value for the :dependencies key (or no dependencies key, which is completely correct and fairly common). More data and steps I used to collect it here: https://github.com/jafingerhut/haironfire#2020-02-1218:42Alex Miller (Clojure team)cool (and nice repo name :)#2020-02-1218:42andy.fingerhutThe most common reason that a project.clj file threw an exception while attempting to use clojure.edn/read to read its contents: regex literals are not valid EDN. Those are reasonably common in project.clj files, for things like jar inclusions/exclusions, or other kinds of file name selection sets.#2020-02-1218:43Alex Miller (Clojure team)that's kind of interesting#2020-02-1218:44Alex Miller (Clojure team)I wonder if you could hack the edn reader to read those into tagged literals or something#2020-02-1218:44Alex Miller (Clojure team)how much additional % does that cover?#2020-02-1218:46andy.fingerhutGood question, which I thought of attempting, but haven't yet. That was the first thing in the project.clj file that caused clojure.edn/read to throw, and there could be others after that, which would only be straightforward to calculate with such a modified version of clojure.edn/read#2020-02-1218:47Alex Miller (Clojure team)such a thing could possibly even make its way into clojure's real edn reader#2020-02-1218:49ghadiregexes aren't portable though#2020-02-1218:49Alex Miller (Clojure team)I didn't say make it into edn#2020-02-1218:50Alex Miller (Clojure team)there are lots of things the clojure edn reader does beyond the edn spec#2020-02-1218:50ghadihmmm.jpg#2020-02-1218:50Alex Miller (Clojure team)and I'm not suggesting reading it into java pattern, but into a tagged literal#2020-02-1218:50ghadisame thing for the var dispatch I was talking to you about#2020-02-1218:50Alex Miller (Clojure team)#regex or whatever#2020-02-1218:51Alex Miller (Clojure team)less sure on vars if that makes sense as you're then connecting to code#2020-02-1218:53ghadi(tangent, but I want to have the choice of whether to connect to code or read as tagged)#2020-02-1219:00dominicmEdamame can read regex I think#2020-02-1219:04andy.fingerhutBased on the numbers I have seen, if the project.clj files that threw exception due to regex are edn-readable in all other ways except the regex literals, I think the most the numbers above could improve would be from current 91.8% up to close to 95% or 96%, but that would be noticeable, I think.#2020-02-1219:06andy.fingerhutSounds like a straightforward thing to hack up locally on my system. I will let you know if I try it out. If anyone hacks up a version of clojure.edn/read before I do, let me know.#2020-02-1219:08ghadihttps://github.com/clojure/tools.reader/blob/master/src/main/clojure/clojure/tools/reader/edn.clj#L345-L354#2020-02-1219:08ghadi@andy.fingerhut you could switch to tools.reader and monkey patch that to read regex + vars#2020-02-1219:09andy.fingerhutCool, that may be the quickest way.#2020-02-1219:09ghadinaughty, but straightforward#2020-02-1219:09andy.fingerhutMaybe I should take the opportunity to see if clojure.edn/read and tools.reader edn reader behave differently on any of these 11,736 files 🙂#2020-02-1221:04sogaiuon a tangential note perhaps, regarding regexes -- there's the regal thing that expresses regexes using plain clojure data: https://github.com/lambdaisland/regal#2020-02-1221:06Alex Miller (Clojure team)this is about existing data in project.clj files, so I don't think that helps anything#2020-02-1221:07sogaiuyou're talking about converting regexes as well as whether the result is valid edn, no?#2020-02-1221:08sogaiuanyway, for future cases, it could be useful for folks to express their regex using regal and having a file that is more likely to be edn?#2020-02-1221:23andy.fingerhutThis is experimental exploration here, wondering how we might use a secure safe reader, either existing, or perhaps slightly tweaked, to read existing project.clj files.#2020-02-1221:24andy.fingerhutI doubt we could ever convince hundreds of people to update the project.clj files to use a new regex representation, for something that already works for them today.#2020-02-1221:57sogaiusure -- just musing about future possibilities. sorry for the noise.#2020-02-1222:03andy.fingerhutNo worries. Just hoping the context makes it clear why we're talking like we do 🙂#2020-02-1222:43sogaiuthanks for your ongoing patience and explanatory efforts 🙂#2020-02-1221:47andy.fingerhutFraction of the 11,736 project.clj files that become 'good' in my experiments (readable using clojure.edn/read without throwing exceptions, defproject form is first, value of dependencies key looks reasonable) goes up from 91.8% to 95.7% with a hacked up version of EdnReader class that can read regex literals. Details here: https://github.com/jafingerhut/haironfire/blob/master/README-extended-edn-reader.md#2020-02-1221:52andy.fingerhutAs mentioned in #clojure-dev channel discussion, my 'good' evaluation here currently has no checking for whether the dependencies list is actually the one used by the project, e.g. no checking for or evaluating the effects of Leiningen plugins, e.g. Lein-modules. There are also approximately another 1,000 git repos associated with Clojars.arg artifacts that have no project.clj nor deps.edn files in their home directory. Most of those probably have some kind of hierarchical structure to their files, with one or more independent Leiningen projects in the same source repo. I haven't tried analyzing those.#2020-02-1222:13deletedI mean, beyond the regex literals#2020-02-1222:18andy.fingerhutThe results I linked to show number of files that give exception messages of various kinds. The next most common non-EDN things, or at least the ones that occur first in the project.clj files, are mentioned explicitly in my commentary text here: https://github.com/jafingerhut/haironfire/blob/master/README-extended-edn-reader.md#2020-02-1222:18andy.fingerhutShort answer appears to be "in-line eval'able code inside the defproject form"#2020-02-1222:19andy.fingerhutOne could pretty easily extend an EDN-style reader to read those things, too, but I suspect the return-on-time-invested is getting fairly low.#2020-02-1222:56borkdude> One could pretty easily extend an EDN-style reader to read those things, too, but I suspect the return-on-time-invested is getting fairly low.
@andy.fingerhut That's exactly what edamame already is.#2020-02-1223:00borkdudee.g.:
(parse-string "#=(+ 1 2 3)" {:read-eval true})
;;=> (read-eval (+ 1 2 3))#2020-02-1223:00borkdudebut :read-eval can also be a function that transforms the value in any way you want#2020-02-1223:01andy.fingerhutUnderstood. Part of the context of this discussion, I believe, is what dependencies outside of itself and Clojure that Alex is willing to take on for tools.deps.#2020-02-1223:02andy.fingerhutI do not know his detailed thoughts on the matter, other than some nuanced approximation of "as few as possible".#2020-02-1223:03borkdudeYeah, I understand. I guess you could also have a "forgiving" EDN parser that just skips over what it cannot read. If that leaves a valid :dependencies vector, that's probably all that counts.#2020-02-1223:05andy.fingerhutI suspect the harder part of getting a valid dependencies sequence (one thing I learned is Leiningen accepts, and a few project.clj files actually use, lists or even a map for the value of :dependencies ), is the existence of git repos with a project.clj file in the root directory, and then several more in subdirectories, with interactions between their contents, e.g. via Lein-modules plugin or other mechanisms.#2020-02-1223:05borkdudeso what is the summary of your research, what's the percentage of EDN-readable project.cljs?#2020-02-1223:06andy.fingerhut91.8% of 11,736 project.clj files that are in the root directory of git repos, and those were selected because they do not also have a deps.edn file in the root dir, are clojure.edn/read readable in Clojure 1.10.1. If you extended clojure.edn/read to not throw an exception on encountering a regex literal, it goes up to 95.7%#2020-02-1223:08andy.fingerhutBut as I mentioned two comments ago, the percentage that are EDN-readable isn't all that one cares about if one wants to get correct dependencies from a Leiningen source repo.#2020-02-1223:08borkdudevery cool. I assume you communicated this summary to Alex as well?#2020-02-1223:08andy.fingerhutEarlier today in this same channel.#2020-02-1223:08borkdudetrue, but maybe those multi-project.clj repos are also in the <10% range?#2020-02-1223:08andy.fingerhutI haven't tried to determine that yet.#2020-02-1223:09borkdudeI haven't had time to read slack today. Nice progress!#2020-02-1223:10andy.fingerhutSure, it's been fun hacking away. Also makes me realize that the actual question is harder to answer 🙂#2020-02-1305:45rbergerDoes seancorfield/clj-new have a way to specify copying a binary file from the template source dir to the target dir? The doc kind of says that the render function without a data argument would do that, but it looks like it treats the file as text with linefeeds and corrupts the binary data. Is there a way to do this that still fits into the ->files body?#2020-02-1317:32seancorfield@rberger That sounds like a nice enhancement! Feel free to open a GitHub issue. Do you know if either lein-new or boot-new support such a thing? (`boot-new` was originally based on lein-new and clj-new was forked from boot-new but I haven't tracked their progress much since).#2020-02-1318:01rbergerI haven't used the others {lein,boot}-new I'll open a github issue. I guess in the short term I could create another renderer for binary files that would do the copy#2020-02-1318:03seancorfieldFeel free to provide a PR with that enhancement too.#2020-02-1318:08seancorfield@rberger Just looked in the code and there's a raw-resourcer which I suspect can be used to copy binary files. I can't say I remember trying that but it's worth a shot.#2020-02-1318:09rbergerIt wasn't clear to me when I looked at that, but will look at it today. Thanks!#2020-02-1318:14seancorfieldhttps://github.com/search?q=raw-resourcer&type=Code -- looks like quite a few Leiningen templates use that so that should point you in the right direction#2020-02-1321:21rbergerLooks like the raw-resourcer does the trick! No need for any changes in clj-new. Just a couple of lines in my template code (https://github.com/omnyway-labs/chromex-shadow-template)
By the way for some reason https://cljdoc.org/d/seancorfield/clj-new is having problems building the docs for the most recent couple of releases of clj-new.#2020-02-1321:25seancorfieldThanks for the heads up. Weird error Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.7.5#2020-02-1321:26seancorfieldHmm, it's leftover in the pom.xml by accident 😞#2020-02-1321:41seancorfieldI've pushed 0.9.0 with that fix but the doc system build process is broken right now (by changes to Clojars CDN setup). Once that's fixed, I'll rebuild the clj-new docs.#2020-02-1402:01rbergerThanks!#2020-02-1313:25Andreas S.Hello I try to run clj on windows in cygwin but I get:#2020-02-1313:25Andreas S.$ clojure
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main#2020-02-1313:25Andreas S.Any ideas?#2020-02-1313:33delaguardotry clojure -Spath you should get a string with all the jars including clojure-%version%.jar then go check if this jar is actualy available#2020-02-1313:36Andreas S.If I run "clojure -Spath" I get the same error#2020-02-1313:36borkdude@andreas.scheinert We had someone here a while ago who has an older version of clojure on his path. What do you see when you type clojure -h#2020-02-1313:37Andreas S.I have a file here: /usr/local/lib/clojure/libexec ==> clojure-tools-1.10.1.507.jar*#2020-02-1313:38Andreas S.borkdude here is version "clojure -h" print out#2020-02-1313:38Andreas S.Version: 1.10.1.507#2020-02-1313:38borkdudethat's good then#2020-02-1313:40Andreas S.I know its a bit side stepping but, ok I took the effort and installed it on powershell#2020-02-1313:40Andreas S.clj and clojure give me now a REPL#2020-02-1313:40Andreas S.but#2020-02-1313:41Andreas S.how do I run the deps code from here in powershell? https://github.com/clj-python/clj-template#2020-02-1313:41Andreas S.clj -Sdeps '{:deps {seancorfield/clj-new {:mvn/version "0.8.6"}}}' \
-m clj-new.create \
\
appcompany.funapp
#2020-02-1313:42Andreas S.I figured I remove all the \ and create a long single line of command, but then that still gives me:#2020-02-1313:42Andreas S.Error while parsing option "--config-data {:deps {seancorfield/clj-new {:mvn/version 0.8.6}}}": java.lang.NumberFormatException: Invalid number: 0.8.6#2020-02-1317:32seancorfield@rberger That sounds like a nice enhancement! Feel free to open a GitHub issue. Do you know if either lein-new or boot-new support such a thing? (`boot-new` was originally based on lein-new and clj-new was forked from boot-new but I haven't tracked their progress much since).#2020-02-1317:36dominicmI notice that resolve-deps doesn't mention that it will be concurrent even if :threads isn't provided. Should it?#2020-02-1317:36seancorfieldAh, follow this guide for Windows https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#escaping-quotes#2020-02-1317:40seancorfieldEven following that Windows guide didn't work for me 😞#2020-02-1317:41dominicmmake-classpath-roots mentions the system separator, but it doesn't really make sense in the context.#2020-02-1317:42seancorfield@andreas.scheinert This worked for me:
> clj -Sdeps '{:deps {seancorfield/clj-new {:mvn/version ""0.8.6""}}}' -m clj-new.create appcompany.funapp
So, double "" inside '#2020-02-1317:47seancorfield(that said, even with Python 3.7 installed from the Windows Store, I can't actually get the new project to run but that should be taken up in #libpython-clj at this point)#2020-02-1411:42Andreas S.Thank you sean for replying! Yes the quoting did the trick but I am stuck kat the same point as you I couldn't get windows python to work with libpython-clj#2020-02-1320:33samFor private git repo deps on Windows, is only Pageant supported as the ssh agent for authentication or can OpenSSH also work?#2020-02-1321:03Alex Miller (Clojure team)try it and see?#2020-02-1321:03Alex Miller (Clojure team)I know that pageant works, don't know about openssh#2020-02-1321:13sam@alexmiller I have been trying. It seems to be attempting to use OpenSSH. I can tell because the specific error message I get changes depending on whether github is added to .ssh/known_hosts. However, even though I can use git on Powershell to clone a private git repo, I get an error when trying to run clj with a private github dep. The lowest part of the stack trace is just:
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:126)
... 45 more
which doesn’t give me a ton to go on. It’s been really painful to debug.#2020-02-1321:13Alex Miller (Clojure team)well I think I would say, no it doesn't work then#2020-02-1321:14samIt’s been finicky to get it working even with Pageant. So I’m never sure whether something just doesn’t work, or if I’m just doing something slightly wrong#2020-02-1321:14Alex Miller (Clojure team)ssh is generally tricky, particularly with access to the wide world of credential providers#2020-02-1321:16Alex Miller (Clojure team)there are some options (including shelling out to just use git) which we have explored. newer versions of jgit use apache mina instead of jsch. I have spent some time working on evaluating these alternatives but haven't had time to really get to the end of that process#2020-02-1321:16Alex Miller (Clojure team)another option would be to clone, then use as a local repo instead#2020-02-1321:17Alex Miller (Clojure team)or to clone, build and install a local artifact, then use that#2020-02-1321:20samOkay, thanks. One other question for you. I’ve resorted to trying OpenSSH on one of our dev’s windows machines only because we couldn’t get deps working with Pageant on that machine. I have no idea why it’s not working because we’ve been able to get it working on other Windows machines. Based on your experience, do you have any idea of any “gotchas” we might be running into? The error message is the same as above.#2020-02-1321:21samIf not, that’s okay, I know that’s a very vague question. Just wondering if anything comes to mind of the top of your head#2020-02-1321:22Alex Miller (Clojure team)I don't unfortunately. the jsch stuff is a bit flaky#2020-02-1321:26samOkay, well thanks anyway. Btw, just wanted to say I was at StrangeLoop in September and really enjoyed it. Thanks for doing that.#2020-02-1321:27Alex Miller (Clojure team)great! thx#2020-02-1321:24ghadiif you're using any ssh agent, make sure the SSH_AUTH_SOCK variable is set so that Clojure (via JSch) knows where the agent is#2020-02-1321:27samcool, thank you, I’ll look into that.#2020-02-1322:01samAre you sure this is applicable for Windows? I can’t find much information about it in the context of Windows, and Clojure seems to work fine on my windows machine using Pageant as the ssh-agent without this env variable being set.#2020-02-1322:03ghadissh clients discover where an agent is running by consulting that environment variable#2020-02-1322:04ghadithat's what jsch does, too.... sorry Windows just adds a lot of extra wildcards because of all the various linux-y environments#2020-02-1322:04ghadipaging @U04V70XH6#2020-02-1322:05ghadijust in case he knows#2020-02-1322:11seancorfieldI know nothing about ssh on windows, sorry #2020-02-1322:12seancorfieldI only ever use ssh stuff via Ubuntu / WSL on windows 😁#2020-02-1322:15samThanks anyway!#2020-02-1400:30andy.fingerhutI've added one section to the README for my haironfire repo here: https://github.com/jafingerhut/haironfire#some-leiningen-projects-can-depend-upon-multiple-files-andor-tools-to-determine-the-actual-dependencies. merely mentioning some challenges of calculating accurate dependencies for Leiningen projects, which I have not attempted to quantify in my work how often those tricky cases even arise in the projects I've analyzed. I don't have any plans to try to analyze those things, but others are certainly welcome to if they are interested, using any or all of the code in that repo if it helps. Perhaps the biggest hole in the 91.8% and 95.6% numbers I reported for EDN-readable project.clj files are: (a) Those were for repos with a project.clj file in the home directory. Outside of the 100% number is about another 10% of repos that have no project.clj nor deps.edn file in the home directory, many of which probably contain project.clj files in subdirectories that I haven't attempted to find or read. And (b) the difficulties in determining dependencies even after you read a single project.clj file, if there are tools like Lein-modules or similar things in use, again which I haven't tried to determine which projects might use such things.#2020-02-1400:32andy.fingerhutOr in summary, I will now more fully understand if the decision is made not to have tools.deps read project.clj files 🙂#2020-02-1407:33dominicmSomething just surprised me while I was writing up a repro for 132, this dependency tree resolves without error, because somehow B and B compare as the same version, but I don't understand how because there's no compare-version for :local :local (yet, that's what my patch does).
deps.edn
+--A {:local/root "/A"}
| |
| +- B {:local/root "/B"}
+--C {:local/root "/C"}
|
+- B {:local/root "/B"}
It doesn't when there's a git in the way for A/C. I think the cause is https://github.com/clojure/tools.deps.alpha/blob/1feb495adfc5c72c9c5fc9a64917a6bab2203c1a/src/main/clojure/clojure/tools/deps/alpha.clj#L149 not using (= 0 (compare-version a c)) .#2020-02-1407:53dominicmNothing to do with git, this equivalence comparison just already exists. So my changes don't help. I could have sworn I tested it...#2020-02-1407:54dominicmOh, I did. But I was being very, very stupid (= (:local/root coord-x) (:local/root coord-x)) comparing x to x 🙂#2020-02-1418:42dominicmI wrote a project to visualize the relationship between your project and it's dependencies. It currently relies on the deps.edn format in order to function. Here's the project: https://github.com/SevereOverfl0w/vizns#2020-02-1419:22andy.fingerhutLooks like a useful thing to add somewhere on this wiki page, if it is not already there: https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-02-1419:33borkdudedone#2020-02-1419:54dominicmThank you 😚#2020-02-1711:33teodorluThis is really cool! Fantastic way to confirm that your code is all spaghetti after all.
Some quick notes on usability:
• I first naïvely ran it in a folder without a deps.edn file, and spent some time figuring out why it didn't work.
• Then I reran it in an actual project, only to find out I was making it crash because I hadn't supplied any main args
• Then I figured I needed to sudo apt install graphviz, because of the dependency on dot.
Then all was fine!#2020-02-1712:21dominicmI don't have the headspace to pick those up yet, but I'd welcome a PR if you have time. Thanks for the feedback, I appreciate it!#2020-02-1712:26teodorluI created https://github.com/SevereOverfl0w/vizns/issues/4 🙂#2020-02-1420:14Alex Miller (Clojure team)@dominicm the readme example has tools.deps.graph for the main class#2020-02-1420:36dominicmD'oh, used that as inspiration. I thought I'd got it all, will fix, thanks.#2020-02-1420:16Alex Miller (Clojure team)I assume that should be io.dominic.vizns.core#2020-02-1420:18Alex Miller (Clojure team)also, it looks like you happened to rely on make-classpath-roots which I added in tda 0.8.666 (which I never announced) then made public in the release earlier today#2020-02-1420:18Alex Miller (Clojure team)I didn't expect anyone would make use of it in interim :)#2020-02-1420:19Alex Miller (Clojure team)seems like you could have just used make-classpath though?#2020-02-1420:49dominicmI could have split the string, yes. It was nice to think of all the saved cycles of not doing that.#2020-02-1420:52Alex Miller (Clojure team)I sent you a pr#2020-02-1421:00dominicmOh, that's a better solution. Thanks for the PRs, makes it mergeable from my phone. The final one won't merge at the moment, but I can resolve from a keyboard :)#2020-02-1420:20Alex Miller (Clojure team)there's some ongoing discussion between rich and I in that area about structured classpath data and I'm not ready to commit to anything there yet#2020-02-1420:21borkdudemaybe the tool should just deal with a classpath as input so users can decide which tool they use to create it (`clojure -Spath`, lein classpath, etc)?#2020-02-1420:50dominicmI want to do that, but then I have to identify classpath entries back to their library name.#2020-02-1420:51dominicmI'd need something slightly different to a classpath. Haven't figured out what that looks like yet from an agnostic point of view.#2020-02-1420:55dominicmGit dependencies are pretty difficult to name.#2020-02-1421:04borkduderight!#2020-02-1420:21borkdude(I haven't looked at any details so I don't know if that's going to make sense)#2020-02-1420:32borkdudefor the project.clj git dep problem, one maybe silly but effective solution could be to let tools.deps clone the project, then let it invoke lein classpath on that project, so lein can deal with itself, and write that classpath to a file which is cached, so next time tools.deps wants to know the classpath that that dep is going to produce, it doesn't have to invoke lein again#2020-02-1420:33borkdudehaven't really thought this through, feel free to dismiss it. have a great weekend. 🍻#2020-02-1420:35Alex Miller (Clojure team)I think it's interesting to consider a dep type that could provide it's deps explicitly like that#2020-02-1420:35Alex Miller (Clojure team)hadn't thought about it before#2020-02-1420:48andy.fingerhutThat still requires eval-level trust in projects you depend upon, at dep resolution time, yes? I do not know if it is any kind of clear bright line to avoid that -- just asking to confirm that the proposed idea still does do eval of project.clj files, even if you never require their code.#2020-02-1421:02borkdudetrue, it will eval. it could potentially be a solution for lein projects with java sources as well btw.#2020-02-1421:04borkdudeI guess the user of a git dep can check the project at given sha if it's deemed safe#2020-02-1421:06borkdudeRunning library code from the wild is going to be eval'ed anyway, so you should in theory always be checking the code of a dep before you run it?#2020-02-1421:15andy.fingerhutIt would be safest, yes. There is a lot of web of trust kinds of things going on, for sure, and the common practice that if something is discovered bad, it will be publicized. That all works with your idea, too, still. I'm only perhaps nit-picking that deps resolution time is another eval-time opening this creates, over and above require'ing the code and running it. To take advantage of it, a malicious party still needs to become trusted, i.e. depended upon by other projects, so in that sense it isn't a new opportunity for attack.#2020-02-1509:38penryuI'm used to seeing maven dep versions in deps.edn as {:mvn/version "0.4.490"}}, but I've found one that specifies deps with #:mvn{:version "0.4.490"}. Looks like a reader macro, but I haven't seen it before.#2020-02-1509:39borkdudethat's called a namespaced map#2020-02-1509:40penryuit looks like it's equivalent to the other syntax; are they interchangeable?#2020-02-1509:40borkdudesee "Map namespace syntax" here: https://clojure.org/reference/reader#_map_namespace_syntax
it's supported since 1.9 and yes, they are interchangeable#2020-02-1509:41penryuAh, you can use it to factor out a common namespace in map keys?#2020-02-1509:43penryuThat's exactly what I was missing. Thanks, @borkdude.#2020-02-1517:08borkdudesince plk is a kind of port of the clojure bash script and is listed here: https://github.com/clojure/tools.deps.alpha/wiki/Tools#runtimes
would it make sense to add deps.clj there as a port of the clojure bash script to clojure itself?#2020-02-1517:09Alex Miller (Clojure team)Yep#2020-02-1517:10borkdudedone#2020-02-1517:11borkdudeI was just informed of this project: https://github.com/benedekfazekas/morpheus#2020-02-1517:11borkdudeMight make sense to add here: https://github.com/clojure/tools.deps.alpha/wiki/Tools#deps-management, since that also lists other viz tools (cc @benedek)#2020-02-1517:13benedekdone#2020-02-1611:07kwladyka~/L/M/c/s/P/l/pdf-converter clj -Aoutdated
| Dependency | Current | Latest |
|---------------------------+----------+----------|
| org.clojure/clojure | 1.10.0 | 1.10.1 |
| org.clojure/clojurescript | 1.10.520 | 1.10.597 |
! ~/L/M/c/s/P/l/pdf-converter clj -Aoutdated -u
Updating: deps.edn
org.clojure/clojure {:mvn/version "1.10.0"} -> {:mvn/version "1.10.1"}
org.clojure/test.check {:mvn/version "0.9.0"} -> {:mvn/version "0.10.0"}
com.cognitect/test-runner {:sha "028a6d41ac9ac5d5c405dfc38e4da6b4cc1255d5"} -> {:sha "f7ef16dc3b8332b0d77bc0274578ad5270fbfedd"}
~/L/M/c/s/P/l/pdf-converter clj -Aoutdated
| Dependency | Current | Latest |
|---------------------------+----------+----------|
| org.clojure/clojurescript | 1.10.520 | 1.10.597 |
~/L/M/c/s/P/l/pdf-converter #2020-02-1611:08kwladykaHow https://github.com/Olical/depot work?#2020-02-1611:08kwladykaWhat is the logic here?#2020-02-1611:08kwladyka{:paths ["resources" "src"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}}
:aliases {:test {:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "0.10.0"}
me.raynes/fs {:mvn/version "1.4.6"}}}}
:runner {:extra-deps {com.cognitect/test-runner
{:git/url ""
:sha "f7ef16dc3b8332b0d77bc0274578ad5270fbfedd"}}
:main-opts ["-m" "cognitect.test-runner"
"-d" "test"]}}
deps.edn#2020-02-1611:16kwladykait is showing something different to update and update something different#2020-02-1611:16kwladykaand what cljs doing here?#2020-02-1611:37dominicmMaybe the first one is looking at transitive dependencies when it shouldn't be?#2020-02-1611:38kwladykaWhat do you use for updates? I found there is also vulcan#2020-02-1611:46dominicmI don't use anything :)#2020-02-1611:49borkdude@kwladyka I'm not using it myself, but this hacky script also works: https://github.com/borkdude/babashka/#list-outdated-maven-dependencies#2020-02-1611:51borkdude(for maven deps)#2020-02-1618:17seancorfieldI would never run a tool that auto-updated my deps. I used Depot for a while but the 2.x version doesn't work with our setup (because it processes deps files directly instead of relying on t.d.a. to process all the aliases and override etc via a non-user deps.edn specified via CLJ_CONFIG).#2020-02-1618:19seancorfieldLike @borkdude I created a script instead -- it runs clojure -Stree and massages the output, then creates a deps.edn with all the :mvn/version deps changed to "RELEASE" and re-run clojure -Stree against that, and then compare the top N levels of those trees.#2020-02-1710:00Michaël SalihiThis script is available on your Github ?#2020-02-1710:01borkdude@UFBL6R4P3 There is one variation available here:
https://github.com/borkdude/babashka/#list-outdated-maven-dependencies#2020-02-1710:02borkdudeI made it based on the idea of Sean#2020-02-1710:17Michaël Salihi@borkdudeNice script, to use and also for learning beautiful Clojure scripting.
Babashka rocks! Thank you!#2020-02-1710:23borkdudeThanks 😊#2020-02-1717:56seancorfield@UFBL6R4P3 Our script at work is somewhat proprietary and heavily tied to how our monorepo works (30 subprojects, each with their own deps.edn and with an overall deps.edn that is used instead of the ~/.clojure user-level one, via the CLJ_CONFIG environment variable).#2020-02-1718:45Michaël Salihi@U04V70XH6 I understand. Thx for the information, I don't know CLJ_CONFIG.
New thing in my baggage!#2020-02-1707:13benedeka tool that creates a PR automatically would be nice i think. added benefit is all the tests run if you have some autobuild set up. such tools exist for other ecosystems i think#2020-02-1910:08souenzzoFrom
https://github.com/portkey-cloud/cfn-yaml/issues/1
How I supposed to import a project withtout a clear ArtifactID?
Consider:
I publish my/lib {:rev :v1} that requires cfn-yaml {:git/url "" :sha v1}
Someone else import my/lib {:rev :v1} portkey-cloud/cfn-yaml {:git/url "" :sha v2}
It will result in git deps.#2020-02-1915:03Alex Miller (Clojure team)it kind of is with the advice of
github-yourname/time-lib
#2020-02-1914:15dominicmIf repos were the canonical source of what id they represented, that would solve this. It would also make the deps.edn format very different.#2020-02-1914:18Alex Miller (Clojure team)it would also break forking use cases etc#2020-02-1914:19Alex Miller (Clojure team)it was long considered#2020-02-1915:02souenzzoCan I explicit this good practice in here?
https://www.clojure.org/guides/deps_and_cli#_using_git_libraries
Something like
Is a good practice explicit a ID in documentation/README case ou pretend to distribute just via git deps.#2020-02-2108:50teodorluHey! 👋
Any idea on why this doesn't work?
$ clj -e '(doc ns-map)'
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: doc in this context
Full report at:
/tmp/clojure-135646063998456412.edn
#2020-02-2108:58dominicmYou need to require clojure.repl, doc is defined there, not in core#2020-02-2108:59dominicmuser.clj requires clojure.repl for you#2020-02-2113:54Alex Miller (Clojure team)No, Clojure.main does this in the repl setup#2020-02-2109:00vlaaadI would guess it would be more correct to say that starting repl requires clojure.repl for you#2020-02-2109:03vlaaadI would guess even more correct thing to say that clojure.main/repl-requires are required when you start a repl https://github.com/clojure/clojure/blob/38524061dcb14c598c239be87184b3378ffc5bac/src/clj/clojure/main.clj#L355#2020-02-2109:06dominicmQuite right, I'm not sure why I thought otherwise. Brain fart.#2020-02-2113:14Wes HallMaking some increasingly half-hearted attempt to get DynamoDBLocal running to start embedded for tests, but it seems to depend on sqllite4java which exists in the mvn repos but with <packaging>dylib</packaging>. Am I right in assuming that tools-deps doesn't have support for downloading dylib dependencies?#2020-02-2113:48kszaboI prefer to use docker-compose to set up local service dependencies:
version: '2'
services:
dynamodb-local:
image: 'dwmkerr/dynamodb'#2020-02-2113:48kszabohttps://hub.docker.com/r/dwmkerr/dynamodb#2020-02-2113:49kszabobut this is not related to #tools-deps this way#2020-02-2114:20Wes Hall@thenonameguy Yeah, I will probably move to something like that. Dynamo local, at least in theory, does support embedded running so should be possible to throw up an in-memory version in a text fixture and tear it down afterwards, but the dependency on sqllite (and thus native libraries), is a bit of a hurdle. I'll probably give up, but just thought I would check that I am not missing a route I could at least try, but I think the docker-compose option is probably the way to go. Thanks.#2020-02-2114:21Wes HallDownloading native (thus platform dependent) libraries, is a bit awful anyway, I wouldn't be surprised if it's not supported.#2020-02-2114:33teodorlu@vlaaad @dominicm @alexmiller thanks for the help clearing that up! I realize why doc shouldn't work above, and I get the expected result when I require it as I "should" do.
$ clj -e "(require '[clojure.repl :refer [doc]]) (doc ns-map)"
-------------------------
clojure.core/ns-map
([ns])
Returns a map of all the mappings for the namespace.
#2020-02-2117:14rickmoynihanNot sure if this is a known regression or not… but I was using some private S3 buckets as mvn repos; with a URL of the form s3p://<bucket>/<directory>/
I’ve been using this for years (previously with leiningen) and found earlier in the week that I could no longer resolve deps via the clj tool.
I suspect this was caused by the recent changes in t.d.a to using the cognitect aws lib; but I can’t be certain.
Anyway I’ve just tried changing the bucket URL’s to be of the form s3:// and it now works. No idea what the difference is in the protocols; but I’ve seen both forms over the years.#2020-02-2117:17gerreds3p is for hadoop/presto it seems.#2020-02-2117:18gerrednot an official protocol. AWS has a few other protocols for similar reasons (with EMR specially): s3a and s3n come to mind.#2020-02-2117:19gerredhttps://github.com/s3-wagon-private/s3-wagon-private i see this of course#2020-02-2117:19gerredso maybe this was only a wagon thing.#2020-02-2117:21gerredit looks like it's even gone from newer versions of Wagon#2020-02-2117:22rickmoynihanyeah iirc tools.deps used the s3 wagon until recently#2020-02-2117:23gerredoh no it's still there#2020-02-2117:23gerredhttps://github.com/s3-wagon-private/s3-wagon-private/blob/master/src/main/java/org/springframework/build/aws/maven/PrivateS3Wagon.java#2020-02-2117:23gerredyeah#2020-02-2117:27dominicmAlex did ask to know if anything broke. I think compatibility was the intention.#2020-02-2117:33rickmoynihanYeah that’s why I’m mentioning it.
The (end user) fix is trivial (just change to use s3://) once you know what the problem is though.#2020-02-2117:53Alex Miller (Clojure team)s3p has never been a documented supported protocol for clj#2020-02-2117:54Alex Miller (Clojure team)So was working before by accident, not by intent#2020-02-2117:54Alex Miller (Clojure team)s3p is an invention of the spring aws wagon#2020-02-2117:55rickmoynihanAh interesting, I think I just copied my URLs from lein and didn’t think anything of it when they worked#2020-02-2117:55rickmoynihanAnyway it’s not a problem, I just thought I’d mention it incase it happens to others#2020-02-2405:54onetomI'm trying to switch to Google's Maven Central mirror.
I've added a mirror section to my ~/.m2/settings.xml but I'm not sure if it's used by clj.
How can I check what actual URL is it downloading from when it says:
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar from central
I've tried -Sverbose and -Sdescribe but neither of those options provide an answer#2020-02-2413:52Alex Miller (Clojure team)I don’t think there is any way to see that, but there is code with the intent to use mirrors in settings#2020-02-2413:52Alex Miller (Clojure team)Someone else from Asia walked down this same path a few months ago and it was working#2020-02-2406:00EmmanuelOgaperhaps you need to also add it to the deps.edn map?#2020-02-2406:00EmmanuelOgahttps://clojure.org/reference/deps_and_cli#_procurers#2020-02-2406:01onetomI'm trying to move it out from my deps.edns becaues mirror settings are not project specific#2020-02-2406:02EmmanuelOgawhat if you put it in the global deps.edn?#2020-02-2406:02EmmanuelOga$HOME/.clojure/deps.edn#2020-02-2406:04onetomthen pure maven tools wouldn't benefit from this setting#2020-02-2406:04onetomplus im not sure what's the semantic difference between overriding the repo called "central" vs specifying mirrors instead#2020-02-2406:06onetomit's also a bit unclear how the :mvn/repos key is merged into the hardwired default values, because i haven't found a way to print the effective value of it using clj#2020-02-2406:08onetomi had the impression that the :mvn/repos key in my project's deps.edn overrides the value coming from ~/.clojure/deps.edn but i haven't found this being documented#2020-02-2406:14onetomthe only reason i suspect that the <mirror> setting is taken into account is because the installation of my deps from scratch takes 34s instead of 51s#2020-02-2406:34EmmanuelOgaheh nice#2020-02-2406:36EmmanuelOgaare you working with a setup that for some reason needs toperform installs from scratch often?#2020-02-2407:02onetomim in hong kong and im on crappy mobile connections often, so using mirrors make a big difference even if im not doing from-scratch installs.
on the other hand im also experimenting with building self-contained developer workstation setups and then i want to test from-scratch builds sometimes just to make sure everything still works.
a datomic upgrade also pulls in quite some dependencies and our projects might be on slightly different versions of common libraries and all this adds up...
just to demonstrate the absolute scale of problems:
• im taking a 30minute ferry ride twice a day to work and i could spend about 20minutes of that working
• i might need to run a nix package or maven update every week probably which would amount to downloading ~200MB. that can take 10minutes; half of my travel time is wasted.
• if i use a mirror and it just takes 5 minutes to upgrade, only 1/4th of my travel time is wasted.
but generally many things are significantly slower from asia, so even if i run something on a singapore/tokyo/hong kong aws or digital ocean server, the download time differences add up to minutes compared to US servers.#2020-02-2407:03onetomif u consider that multiple ppl in the company suffer that time waste from time to time, then it's a non-negligible factor#2020-02-2407:05onetomand then i haven't even talked about ppl in china-proper, behind the GFW, who often have to go thru very slow vpns.
for them an asian mirror might be several times faster or not even being blocked, so they dont need a vpn to access...#2020-02-2407:07EmmanuelOgaoh wow ok setting up that mirro rsounds like a worthy goal#2020-02-2407:07onetombecause of these circumstances, im very excited to see parallel downloads in tools.deps!
now it performs just as fast as npm2/yarn/pnpm for example 🙂#2020-02-2407:07onetomone less critique i have to listen to from former JS ppl whom im trying to teach clojure to...#2020-02-2407:09EmmanuelOgawould it make sense to setup some sort of system wide caching proxy? quick google found this thing: http://proxychains.sourceforge.net/#2020-02-2407:10EmmanuelOgaI imagine there could be a way to make it so every time you open an http connection in any program on your system it would go through your proxy#2020-02-2407:10EmmanuelOgaI'm not very good at this kind of thing though so maybe it is just a stupid idea hah#2020-02-2407:11onetomproxychains doesnt do caching, but yeah, it would make sense to setup some proxies, but every ecosystem needs different kind of proxies and then some of the deps we need are private because they are our own, in-house code, or it's datomic.
setting up proxy for such an environment is not trivial at all.#2020-02-2407:11EmmanuelOgagotcha#2020-02-2407:12onetomive experimented with various maven mirroring solution, like jfrog and artifactory, etc, but that doesn't solve brew, macos, git, nix, npm caching...#2020-02-2407:16onetomfor proxychains, i've actually made a nix package:
{ stdenv, fetchFromGitHub } :
stdenv.mkDerivation rec {
pname = "proxychains";
version = "4.14";
src = fetchFromGitHub {
owner = "rofl0r";
repo = "proxychains-ng";
rev = "v${version}";
sha256 = "03wk2xpxwc7kwlq6z9jf9df1vlh6p0dn0kf826fh1k7nfaa8c4py";
};
postPatch = ''
# Temporary work-around; most likely fixed by next upstream release
#sed -i Makefile -e '/-lpthread/a LDFLAGS+=-ldl'
'';
meta = {
description = "Proxifier for SOCKS proxies";
homepage = ;
license = stdenv.lib.licenses.gpl2Plus;
platforms = stdenv.lib.platforms.all;
};
}
and using this config file for it:
[ProxyList]
socks5 127.0.0.1 1080
u can just ssh -D 1080 some.aws.basition.host to establish a tunnel, then
export MYSQL_HOST=
proxychains4 -f proxychains.conf mysql --host=$MYSQL_HOST --user=root --password -e 'show databases'
would allow accessing an RDS instance using its internal aws hostname, although the mysql cli tool doesn't support socks proxies.
just showing, to clarify what proxychains is for#2020-02-2407:19onetomalso, this kind of proxying over SSH is also said to be disrupted across the GFW.
i find it a bit hard to believe, but that's what i've heard...#2020-02-2407:21onetomwhile all this might sound off-topic on this channel, i think it's actually not, because these kind of scenarios are the reasons, why supporting various directives in maven settings.xml or providing parallel downloads is important.
it's just ppl suffering from these issues don't really have a voice to raise them, because they don't speak good english and they are not aware where should they speak up...#2020-02-2407:24EmmanuelOgawould you say clojure is popular in hong kong?#2020-02-2407:24onetomwell, im personally making sure it is ;D#2020-02-2407:26EmmanuelOganice. Good luck with that 🙂#2020-02-2407:26onetomi've introduced clojure into almost all companies i've worked at in the past ~5years https://www.linkedin.com/in/onetom/
im not going to meetups often lately, but when i do, i also sow the seeds of clojure amongst the audience 😉#2020-02-2407:27EmmanuelOgado you speak chinese and/or cantonese?#2020-02-2407:28onetomnope, only a bit of thai, but not at a technical level; im hungarian.
i've started to shake up the hungarian clojure community lately too on #clojure-hungary for example#2020-02-2407:29EmmanuelOgacool#2020-02-2601:34cflemingAm I correct in thinking that this commit is what’s needed to read the authentication for a particular repo from settings.xml? https://github.com/clojure/tools.deps.alpha/commit/658074337ce9577c9f1298fd5146a3029308748c#2020-02-2601:35Alex Miller (Clojure team)Yes, modulo any later changes#2020-02-2601:36cflemingOk, I’ll look at that. Seems pretty magic in there. It’s odd that that functionality isn’t part of the resolver itself, it seems to live in a Maven jar.#2020-02-2601:36Alex Miller (Clojure team)Don’t get me started#2020-02-2608:03dominicmI think I ported something I found in the maven cli code. Was a bit of a hunt.#2020-02-2620:29neupshHello, I have @seancorfield’s deps.edn file from https://github.com/seancorfield/dot-clojure in my ~/.clojure/deps.edn and I am running clj -A:rebel:deps to start a clojure repl with rebel-readline and tools.deps.alpha for adding libraries from within the repl. So, to be able to add library, I will have to (require '[clojure.tools.deps.alpha.repl :refer [add-lib]]) before I can call add-lib. Is there a way to auto require this in the user namespace so that when I start the repl, I have it available there?#2020-02-2620:49seancorfield@neupsh You have to split the resource/main opts:
> clj -R:rebel:deps -e "(require '[clojure.tools.deps.alpha.repl :refer [add-lib]])" -m rebel-readline.main
[Rebel readline] Type :repl/help for online help info
user=> (add-lib 'seancorfield/next.jdbc {:mvn/version "RELEASE"})
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
Downloading: seancorfield/next.jdbc/maven-metadata.xml from
true#2020-02-2620:50seancorfieldYou could add a new alias that combines all of that -- you'll need , instead of whitespace in the -e option if you put it in deps.edn tho'.#2020-03-0122:50samI wrote a script to aot-compile classes in my project, which I have configured to run automatically when necessary via aliases. e.g.
:test {:extra-paths ["test" "test-resources"]
:extra-deps {com.cognitect/test-runner {:git/url ""
:sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}}
:main-opts ["-i" "aot.clj"
"-m" "cognitect.test-runner"]}
I tried adding logging to my aot script using clojure.tools.logging, and found that tests started failing with java.lang.NoClassDefFoundError: clojure/tools/logging/impl/LoggerFactory
I don’t totally understand what’s going on, but I think it has something to do with the fact that the clojure compile function transitively compiles only namespaces that haven’t already been loaded. So, because I have already loaded clojure.tools.logging before calling compile, it does not get transitively compiled even though the namespaces I’m compiling depend on it.
Still, I wasn’t sure that this situation should cause a NoClassDefFoundError, since the clojure source code should still be available on my classpath, but I think it might be due to this issue: https://clojure.atlassian.net/browse/CLJ-1544. If I understand correctly, I think it’s important in my case for clojure.tools.logging to be compiled since the calling code is compiled.
My workaround was to add an explicit (compile 'clojure.tools.logging) to the script.
So my questions are:
1. Does my understanding of the problem make sense?
2. Is there an issue here with compile that should be addressed? Assuming my understanding of all this is correct, I wish compile had options about how it should handle dependency namespaces, or at least that the documentation specified this behavior (if it does, I couldn’t find it).#2020-03-0123:09seancorfield@samwagg0583 AOT has a number of serious "gotchas" and it's best to avoid it unless you absolutely need it (it's "safe" as the very last step before you build an uberjar for deployment as a complete application, but most other uses can trip you up).#2020-03-0123:10Alex Miller (Clojure team)I don’t think this is CLJ-1544#2020-03-0123:11Alex Miller (Clojure team)Compile is transitive but it can’t necessarily “see” everything and you may need to explicitly compile things sometimes. All that is normal.#2020-03-0123:12seancorfieldWhen code is (partially) AOT'd and then it attempts to load in (and compile on-demand) more code, you can get mismatches between the classes that the class loaders can see. Is that a fair description @alexmiller?#2020-03-0123:14Alex Miller (Clojure team)It’s more specific than that#2020-03-0123:15Alex Miller (Clojure team)Sorry, I gotta run #2020-03-0123:13Alex Miller (Clojure team)Your issue with having it already loaded is something to consider and you may need to change your compilation process to account for that#2020-03-0123:17seancorfield@samwagg0583 What is driving your need to AOT code for testing here?#2020-03-0123:58samWe are using a java library for distributed data pipelines called Apache Beam and running in Google Cloud Dataflow. Running a pipeline involves running code on a local JVM that uploads serialized java objects to remote JVMs. De-serialization on the other end requires class files, hence the AOT.#2020-03-0123:59samMy workaround of explicitly compiling classes that are already loaded seems to work okay. I just wanted to better understand the issue.#2020-03-0200:06seancorfieldOuch. Yeah, you don't have much of a choice I guess. Tho' I thought I'd read about a Beam-related library that avoided AOT recently?#2020-03-0200:06seancorfieldYeah https://github.com/atdixon/thurber#2020-03-0202:59samThanks for this! We have looked into existing clojure libraries for Beam in the past, but I wasn’t aware of this one. I’ll definitely have to check it out further.#2020-03-0203:33seancorfieldIt's pretty new. I can certainly understand the authors' desire to avoid AOT for this 🙂#2020-03-0214:16sparkofreasonI've been using thurber, so far very successfully. Aaron has also been very responsive to issues and questions. It's still a bit raw, but useable, and I like the philosophy of keeping Clojure wrappers of Java APIs as thin as possible. Makes it a lot easier to figure out how to go from Java-based docs etc. to the Clojure equivalent.#2020-03-0217:33aviTIL!#2020-03-0200:07seancorfield> AOT Nothing
Fully dynamic experience. Reload namespaces at whim. thurber's dependency on Beam, Clojure, etc versions are completely dynamic/floatable. No forced AOT'd dependencies, Etc.#2020-03-0219:20fabraohello all, using deps.edn is there any way to load dependency dynamic as boot does?#2020-03-0219:20ghadiwhat does that mean @fabrao?#2020-03-0219:20borkdudehe might be looking for the add-lib stuff#2020-03-0219:20borkdudehttps://insideclojure.org/2018/05/04/add-lib/#2020-03-0219:21fabraoHumm, I think it´s what is I want, let me check#2020-03-0219:22borkdudeit is in a branch, not officially out yet, but might be later#2020-03-0219:24fabraoso, for now only include new lib in :deps to include new lib and restart repl?#2020-03-0219:28seancorfield@fabrao My dot-clojure file has an alias that brings in a recent add-lib branch of tools.deps.alpha and notes on how to use it to dynamically load new libraries into your REPL without a restart.#2020-03-0219:29seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L144-L147#2020-03-0219:30seancorfieldI use that quite a bit to load extra libraries while I'm working.#2020-03-0219:31seancorfieldYou can see how I use it with next.jdbc to bring in all of its test dependencies into the REPL while I'm working on other projects https://github.com/seancorfield/next-jdbc/blob/master/test/next/jdbc/test_fixtures.clj#L133-L149#2020-03-0219:35fabraoI have to change it to use with cider#2020-03-0411:14practicalli-johnTo use a deps.edn alias with Cider (Emacs), either use the universal constant to edit the jack-in command or create a .dir-local.el file and set the cli to run an alias
https://practicalli.github.io/blog/posts/cider-jack-in-to-clojure-cli-projects-from-spacemacs/#2020-03-0219:35fabraoin emacs#2020-03-0219:50fabraoit seems that it works only in REBL enviroment#2020-03-0219:55seancorfieldNo, it works in a plain REPL.#2020-03-0219:56seancorfield(in fact, it specifically does not work in REBL because REBL does not use a DynamicClassLoader)#2020-03-0219:57seancorfieldYou don't need to change anything -- it's pure Clojure and works with any plain REPL, as long as you had the add-lib branch dependency on your classpath when you start the REPL.#2020-03-0219:58seancorfieldI just start a REPL with clj -A:deps (and other options for my use). If you're using Leiningen, you'd need something that pulls in Git dependencies (I do not use Leiningen).#2020-03-0220:04Alex Miller (Clojure team)I thought we fixed the rebl thing?#2020-03-0220:04Alex Miller (Clojure team)maybe I just filed tickets to fix it generically in prepl and socket repl#2020-03-0220:05Alex Miller (Clojure team)don't remember now#2020-03-0220:05ghadiyeah tickets filed for DCL#2020-03-0220:05ghadione ticket, I think#2020-03-0220:05Alex Miller (Clojure team)yeah#2020-03-0220:05ghadihttps://clojure.atlassian.net/projects/CLJ/issues/CLJ-2540#2020-03-0220:06Alex Miller (Clojure team)yeah, it's in the chute for 1.11#2020-03-0411:14practicalli-johnTo use a deps.edn alias with Cider (Emacs), either use the universal constant to edit the jack-in command or create a .dir-local.el file and set the cli to run an alias
https://practicalli.github.io/blog/posts/cider-jack-in-to-clojure-cli-projects-from-spacemacs/#2020-03-0312:21geraldodevhow we can programatically get the list of loaded aliases ?#2020-03-0312:27vlaaad(.getAliases *ns*)#2020-03-0312:38vlaaadoh, sorry, this is tools-deps, I thought you are asking about namespaces#2020-03-0312:39borkdudeThis would be the public API for that I think (I know, this isn't about tools.deps):
user=> (require '[clojure.string :as str])
nil
user=> (ns-aliases *ns*)
{str #object[clojure.lang.Namespace 0x301d8120 "clojure.string"]}#2020-03-0312:41vlaaadah, right, forgot about that one 😄#2020-03-0312:53vlaaadback to original question @U0516053R :
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "c5b5b7aab3a485fcfce8f3f0f3f1f7"}, :content ("[email protected]")}#2020-03-0312:54vlaaadI don’t think there is a good and reliable way to figure it out, because clj launches a new process with classpath, so no alias information is really available to the jvm#2020-03-0312:55vlaaadwith that said, you can lookup your PID, lookup parent PID, run ps (linux and mac only) to find information about parent PID and extract some info from that: as you see, :out contains invocation string#2020-03-0312:59vlaaadnote: ProcessHandle is available since JDK 9, previous JDKs will require some reflection and googling to get pids..#2020-03-0313:06geraldodev@U47G49KHQ thank you#2020-03-0313:35Alex Miller (Clojure team)Can you back up to what you are trying to do?#2020-03-0317:04geraldodevI'd like to take code action based on the existence of dev alias when connecting to database.#2020-03-0317:05borkdudemaybe use an environment variable for that?#2020-03-0317:13Alex Miller (Clojure team)yeah, I would add something extra that is explicit for that case, don't rely on all that process stuff above#2020-03-0317:13Alex Miller (Clojure team)system property, include dev resources in classpath, etc#2020-03-0318:41dominicmI use system props because they go into jvm-opts and they compose#2020-03-0317:21geraldodevThank you @alexmiller and @borkdude.#2020-03-0603:25devnIs there a good guide or rationale on http://tools.deps vs other options? Several beginners have expressed confusion about lein v http://tools.deps, and lamented the fact indirectly that they’ve now needed to learn both tools while leveling up, but that almost all of their professional work has involved lein.#2020-03-0603:27devnI’m actually confused myself as a long time clojure user. Why did we need another half-finished build tool again? I think I personally understand some of the why, but there’s no doubt it’s contributed confusion to would-be clojure users.#2020-03-0603:33devnThere are numerous answers here that will be acceptable to me, a long-time clojure user, but the fragmentation of beginner tutorials sucks.#2020-03-0603:34devnSorry to be a downer. This ship has sailed, but I have to let y’all know that new users are frequently wondering why they have to learn two build tools.#2020-03-0603:37andy.fingerhutOne answer is that tools.deps is not a build tool. Some more discussion comparing Leiningen vs. tools.deps capabilities here: https://clojureverse.org/t/is-there-a-sales-pitch-for-switching-to-deps-edn-from-lein-in-2020/5367#2020-03-0603:38devnSo, that’s kind of pointing directly at the problem.#2020-03-0603:40andy.fingerhutI can point you at resources that you may or may not find useful to point beginners at. I've got no horse in this race.#2020-03-0603:44Bobbi TowersI think David Nolan made a pretty good case here https://www.youtube.com/watch?v=F0Lv53lop2g#2020-03-0603:45devnI would appreciate any handy resources, and I’ll cop to soapboxing a bit here because it pains me to see beginners who are serious learners being legitimately confused by the ecosystem.#2020-03-0603:47devnI remember when I started, and people were using ivy and gradle and maven. Lein came into fashion, then cake was absorbed into lein.#2020-03-0603:50seancorfield> Why did we need another half-finished build tool again?
CLI/`deps.edn` is very specifically not a build tool.#2020-03-0603:50Alex Miller (Clojure team)^^#2020-03-0603:51devnyou can explain this all day to new users #2020-03-0603:51seancorfieldNew users do not need a build tool. They need a REPL. They need to run programs.#2020-03-0603:51devnthat’s nonsense imho#2020-03-0603:52devnlike you can be right, but it doesn’t help new user confusion #2020-03-0603:53seancorfieldThe "problem" is that most books and lots of online tutorials that have been around for ages all show Leiningen as part of the first steps in using Clojure. We're in a transition period right now because Leiningen has been around for years and new tooling is appearing. This happens any time a tech community has new tooling come up to replace older, existing tooling.#2020-03-0603:53devnstop right there#2020-03-0603:53devnYou said it’s not a build tool #2020-03-0603:54devnWhat transition?#2020-03-0603:54seancorfieldThere was a time when Boot was gaining ground and folks complained about that too.#2020-03-0603:54seancorfieldLeiningen is a "sledgehammer": it starts a REPL, it runs programs, it bakes, it broils, it takes out the trash.#2020-03-0603:55seancorfieldI see beginners tripping over problems with Leiningen all the time because of plugins they've added to ~/.lein/profiles.clj because it's "easy" rather than "simple".#2020-03-0603:55devnI expect to be on the losing end of this, but I have to really underscore the confusion for new users. I feel it’s my duty. I know that may sound hyperbolic, but this is coming from a place of love.#2020-03-0603:55devnNew users do not understand this subtle distinction you’re making.#2020-03-0603:55Bobbi TowersNot needing Leiningen to get started is a huge plus for new users#2020-03-0603:55seancorfieldA lot of us here are on the front lines in #beginners all the time, helping beginners.#2020-03-0603:56devnyes, and I’ve heard directly from some of them who wonder why anyone is bringing up http://tools.deps, because all of their work is in lein world #2020-03-0603:56devnThat’s exactly the confusion. #2020-03-0603:57seancorfieldThe official Clojure Getting Started docs lead people to the CLI and deps.edn.#2020-03-0603:58devnyes, why? The vast majority of existing production clojure projects don’t use it.#2020-03-0603:58seancorfieldThe latest edition of Programming Clojure uses the CLI and deps.edn I believe (right @alexmiller?) and the ClojureScript Getting Started guide uses the CLI and deps.edn.#2020-03-0603:58seancorfieldTech changes over time. There's nothing strange about that.#2020-03-0603:58seancorfieldLook at all the changes in the Java world over the years... beginners now do stuff completely differently than beginners five, ten, fifteen years ago...#2020-03-0603:59devnI’ve said my piece. I don’t think it’s helpful. I think it’s confusing for new users.#2020-03-0603:59seancorfieldWould you prefer tooling never changed and never improved?#2020-03-0603:59devnthat’s hyperbolic#2020-03-0603:59devnI’m telling you what I know here. Anecdotal as it may be...#2020-03-0604:00seancorfieldI have anecdotes too -- that many beginners find CLI and deps.edn to be much simpler to learn and use.#2020-03-0604:02devnOk, what is depstar, and why does it exist?#2020-03-0604:03seancorfieldAsk the folks who wrote it? 🙂#2020-03-0604:03devnYou maintain a fork of it#2020-03-0604:04devnno?#2020-03-0604:04devnI’m not trump to gotcha anybody#2020-03-0604:04devnBut that project alone should act as sufficient evidence of the confusion #2020-03-0604:04seancorfieldYes, I found the initial version useful enough that I wanted to fix a few bugs in it, and folks seem to like my fork of it.#2020-03-0604:05devnIt uberjars.#2020-03-0604:05devnIt’s. A. Build. Tool.#2020-03-0604:05seancorfieldHardly.#2020-03-0604:05seancorfieldIt's not ant, not maven, nor gradle etc -- those are build tools.#2020-03-0604:06seancorfieldLein and Boot are build tools.#2020-03-0604:06devnagain, this isn’t for me #2020-03-0604:06devnI know what you’re trying to say#2020-03-0604:07devnBut for beginners this is nonsense#2020-03-0604:07devnI’ll leave it there#2020-03-0604:10Alex Miller (Clojure team)I'm confused what you would find a satisfactory answer to these queries?#2020-03-0604:11seancorfield@devn ^#2020-03-0604:12seancorfield@alexmiller Programming Clojure uses the CLI and deps.edn in the latest edition, right?#2020-03-0604:13Alex Miller (Clojure team)yes#2020-03-0604:14seancorfieldAnd it doesn't mention "uberjar" anywhere in the book... and only has a couple of mentions of "jar" (I just looked).#2020-03-0604:15seancorfieldSo a beginner can read that book end-to-end and not worry about build tools...#2020-03-0604:15seancorfield(that was what I suspected but wanted to confirm by searching the PDF)#2020-03-0607:40pyrYou could look at it from the angle that there's value in having a lower level toolbox for builds. There is tremendous value as well in having a build tool with conventions (a-la lein). We can hope for a new tool or an evolution of lein, bringing back the baked in conventions which relies on tools-deps#2020-03-0607:52sogaiucheck out the latest insideclojure?#2020-03-0608:15vlaaad“tools.build - shh, top secret”? 🙂#2020-03-0608:50vlaaadA bit late to the flamewar of tools-deps vs build tools, I wanted to repeat once again how much I love clj compared to lein. It all boils down to how you run java programs. To answer the question “How do you run java programs?” one needs to have a look at java command: what arguments it accepts, what are actual, real semantics of running java programs. Actual, real semantics of running java are:
• jvm options like maximum memory (`-Xmx512`) or system properties (`-Dclojure.spec.skip-macros=true`)
• classpath: sequence of directories to look up resources (both .class files and any other type of file that might be needed for running programs) — either as a directories on file system or as archived directories (aka jar files)
• main class to run
• arguments to main class
Everything else is non-essential complexity that is imposed by a build tool.
Some of this added complexity has a lot of value: specifying dependencies to automatically build classpath, for example.
Some of it does not. For example, lets look at source path and resource path separation in leiningen. It’s accidental: you can put your source code in resource files and it will work. What good does it give you besides obscuring actual, real semantics of running java program? Lein plugins, for example, are just wrong and unnecessary abstraction: you could create a separate profile that has a dependency on a tool of your choice and run it. Automatically picked up leiningen profiles for me are still hard to understand: clj’s aliases are much more intuitive and predictable. clj command line options map much better to java command line options. Case in point: yesterday I wanted to run a lein command with system property provided from command line. I could not do it: you have to specify jvm options in :jvm-opts in project.clj, there is no way (at least that I know of) to specify them for lein run invocation.
Beginners coming from java land with equally complected tooling might have a hard time to adapt to tools-deps, but that’s because that tooling equally obscures the actual, real semantics of running java programs.
Break the cycle. Rise above. Focus on simplicity.#2020-03-0608:56dominicmhttps://clojurians.slack.com/archives/C6QH853H8/p1583465670122100
This message was the one I think is quite interesting. The answer is quite easy "there are 2 major competing entrypoints into clojure". I'm sure anybody can understand that. Why is there multiple battery brands that exist, etc. Because they each believe they offer something better than the other.
It's true that there's a cost to having 2 things that both provide the entrypoint. Enough people seem to think that cost pales in comparison to the benefits provided by tdeps. But those benefits are not easily rendered to someone who isn't versed enough in clojure to understand the bigger picture of elements playing out.#2020-03-0609:18teodorluHey @devn,
I think you're bringing some good points. I read your comments above, and I'd like to respond. I hope to be able to bring some civility.
• Personal opinion: getting started with Clojure isn't a perfect experience for new users. That you might encounter project.clj and deps.edn is evidence of that -- learning a single tool would be easier.
• Personal judgement: A common Clojure opinion is that "build tool" is a too broad category, in the sense that Gradle, Maven or NPM works. deps.edn is part of that solution. The "full solution" isn't known yet. deps.edn implementing part of that solution opens the space for other actors.
• Personal experience: A simple / pure data deps.edn is great to work with. I can download the dependencies without having to buy into how you'd like to build your project. Too much complex stuff in the project build pipeline can cause the project build pipeline to become opaque / magic.
Some questions:
1. Do you feel like the above points bring any value to the discussion you'd like?
2. Do you have a suggestion on what work should be prioritized in order to improve new user experience?
Thanks!
Teodor#2020-03-0613:43devnOof, hearing “flame war” and “bring some civility” makes me feel pretty terrible. I talked this out with Alex for awhile last night, and Teodor, I think the second question is one I need to pose to those that have expressed their frustration to me.#2020-03-0615:30devnI sincerely apologize if I came off like a jerk. I was out of sorts last night and did a poor job of broaching the topic and keeping it constructive. facepalm #2020-03-1011:36teodorluYet, I think digging into frustration can be fruitful for improving the state!
I appreciate that you're spending the effort 🙂#2020-03-0615:50seancorfieldIt's definitely unfortunate that Clojure didn't have something like the CLI and deps.edn from "day one" and we ended up with lein as the default : it is easy but not simple in the Clojure sense. At work we started with Leiningen because there really was no choice back in 2011. We switched to Boot in 2015, and the CLI and deps.edn at the end of 2018 I think. It's a much simpler setup, even managing a 90k line monorepo with thirty subprojects. I'm very glad that our tooling continues to evolve. Yes, it means that beginners who start from most books and tutorials today see Leiningen there but then see CLI and deps.edn in new books, new tutorials and on the official site. That's why I consider this a transition : at some point Leiningen will be the minority tooling I suspect or will at least be seen as "the old way" and the question of "which should I use?" will go away, or at least have an obvious answer. #2020-03-0615:52seancorfieldThe transition from monolithic Contrib to new Contrib was similarly confusing to beginners (but the community was much smaller then so the pain was less). #2020-03-0615:53seancorfieldThe introduction of Spec caused similar confusion I think (should I use Schema? Spec? Typed Clojure?).#2020-03-0615:55seancorfieldAnd it's not like we don't have a bunch of options in other areas : which JSON library? Which routing library? Which web server? Learning Clojure is littered with such choices #2020-03-0618:56devnI think I may just fundamentally disagree that simple > easy for new users. I agree there are a lot of choices and that this is just the nature of the beast, but for instance, getting a basic project structure with a working test runner is a whole thing, and IMO it needn’t be this way. test-runner and clj-new exist, but they both are extra steps to simply having a basic working project structure, and then people hit the “ok how do I build an uberjar” step and now they have 3 more options to choose from that I’m aware of. I hadn’t heard about tools.build and am interested in learning more if anyone has a link handy.
lein has its warts certainly. At a certain point you definitely begin fighting with the tool. For instance, checkouts, tasks, the craziness of cljsbuild’s 9000 options, trampoline, passing java opts, and many more, but for new users you could type lein new to get a common project structure, add a dependency, write a test, type lein test, and it just worked.#2020-03-0619:00devnI know this isn’t what http://tools.deps wants to be, but I think introducing it to new users on getting started is a rougher-than-necessary entry point without a bit of ease sprinkled in.#2020-03-0619:06Alex Miller (Clojure team)I think there are multiple phases. clj is easy, probably easier than lein imo, if you want to just use a repl and write some clojure and use some deps, etc. Beyond that is "make a simple lib/app". Lein is easier in that zone (for now). And then beyond that you get to more bespoke builds, and actually I would again say clj is easier to fully customize because you are not constrained to whatever flags and switches and lifecycles lein makes available.#2020-03-0619:07Alex Miller (Clojure team)more to come in the near future for the middle zone#2020-03-0619:08Alex Miller (Clojure team)in the interim, something like sean's template (and others) can give you a slightly opinionated set of choices for making a new project based on clj. It's harder, but it's not like 10x harder, it's like 1.5x harder.#2020-03-0619:25dominicmYou can make easy from simple, can't make simple from easy.#2020-03-0619:29Alex Miller (Clojure team)more importantly, simple and easy are different dimensions not a vs b.#2020-03-0619:30Alex Miller (Clojure team)so really, you can go simple+hard -> simple+easy, but you can't go complex+easy -> simple+easy.#2020-03-0619:47devnUnderneath some of what I said last night, there is some personal fatigue from conversations I’ve had where people seem to use simple v easy as a way of downplaying aesthetic or ergonomic concerns by tossing those two words out without addressing the human side. New users should have it easy, full stop. To the detriment of all other things? Of course not. It’s not an either/or.
For example, I have a really difficult time imagining how having clj new and clj test would destroy the aims of the project.#2020-03-0619:47devnI’m not saying this is what should be done necessarily, but there’s no point in being pure to the detriment of new users.#2020-03-0619:58Alex Miller (Clojure team)I certainly agree that simple does not mean hard#2020-03-0619:59Alex Miller (Clojure team)clj new and clj test seem like making clj into something that it's not though (a build tool). it's a tool for running programs. template gen and test runners are programs.#2020-03-0620:00devntotally agree#2020-03-0620:00Alex Miller (Clojure team)having ways to discover useful programs to run or templatize/kickstart how to do so seem like good things#2020-03-0620:01devnbut like, should I just make a thing called dmk that does all of the clj stuff and adds two commands for new users?#2020-03-0620:01devnthat seems bizarre to me?#2020-03-0620:02Alex Miller (Clojure team)well I wouldn't do that :)#2020-03-0620:02devnhaha, I’m asking then honestly, where would something like that go? It’s never in the cards?#2020-03-0620:03Alex Miller (Clojure team)I don't have a pat answer to that - finding one is a design process#2020-03-0620:03Alex Miller (Clojure team)these things literally exist now#2020-03-0620:03Alex Miller (Clojure team)as programs that you can run now with clj#2020-03-0620:04Alex Miller (Clojure team)and as templates that you can create/install#2020-03-0620:05Alex Miller (Clojure team)is the problem that we have not done a good job explaining that or that that is not easy enough. and if the latter, what would make it so?#2020-03-0620:06Alex Miller (Clojure team)like if this page included those examples https://clojure.org/guides/deps_and_cli would that be sufficient?#2020-03-0620:08Alex Miller (Clojure team)if you added a bit of stuff to your ~/.clojure/deps.edn, then it's literally clj -A:test or clj -A:new etc which seems pretty close to what you have above#2020-03-0620:08vlaaadThere were some discussions on clojureverse to make such a simple tool, even some code was written: https://github.com/Lokeh/plum https://clojureverse.org/t/idea-a-tool-for-helping-developers-get-started-using-clojure-cli-tools-and-deps-edn/4270#2020-03-0620:09devnIt kind of reminds me of clj-rewrite and the evolution of “just update my deps” or “just add the damn test runner”#2020-03-0620:10devnBrb#2020-03-0620:15mike_ananev@devn try this https://github.com/redstarssystems/pbuilder here you can find templates, easy commands and more#2020-03-0620:19mike_ananevin templates I use make utility, so it is very easy to perform different tasks#2020-03-0620:34seancorfieldIf someone does add the :new alias to create a new app or lib project, the generated project includes the :test, :jar/`:uberjar`, :install/`:deploy` aliases so it all becomes as "easy" as lein for basic usage. So the missing piece is really only that initial project creation piece I think.#2020-03-0620:36seancorfieldStrangely, when we switched from lein to boot at work, "new" wasn't a thing for Boot either and I ended up forking lein-new (which wasn't a thing for Leiningen in the very early days, by the way!) into boot-new and it was quite some time before that was integrated into the core tool.#2020-03-0620:36devnHeh, I mean, here we are. http://tools.deps is not a build tool but there are 3 choices for making an uberjar. It’s too simple to add any porcelain commands so there are at least three things that add that type of functionality. That’s my gripe I guess.#2020-03-0620:36seancorfieldSo there's a history of tooling not including an "easy" way to start a new project from scratch 🙂#2020-03-0620:37devnOh yeah! I actually worked on the boot lein template thing I think...#2020-03-0620:37seancorfield> 3 choices for making an uberjar
Oh, c'mon! There are far more than just three choices! :rolling_on_the_floor_laughing:#2020-03-0620:37devnLong time ago#2020-03-0620:37devnhaha#2020-03-0620:38seancorfieldI think far more beginners are stumped by the "how do I build a simple web app" question -- given the plethora of choices they get offered at that point. And heaven help them if someone says lein new luminus myapp and then they have to figure out what that produces!#2020-03-0710:27practicalli-johnI can only share my experiences, but build tools and running a repl are fairly low on the confusion list for those new to coding with Clojure. At ClojureBridge London we have introduced over 300 students who are either new to coding or on their first language. We had been using Leiningen for a long time and now use Clojure CLI and Leiningen,
Clojure CLI is far quicker to install and run, deps.edn has less in it so is easier to explain. As there are numerous plugins (aliases), it's much easier to gradually introduce new features into how Clojure is used. We only really use clj-new and figwheel-main You can also have both Leiningen and deps.edn configuration in a project, so there is a very simple migration (especially if there are relatively few dependencies).
If you have invested a lot of time learning Leiningen or rely heavily on Leiningen plugins that do some magic, like lein-ring, then switching to deps.edn configuration does require a bit of effort, but you don't have to switch.#2020-03-0805:09Bobbi TowersSounds like a great community! I often feel like an oddball for (besides all the usual reasons) choosing to dive into Clojure without significant experience in another language. It has been an interesting ride, and I would love to talk to others who have similar experiences.#2020-03-1011:49teodorluI'd like to add some comments to the discussion on deps.edn's ease of use:
• To me, deps.edn seems like the right platform to make build tools on. It standardizes a part that can and should be standardized.
• How to build a user-friendly product on top of that can then be tacled as the next problem. It would make sense to converge on standard build tooling, but that, to me, seems like the next problem "up the chain".
The Clojure way has seemed to me like "doing the right thing, and letting it take the time it requires". I'm eager to see what build tooling we can have on top of deps.edn when some time has passed.#2020-03-1011:51teodorluRelevant dichotomy: bottom up/ top down. Bottom-up: small functions, composition. Platform first. Common with Clojurians. Top-down: focus foremost on the end-user experience, and try to improve that incrementally. Common in JS-land.
To me, bottom-up rhymes with simple-first, whereas top-down rhymes with easy-first.#2020-03-1017:59rickmoynihanInteresting observations. I guess neither top-down nor bottom up guarantee either simplicity or ease in the final result; but yeah I think what you’re saying generally holds. And you can also do both together and meet in the middle. As a heuristic though if you wanted simplicity you’d tend to start bottom up, and if you wanted ease you might prefer top-down — though they’re not necessarily mutually exclusive.#2020-03-1109:09slipsetOne of the things that I find interesting in this perspective is that Clojure has slurp/spit whereas I’ve never figured out how to read a file in Java (prior to Java7)
Here it seems like Clojure has taken the top down approach, whereas Java took the bottom up approach with the streams and readers and what not.#2020-03-1109:21rickmoynihanNah, clojure also has the simple approach too with slurp makes the common case easy.#2020-03-1120:09Lone Rangerno sure if this is a tool-deps specific question exactly, does anyone have any suggestions on how to troubleshoot this?
$ clj -R:nREPL figwheel.main -bo min
Cloning:
#2020-03-1120:10Lone RangerI am able to clone it directly with git#2020-03-1120:10Alex Miller (Clojure team)what os?#2020-03-1120:10Lone Rangerred hat >.<#2020-03-1120:11Lone Ranger$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL=""
BUG_REPORT_URL=""
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
#2020-03-1120:11Alex Miller (Clojure team)it uses the ssh agent to obtain identities#2020-03-1120:11Alex Miller (Clojure team)so - is that running?#2020-03-1120:11Alex Miller (Clojure team)ssh-agent#2020-03-1120:11Alex Miller (Clojure team)if so, does it have your identity ssh-add -l#2020-03-1120:11Alex Miller (Clojure team)if not, add it#2020-03-1120:12Lone Rangerchecking#2020-03-1120:12Alex Miller (Clojure team)those cover 90% of these issues#2020-03-1120:13Alex Miller (Clojure team)https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent is a good overview of those steps too#2020-03-1120:14Lone Rangerappreciated. I turned it on and got a new error. Trying to figure out how to add my key now.#2020-03-1120:15Alex Miller (Clojure team)ssh-add ~/.ssh/id_rsa
#2020-03-1120:15Lone Ranger$ clj -R:nREPL figwheel.main -bo min
Cloning:
#2020-03-1120:16Lone Rangerdoes that seem like a different error?#2020-03-1120:16Alex Miller (Clojure team)yeah, that seems bad#2020-03-1120:16Lone Ranger(after following steps of turning on ssh-agent and adding key)#2020-03-1120:16Lone Rangerhahahaha#2020-03-1120:16Alex Miller (Clojure team)might be in how you're running ssh-agent#2020-03-1120:16Lone RangerI believe you#2020-03-1120:16Alex Miller (Clojure team)you've walked off my knowledge in this area :)#2020-03-1120:17Lone Rangeralright! good start. much appreciated#2020-03-1120:19Alex Miller (Clojure team)sudo mount /tmp -o remount,exec internet suggests maybe /tmp needs exec ?#2020-03-1120:20Lone Rangerhmm#2020-03-1120:21Alex Miller (Clojure team)or set a different tmp dir#2020-03-1120:22Alex Miller (Clojure team)clj -.tmpdir=/my/tmp -R:nREPL figwheel.main -bo min#2020-03-1120:23Alex Miller (Clojure team)the git ssh stuff happens through jsch which talks to ssh-agent via jna. it appears jna will make and use stuff in the tmp dir#2020-03-1120:24Alex Miller (Clojure team)different lib, same problem: https://support.cloudbees.com/hc/en-us/articles/215281717-Jenkins-fails-to-start-with-JNA-error and pretty good writeup of the alternatives#2020-03-1120:24Lone Rangerinteresting#2020-03-1120:24Alex Miller (Clojure team)I've never seen this before, so you get a cookie#2020-03-1120:24Alex Miller (Clojure team)🍪#2020-03-1120:24Lone Rangerhahahahahaha#2020-03-1120:24Lone Rangeryesssss#2020-03-1120:26Lone Rangeralmost done setting all this up jeeze#2020-03-1120:27Lone Rangernow I'm getting UnknownHost key, but at least we're making progress in some direction#2020-03-1120:28Lone Rangerof course I see there are 6 ssh-agents running now 😑#2020-03-1120:33Alex Miller (Clojure team)well you might end up running into https://clojure.atlassian.net/browse/TDEPS-104#2020-03-1120:35Lone Rangerhttps://stackoverflow.com/questions/2003419/com-jcraft-jsch-jschexception-unknownhostkey#2020-03-1120:35Lone Rangerthat fixed it#2020-03-1120:35Alex Miller (Clojure team)🍪#2020-03-1120:36Lone Ranger😮 holy snap#2020-03-1120:37Lone Rangerthat's goin on the fridge#2020-03-1120:37Lone Rangerokay last problem, most of the things downloaded#2020-03-1120:38Lone RangerError building classpath. Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.1
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.1
#2020-03-1120:38Lone Ranger@alexmiller is there a way to comment on that issue?#2020-03-1120:39Lone Rangeror just... include that link in there I guess#2020-03-1120:39Lone Rangerspecifically this is what I did:
$ ssh-keyscan -H -t rsa >> known_hosts
#2020-03-1120:42Alex Miller (Clojure team)done#2020-03-1120:50Lone Rangerlast piece of the puzzle was a stale .m2 repo that had wrong perms#2020-03-1120:51Lone Rangerdo you know where the git stuff gets cloned to? I don't see it in .m2#2020-03-1120:56Lone Ranger.gitlibs#2020-03-1120:56Lone Rangersometimes it helps if I just post the questions and then get embarrassed and try harder#2020-03-1121:31Alex Miller (Clojure team)some of these questions were answerable at https://clojure.org/reference/deps_and_cli btw#2020-03-1216:04danielcomptonI'm wanting to write a small Java file and use it in my Clojure app using deps.edn and clj . Is there anything similar to :java-source-paths or another way I can accomplish the same goal? My other option is to write this and create a JAR for it and deploy it somewhere or keep it as a local resource#2020-03-1216:05dominicmNo equivalent. No compilation done by tdeps.#2020-03-1216:06dominicmYou can use javac and have it output to target, and run that shell script before starting clj#2020-03-1216:18danielcomptonMight be easier to rewrite the algorithm to Clojure 🙂#2020-03-1216:39Alex Miller (Clojure team)there are some tools.deps add-ons that can compile as well, like badigeon#2020-03-1216:48mike_ananev@danielcompton try this https://github.com/redstarssystems/pbuilder#2020-03-1216:48mike_ananevit's a wrapper for badigeon#2020-03-1216:49borkdudemaybe a silly question, but why does badigeon need a wrapper? it's already a tool?#2020-03-1216:49mike_ananevalso there is a template for application https://github.com/redstarssystems/app-template#2020-03-1216:50mike_ananev@borkdude no, badigeon is not a ready tool. it is low level api for managing project distributed as sources, which need to be adapted.#2020-03-1216:51borkdudeah ok, I see now#2020-03-1217:52dominicmJavac is pretty good though#2020-03-1220:48Alex Miller (Clojure team)@dominicm if you are suggesting that libraries should maintain a directory full of compiled classes, I think that is a bad idea and you should not do that#2020-03-1220:51Alex Miller (Clojure team)if you want to distribute compiled artifacts, you can already do that by putting them in a jar. we don't typically do that in Clojure because the source is so much more flexible.#2020-03-1220:53dominicmI'm not. I'm saying all of the infrastructure around this, code laying around to do the compilation on startup.#2020-03-1220:53imre@alexmiller if you call (compile 'foo) twice, will the second be almost a no-op as it sees it's already compiled?#2020-03-1220:57Alex Miller (Clojure team)yes#2020-03-1220:58Alex Miller (Clojure team)you can try it in that project I posted if you like#2020-03-1221:03imrecheers#2020-03-1316:35Lone RangerI'm back with more git issues#2020-03-1316:36Lone RangerCloning:
#2020-03-1316:36Lone Rangertrying to help a coworker get it running#2020-03-1316:36Lone Rangerthis is on mac#2020-03-1316:37Lone Rangerso far we've done the following#2020-03-1316:37Lone Rangerif [ -z "$SSH_AUTH_SOCK" ] ; then
eval `ssh-agent -s`
ssh-add
ssh-add -k ~/.ssh/id_rsa # add keys -- did this for all private keys
# add github keys
ssh-keyscan -H -t rsa >> ~/.ssh/known_hosts
fi
#2020-03-1316:39Lone Rangergit clone works#2020-03-1316:40ghadido they also have ~/.ssh/config entries? @goomba#2020-03-1316:41Lone RangerI haven't checked -- is that important?#2020-03-1316:42Lone Rangerthere is no ssh config#2020-03-1316:42Lone Ranger#2020-03-1316:43Lone Rangerwow#2020-03-1316:43Lone Rangerrenamed stuff and it worked#2020-03-1316:44Lone Rangerhow on earth did you know that#2020-03-1316:47Alex Miller (Clojure team)what error are you getting?#2020-03-1316:47Lone Rangerorg.eclipse.jgit.api.errors.TransportException: #2020-03-1316:47Lone Rangerit was tied to the .ssh/config somehow#2020-03-1316:51Lone Ranger(it's working now, thank you @ghadi)#2020-03-1316:51ghadiJSch, the underlying library used, tries to read the .ssh/config for some things instead of consulting the running ssh-agent#2020-03-1316:53Lone RangerI notice that https://github.com/clojure/tools.gitlibs says it supports:
• Full sha (40 chars)
• Prefix sha (sufficiently unique in the repo)
• Tag name
• Branch name
#2020-03-1316:53Lone RangerI don't suppose :git/url supports tags/branches, and/or would a PR to that effect be welcome?#2020-03-1316:54Lone RangerI'm using this extensively and I have 11 (and growing) repos (we're using enterprise git so I can't publish to clojars) and being able to tags/branches would be nice#2020-03-1316:55Alex Miller (Clojure team)so as above tools.gitlibs does support tags or branches#2020-03-1316:55Alex Miller (Clojure team)clj intentionally supports only full shas#2020-03-1316:56Lone Rangerbecause tags/branches are mutable ?#2020-03-1316:56Alex Miller (Clojure team)yes, it totally breaks classpath caching#2020-03-1316:56Lone Rangertags too?#2020-03-1316:56Alex Miller (Clojure team)same#2020-03-1317:13dominicmWell, tags are meant to be immutable. You have to force push them.#2020-03-1317:27Alex Miller (Clojure team)you see how somebody else could break your build with that though, right?#2020-03-1317:45dominicmOh yeah. I wasn't sure how defensive we were being really. I understand that one changes future builds, and one is cryptographically difficult to change future builds. If you want cryptographic protection then it's okay.#2020-03-1318:06dominicmMaven central do that through policy, but there's no requirement that happens though (I think)#2020-03-1316:57Lone Rangergotcha... bummer#2020-03-1316:57Lone RangerI guess including a "force recalculate" directive would be out of the question?#2020-03-1316:57Alex Miller (Clojure team)that does exist (-Sforce) but you would need to basically do it on every clj call#2020-03-1316:58Alex Miller (Clojure team)I would not rule out some way to support this use case, but would want to start from the big picture#2020-03-1316:59Lone Rangergotcha. I'm just trying to make the workflow for this work in a less error-prone way.#2020-03-1316:59Alex Miller (Clojure team)the other similar but different thing people have been poking at is monorepo with embedded local deps#2020-03-1317:00Lone RangerI'm gonna DM you the details that I can't put here on what the situation is#2020-03-1317:01Alex Miller (Clojure team)another you might want to consider is using a private shared repo (either something like nexus or even a private s3 repo)#2020-03-1317:02Lone Rangerso for each of those repos you see ... every time I make a commit, I need to update the shas. And then I need to update the READMEs for each one. And god forbid they have dependencies on one another --- need to do all that by hand#2020-03-1317:02Alex Miller (Clojure team)I get it - I've used a private shared repo in this situation#2020-03-1317:02Lone Rangeris that like a "private clojars"?#2020-03-1317:02Alex Miller (Clojure team)yeah#2020-03-1317:02Lone Rangerok -- that would be a much better solution, yes#2020-03-1317:02Alex Miller (Clojure team)maven repos can be just a file system#2020-03-1317:03Alex Miller (Clojure team)and private s3 repos are exactly that#2020-03-1317:03Lone RangerI did not know that. How do you specify an alternate .m2?#2020-03-1317:03Alex Miller (Clojure team)nexus is actually an app and can do a lot more for you (like mirroring all your external deps in case maven central is unavailable or something)#2020-03-1317:03Alex Miller (Clojure team)you don't need an alternate .m2#2020-03-1317:04Alex Miller (Clojure team)for this#2020-03-1317:04Alex Miller (Clojure team)if you're asking separately, you can use a :mvn/local-repo key in your deps.edn#2020-03-1317:04Lone Rangerinnnteresting#2020-03-1317:04Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli has docs on private s3 access#2020-03-1317:04Lone Rangerokay, thank you, I'll look into that. 🙏#2020-03-1317:05Alex Miller (Clojure team)for deployment, you'll need something to make artifacts and deploy (really just an s3 copy) and there are some clj tools out there https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-03-1317:08Lone Rangerwhich category on tools would that be under?#2020-03-1317:09Lone Rangerversioning and releasing?#2020-03-1317:10Alex Miller (Clojure team)which part?#2020-03-1317:10Alex Miller (Clojure team)making an artifact -> packaging#2020-03-1317:11Alex Miller (Clojure team)but I guess also some of versioning and releasing#2020-03-1317:11Lone Rangerno I meant a tool for where to host#2020-03-1317:11Lone Rangerthe files#2020-03-1317:11Lone Rangeri.e. nexus or similar#2020-03-1317:12Alex Miller (Clojure team)oh, I don't think you'll find that theree#2020-03-1317:12Lone Rangerjust making sure I'm not crazy#2020-03-1317:12Alex Miller (Clojure team)but you could look at any maven repository docs. https://www.deps.co/ is built by Daniel Compton from the Clojure community#2020-03-1317:13Lone Ranger😛 well I am technically a little @goomba#2020-03-1317:13Alex Miller (Clojure team)https://www.sonatype.com/product-nexus-repository#2020-03-1317:13Alex Miller (Clojure team)https://jfrog.com/artifactory/#2020-03-1317:13Alex Miller (Clojure team)those are common ones#2020-03-1317:14Alex Miller (Clojure team)I think there's a thing built into git too?#2020-03-1317:15Lone RangerWHAAAT#2020-03-1317:15Lone Rangeryay that's a tool I'm actually allowed to use!#2020-03-1317:15Alex Miller (Clojure team)https://dzone.com/articles/using-github-as-maven-repository#2020-03-1317:15Alex Miller (Clojure team)just googling, not sure that particular article is the platonic ideal#2020-03-1317:17Alex Miller (Clojure team)there's some tool for this too that I can't remember the name of#2020-03-1317:18Alex Miller (Clojure team)https://jitpack.io/#2020-03-1317:40dharriganI use Nexus OSS#2020-03-1317:40dharriganworks well#2020-03-1318:01Alex Miller (Clojure team)ditto#2020-03-1319:28lilactowna user reported an error trying to use git deps for a lib I wrote:
https://github.com/Lokeh/helix/issues/5#issuecomment-598850716
does anyone know offhand, based on the message, whether this is a user error or something with my lib?#2020-03-1319:32borkdudemaybe he can try -Sforce?#2020-03-1320:00hiredmanthey should delete .gitlibs/libs/helix#2020-03-1320:12hiredmanor maybe .gitlibs/libs/Lokeh/helix#2020-03-1320:15hiredmanit is a jgit error you get when trying to clone into a none empty directory https://github.com/eclipse/jgit/blob/cf60f26d26db2dc4f2d46264d5417e51c4e05c91/org.eclipse.jgit/resources/org/eclipse/jgit/internal/JGitText.properties#L126#2020-03-1515:23teodorluI'd like to comment on the previous discussion of referencing with tags.
• Personal opinion: a utility to rewrite a git ref (HEAD, some-tag, some SHA letters) into a full-blown 40-char SHA might help ease of use. That may be a task for a more high-level tool, though; it seems like more of a UI concern. Perhaps an alias.
• Personal opinions: refs should be resolved immediately, and only SHAs make it into the data
clj -A:install-dep --git-url --git-ref HEAD
I'm guessing this might not be novel ideas, as I feel that tools.deps already embodies this kind of thinking.#2020-03-1516:24Alex Miller (Clojure team)Have you looked at -Sresolve-tags option#2020-03-1516:25Alex Miller (Clojure team)It resolves tags to shas in your deps.edn#2020-03-1618:43teodorluYou're way ahead of me 💯#2020-03-1523:19Jag GunawardanaI want to include my database migrations in my Uberjar. I am using juxt edge with onejar to build. In lein resource-paths did this. I am using ragtime with edn files. The only way I can get this to work is using “jar uf...” to update the jar after building it. There must be some deps .edn magic?#2020-03-1523:41seancorfield@jag I've no idea how Juxt/OneJar handle that but with depstar you would just use an alias that adds the resources (via :extra-paths) when you ask it to build the JAR.#2020-03-1523:42seancorfieldWe build uberjars at work using depstar that include DB migrations, HTML templates, and all sorts of non-code stuff.#2020-03-1608:20dominicm@jag they should be included by default unless they're not in a default alias. If they aren't you can put "-A:alias" to pack.#2020-03-1615:59Jag GunawardanaThanks for the suggestions, looks like moving to depstar seems to be the best long term solution. I like the approach taken in juxt edge project, but the docs around build are a bit too concise.#2020-03-1616:40dominicmSorry about that, is there somewhere I can improve them?#2020-03-1808:53Jag GunawardanaI think a few more use cases for some of the tools, or general cases would work e.g. “How to include resources in the JAR file”. I ended up using Sean’s fork of depstar with edge.
clojure -A:prod:depstar -m hf.depstar.uberjar myjar.jar
and running
java -cp myjar.jar clojure.main -m edge.main
That seems to work fine.
I can add something to the docs on this if it helps?#2020-03-1810:29dominicmI'd love to take a look at a pr if you have an idea for this :)#2020-03-1915:37Eamonn SullivanHi, probably a quick question: Can I use a private github repo as a dependency in deps.edn? I'm trying, but getting Authentication is required but no CredentialsProvider has been registered. If it is supported, how do I provide my credentials (a github access token, I guess)?#2020-03-1915:39ghadiyou can, but through ssh:// + an ssh-agent, not via the https:// url#2020-03-1915:45Eamonn SullivanI've been trying /cdn-cgi/l/email-protection links, which work from the command-line (`git clone /cdn-cgi/l/email-protection`), but when I try that in a deps.edn I get invalid privatekey. I must be missing something simple...#2020-03-1915:50Eamonn SullivanSorry, I thought I hadn't run ssh-agent and it was working, but I was actually in the wrong directory. 😞#2020-03-1915:53ghadiplease describe exactly what you did, and what the result was#2020-03-1915:57Eamonn SullivanI added to the dependencies. My private repo is the last one listed below. I then try to run clj and the result is Error building classpath. #2020-03-1916:01ghadiwhat is the output of ssh-add -l#2020-03-1916:01ghadiand is your key an ed25519 key?#2020-03-1916:02Eamonn Sullivanhere it is.#2020-03-1916:03Eamonn SullivanAnd, no, it's not an ed25519 key, as far as I can tell (they both say RSA).#2020-03-1916:06ghadiif you have a bunch of entries like defaults in $HOME/.ssh/config consider moving that file out of the way and retrying#2020-03-1916:07ghaditemporarily#2020-03-1916:07ghadithe ssh library prioritizes that file even when there is an ssh-agent present#2020-03-1916:09ghadi(unfortunately)#2020-03-1916:09Eamonn SullivanAh, that worked. I need those entries in config for various things, so I wonder what I need to do to not confuse things.#2020-03-1916:12Eamonn SullivanBut, anyway, I can now build, so that's a big improvement. Thank you!#2020-03-1916:13ghadimy pleasure#2020-03-1921:39samI’m getting an unexpected transitive dependency resolution in my deps.edn project. This is what my dependency tree looks like:
my-project -> direct-dep -> trans-dep1 (version 1)
\
-------> trans-dep2 -> trans-dep1 (version 2)
• direct-dep is a git dependency
• direct-dep has a pom.xml file that serves as the manifest
The problem is that trans-dep1 (version 2) is on the classpath instead of version 1#2020-03-1921:41samAm I correct in thinking that version 1 of trans-dep1 should be the expected resolution (assuming I’m describing this correctly and am not doing something dumb I’ve overlooked)#2020-03-1921:43ghaditools.deps prefers latest versions first#2020-03-1921:52samOh thanks, I totally missed that.#2020-03-1921:53samIs there a rationale for that written somewhere?#2020-03-1921:56Alex Miller (Clojure team)the long form is Rich's "Spec-ulation" talk https://www.youtube.com/watch?v=oyLBGkS5ICk#2020-03-1921:57Alex Miller (Clojure team)and I've done a couple talks that cover it as well, most recently (https://www.youtube.com/watch?v=7CM7Ef-dPWQ)#2020-03-1921:57samAwesome, thanks. I appreciate it.#2020-03-1921:58Alex Miller (Clojure team)I don't know if I have a good written thing to point at#2020-03-1922:06samNp, I think I’ve seen the spec-ulation talk. I think I vaguely remember this. In any case, I can imagine an argument that chosing the latest version is simpler, easier-to-understand behavior compared to the behavior in lein/mvn that I was expecting.#2020-03-1922:25seancorfieldThe latest episode of the ClojureScript Podcast is very good https://clojurescriptpodcast.com/ -- Alex talks about tools.deps and covers this specific issue...#2020-03-1922:30Alex Miller (Clojure team)oh yeah, that too. we recorded it months ago so it's hard to remember that. :)#2020-03-1922:40seancorfieldIt's a really good all-round discussion about tools.deps and the new CLI stuff. Definitely worth pointing folks at for any Qs regarding the "philosophy" of it (including the SemVer stuff).#2020-03-2015:38grzmI have a project which has a deps.edn dependency to a private https://github.com repo, Clojure deps is failing to pull the dependency with:
Cloning:
Seems like it’s an issue with my SSH keys, but I’m a bit puzzled as I can clone this same dependency from https://github.com from the command line with git clone without any problem, and the necessary keys show up when using ssh-add -l. Hints on what I might be doing wrong?#2020-03-2015:40Derekhttps://clojurians.slack.com/archives/C6QH853H8/p1584634015008000#2020-03-2023:53colinkahnIs this the best guide to use for getting running on Windows? https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#2020-03-2109:16practicalli-johnthe #clj-on-windows channel can also provide help with this#2020-03-2201:22colinkahnOh nice, thanks!#2020-03-2023:57sogaiuit definitely has relevant info that people have referenced here from time to time.
the scoop option mentioned at the bottom of the page appears to be somewhat popular. would recommend checking that out.#2020-03-2023:58sogaiui have been using that successfully.#2020-03-2123:29practicalli-johnHow would I include the Java sources in the :path of my deps.edn configuration?
The sources are currently in a src.zip file, from the install. Is it as simple as using "src.zip" to the :path configuration? Or would I need to extract and a the specific directories relative to the Clojure project?
Also wondering how to include these sources to multiple projects, either using an alias in ~/.clojure/deps.edn or creating a symbolic link to the src.zip file (or extracted directory).#2020-03-2123:49andy.fingerhutNeither the clj nor clojure commands will do anything with Java sources, unless you somehow use them to run a program that looks for and does something with Java source files.#2020-03-2123:51andy.fingerhute.g. if you are using something like mevyn program to compile Java source files: https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-03-2201:07potetmOR you can compile the src w/ javac into target/classes and add "target/classes" to your :paths#2020-03-2206:40mike_ananev@U05254DQM try https://github.com/redstarssystems/pbuilder#2020-03-2207:34dominicmI think the src.zip is referring to the ones that come with the jdk. They're used by cider to allow you to jump to source on java code.#2020-03-2207:35dominicm@U05254DQM I think zip files are fine on the classpath. A :local/root should be fine.#2020-03-2208:41practicalli-john@U09LZR36F that sounds what I am looking for, thank you.#2020-03-2211:15practicalli-johnIt works. I have a deps.edn as follows
{:path
["src" "resources"]
:deps
{org.clojure/clojure {:mvn/version "1.10.1"}
;; Java 11 sources
java-sources {:local/root "/usr/lib/jvm/openjdk-11/lib/src.zip"}}}
I can now browse java sources in Spacemacs with clj-find-var . Thanks everyone.#2020-03-2213:09dominicmRemember that the location will vary for everyone :)#2020-03-2211:14borkdudeEnjoyed the podcast Alex, thanks for taking the time! https://clojurescriptpodcast.com/#2020-03-2211:19pithylessTrying to debug this kind of warning:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:~/.m2/repository/org/slf4j/slf4j-nop/1.7.25/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:~/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
When I run clojure -Stree I do not see any mention of slf4j-nop. Any ideas on how I can find out what is pulling in this dependency (so I can add an :exclusion)?#2020-03-2211:26pithyless^ Nevermind, PEBCAK.#2020-03-2414:07Jakub Holý (HolyJak)Hi! I have added :mvn/repos {"my-company" {:url "...."}} but I want deps to also resolve dependencies from my maven local repo. How do I add it, preferably w/o hardcoding the path of my home directory?#2020-03-2414:08Alex Miller (Clojure team)it will already do that#2020-03-2414:08Alex Miller (Clojure team)assuming you mean ~/.m2/repository - that's actually not a repo, but a local cache of external repos#2020-03-2414:10Alex Miller (Clojure team)as a cache, it's checked first though so you can install stuff locally that you never deploy and it will be used#2020-03-2414:15Jakub Holý (HolyJak)thanks, that is exactly my use case. I found out that the jar was wrongly published to the local cache.#2020-03-2423:29athomasoriginalTerminology is always as touchy topic, but would it be fair to refer as tools.deps.alpha as a package manager? If not, how would you describe the distinction?
The goal for myself to group clojure tools into buckets that fit my mental model. I am happy to update my model, but I just want to see if my current understanding fits 🙂#2020-03-2423:36andy.fingerhutI do not know of any precise definition of the term "package manager" that everyone would agree with, across different programming languages, but it seems to me that tools that are called by that name often do more things than tools.deps.alpha does.#2020-03-2423:36andy.fingerhutThis is too many words, but the http://clojure.org page introducing tools.deps.alpha describes it thus: "a library providing a functional API for resolving dependency graphs and building classpaths that can utilize both Maven and other providers of code or artifacts"#2020-03-2423:37andy.fingerhutPerhaps calling it a "dependency resolver" briefly captures the essentials?#2020-03-2423:37andy.fingerhuthttps://clojure.org/reference/deps_and_cli#2020-03-2423:40athomasoriginalMy thoughts exactly, Andy. I agree that it would be a stretch to compare it as this would lead to further comparisons to other tools like npm#2020-03-2501:44Alex Miller (Clojure team)tools.deps is fundamentally about making classpaths, and incidentally about transitive dependency resolution#2020-03-2501:45Alex Miller (Clojure team)I would not call it a package manager per se#2020-03-2501:45Alex Miller (Clojure team)clj is about running programs#2020-03-2510:21rickmoynihanI’ve been meaning to speak about this issue for a while; not sure if others have run into it or not.
I have a private library in a private github repo; that we’re requiring via tools.deps.
Every so often, in circumstances I don’t fully understand, but it usually appears to be when I do things like change branches (and I suspect the sha in that dep has changed) I get the following exception:#2020-03-2510:21rickmoynihan#2020-03-2510:25rickmoynihanWhen it occurs I usually spot that my ssh-agent is somehow no longer running; so restart that but then get:#2020-03-2510:25rickmoynihan#2020-03-2510:27rickmoynihanThen clearing the lock file fixes it#2020-03-2512:19Alex Miller (Clojure team)We’ve had people report this but I’ve never been able to repro it#2020-03-2514:00rickmoynihanIt happens to me quite often#2020-03-2514:05rickmoynihanthere are a few other things that might be relevant:
1. some-private-lib is actually included in our deps.edn multiple times; with different :deps/root s
2. In dev I start a bunch of services with overmind a Procfile process manager. It boots them in parallel. One of those services is my clj repl; another is shadow-cljs that references the same deps.edn; so when resolving deps two processes are occaisionally concurrently resolving/fetching deps in parallel.#2020-03-2514:07rickmoynihanI suspect 2 might be problematic… so sometimes try and resolve the deps first; then run overmind… though I do at times forget — it usually seems to work without any apparent problems, though it does seem a bit dangerous on my part.#2020-03-2514:09Eamonn SullivanI saw the first error just the other day. I had two small projects, one using a github ref using one hash (that checked out the repo fine) and then moved to another with a newer hash (but the same repo) and saw the above error when trying to build. I went into the directory and deleted the repo and tried again and it worked. Is that supposed to work?#2020-03-2514:11rickmoynihanI think I’ve seen it a similar scenario to that too#2020-03-2514:11rickmoynihanbut this one is definitely within a single project — though changing branches; with a changing sha on the dep which probably amounts to the same thing#2020-03-2514:16Alex Miller (Clojure team)with the newer clj's, resolution happens in parallel, so even in a single project, you could encounter that#2020-03-2514:19Alex Miller (Clojure team)you can turn that off with -Sthreads 1#2020-03-2514:42rickmoynihanOk but this is in a single project; but with two concurrent processes potentially resolving the same deps with tools.deps. Is that safe if the processes aren’t coordinated?#2020-03-2515:12Alex Miller (Clojure team)don't know#2020-03-2512:20Alex Miller (Clojure team)The first error that is#2020-03-2513:36Alex WhittHi,
I love the ability of tools-deps to target git repos at a certain hash. I've operated under the assumption that the hostname of the repo will stay the same forever for private repo dependencies. But that's proven not always to be the case.
If a repo moves to a new hostname, it breaks every historical project state that referenced it.
Yocto has a feature called a "repo manifest" that allows the repository configuration to be a lot more fluid, and effectively solves this problem. Does tools-deps have an equivalent option?#2020-03-2513:38Alex Miller (Clojure team)no#2020-03-2513:39Alex Miller (Clojure team)what is the repo manifest?#2020-03-2513:39Alex Miller (Clojure team)got a link or something?#2020-03-2513:47Alex WhittI think this is what I'm talking about:
https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#repo-fetcher
And an example:
https://github.com/texierp/yocto-bsp-manifest/blob/master/default.xml#2020-03-2513:49Alex WhittI suppose there are workarounds for the issue at the OS-level, but this would be a pretty slick feature.#2020-03-2514:05dominicm@alex.joseph.whitt is this like having a proxy or alternative source for a repo? I'm struggling to understand.#2020-03-2514:16Alex WhittI'm not a Yocto user myself, but I believe the idea is that you can have a locally-defined manifest of repo sources. For each dependency, Yocto will look through them all in order and try to resolve the dependency in each.
That solves the problem of trying to build an old version of your project after your git server hostname changes. The problem is really only sticky when you have multiple internal Clojure dependencies that all point to the same hostname. You have to go through each dependency, branch off of the contemporary commit, update the hostname, commit, then update the hashes on projects that pull in that dependency. Kind of hellish unless you redirect the old hostname at the DNS level or something.#2020-03-2514:21Alex Miller (Clojure team)I mean, you are defining a lib and a coordinate (with git config) in your deps.edn. You can change that coordinate/git config but keep the lib name to point it elsewhere.#2020-03-2514:26Alex Miller (Clojure team)I guess there could be a layer of indirection added in front of the git url#2020-03-2514:26Alex Miller (Clojure team)needs a lot more thought#2020-03-2514:51dominicmAnother use case for this: the bank I work at has a github mirror which is both faster and allows clones from machines that otherwise can't.
This would be useful for us.#2020-03-2610:31daveliepmannHi folks, I've got a head-scratcher with transitive dependencies. I found a workaround but it doesn't seem right.
We have a new library which depends on a Java lib living in the BeDataDriven public repository: http://docs.renjin.org/en/latest/library/project-setup.html
We wish to consume this library in a separate project. However, we get an error when starting a REPL via clojure CLI (or CIDER):
Error building classpath. Could not find artifact org.renjin:renjin-script-engine:jar:3.5-beta43 in central ()
This error goes away if I include the BeDataDriven repo to the consumer's :mvn/repos. Like I said, this is a fix but it doesn't feel correct. Is this by design?
I've created a minimal reproduction (three tiny files) of the issue in this repo: https://github.com/daveliepmann/minimal-transitive-deps-example The workaround is commented out. I get the error above when running clojure from the command line in min-consumer directory. I am running the latest clojure from brew.
The actual lib is at https://github.com/appliedsciencestudio/rdata/ and the consumer deps.edn fails with both git and local deps:
clojure
appliedsciencestudio/rdata ;; {:local/root "/Users/daveliepmann/src/forks/rdata"}
{:git/url ""
:sha "6dff85aa7b83a9ce85eb8d70b0672a3adf0dc7f2"}
Prior art: this issue looks like it could be related to https://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-127?filter=allissues&orderby=priority%20DESC&keyword=transitive but that seems to have been fixed.#2020-03-2610:45dominicmI don't think repos are conveyed across deps.edn projects. They're considered local like :aliases#2020-03-2610:47daveliepmannRight, but the consumer lib doesn't know, or in my mind shouldn't care, that the lib it's requiring uses a repo that's not Maven or Clojars.#2020-03-2610:47dominicmhttps://clojure.atlassian.net/projects/TDEPS/issues/TDEPS-46?filter=allissues&orderby=priority%20DESC&keyword=transitive @daveliepmann#2020-03-2610:48daveliepmannYup that looks like it, thanks. I missed it in my Jira search.#2020-03-2610:48dominicmI can think of a couple cases where that's not true. Banks being one, repos are not transparent there. There's also the risk of someone adding a repo that them causes shadowing. I'm not sure how maven works for this.#2020-03-2611:45rickmoynihanI might be misunderstanding the problem here - but isn’t it a security risk for a transitive dep to specify a new :mvn/repo; if we’re not careful might it not cause tools deps to fetch other deps from there too?
Granted 3rd party deps are always a security problem; but this suggested change in behaviour seems like it would obfuscate things further and potentially makes auditing harder?
I think the usecase is interesting; but would it not be safer for libs containing :mvn/repos to cause tools.deps to fail with a message saying “add X to :mvn/repos”?#2020-03-2611:49rickmoynihanAlso couldn’t it open up new attack vectors? e.g. an attacker could possibly set it to a private repo you own; and cause you to inadvertantly package secret deps in published binaries etc? Kind of like a CSRF attack.
Admitedly this is a fanciful attack — and it would require you to depend on an attackers library either explicitly or transitively -- but attacks usually are elaborate.#2020-03-2611:55dominicmThat's right. But maven allows this, so it might already be the expected model.#2020-03-2612:22Alex Miller (Clojure team)Yes, this should work, just not handled right now#2020-03-2613:00ghadiThat attack vector sounds realistic#2020-03-2613:09Alex Miller (Clojure team)it needs some thought. imo, this is an under-considered part of the Maven model in general. repos are identified by name, and those names are just totally arbitrary and unique solely by rarity/convention.#2020-03-2613:10Alex Miller (Clojure team)but Maven does certainly let you add repos in transitive deps for this case. I'm not sure if their use is "scoped" to that dep and below though#2020-03-2613:46rickmoynihanyeah was wondering about this too… seems less risky to scope down the subtree; than apply globally.#2020-03-2613:12Alex Miller (Clojure team)they are somewhat shielded from this by using a top-down, breadth-first, first one found wins strategy as "parents" in the tree have already been resolved before the transitive repo is encountered#2020-03-2613:16Alex Miller (Clojure team)that's kind of also true with tools.deps (resolution happens top-down), but selection can be modified during the traversal and the "source" is not tracked right now in any way#2020-03-2703:39dorabUsing Sean Corfield's fork of depstar, and running the command clj -A:uberjar x.jar, I get
Building uber jar: x.jar
{:warning "clashing jar item", :path "META-INF/services/com.fasterxml.jackson.core.JsonFactory", :strategy :concat-lines}
{:warning "clashing jar item", :path "META-INF/services/com.fasterxml.jackson.core.JsonFactory", :strategy :concat-lines}
I know it is just a warning, but what do I do to fix it? My deps.edn is
{:paths ["src"] ;until TDEPS-52 is fixed
:aliases
{:test
{:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "RELEASE"}}}
:runner
{:extra-deps
{com.cognitect/test-runner
{:git/url ""
:sha "76568540e7f40268ad2b646110f237a60295fa3c"}}
:main-opts ["-m" "cognitect.test-runner" "-d" "test"]}}
:deps
{
cheshire {:mvn/version "5.10.0"}
http-kit {:mvn/version "2.3.0"}
org.clojure/clojure {:mvn/version "1.10.1"}
}
}
If this is not the appropriate channel, please redirect me. Thanks.#2020-03-2703:49seancorfieldYou don't do anything to fix it. It's just warning you, so you know that the same file is being copied into your JAR multiple times (and it is handling it by concatenating the files). Jackson is a horribly problematic library -- it is the cause of most of the version conflicts we see people run into these days but unfortunately it's also fairly ubiquitous so...#2020-03-2703:50seancorfieldI'm a bit surprised you're seeing the message twice -- that implies three copies of the file are being found on the classpaths...#2020-03-2703:52seancorfieldThe 1.0.x version of depstar (currently on master) will have a -S / --suppress-clash option to hide that warning message(!) but you can currently use the -v / --verbose option to learn more about what it is bringing in @dorab#2020-03-2704:01seancorfieldOK, I've repro'd based on your deps.edn and it looks like each of the three Jackson libraries being pulled in by Cheshire all contain that same file.#2020-03-2704:08seancorfieldYup, seems like each of the Jackson JAR files (core, databind-cbor, and databind-smile) all contain that file and it has a class name in it that matches one in the library it comes from. I'm fairly certain this is harmless and "expected behavior". Perhaps @ghadi can confirm when he is next around, since he's the original author of the first version of depstar...?#2020-03-2704:09seancorfield(this seems like another reason to try using clojure.data.json instead of cheshire 🙂 )#2020-03-2704:17dorabThanks @seancorfield. I will try to use clojure.data.json instead.#2020-03-2704:21dorabIf the same file is being requested multiple times, could depstar just include it once? Sorry if I'm being naive.#2020-03-2704:31seancorfieldNo, the same named file is included in three different libraries.#2020-03-2704:32seancorfieldAs I said above, Cheshire pulls in Jackson, and that means you get its core, cbor, and smile libs (dataformat, not databind, my bad -- rest of it still applies). Each of those three libs includes the same file. depstar concatenates the lines in each file to create a single file in the resulting JAR.#2020-03-2704:36dorabGotit. Thanks.#2020-03-2704:37seancorfieldInterestingly, dataformat-smile and dataformat-cbor are both deprecated and archived. Ah, looks like dataformat-binary is the replacement for both of those now. Sheesh, that is one weird, troublesome library...#2020-03-2715:47fabraoHello all, how to convert this lein uberjar
:main ^:skip-aot app.core
:uberjar-name "project.jar"
:profiles {:uberjar {:aot :all}}#2020-03-2715:47fabraoto deps.end alias?#2020-03-2716:31seancorfield@fabrao https://github.com/seancorfield/depstar#2020-03-2716:31seancorfieldIt has an example in the readme
# generate pom.xml (or create it manually)
clojure -Spom
# build the uberjar with AOT compilation
clojure -A:depstar -m hf.depstar.uberjar MyProject.jar -C -m project.core
# Main-Class: project.core
java -jar MyProject.jar
#2020-03-2716:31seancorfield(use project.jar instead of MyProject.jar and app.core instead of project.core)#2020-03-2716:33seancorfieldYou'll need to edit the pom.xml generated by clojure -Spom to add more stuff. Take a look at a pom.xml file generated by clj-new (if you created your project with a recent version of clj-new and the app template, then the pom.xml file will probably be almost 100% of what you would need for depstar and deployment to Clojars anyway).#2020-03-2717:22fabrao@seancorfield I tried -A:uberjar from your dot-clojure but I didn´t know about pom.xml#2020-03-2717:23fabraoso I creat project.clj to generate jar#2020-03-2818:21rgmapologies if this is in the docs, but is there a way to give a path to a deps.edn, instead of assuming it's in the current working directory?#2020-03-2818:22rgm(I'm coexisting with a bunch of Python and I'd prefer to have one deps.edn someplace else and just use long Makefile commands to get at the clojure libs)#2020-03-2818:25Alex Miller (Clojure team)no#2020-03-2818:34rgmdrat.#2020-03-2818:34rgmsymlinks it is ¯\(ツ)/¯#2020-03-2818:36rgm(or rather, I'll just cd other_place && clojure ... to keep my Python colleagues from tripping over the clojure stuff).#2020-03-2818:40seancorfield@rgm One thing you can try is specifying CLJ_CONFIG=/path/to/folder clojure ... where your deps.edn file is /path/to/folder/deps.edn -- this treats that deps.edn file as the user-level one, ignoring your ~/.clojure/deps.edn (and it would still pick up a deps.edn file in the current folder).#2020-03-2818:40seancorfield(so you'd get system deps, CLJ_CONFIG deps, current/project deps)#2020-03-2818:41rgmoh, that's a great trick ... I don't have a ton of deps#2020-03-2818:41rgmdon't really even need a current folder one.#2020-03-2818:41rgm(which is what got me going down this path, to keep things tidy).#2020-03-2818:41seancorfieldWe use this approach in our monorepo at work, so we can have a "standard" user-level deps plus our current subproject deps (and ignore any developer-level user deps on different machines).#2020-03-2818:46rgmright, and because it's env, I can just manage it through settings in Makefile, or .env, or direnv, or ... this is great.#2020-03-2818:54rgm(Can I just say you and Alex are killing it, and it’s much appreciated?)#2020-03-2819:13borkdude@rgm fwiw deps.clj (an unofficial clj script re-implementation, or the binary version deps.exe) does have an option for this#2020-03-2819:14borkdudehttps://github.com/borkdude/deps.clj#extra-features#2020-03-2819:41rgmI hereby include you in the "killing it" set.#2020-03-2919:04joseis there an example about how to add a java file to my clojure project? In leiningen there is a :java-source-path , but I think there is nothing similar for tools-deps#2020-03-3005:20mike_ananev@U4NGX0FHN also you may try this https://github.com/redstarssystems/pbuilder#2020-03-3013:38josethanks, I didn't know about pbuild, maybe add it to https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging ? I think I found an error on pbuilder docs, I'm opening a github issue#2020-03-3000:16sogaiu@jlle iiuc, compiling java is not in the scope of tools.deps -- may be something under here might be relevant? https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging#2020-03-3000:32potetm@jlle You can also call javac from the cli.#2020-03-3013:34joseI found that badigeon / pbuilder are good for my use case#2020-03-3014:12borkdude$ clj -Sdeps '{:aliases {:foo {:main-opts ["-e" "(ns,foo,(:require,[clojure.java.shell,:refer,[sh]])),(sh,\"javac\",\"Foo.java\"),(shutdown-agents)"]}}}' -A:foo
😜#2020-03-3014:13borkdude(it took me only 10 minutes to figure out the correct nesting and seancorfield-commas on the command line)#2020-03-3109:54borkdudehttps://theronic.github.io/depsy/#2020-03-3116:00jjttjjIs there a way to set the filename clash strategy with depstar? I'd like my data_readers.cljc to be merged#2020-03-3116:00jjttjjActually, it looks like that is done for the .clj but not cljc https://github.com/seancorfield/depstar/blob/9860bd5b510418fcc24d2660c8526bec7d53e7c1/src/hf/depstar/uberjar.clj#L61#2020-03-3116:48seancorfieldFeel free to open an issue on GitHub to add support for data_readers with additional file extensions. Seems like a good idea to add that. @U064UGEUQ#2020-03-3117:12jjttjjWill do!#2020-04-0108:34pezWhat is the equivalent of Leiningen's :local-repo "local-m2" in deps.edn?#2020-04-0108:38delaguardo:mvn/local-repo key#2020-04-0108:58pezIt doesn’t seem to work as I expect it to. I don’t know what to put as the url. #2020-04-0112:18Alex Miller (Clojure team)It takes a string (a path)#2020-04-0112:19Alex Miller (Clojure team)Which can be either relative or absolute#2020-04-0112:21pezThanks. I shall experiment some more, maybe I am not looking at the right places for the right result... Also, right now I'm using lein-tools-deps so maybe that complicates things.#2020-04-0112:22Alex Miller (Clojure team)I suspect that probably won’t work#2020-04-0108:38kszabohttps://github.com/practicalli/clojure-practicalli-content/issues/24#2020-04-0112:21Alex Miller (Clojure team)This seems wrong as it is relative to the current directory. So if you are sharing it, you need an absolute path#2020-04-0111:19cursorkI think I'm just having a stupid moment... But I'm not sure where I can use :jvm-opts in deps.edn . I'm currently using it in a dev alias and calling clj -A:dev which works fine. But the entire purpose of this code is to interoperate via JNI to an underlying dynamic lib, so ideally I'd like to have :jvm-opts ["-Djava.library.path=..."] always be set to a path I control.
Using System/setProperty doesn't seem to have an effect as I guess the initial java.library.path at startup is used.
Is the only option to have an alias for the path, and always invoke it? And then other aliases as usual?#2020-04-0111:34cursorkMaybe I'm answering my own question... If there's a .dylib or .so I'm expecting to use, the location should be specified for each env it's deployed on. As it could be different between developer's machines as well. So maybe we all have a ~/.clojure/deps.edn with some profile with the correct path. A common name of :shared-libs or something might make that easy to work with.#2020-04-0111:44kszaboyeah, jvm-opts only works with aliases#2020-04-0111:46kszabohttps://clojurians-log.clojureverse.org/tools-deps/2018-11-12#2020-04-0111:46kszabo> https://clojurians.slack.com/team/U064X3EF3https://clojurians-log.clojureverse.org/tools-deps/2018-11-12/1541986798.430300It’s missing, no philosophical objection#2020-04-0111:47cursorkThanks for the confirmation 🙂#2020-04-0112:22Alex Miller (Clojure team)I suspect that probably won’t work#2020-04-0212:59rfhayashiHi! I'm trying to invoke tools.deps directly from clojure (more specifically clojure.tools.deps.alpha/resolve-deps) but I get a
Cannot access s3p://... using the registered transporter factories: HttpTransporterFactory, S3TransporterFactory, FileTransporterFactory
error. If I run a clj -Sdeps with the same content it works fine. Is there any library that I should include in the classpath to make s3 repos work? Or any environment variable or system property that I should set? I'm using org.clojure/tools.deps.alpha {:mvn/version "0.8.677"}.#2020-04-0213:06Alex Miller (Clojure team)@rfhayashi did you try s3:// ? that's the only documented protocol for s3 access (s3p did work in some older versions but has never been supported)#2020-04-0213:07Alex Miller (Clojure team)you should not have to do anything special to get s3 to work#2020-04-0213:16rfhayashi@alexmiller it does work. thanks!#2020-04-0213:54kszabois there any support for modifying the bootclasspath, like:
https://github.com/sunng87/lein-bootclasspath-deps
does for leiningen? It would be nice to use some special case of :boot-deps so that I don’t have hack it together with downloading jar files locally from maven and using :jvm-opts directly#2020-04-0213:54kszabothis seems to be in the focus area of tools-deps of invoking java with the correct dependencies ready#2020-04-0214:02Alex Miller (Clojure team)you should not modify the bootclasspath in my opinion, so I'm not going to make that any easier#2020-04-0214:06kszabothere are legitimate use-cases, like enabling alpn for jdk8 for http2:
https://www.eclipse.org/jetty/documentation/current/alpn-chapter.html
I agree that it should be avoided when possible, but now I need to hack around deps.edn because of the lack of support#2020-04-0214:14Alex Miller (Clojure team)you can make an alias that produces the boot classpath you need, then use clj -Spath to generate the classpath from it, so there is no need for manual jar downloading#2020-04-0214:14Alex Miller (Clojure team)and then put that path in your :jvm-opts#2020-04-0214:15kszaboclj -Spath would produce an absolute path to my main maven repo, which is hosted in my home directory#2020-04-0214:16kszabomeaning the resulting :jvm-opts needs to be scripted on each machine#2020-04-0214:16Alex Miller (Clojure team)an alternative would be to set :mvn/local-repo to fetch it into a subdirectory of the project, then use relative paths in the jvm-opts#2020-04-0214:17kszabowouldn’t that apply to all maven deps? I haven’t used that one yet#2020-04-0214:17kszaboI don’t want to download maven again 😂 #2020-04-0214:18Alex Miller (Clojure team)you would just use this for what you need in the bootclasspath#2020-04-0214:19kszaboI see#2020-04-0214:19kszabothanks for the help#2020-04-0221:29jaimeI'm using clojure cli / deps.edn. Is there a way to define env vars following the 12 factor app?
Looking into https://github.com/weavejester/environ but it seems like it does not support deps.edn (for instance, if your using lein. you can have .lein-env)#2020-04-0221:35seancorfield@jaime.sangcap (System/getenv "varname") is all you need to get an environment variable's value so I'm not sure what you're asking in the context of CLI/`deps.edn`?#2020-04-0221:36seancorfieldPerhaps "Can you have environment variables in your deps.edn?" No. It's a pure EDN file, not executable code, and has no substitutions made.#2020-04-0221:43jaimeI'm wondering if there is library that can load env var from file. And you can configure where the file is in the deps.edn.
Right now I'm only using intellij to set env vars.
What I'm thinking is there should be a example file (checkin to version control) that has all the required env var
Then locally, the library will check the example file against your env file for missing env var#2020-04-0315:35jeff.terrellThis might be at least partially what you're looking for: https://github.com/yogthos/config#2020-04-0316:21jaimecool! Yes I think can use this. Thanks 🙂#2020-04-0221:44jaimeBefore loading them and throw error is you missed to set something#2020-04-0221:44seancorfieldSorry, I have no idea what you mean by any of that.#2020-04-0221:45jaimeThanks! I think I'm mixing different things#2020-04-0221:46seancorfieldEnvironment variables are set at the O/S level before a process is started -- I would imagine IntelliJ is setting the environment variables before starting the REPL or run process.#2020-04-0221:47seancorfieldYou can't set environment variables from inside a process. You can set JVM properties tho'...#2020-04-0221:49seancorfieldBTW environ is just a library. It can be used from any process run by CLI/`deps.edn`.#2020-04-0221:49seancorfieldIt just happens to have existing plugins for lein and boot that call it in particular ways.#2020-04-0221:54seancorfield(but I would caution you against storing anything in deps.edn that isn't part of the official syntax -- deps.edn is not intended to be a "general configuration file -- which project.clj and build.boot tend to be)#2020-04-0307:38practicalli-john@jaime.sangcap Storing environment variables in a file that you then version control is the best way to add a huge security hole in your application and isn't following 12 factor app approach.
If you are using a service like Heroku, then that service manages the Environment variables securely for each application and you can the just read them in from Clojure as Sean mentioned or using environ.#2020-04-0308:56jaimeI agree. Thank you for pointing it.
However, what I meant was commit in version control an example file. Let's say .env.example , this will contain all the env vars that are needed by your application.
APP_PORT=8000
DATABASE_URL=
AWS_ACCESS_KEY_ID
SOME_PATH=/temp
Notice that env vars don't have values, but are there so that when other devs can copy the .env.example and create .env and provide the values.
I think an advantage of this one is you can check the .env file if there are missing keys from .env.example and throw an error before the app starts to prevent it from running in unpredictable state.#2020-04-0307:41teodorluOption: store config defaults in an .edn file, and create a custom read-config function that reads an env var if it is present, otherwise defaults to the file.
Then you can set defaults, and override them with environment variables.
Note: env vars are named by strings. We often like to use keywords as identifiers in Clojure. You might want to set a naming scheme that lets you query with keywords rather than strings.#2020-04-0308:28dominicmSounds like aero :)#2020-04-0309:37teodorluAero seems to make all of this possible to do from within the config file. README[1] excerpt:
{:port #or [#env PORT 8080]}
[1]: https://github.com/juxt/aero#2020-04-0309:52jaime@seancorfield Yes I think I got that one. Given I have this deps.edn
{:deps
{org.clojure/clojure {:mvn/version "1.9.0"}}
:paths ["src"]
:aliases {:dev {:extra-paths ["dev"]
:extra-deps {nrepl/nrepl {:mvn/version "0.6.0"}}
:main-opts ["-m" "nrepl.cmdline"]}}}
I could imagine intellij when you add env var DATABASE_URL to it, it will just run using DATABASE_URL=jdbc-url clj -A:dev
Looking at environ code, all functions are private 🙂
Maybe I could just do this. (copied from this convo https://github.com/weavejester/environ/issues/14)
(require '[environ.core :as environ])
(defonce env (merge (read-my-custom-config-file) environ/env)
Where config file is in .gitignore and will be used in local dev.
Thanks to @teodorlu for the idea#2020-04-0312:35Alex Miller (Clojure team)I’ve done something very similar to that in a real app#2020-04-0313:28jjttjjIs there a "recommended" way to have a repl start in a particular namespace and load that file?#2020-04-0313:29jjttjjI've been trying various arrangments of this but none seem to really be what I'm going for
:main-opts ["-i" "src/jt/jt15.clj"
"-e" "(in-ns,'jt.jt15)"
"-r"]#2020-04-0313:31jjttjjnevermind, had a typo in the namespace, this is working#2020-04-0400:14cflemingHow does deps handle snapshot versions? There doesn’t seem to be any setting for how often repos should be checked for new snapshots. Is there even any concept of a snapshot version in deps?#2020-04-0400:44seancorfieldI thought the underlying Maven library checked snapshots once a day?#2020-04-0400:45cflemingI’m not sure. It can be configured, using lein it can be set to :daily, :always or :never: https://github.com/technomancy/leiningen/blob/stable/sample.project.clj#L117#2020-04-0400:45cflemingDeps doesn’t seem to have a knob for that and I’m not sure what the default is.#2020-04-0401:09hiredmanI think snapshot builds may be fundamentally incompatible with how tools deps caches the built classpath#2020-04-0401:11hiredmanOr, I guess what is more likely is tools deps checks every time it needs to rebuild the cache, but if the classpath is cached and doesnt need rebuilding you'll never get a new snapshot#2020-04-0414:20Alex Miller (Clojure team)This#2020-04-0401:21seancorfieldHmm, yeah, that makes sense. I guess the "knob" there is that you can -Sforce a rebuild of the classpath. I know Alex has said in the past that "RELEASE" and "LATEST" fly in the face of the stability that deps.edn is aiming for...#2020-04-0405:20cflemingYeah, I did wonder if deps might be philosophically opposed to them. If hiredman is correct that explains what some people are seeing in Cursive (very slow launches of REPLs) since Cursive doesn’t use the classpath caching.#2020-04-0413:46ghadiRELEASE & LATEST and origin/master are mutable references
SHAs are values#2020-04-0414:16Alex Miller (Clojure team)Snapshots are checked daily by default and there is no knob to change that atm#2020-04-0414:17Alex Miller (Clojure team)-Sforce will NOT force a recheck (although I’ve thought about making that flag also trigger update always so it would)#2020-04-0414:17Alex Miller (Clojure team)The only real way to force a snapshot to update right now is to rm from your m2 repository AND -Sforce#2020-04-0414:19Alex Miller (Clojure team)I don’t see how snapshots would be any more likely to cause slowness than any other version#2020-04-0420:13cflemingThis can happen with lein if it’s configured to check for new versions always. That means every project resolve will hit the remote repo for every snapshot dep.#2020-04-0423:46Alex Miller (Clojure team)Ah, yeah. Should just be daily w deps#2020-04-0419:39athomasoriginalCompiling a CLJS program and running a CLJS repl is straightforward enough with
clj --main cljs.main --compile ns.name.here --repl
However, if I want to run a cljs socket repl so I can connect my editor to it (Atom) I cam up with these two options
Option 1
; compile cljs
clj --main cljs.main --watch "src" --compile ns.name.here
; run a socket repl
clj -J-Dclojure.server.browser="{:port 4444 :accept cljs.server.browser/repl}"
Option 2
; do both in one command
clj -J-Dclojure.server.browser="{:port 4444 :accept cljs.server.browser/repl}" -- --main cljs.main --watch "src" --compile ns.name.here
I’m now wondering if there is a better option which I missed? 🙈#2020-04-0507:55didibusOne thing I once in a while wish for is a way to have tools.deps list out all of the available aliases.#2020-04-0507:57didibusLike it be nice if clj -A would list out the available alises instead of throwing an error. And same thing could happen for all other alias type, clj -R to list available resolve-deps alias, etc.#2020-04-0510:38borkdude@didibus since it's EDN it's very easy to do this yourself?#2020-04-0513:52Alex Miller (Clojure team)I like that idea#2020-04-0513:53Alex Miller (Clojure team)Let me think about the impl a little#2020-04-0517:11vlaaadI'm pretty sure Cursive does that#2020-04-0522:05cflemingYeah, Cursive shows all aliases and also whether they’re system or project ones. It doesn’t distinguish between system and user at the moment since the only way to determine that currently with deps is counting the config files, which is a little gross.#2020-04-0611:21pezI am trying to move a project over from Leiningen to deps. In the project.clj there is a top level :exclusionsvector. Anyone familiar enough with both lein and deps to hint me where I should look up how to translate it to deps?#2020-04-0612:27Alex Miller (Clojure team)There is no support for global exclusions right now#2020-04-0612:27Alex Miller (Clojure team)But you can repeat it at each top level dep if needed #2020-04-0612:32pezThanks! We'll see if needed. 😃#2020-04-0611:26vlaaadhmm, I think I saw something about using :override-deps with nil values…#2020-04-0611:30mgrbyteI am trying to figure out how to create an uberjar with exlusions (This achieved with the :provided profile in lein). I'm using depstar atm. Has anyone found a way to do this?#2020-04-0611:45rickmoynihan@pez: it’s translates directly:
some/dep {:mvn/version "3.0.0"
:exclusions [ch.qos.logback/logback-classic]}#2020-04-0612:22teodorluNote: deps uses different dependency resolution from Lein (which relies on Maven). It might make sense to check if you need the exclusions after moving to deps.
The ClojureScript podcast S3E4[1] with @alexmiller provides some discussion on this.
[1]: https://clojurescriptpodcast.com/#2020-04-0612:27imreWould it go against deps philosophy to add support for something like provided? To me that (although it could perhaps be better named) communicates for (some of) its functionality, this project needs xx/yy, however it is left to the consumer to provide those. Or is a dev alias with those dependencies the preferred way?#2020-04-0612:28Alex Miller (Clojure team)The latter#2020-04-0612:30imreUnderstood, thanks#2020-04-0612:30pez@rickmoynihan yes, those exclusions work. This is a top-level key that Leiningen supports. I am not even sure what it does, only that I fail to get my project running and thinking it might be this. But after trying with @vlaaad’s suggestion, indicates I also have some other error in my config.#2020-04-0616:52seancorfield@pez FWIW, we used exclusions heavily with both lein and boot but once we switched to deps.edn we were able to get rid of nearly all our exclusions (I think we still have a couple of exclusions of ClojureScript in our backend deps because we just don't need/want that brought in).#2020-04-0617:10pez@seancorfield, Thanks! I'll listen to that podcast that @teodorlu mentioned and see if I can figure out why. 😃#2020-04-0621:08mikerodhttps://insideclojure.org/2018/05/04/add-lib/ was this feature never added?#2020-04-0621:11hiredmanhttps://github.com/clojure/tools.deps.alpha/tree/add-lib#2020-04-0621:24seancorfield@mikerod Recent commits have made add-lib depend on Java code in that repo so you can no longer depend on it via git. I built a local version to use, so I could work with the most recent features & fixes. Alex has talked about some form of add-lib ending up in core or part of the REPL or something... all TBD at this point.#2020-04-0621:25mikerodAh ok. Thanks for the details. #2020-04-0621:25seancorfieldMy "dot-clojure" file (`deps.edn`) has a :deps alias that depends on a git SHA prior to the Java code dependency.#2020-04-0621:26mikerodI did see that one. #2020-04-0621:26mikerodBut guess it requires some more. #2020-04-0621:27seancorfieldWell, it can be used as-is -- and it generally works just fine. I used it quite a bit in that form for loading new libraries into my running REPL (since I tend to start a REPL and run it for days).#2020-04-0714:49mikerodI believe I read there were issues in newer JVMs around trying to add to URL classloaders or something of that sort. This has been worked around in this branch? I’m going to look a bit at the impl because I’m curious what is being done.#2020-04-0714:50mikerodOf relevance is this https://github.com/tobias/dynapath#note-on-urlclassloader#2020-04-0714:50mikerod(which is used by pomegranate)#2020-04-0715:58seancorfield@mikerod If you create a dynamic class loader, you can load new libraries/classes into it: that's how we embed Clojure at work into a legacy app and how add-lib works (you can only use it in a context that has a DCL -- for example, Cognitect's REBL does not, so at work we have a small Clojure script to start REBL's UI after setting up a DCL).#2020-04-0716:04mikerodah ok. that makes sense - thanks for the extra details#2020-04-0621:28seancorfield(I'm using a variant of 0.8.667 built locally now)#2020-04-0621:36Alex Miller (Clojure team)I'm actually working on it RIGHT NOW :)#2020-04-0621:36Alex Miller (Clojure team)Rich and I spent an hour this morning talking about it#2020-04-0621:38andy.fingerhutQuick. Everybody don't bother Alex! 🙂#2020-04-0621:44seancorfieldI deliberately didn't @ you so I wouldn't disturb you 🙂#2020-04-0622:02mikerodOh nice! #2020-04-0622:03mikerodOk. I appreciate all the input.#2020-04-0708:33penryuIs there anything in tools.deps that approximates lein's :java-source-paths?#2020-04-0708:36vlaaad@penryu no, tools-deps does not compile java, it only builds class paths and runs java programs#2020-04-0708:37vlaaadyou can compile java separately, release it to clojars and use as a maven dependency 🙂#2020-04-0708:38vlaaadthere are tools on top of tools-deps, like https://github.com/EwenG/badigeon — they can compile java sources#2020-04-0708:39penryuThanks, @vlaaad. In this case, I'm going to implement some of the core functions in java, and provide a clojure wrapper.#2020-04-0708:39penryuAh, I'll take a look at badigeon#2020-04-0709:28mike_ananev@penryu or you can take high level wrapper for Badigeon
https://github.com/redstarssystems/pbuilder#2020-04-0719:35teodorluIs it possible to override :main-opts from the command line?#2020-04-0719:46teodorluContext: standard :test:runner from Sean's clj -A:new setup. Trying to make Docker download test dependencies into the image without running stuff early on to improve caching.#2020-04-0720:01Alex Miller (Clojure team)main-opts concat rather than override#2020-04-0720:01Alex Miller (Clojure team)so no if I understand your question#2020-04-0720:02Alex Miller (Clojure team)but you can always make a new alias that does what you want#2020-04-0720:08teodorluThat works. Thanks!#2020-04-1014:59borkdudedeps.clj, a community re-implementation of the clojure bash script in clojure, supports http proxy settings picked up from the environment:
https://github.com/borkdude/deps.clj#proxy-environment-variables
This was already supported for a while, but now it's documented.#2020-04-1021:47seancorfieldseancorfield/depstar {:mvn/version "1.0.94"} -- now merges data_readers for cljs and cljc extensions; adds -S/`--suppress-clash` option to suppress file clash warnings (some people find these information messages too annoying 🙂 ) https://github.com/seancorfield/depstar#2020-04-1021:48seancorfieldseancorfield/clj-new {:mvn/version "1.0.199"} -- mostly documentation updates; generated projects now use latest dependencies. https://github.com/seancorfield/clj-new#2020-04-1211:50afhammadHi, having issues importing java classes (generated by protobuf) from another folder in the monorepo using deps.edn . I've tried both adding the paths to :paths of my project and creating a deps.edn in the protos folder and adding that as a local dep. In both cases the classpath shows up when doing clj -Spath but starting a repl fails with ClassNotFoundException#2020-04-1212:10practicalli-john@afhammad have you tried {:local/root "/path/to/java/classes"} ? I used this to import java sources into a deps.edn project, using an alias, e.g. https://github.com/practicalli/clojure-deps-edn/blob/master/deps.edn#L63 Perhaps that will work for classes too, as it add them onto the classpath.#2020-04-1213:30afhammad@jr0cket does it have to be a zip? It's currently a folder with .java files. This worked using boot's :source-paths https://github.com/boot-clj/boot/wiki/Boot-Environment#env-keys#2020-04-1213:31mpenethttps://clojure.org/guides/dev_startup_time there are some hints here#2020-04-1213:51Alex Miller (Clojure team)If you have Java files, you need to javac compile them into classes#2020-04-1213:52Alex Miller (Clojure team)If you have classes, what you say above should work, but make sure the classes are in a directory structure that matches their package name (just like Clojure source)#2020-04-1219:11afhammadThanks @alexmiller. Precompiling worked. Is there a nice way of doing it as part of the deps.edn setup? One reason is it requires passing in a .jar dependency in order to compile which is already a dep of the project, but having the javac compilation as a separate step means I have to worry about fetching that dependency myself.#2020-04-1219:26Alex Miller (Clojure team)it will always be an extra step, but there are some helper dep tools out there to do that#2020-04-1219:27Alex Miller (Clojure team)one option is to portion the java stuff into its own project that you can build (with mvn or something else), deploy it, and depend on it#2020-04-1219:29afhammadyeh, thats one way to go, thanks#2020-04-1607:05flipmokidHi. What's the best way of adding java source to a deps.edn project? I have one java file that will be used as the entrypoint to the code that will call into the clojure code. It's there because Azure Function lib uses reflection on the method's arguements to determine how to convert data before passing it in and looks at the parameterised types used.#2020-04-1607:09andy.fingerhutI do not know the best way, as I haven't personally used this in a project before. I can tell you that this project claims to support the compilation of Java source files in a project that uses deps.edn. Exactly how, and whether it suits your purposes, is best determined by you: https://github.com/EwenG/badigeon#2020-04-1609:42flipmokidThanks @U0CMVHBL2 I'll give it a try.#2020-04-1611:31mike_ananev@U0C6HAPFU or you can try wrapper for it
https://github.com/redstarssystems/pbuilder#2020-04-1616:38flipmokidThanks @U097654L8, that looks really good.#2020-04-1810:39Eamonn SullivanHi all, I have a small experimental Clojure library (https://github.com/eamonnsullivan/github-search) that I'd like to turn into a mono-repo that produces both a jar and an npm package (shadow-cljs), with some common code (.cljc). Is this possible in tools-deps? Has anyone tried this and can point me to an example I can study?#2020-04-2107:08rbergerWhat would the equivaliant to the function leiningen.core.main/abort be in a program started by clj tool? My googling for a solution has been unsuccessful.#2020-04-2107:17dominicmI don't know the function, but I'd guess System/exit or throw based on the name.#2020-04-2107:18dominicmWhat behavior do you want?#2020-04-2107:22rbergerI guess Print msg to standard err and exit with a value of 1. And propler terminate anything and supply proper messages... I did use System/exit for now. I’ve been translating software that use to use lein to use tools.deps toolchain and looking for things like this.#2020-04-2111:15practicalli-johnI have shared my user level ~/.clojure/deps.edn file, which includes aliases for all the common tools I use with Clojure projects. Instructions for using the aliases are in the README and in the relevant :alias sections. For some aliases that have optional arguments I have also created additional alias names to capture the most common options, to save the amount of details that need to be remembered. Any other useful aliases are most welcome and thanks to Sean Corfield for his collection of aliases.
https://github.com/practicalli/clojure-deps-edn
Thank you.#2020-04-2116:00seancorfield@jr0cket Is yours a superset of mine at this point, or is a selection of mine (plus your own additions)? Maybe I'll mine yours for extra stuff to add to mine 🙂#2020-04-2117:26practicalli-johnThere is a substantial cross-over of libraries includes in the aliases. You have some I have not tested yet. I have some not included.
There is some divergence in naming of aliases.
Happy to work on convergence if there is interest / value.#2020-04-2119:35jrwdunhamHi all. Looking for some help here. I'm using tools-deps and my deps.edn file has a :repl alias with an :override-deps that we can summarize as {tegere {:local/root "/path/to/tegere" :deps/manifest :deps}}. The problem seems to be that the transitive dependencies of my local deps are not present in my classpath. I can see by running clj -R:repl -Stree that I have a dependency path like tegere/tegere > com.cemerick/pomegranate 1.1.0 > org.apache.maven/maven-resolver-provider 3.5.3 > javax.inject/javax.inject 1. However, if I ls -l /root/.m2/repository/ I can see that my top-level :deps dependencies are present but the dependencies of my local [:repl :override-deps] (e.g., javax.inject, pomegranate) are not. Any advice? Thanks in advance.#2020-04-2119:47Alex Miller (Clojure team)can you provide your clj version (`clj -Sdescribe`), a deps.edn to repro, and clj -Stree output?#2020-04-2119:57jrwdunhambash-5.0# clj -Sdescribe
{:version "1.10.1.502"
:config-files ["/usr/local/lib/clojure/deps.edn" "/root/.clojure/deps.edn" "deps.edn" ]
:config-user "/root/.clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/usr/local/lib/clojure"
:config-dir "/root/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
bash-5.0# cat deps.edn
{:deps
{cheshire {:mvn/version "5.8.1"}
clojure.java-time {:mvn/version "0.3.2"}
clj-http {:mvn/version "3.10.0"}
com.pubnub/pubnub-gson {:mvn/version "4.25.0"}
com.taoensso/timbre {:mvn/version "4.10.0"}
nrepl/nrepl {:mvn/version "0.6.0"}
cider/cider-nrepl {:mvn/version "0.22.0"}
hickory {:mvn/version "0.7.1"}
org.clojure/tools.namespace {:mvn/version "1.0.0"}
vbclient
{:git/url "#2020-04-2119:57jrwdunham#2020-04-2120:03Alex Miller (Clojure team)and clj -Spath doesn't include pomegranate and javax.inject?#2020-04-2120:06Alex Miller (Clojure team)oh, did you try -Sforce to force re-create your classpath?#2020-04-2120:06Alex Miller (Clojure team)changes in local deps' deps.edn files do not trigger a classpath cache refresh so you might be seeing something old there#2020-04-2120:10jrwdunhamThose deps are present when I run clj -Spath#2020-04-2120:10jrwdunham$ clj -R:repl -Spath
src:/root/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:...
#2020-04-2120:11jrwdunhambut there is nothing at those locations on disk#2020-04-2120:11jrwdunhambash-5.0# ls /root/.m2/repository/javax
ls: /root/.m2/repository/javax: No such file or directory
#2020-04-2120:19jrwdunhamOdd, if I run clj -R:repl -Sforce -Stree then it seems to work...#2020-04-2120:19jrwdunhambash-5.0# ls /root/.m2/repository/javax
inject
#2020-04-2120:20jrwdunhamthing is, I'm trying to build a Docker image, and having the following RUN statements in my Dockerfile is not having the same effect:
RUN clj -R:repl -Sforce -Stree
RUN clj -R:repl -Sforce -Stree
#2020-04-2121:23jrwdunhamWell, after removing the .cpcache/ dirs on my host machine and ensuring I have a Docker COPY that ensures my local :override-deps paths are available at build time (d'oh!), and running a Docker build with --no-cache, all seems to be working as expected.#2020-04-2121:24jrwdunhamThanks for your help @U064X3EF3 (even if the issue seems ultimately to have been my Docker misconfiguration)—I at least learned more about how to use tools-deps, and maybe this will help somebody else.#2020-04-2121:26Alex Miller (Clojure team)if you have a cached classpath, it does not currently verify that the dirs/files on the classpath actually exist, so manual mucking with ~/.m2 (either by the filesystem or your container config) can lead to some confusing results. -Sforce will usually help you there#2020-04-2121:26jrwdunhamcool. gtk#2020-04-2121:27Alex Miller (Clojure team)it's a little tricky to implement this check as it would need to be in the bash script#2020-04-2121:27Alex Miller (Clojure team)and it requires some interpretation b/c missing path dirs is fine and pretty common#2020-04-2216:32Lone RangerOkay finally have my private maven all set up on artifactory, verified that the code is deployed there. Choking on how to include that in the deps.edn for the project, though.
:deps
{com.business.projname {:mvn/version "0.1.5"
:mvn/repos {"central" {:url ""}}}
I assume that is not correct#2020-04-2216:33Lone Rangeron
$ clj -m figwheel.main -bo dev
Error building classpath. Could not find artifact com.business.projname:com.business.projname:jar:0.1.5 in central ()
#2020-04-2216:36Lone Rangerah, think I got it. well at least it's complaining it can't file the file at the right location#2020-04-2216:36Lone Rangernew error message, I'll call that a success#2020-04-2216:37imreIsn't :mvn/repos supposed to be a top-level key?#2020-04-2216:38Lone Rangeryep. and I needed so use deps correctly#2020-04-2216:39Lone Ranger:mvn/repos {"business" {:url ""}}
:deps {com.business/projname {:mvn/version "0.1.5"}}
#2020-04-2216:39Lone Rangernow at least I get a new error with that#2020-04-2216:39Lone Ranger😄#2020-04-2216:41Lone Rangerah okay, no error message now, works exactly as desired#2020-04-2216:41Lone Rangeryay#2020-04-2216:41Lone Rangersuspiciously painless#2020-04-2306:11ikitommiwhen adding dependency using :git/url, using clojure.tools.namespace.repl/refresh seems to refresh also namespaces from those dependencies. Is there a way not to do that?#2020-04-2306:12ikitommiadded dependency to malli and it pulled sci, refreshing a lot of non-project ns’es:
:reloading (sci.impl.namespaces sci.impl.utils sci.impl.destructure sci.impl.fns sci.impl.interop sci.impl.doseq-macro sci.impl.exceptions sci.impl.max-or-throw sci.impl.for-macro sci.impl.analyzer sci.impl.readers sci.impl.parser sci.impl.interpreter sci.core malli.core malli.provider malli.json-schema malli.transform malli.edn malli.error malli.mermaid malli.swagger user sci.impl.main malli.util malli.generator)
#2020-04-2306:25Alex Miller (Clojure team)I don't know enough about refresh to say#2020-04-2306:26Alex Miller (Clojure team)I think refresh looks at directories on the Java classpath, and git deps will cause dep paths to be there#2020-04-2306:27Alex Miller (Clojure team)refresh doc string says "The directories to be scanned are controlled by 'set-refresh-dirs'; defaults to all directories on the Java classpath."#2020-04-2306:27Alex Miller (Clojure team)so you could use that to narrow the scope#2020-04-2306:28Alex Miller (Clojure team)(clojure.tools.namespace.repl/set-refresh-dirs "src")
#2020-04-2306:28Alex Miller (Clojure team)might be sufficient#2020-04-2306:28ikitommiwhen using :mvn dependency, the code is in jars and because of that, not in scope of refreshing?#2020-04-2306:29Alex Miller (Clojure team)right#2020-04-2306:30ikitommiset-refresh-dirs did the trick, thanks!#2020-04-2306:30Alex Miller (Clojure team)when in doubt, read the docs :)#2020-04-2306:50ikitommiwould be great if this worked out-of-the-box, e.g. convention from DEPS to which folder the pulled deps are in, that is automatically excluded in TNS.#2020-04-2307:06Alex Miller (Clojure team)by "box" I assume you mean tools.namespace?#2020-04-2307:07Alex Miller (Clojure team)I think I would invert it - only include src dirs that are relative to the current directory and exclude others#2020-04-2307:07Alex Miller (Clojure team)tns doesn't know anything about deps or git deps or whatever#2020-04-2306:30ikitommiset-refresh-dirs did the trick, thanks!#2020-04-2308:29Jakub Holý (HolyJak)Hi! Is there support for adding an optional dependency, without specifying its version? clj-http has an optional dependency on cheshire and I would like to include it, using whatever version this version of clj-http wants. So instead of cheshire/cheshire {:mvn/version "5.8.1"} I want just cheshire/cheshire . Is this possible? Thank you!!!#2020-04-2311:27borkdudeI don't think it's possible since clj-http doesn't prescribe the cheshire version as a dependency, it's only available at development time for clj-http I think?#2020-04-2312:28Alex Miller (Clojure team)No, this is not possible, you’ll need to pick a version#2020-04-2314:55Laverne SchrockI feel like I've heard mention of others running into this:
Cloning: <actual repo name snipped>
Error building classpath. Destination path "ops-cli" already exists and is not an empty directory
org.eclipse.jgit.api.errors.JGitInternalException: Destination path "ops-cli" already exists and is not an empty directory
at org.eclipse.jgit.api.CloneCommand.verifyDirectories(CloneCommand.java:257)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:189)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:49)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68)
at clojure.tools.gitlibs.impl$ensure_git_dir$fn__955.invoke(impl.clj:109)
at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:105)
at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
at clojure.tools.deps.alpha.extensions.git$eval999$fn__1001.invoke(git.clj:41)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:189)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:169)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:237)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:217)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invokeStatic(make_classpath2.clj:55)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invoke(make_classpath2.clj:47)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:81)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:72)
at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:101)
at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:95)
at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:146)
at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:118)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
Unfortunately, when I was trying to figure out what the reproduction steps were, the problem suddenly went away. I can however say, that before it started working I was able to confirm that ~/.gitlibs/libs/.../ops-cli existed with a few SHA directories, but not the one corresponding to the one I was using in my deps.edn. This was running version 1.10.1.492 of clj/clojure on Linux.
Would it be possible to add some logging here: https://github.com/clojure/tools.gitlibs/blob/master/src/main/clojure/clojure/tools/gitlibs/impl.clj#L109 so that in the future we can see what the original Throwable was?#2020-04-2315:01Alex Miller (Clojure team)I have some good guesses at this point what the problem is#2020-04-2315:02Alex Miller (Clojure team)I think it's related to having two threads/processes cloning the same lib in parallel#2020-04-2315:03Alex Miller (Clojure team)because deps expansion is now parallel this is much easier to encounter now#2020-04-2315:03Alex Miller (Clojure team)and/or it could even just be bugs in the much older version of jgit we're using#2020-04-2315:04Alex Miller (Clojure team)I have a long paused set of work in a branch to update to latest jgit etc and I wanted to do that first#2020-04-2315:05Alex Miller (Clojure team)if it is the parallel stuff, you can set -Sthreads 1 to turn it down#2020-04-2315:53Laverne SchrockI'll give that a try if I run into it again#2020-04-2315:06Alex Miller (Clojure team)probably longer term to actually "fix" this, needs more thought#2020-04-2405:44lispycloudsHello, what is a recommended/nice way of organizing a mono-repo like project of multiple tools.deps backed projects in terms of being able to run all tests and produce JARs/docker images/run tests for each from the root of the project? I have something like this, using Makefiles:
.
├── Makefile
├── docker-compose.yaml
├── consumer
│ ├── Dockerfile
│ ├── Makefile
│ ├── deps.edn
│ ├── src
│ │ └── queue
│ │ └── consumer.clj
│ └── test
│ └── queue
│ └── consumer_test.clj
└── producer
├── Dockerfile
├── Makefile
├── deps.edn
├── src
│ └── queue
│ └── producer.clj
└── test
└── queue
└── producer_test.clj#2020-04-2406:33dominicm@rahul080327 I'd recommend looking at juxt edge which is my implementation of a monorepo. If you are looking to share makefiles, you probably want the include instruction in make (not shown in edge)#2020-04-2406:53lispycloudsThis looks really neat @U09LZR36F!
I was considering going the shell approach but was thinking the need for manual orchestration and error handling might be complex hence opted for Make. But will try out your organized approach! Thanks! 😄#2020-04-2406:34dominicmIn edge I solve the reuse problem with bin scripts at ../bin/eftest, etc#2020-04-2413:58zaneIs it possible to use clj and friends private repositories via a password-protected SSH key?#2020-04-2414:00ghadithere's no interactive ssh with the tools @zane#2020-04-2414:00ghadissh-add the key to your running ssh-agent#2020-04-2414:00ghadithen you can run clj#2020-04-2414:02zane@ghadi Yes, I did that but I’m getting an error (`USERAUTH fail`) when I try it with a password-protected SSH key.#2020-04-2414:03zaneWorks when I use a SSH key that does not have a password.#2020-04-2414:03ghadiif you run ssh-add -l you see the key you intended to add?#2020-04-2414:04zaneI believe I added it, yes. Let me run through it one more time to be sure.#2020-04-2414:04zaneBut it sounds like what you’re telling me is that this ought to work. 🙂#2020-04-2414:04Alex Miller (Clojure team)I don't think it should work#2020-04-2414:05Alex Miller (Clojure team)I don't think password protected ssh keys are supported in the set of stuff we're running#2020-04-2414:05zaneAh, okay.#2020-04-2414:05ghaditotally should work#2020-04-2414:05ghadiit prompts for pw when adding to the ssh-agent#2020-04-2414:05ghadiit's non-interactive after that#2020-04-2414:06ghadiif you see the key in your agent, make sure it doesn't appear in ~/.ssh/config#2020-04-2414:06ghadiJSch, the underlying ssh library, is stupid w.r.t. understanding .ssh/config when there is also an ssh-agent present#2020-04-2414:07ghadi(It prefers to read the key directly from the file, if it knows about it)#2020-04-2414:07ghadi(Rather than consulting the agent)#2020-04-2414:08ghadiSo, what's in your ssh agent (`ssh-add -l`) and what's in your .ssh/config?#2020-04-2414:09zane% ssh-add -l
4096 SHA256:… /Users/zane/.ssh/id_rsa (RSA)
% cat ~/.ssh/config
Host *
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_rsa
#2020-04-2414:09ghadiok comment out that block in your .ssh/config#2020-04-2414:09ghaditemporarily#2020-04-2414:09ghadithen run clj again#2020-04-2414:10zaneclj succeeds, but I think it’s pulling from a cache.#2020-04-2414:10zanei.e. It’s using the copy of the repository I downloaded when I was using a passwordless SSH key.#2020-04-2414:11ghadiremove that from ~/.gitlibs to verify#2020-04-2414:11zaneTrying to remember how to clear that cache.#2020-04-2414:11zaneRad. Thanks.#2020-04-2414:11ghadishould be in two places in there#2020-04-2414:11ghadi~/.gitlibs/{_repos and libs}#2020-04-2414:11zaneAny harm in just blowing away ~/.gitlibs entirely?#2020-04-2414:12ghadidunno how much you have in there#2020-04-2414:12ghadibut anything will be redownloaded if you have creds#2020-04-2414:12ghadino worse than blowing away ~/.m2/repository#2020-04-2414:13zaneAwesome.#2020-04-2414:13ghadiso did it work?#2020-04-2414:13zaneTrying.#2020-04-2414:13zaneStarted up instantly again.#2020-04-2414:13zaneThat’s odd.#2020-04-2414:14zanei.e. I didn’t see it download the private repository.#2020-04-2414:14ghadiclj -Sforce#2020-04-2414:15zaneWorked.#2020-04-2414:15ghadiNice.#2020-04-2414:15ghadi@alexmiller#2020-04-2414:15zaneThanks!#2020-04-2414:16zaneSo, what should I do with ~/.ssh/config now?#2020-04-2414:16zaneLeave that stuff commented?#2020-04-2414:16ghaditl;dr: https://clojurians.slack.com/archives/C6QH853H8/p1587737218326100#2020-04-2414:16zaneI confess I don’t understand SSH well enough fully understand the implications of that.#2020-04-2414:17ghadiI don't think in your case there is much harm in removing just the IdentityFile portion of that block#2020-04-2414:17zaneGot it. I had hoped that was the case.#2020-04-2414:17ghadibecause ~/.ssh/id_rsa is one of the default searched keys#2020-04-2414:18zaneBut for people with multiple SSH keys that might be an issue, sounds like?#2020-04-2414:18zane(I’m writing up documentation on how to do this for the rest of my team.)#2020-04-2414:20ghadiit depends#2020-04-2414:21ghadido you have 1 SSH key <> per repo?#2020-04-2414:21ghadior 1 SSH key that grants you access to a bunch of repos, but you also have unrelated SSH keys present too?#2020-04-2414:21ghadiThe former is not supported, the latter is, AFAICT#2020-04-2414:22zaneI personally only have this one key, but I imagine others on the team might have more.#2020-04-2414:22zaneI see.#2020-04-2414:22zaneAlso, I just want to confirm one more thing: https://ask.clojure.org/index.php/8725/deps-support-newer-private-file-formats-types-such-ed25519 that clj won’t work with OpenSSH-format keys and that you must use -m PEM when generating them. Is that correct?#2020-04-2414:23ghadiMy personal recommendation is to not use files at all, only put your ssh keys into the agent#2020-04-2414:23zaneI see. I’ll read up on how to do that.#2020-04-2414:24ghadiclj can use fancy newer keys like ed25519 keys as long as JSch is asking the agent to deal with them#2020-04-2414:24ghadiIf JSch is trying to read them from the key files directly -> no bueno#2020-04-2414:24zaneI see!#2020-04-2414:25ghadiall of this is a pain IMHO#2020-04-2414:25ghadigithub style https auth tokens are a simpler model, and offer finer grained authority#2020-04-2414:25zaneBut that’s not supported at the moment?#2020-04-2414:25ghadieasier rotation/revocation#2020-04-2414:26ghadiright, not supported#2020-04-2414:26zaneGot it.#2020-04-2414:26zaneThis is extremely helpful, @ghadi. Thanks again for your time.#2020-04-2414:26zaneI may try to write this up so others can benefit.#2020-04-2414:26ghadia buddy at Github told me <35% of the auth they see is ssh based#2020-04-2414:27zaneRight. I certainly wasn’t using SSH with GitHub until I had to get clj to work with a private repository.#2022-08-1615:53richiardiandreaSorry to barge in on this one but I was searching for answer on how to use the clj cli (and :git/url) against github private repos
Did you end up having that working?#2022-08-1615:53zaneYes.#2022-08-1615:54zaneThe process Ghadi describes worked for me.#2022-08-1615:54richiardiandreaNice, ok I'am going to read up the history then - thank you for confirming!#2022-08-1615:55zaneYou’re welcome!#2022-08-1615:58richiardiandrealast q 😄 are Oauth still unsupported?#2022-08-1615:58zaneThat I don’t know.#2022-08-1616:06richiardiandrea> The supported git url protocols are https and ssh. https repos will be accessed anonymously and require no additional authentication information. This approach is recommended for public repos. ssh repos may be either public or private.#2022-08-1616:51zaneWhat are you quoting there?#2022-08-1616:52zaneI know that there were plans to switch clj over to stop using the JVM implementation of git, which was the root of all of this pain.#2022-08-1616:52zaneIf that migration happened then perhaps https:// works now.#2022-08-1616:53zaneYou could ask Ghadi.#2022-08-1616:53richiardiandreaI am quoting the official docs
https://clojure.org/reference/deps_and_cli
Maybe it did not happen yet? let's ask @ghadi what is the status there :D#2020-04-2414:38dominicmWeird, I always considered https the less-secure approach.#2020-04-2414:41ghaditokens can be ephemeral, ssh identities are long-lived#2020-04-2414:45ghadihttps://developer.github.com/v3/guides/managing-deploy-keys/#2020-04-2414:45ghadigood pros/cons ^#2020-04-2610:29thhellerclj -A:dev
Program 'java.exe' failed to run: The filename or extension is too longAt
C:\Users\thheller\Documents\WindowsPowerShell\Modules\ClojureTools\ClojureTools.psm1:353 char:5
+ & $JavaCmd @JvmCacheOpts @JvmOpts "-Dclojure.libfile=$LibsFile" - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At C:\Users\thheller\Documents\WindowsPowerShell\Modules\ClojureTools\ClojureTools.psm1:353 char:5
+ & $JavaCmd @JvmCacheOpts @JvmOpts "-Dclojure.libfile=$LibsFile" - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : NativeCommandFailed#2020-04-2610:30thhellerwindows powershell. I'm assuming that means the classpath is too long?#2020-04-2610:52dominicmI know people have reported long classpath issues on windows.#2020-04-2610:52dominicmSomeone looked into copying whatever it is mvn does, I think it involved a file with the cp in.#2020-04-2611:09sogaiui think this may be related: https://clojure.atlassian.net/browse/TDEPS-120#2020-04-2611:11sogaiui've summarized a bit of what i found here: https://github.com/sogaiu/clj-pathing-jar#technical-background#2020-04-2812:36Bhouglandtotal newb question, but where does deps.edn install the libraries?#2020-04-2812:40kszabo~/.m2 and ~/.gitlibs#2020-04-2812:40kszaboby default#2020-04-2812:40teodorlu~/.m2 is the same as Maven and Leiningen.
$ find ~/.m2 -iname 'clojure*jar'
/home/teodorlu/.m2/repository/net/mikera/clojure-utils/0.8.0/clojure-utils-0.8.0.jar
/home/teodorlu/.m2/repository/clojure-msgpack/clojure-msgpack/1.2.1/clojure-msgpack-1.2.1.jar
/home/teodorlu/.m2/repository/org/clojure/clojurescript/1.10.520/clojurescript-1.10.520.jar
/home/teodorlu/.m2/repository/org/clojure/clojurescript/1.10.597/clojurescript-1.10.597.jar
/home/teodorlu/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/home/teodorlu/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar
/home/teodorlu/.m2/repository/clojure/java-time/clojure.java-time/0.3.2/clojure.java-time-0.3.2.jar
/home/teodorlu/.m2/repository/com/theoryinpractise/clojure-maven-plugin/1.7.1/clojure-maven-plugin-1.7.1.jar
/home/teodorlu/.m2/repository/com/cemerick/clojurescript.test/0.0.4/clojurescript.test-0.0.4.jar
#2020-04-2812:40Bhouglandthen the trace.edn tellls my project how to find them?#2020-04-2812:42Bhouglandnevermind. that doesn't show a path on my local machine#2020-04-2812:43teodorluBased on your deps.edn file, you get a classpath. Try running clj -Spath in a deps project, and you'll se it refer back into where your libraries are installed.#2020-04-2812:44Bhougland{:version "1.10.1.489"
:config-files ["/home/ben/.clojure/deps.edn" "deps.edn" ]
:config-user "/home/ben/.clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/nix/store/msn2kqpck9c5iin8hgjzm5chpwpm2167-clojure-1.10.1.489-prefix"
:config-dir "/home/ben/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}#2020-04-2812:44BhouglandDoes this look okay?#2020-04-2812:45BhouglandI am questioning the :install-dir because I use Nixos and this path would be immutable#2020-04-2812:48Alex Miller (Clojure team)The install dir isn’t used anymore#2020-04-2812:45teodorluI must admin that I'm not sure about what you're asking.#2020-04-2812:46Bhouglandmy repl can never mind my classpath#2020-04-2812:47teodorluIf you want to use deps.edn the "Nix way", I'm guessing you'll want guaranteed immutable references to what's normally a mutable folder in ~/.m2. But then you'll have to do some major modifications to tools-deps, I think.
Other people might be able to answer better.#2020-04-2812:48BhouglandI don't really want to do it the nix way. I just want to transistion to deps.edn and my repl yells at me that it can never find my classpath#2020-04-2812:49teodorluOh, so you've installed the clojure tool via Nix, it doesn't work, and you just want it to work?#2020-04-2812:49BhouglandYes. But I am wondering if it is nix or my lack of understanding on some point#2020-04-2812:50teodorluHmm. I love Nix, but for Clojure I just use the official installer: https://clojure.org/guides/getting_started#_clojure_installer_and_cli_tools#2020-04-2812:52BhouglandI appologize, earlier I posted Sdescribe not Spath. Spath is showing the .m2 directiory like it should#2020-04-2812:53Bhouglandanother vairable that is probably tripping me up is that I wired together Kakoune and the klipse-repl. I probably did something wrong there.#2020-04-2812:54BhouglandThanks for the help#2020-04-2812:56teodorluDon't think I'll be able to help you with that setup. If you want a simple experience getting started, the official Clojure installer + #calva should be a good place to start.#2020-04-2812:57BhouglandThanks for the support, it helped me target the most likely issue.#2020-04-3018:34dominicm@alexmiller I know it's been discussed before, but one good reason to have access to the resulting deps.edn that built the current JVM is for file watching. I want to automatically configure figwheel to watch my :paths and my :local/root deps, but not my gitlibs (which are dirs on the classpath). Workaround is to filter anything containing ".gitlibs", but I'm not sure of the portability or future-proof of that (as soon as someone needs a config flag for that, the tower falls).#2020-04-3018:35Alex Miller (Clojure team)it's coming...#2020-04-3018:36Alex Miller (Clojure team)that work is done and in a branch but ties into ongoing work on add-lib and tools.build and maybe even some clojure core stuff#2020-04-3018:39Alex Miller (Clojure team)the new term we're using for this is the "runtime basis", which is a superset of the merged deps.edn that also includes your effective alias mods, the lib map and a (datafied) classpath. clj will compute this, cache the basis file, and inject a reference to it via a system property. we've got some additional apis to read and help you use the basis that may end up in clojure proper (the reading part does not depend on tools.deps - it's just data), but those are really just helpful, it's just reading edn and using clojure data#2020-04-3018:40Alex Miller (Clojure team)fyi, there is an env var that defines the gitlibs root, so you could base it on that (GITLIBS)#2020-04-3018:40dominicmAh, interesting. Didn't know that.#2020-04-3018:40Alex Miller (Clojure team)https://github.com/clojure/tools.gitlibs#cache-directory#2020-04-3018:40dominicmI've also just realized that the libfile is sufficient. I can figure out the intersection of classpath dirs & gitlibs from that.#2020-04-3018:41Alex Miller (Clojure team)so, that is currently being injected by clj as I describe above, but that's probably going to be replaced#2020-04-3018:41dominicmI've spent a lot of time this afternoon due to a collision of me trying to remove code that filtered libs out (but indirectly - so I didn't know that was the difference) & a bug figwheel where it can't load certain namespaces that are in watch dirs, and it's caused by a particular dependency we have... and now it's almost 20:00!#2020-04-3018:42Alex Miller (Clojure team)there is a system property clojure.libfile that points to the libfile used to start the current process#2020-04-3018:42Alex Miller (Clojure team)assuming you start with clj of course#2020-04-3018:42dominicmHmm, okay. I'll probably make sure my code explicitly removes the bad dependency then, that way it'll continue working after the clj version bump.#2020-05-0122:43Drew Verleewhats the difference between
-Jopt Pass opt through in java_opts, ex: -J-Xmx512m
-Oalias... Concatenated jvm option aliases, ex: -O:mem
the first is jsut for one option and the later is for multiple?#2020-05-0122:48Drew Verleehttps://clojureverse.org/t/how-to-pass-jvm-opts-to-clj-from-terminal/1940/3?u=drewverlee#2020-05-0122:55hiredmanaliases are a feature of tools.deps, where you can kind of group alternative configurations together and give them a name, and -O lets you apply them#2020-05-0122:55hiredman-J is for passing arguments directly to the command that launches the jvm, typically 'java'#2020-05-0122:56Drew VerleeWhat would it look like to psas multiple arguments to the jvm? each one gets a J-<arg>#2020-05-0122:56Drew Verleeclojure J-<arg> J-<arg>#2020-05-0122:56hiredmanpassing arguments is bad wording on my part#2020-05-0122:57hiredmanit is passing options, flags like -Xmx512m#2020-05-0122:58hiredmanso for example, you could pass -J-server (which is the default these days)#2020-05-0122:58Drew Verleeso for multiple its like clojure ... -J-Xmx512m -j-Xms512#2020-05-0122:58seancorfield(! 613)-> clj -J-Dfoo=bar -J-Dquux=wibble
Clojure 1.10.1
user=> (System/getProperty "foo")
"bar"
user=> (System/getProperty "quux")
"wibble"
user=> ^D
#2020-05-0122:59hiredmanI would put them first#2020-05-0122:59seancorfieldOne -J per argument you want to pass. Beware of white space 🙂#2020-05-0122:59hiredmanwhen you run clojure or clj the argument handling is complicated#2020-05-0122:59Drew Verleenice. i didnt know about system/getProperty.#2020-05-0123:00hiredmansome arguments are for scripts, some for jvms, and some for clojure, and some for your code#2020-05-0123:02hiredmanclj/clojure also have another of different ways to combine aliases, but -A will usually do want you want#2020-05-0123:03hiredman(aliases can also specify jvm options, and -O combines those ignoring the other stuff specified in the aliases)#2020-05-0219:44Audriushello! where is source from the Jar file that is bownloaded by this installer https://clojure.org/guides/getting_started#_installation_on_linux ?#2020-05-0219:49seancorfieldIt's an uberjar built from this https://github.com/clojure/tools.deps.alpha#2020-05-0219:50seancorfieldThe installer's source is here https://github.com/clojure/brew-install (the shell scripts etc) and that readme talks about the components, including the uberjar above @stebokas#2020-05-0310:03victorbHello! I've got an idea for my own extension to tools.deps.alpha, I can see bunch of them in the src/main/clojure/clojure/tools/deps/alpha/extensions/ path. So let's say I now have my X extension written, how can I make tools.deps.alpha use it? I guess I want to end up with something similar to a lein plugin where installing it gives you the ability of using the new extension. I'm not super familiar with Java (mostly been dealing with ClojureScript) so don't expect anyone to give me the straight path to my solution, but some pointers/keywords for what to lookup would be most appreciated!#2020-05-0310:07dominicmUsing it with the library is easy, using it with the cli is not yet possible#2020-05-0310:10victorbI see. Thanks! So for now, if I'd like to publish it to enable others to use it, I would have to publish a forked version of tools.deps.alpha with my extension worked into the codebase?#2020-05-0310:15dominicmNo. They would just require your namespace which extends the multi method#2020-05-0310:18victorbHm, but if this is not exposed to do in the cli, it feels like a workaround at best, only for people who are using deps as a library. Not sure how the common use case looks for using deps (as I'm still mostly using leiningen), but feels like most people would still stick to using the cli. But again, I don't know enough about this to say for sure#2020-05-0310:32dominicmIn that case you'll need to fork the cli#2020-05-0310:34victorbalright. Thanks a lot for the explanations!#2020-05-0312:16Alex Miller (Clojure team)What’s your extension?#2020-05-0312:25victorb@U064X3EF3 fetching dependencies via a content-addressed database (IPFS or similar) and splitting versions into two different versions, one for the interface and one for the implementation#2020-05-0312:32Alex Miller (Clojure team)Including an extension would just require changing the one line in the clojure script that defined the tools classpath#2020-05-0312:33Alex Miller (Clojure team)Well, I guess the extension also needs to be loaded#2020-05-0312:35Alex Miller (Clojure team)clj has been out for 2 or 3 years and this is the first time someone has asked about this so it hasn’t really been a priority#2020-05-0312:50victorbyeah, if I'll fork tools.deps I can see the path forward easily, but for obvious reasons I wanted to avoid that.
Makes sense that no one asked, as it's a bit bleeding-edge stuff to load dependencies from something that is not HTTP/S
Thanks Alex!#2020-05-0313:05dominicmI think it gets more complex if you want to support dependencies which depend on an extension.#2020-05-0314:22victorbHm, yeah, might have to have some sort of bootstrapping available or ship whatever the extension comes with over clojars/maven. Unless I just end up with a external babashka cli util that does the fetching... Not sure yet, exploring the options#2020-05-0316:58Alex Miller (Clojure team)Figuring all that out is why we haven’t done it yet :)#2020-05-0315:44murtaza52what is the lein clean equivalent for cli tools ?#2020-05-0315:57dominicmclj -Sforce #2020-05-0317:03Alex Miller (Clojure team)There isn’t anything to clean#2020-05-0317:04Alex Miller (Clojure team)Force will force a recompute of the cached classpath which is needed in a few special cases but rare#2020-05-0317:05Alex Miller (Clojure team)In lein, there is a build directory with artifacts, but that’s not a thing in clj (YET!)#2020-05-0317:08dominicmDoes that mean it will be a thing in clj, or that it will be a thing in a tools.build main?#2020-05-0317:08dominicm(wondering how integrated these things are)#2020-05-0317:10vlaaadI hope tools.build will have some sha hashes or something for its inputs/outputs caching so it won't require cleaning...#2020-05-0317:14Alex Miller (Clojure team)Not going to get into any details now, sorry :)#2020-05-0317:16dominicmWe'll have to learn how to bribe you. I hear the ops team at facebook has a full wall of alcohol.#2020-05-0412:21tianshuHow can I make clojure cli download jars if Clojure 1.10.1 in ~/.m2 is not found.#2020-05-0412:27Alex Miller (Clojure team)try using -Sforce#2020-05-0412:28Alex Miller (Clojure team)that will force the classpath to be recomputed, not used from cache and should trigger a download#2020-05-0412:31tianshu@alexmiller Thanks! it works#2020-05-0414:59Drew VerleeIs there a way to run the clj/clojure script so it doesn't try to resolve deps first? were getting a classpath error and were hoping to get some diagnostic information via J-XshowSettings:properties, but we hit the "cannot find artifact" error first.#2020-05-0415:07Alex Miller (Clojure team)if you don't resolve deps, you can't make a classpath, and thus can't start a JVM to use that setting, so no?#2020-05-0415:09Drew VerleeThat makes sense. I guess i need to oreint myself, clj assumes you can start a JVM, but for debugging their is some information you can get before that. So i can just run java -XshowSettings:properties.#2020-05-0415:10Alex Miller (Clojure team)is that a thing?#2020-05-0415:11Drew Verleeis what a thing? 🙂#2020-05-0415:11Alex Miller (Clojure team)showsettings#2020-05-0415:11Alex Miller (Clojure team)I see it is - was confused by the help message after nvm#2020-05-0416:25Drew VerleeIf a deps procurer (local/root) points to another deps with a Procurer (e.g mvn/repos) Can i assume that it will read that Procurer or does it just grab the deps?#2020-05-0416:35Alex Miller (Clojure team)if you're asking whether transitive dependency maven repos will be used to resolve dependencies, then no#2020-05-0416:35Alex Miller (Clojure team)maven allows this, but it seems awfully dicey from a security pov so I kind of consider it an anti-feature#2020-05-0416:46Drew Verleeits likely i'm not describing this correctly.
I have a deps.edn with {:deps {datomic {:local/root "../datomic"}} and in "../datomic" the deps reads {:mvn/repos {"http://my.datomic.com" {:url "https..."}} It seems i have to add the mvn repos to the deps with the local/root procurer, even though that deps doesn't directly have a dep on datomic-pro. It has that dependency through the other deps.
@alexmiller as always, thanks for the help.#2020-05-0418:00Alex Miller (Clojure team)Yeah, that’s the scenario I’m describing#2020-05-0418:01Alex Miller (Clojure team)The question is, do you want your transitive deps to make decisions about where you get things on the internet?#2020-05-0418:02Alex Miller (Clojure team)The answer is definitely not an unqualified yes#2020-05-0418:02Alex Miller (Clojure team)So it needs a lot of thought#2020-05-0418:10Drew Verleei understand. thanks alex.#2020-05-0420:05vlaaadokay, I know how to bait Alex Miller to tell us some more about tools-build, watch this:#2020-05-0420:06vlaaadtools-build is a fork of scala's SBT#2020-05-0420:27Alex Miller (Clojure team)do you want me to write about it, or do you want me to finish it? :)#2020-05-0420:43dominicmI was going to make a joke reply, but now I'm trying to decide my actual stance on it... If I know the high level goal I can plan around it and start getting my chickens in a row 🐔#2020-05-0421:18seancorfield@dominicm I'm curious how your chickens would be organized differently based on various possible high level goals of tools.build?#2020-05-0507:35dominicmApparently my ducks would be because I was a bit daft last night#2020-05-0507:45dominicmAnyway. I usually find libraries from core require a shift in thinking or approach. Eg clj forces you to do your own builds, so I'd expect the ecosystem to start building -mains earlier in anticipation.
I have done my best to make sure my build functions can be used in different ways, but I'm betting I missed something.
Knowing whether it solves the java sources problem would allow people using lein currently who want to move off to start doing any prep they need, as I doubt there will be something as straightforward as :java-sources.#2020-05-0515:09seancorfieldI'm not sure what "prep" those lein users could realistically do, even if the goal was stated as "solves the java sources problem"?#2020-05-0515:13seancorfieldIf tools.build can completely replace depstar, I'll be happy to switch to it. That means: build JAR and uberJAR, do AOT, specify the main namespace, Multi-Release JAR support, merging data readers (and several other types of files) in appropriate ways... If it also includes Java compilation, I might change how we build/deploy a couple of OSS Java projects that we currently build from source (manually, as needed) and use local JAR files for.#2020-05-0506:56murtaza52there was a strange bug, I had deleted my .m2 directory some time back, and when I started my nrepl, it would start, but then on compiling a file it would give an error of a missing dep.
I changed the deps.edn file, and this triggered something, and on next nrepl startup, it downloaded the dependency. have never seen such a behaviour before.#2020-05-0513:25Alex Miller (Clojure team)that all is what I would expect#2020-05-0513:26Alex Miller (Clojure team)classpath is computed and then in cached in your .cpcache directory, with pointers into your .m2#2020-05-0513:27Alex Miller (Clojure team)clj script doesn't re-check that those files still exist (it could do so, but there are some performance implications and it's a little messy)#2020-05-0513:28Alex Miller (Clojure team)changing the deps.edn makes the cached cp stale (it checks that the cp file is newer than all the source deps.edn files)#2020-05-0513:29Alex Miller (Clojure team)if you get into this situation again, use -Sforce to force the cp to be recomputed, which will also trigger a redownload#2020-05-0517:35rogerio haruo adachiHey, I'm running into some issues here, regarding transitive dependencies with deps with logging dependencies (cambium + unilog).
Scenario is as follows:
1. I have a shared library that has common libs ( datomic in memory, datomic client, kafka, integrant ... and logging)
2. An application that imports the above shared library
I have made all the necessary exclusions in item 1.
But when I import in item 2, I got conflicts between slf4j many libraries that I have excluded in item 1.
How can I maintain only dependencies required (with exclusions) in item 1 in a transitive way?#2020-05-0518:21Alex Miller (Clojure team)can you share your dep.edn to repro?#2020-05-0519:02rogerio haruo adachiSure.#2020-05-0519:04rogerio haruo adachithis is my libs deps
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
fundingcircle/jackdaw {:mvn/version "0.6.9"
:exclusions [org.slf4j/slf4j-log4j12
commons-logging/commons-logging
log4j/log4j
org.slf4j/slf4j-nop
org.slf4j/log4j-over-slf4j
org.clojure/tools.logging]}
datascript/datascript {:mvn/version "0.17.1"}
willa {:mvn/version "0.1.2"}
com.cognitect/anomalies {:mvn/version "0.1.12"}
com.gfredericks/test.chuck {:mvn/version "0.2.10"}
org.apache.kafka/kafka-streams {:mvn/version "2.3.0"
:exclusions [org.slf4j/slf4j-api]}
org.apache.kafka/kafka-streams-test-utils {:mvn/version "2.3.0"}
io.pedestal/pedestal.service {:mvn/version "0.5.7"}
io.pedestal/pedestal.route {:mvn/version "0.5.7"}
io.pedestal/pedestal.jetty {:mvn/version "0.5.7"}
integrant {:mvn/version "0.7.0"}
integrant/repl {:mvn/version "0.3.1"}
duct/core {:mvn/version "0.8.0"}
com.datomic/client-cloud {:mvn/version "0.8.78"
:exclusions [commons-logging/commons-logging]}
com.cognitect.aws/api {:mvn/version "0.8.408"
:exclusions [org.clojure/tools.logging]}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.689"}
com.cognitect.aws/s3 {:mvn/version "780.2.583.0"}
rewrite-clj {:mvn/version "0.6.1"}
compute/datomic-client-memdb
{:git/url ""
:sha "fd880e70b45a0837f1604aa1230047cb74ec536e"
:exclusions [org.slf4j/slf4j-nop org.slf4j/log4j-over-slf4j org.slf4j/jcl-over-slf4j]}
badigeon/badigeon
{:git/url ""
:sha "dca97f9680a6ea204a2504c4414cafc4ba182a83"
:exclusions [org.slf4j/slf4j-nop
org.slf4j/jcl-over-slf4j]}
org.clojure/tools.cli {:mvn/version "0.4.2"}
cambium/cambium.core {:mvn/version "0.9.3"}
cambium/cambium.codec-simple {:mvn/version "0.9.3"}
spootnik/unilog {:mvn/version "0.7.24"}}#2020-05-0519:06rogerio haruo adachiAnd this is my app Deps
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
my.lib {:mvn/version "0.1.11-SNAPSHOT"} ;; This is where I import my lib and get conflicts over slf4j log4j libs
tick {:mvn/version "0.4.20-alpha"}
integrant/repl {:mvn/version "0.3.1"}
com.gfredericks/test.chuck {:mvn/version "0.2.10"}}#2020-05-0519:56Alex Miller (Clojure team)are you sure you're actually getting the latest version of your snapshot lib? by default, maven will only refresh snapshots once per day#2020-05-0522:37rogerio haruo adachiYes, I have been working with the latest version of my lib (snapshot)#2020-05-0520:13Alex Miller (Clojure team)what conflicts are you seeing?#2020-05-0520:14Alex Miller (Clojure team)I see log4j-over-slf4j under spootnik/unilog and other slf4j stuff under cambium - you should look at clj -Stree and see if what you're getting makes sense#2020-05-0520:16Alex Miller (Clojure team)if you want a more fine-grained analysis, use clj -Strace - that will generate a trace.edn file and you can walk through the :log to see each library it considers in order, whether it included it or not, and why#2020-05-0520:19Alex Miller (Clojure team)as a fallback, including it as an explicit dep in your app deps will take precedence over anything below (none of those exclusions are necessary then)#2020-05-0522:38rogerio haruo adachiThanks @alexmiller the fallback choice might work for now. Mean while I have extracted logging dependencies into a separate library to avoid conflicts.
Thanks for your help#2020-05-0615:16ikitommiAfter removing a directory from my .m2/repository, my clj failed the find dependencies using clj -m nrepl.cmdline -i -C. After doing any modification to the deps.edn , it reloaded the missing libs from clojars. Is there some local cache that caused this?#2020-05-0615:17ikitommie.g. “`deps.edn` unchanged, we have all the deps already”?#2020-05-0615:20delaguardoclass path cache is in .cpcache
and yes, unchanged deps.edn means all dependencies already downloaded#2020-05-0615:22mgrbyteThis came up yesterday too:
https://clojurians.slack.com/archives/C6QH853H8/p1588685140446600
also documented here:
https://clojure.org/reference/deps_and_cli#_classpath_caching#2020-05-0615:47ikitommiThanks, I should have read the history better.#2020-05-0616:49seancorfieldclj -Sforce is your friend here. I've tripped over this too.#2020-05-0619:57kennyWe have been getting this exception fairly frequently on CI builds. Retrying the job succeeds. Any idea what this is?
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1882)
at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:1998)
at java.base/java.util.HashMap.treeifyBin(HashMap.java:767)
at java.base/java.util.HashMap.putVal(HashMap.java:639)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:847)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:659)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:583)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:373)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:494)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.model.building.DefaultModelBuilder.importDependencyManagement(DefaultModelBuilder.java:1299)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:474)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:292)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$eval813$fn__815.invoke(maven.clj:99)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$fn__1230$fn__1231.invoke(alpha.clj:214)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__500.invoke(concurrent.clj:33)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
#2020-05-0620:01hiredmanthat is a race condition when using a hash map from multiple threads, there was a recent-ish change to clj to do some stuff in parallel, and I bet if you disabled that those exceptions would go away#2020-05-0620:04hiredmanyou can stick :threads 1 in your deps.edn somewhere to download single threaded#2020-05-0620:11Alex Miller (Clojure team)No it’s a clj flag #2020-05-0620:12Alex Miller (Clojure team)-Sthreads 1#2020-05-0620:12Alex Miller (Clojure team)(But going to be a resolve-args alias)#2020-05-0620:13Alex Miller (Clojure team)I haven’t seen this error anywhere before#2020-05-0620:16Alex Miller (Clojure team)I will file a ticket to take a closer look#2020-05-0916:31dpsuttoni'm trying to get a repl up and running in the tools.deps repo and running into a class not found exception for clojure.tools.deps.alpha.util.S3TransporterFactory. This seems quite reasonable but I don't see any instructions on how to build the java portion of this project. Does anyone know what incantation i need to provide to have this class on the classpath?#2020-05-0917:20Alex Miller (Clojure team)the project is built with mvn package#2020-05-0917:26seancorfieldAll Contrib libraries are, yes?#2020-05-0917:26Alex Miller (Clojure team)yes#2020-05-0917:29Alex Miller (Clojure team)you can use mvn to get a repl too with mvn clojure:repl but it's a pretty crappy repl#2020-05-0917:30Alex Miller (Clojure team)I typically use it as a maven project in Cursive#2020-05-0917:30dpsuttonah. i'm learning cursive and reflexively imported it as a deps project. i'll figure out how to switch that#2020-05-0917:31dpsuttonthanks much#2020-05-1211:47plexuswhen you have a dependency with a :local/root to another clojure-cli/deps.edn project, is there any way to have it pick up aliases in the deps.edn that is being referenced?#2020-05-1211:48plexusI thought they might "inherit" whatever aliases are in use, but seems that's not the case.#2020-05-1212:25Alex Miller (Clojure team)Currently, no#2020-05-1212:27Alex Miller (Clojure team)What’s the actual problem you’re trying to solve?#2020-05-1407:44plexusSorry for the delay, I've elaborated on my use case on Ask Clojure: https://ask.clojure.org/index.php/9302/allow-transitive-deps-edn-dependencies-to-inherit-aliases#2020-05-1212:28vlaaadoh, oh, there is a relevant issue for that: https://clojure.atlassian.net/browse/TDEPS-116#2020-05-1216:12Jakub Holý (HolyJak)Where does deps download dependencies to (ie which folder to back up for caching on CI)? ~/.m2?#2020-05-1216:15ghadi.m2 for maven, .gitlibs for git dependencies#2020-05-1216:43Alex Miller (Clojure team)#2020-05-1217:12ghadiin Github Actions I use:
- name: Cache Maven
uses: actions/#2020-05-1217:12ghadito handle caching#2020-05-1217:53avi👋 I feel like this might be a FAQ but I did search the mirror in Zulip to no avail, so:
I have a new project that a teammate is trying to work on. (They’re new/rusty to/with Clojure.)
All the Git deps in the deps.edn are specified with https URLs.
For some reason I don’t understand, when my teammate tries to start a REPL, and thereby triggers the retrieval of the deps, they’re getting this error:
Error building classpath. @github.com/cognitect-labs/anomalies: invalid privatekey: [
This is confusing because this dep has this coord:
{:git/url ""
:sha "2decbbb3ffd8e919c67d29f39ec6d920575f65c3"
:tag "0.1.12"
:deps/manifest :deps}
Full deps.edn: https://github.com/FundingCircle/md2c8e/blob/master/deps.edn
Is this a known problem with a known solution? Am I just missing something?
Thanks!#2020-05-1218:03Alex Miller (Clojure team)that's weird looking#2020-05-1218:04Alex Miller (Clojure team)I cloned that repo and was able to start a repl#2020-05-1218:04aviYeah, I’ve been wracking my brain to try to think of something I did wrong here… maybe my teammate’s git config/environment is all wonky?#2020-05-1218:05Alex Miller (Clojure team)well, deps mostly doesn't use your git config at all#2020-05-1218:05Alex Miller (Clojure team)I would definitely re-try with -Sforce just to make sure you don't have a bad cached classpath#2020-05-1218:06Alex Miller (Clojure team)and look at ~/.m2/settings.xml to see if there are any funky repos set up#2020-05-1218:06Alex Miller (Clojure team)and I guess you should look at ~/.clojure/deps.edn in case there is anything in there#2020-05-1218:07Alex Miller (Clojure team)it may be that the error is coming from the guts of jsch under jgit and the url has been translated so that being different than what's in deps.edn is not the issue#2020-05-1218:09aviok, thank you!#2020-05-1218:29Alex Miller (Clojure team)might be something weird seen by the ssh-agent that's totally unrelated, dunno#2020-05-1220:38johanatanhi, is it possible to have a deps.edn alias to shell out to normal bash commands ?#2020-05-1220:39johanatanwe have a preprocessing step that needs to run before figwheel-main is started (for example)#2020-05-1220:39johanatanand i'd like to do: clj -A:preprocess:fig:repl or clj -A:preprocess:fig:build#2020-05-1220:39Alex Miller (Clojure team)Yes, depending on how much pain you’re willing to endure#2020-05-1220:39johanatanhehe, what are my poisons?#2020-05-1220:40Alex Miller (Clojure team)Or maybe not as a single command#2020-05-1220:40Alex Miller (Clojure team)A single clj command line will run one Clojure program#2020-05-1220:40Alex Miller (Clojure team)Clojure can shell our and run anything#2020-05-1220:41johanatanyea, sounds fine. so you're saying to use -m the-main and shell out from the-main ?#2020-05-1220:42Alex Miller (Clojure team)Yeah - either using clojure.shell or actually I find doing the interop with Java’s ProcessBuilder to be pretty trivial#2020-05-1220:42Alex Miller (Clojure team)But if you’re running two commands, why not just write bash or Babashka or something #2020-05-1220:42johanatanit's just one command really: npm run build to build our webpack#2020-05-1220:54johanatanthis did the trick:
(ns my-ns.webpack
(:require
[clojure.java.shell]))
(defn -main []
(let [{:keys [exit out err]}
(clojure.java.shell/sh "npm" "run" "build" :dir "./webpack")]
(assert (= exit 0))
(println out)
(System/exit 0)))
#2020-05-1220:59ghadiwhat's the point#2020-05-1221:00ghadi@johanatan what do you gain doing this in clojure?#2020-05-1221:02johanatani gain the ability to do: clj -A:preprocess:fig:repl or clj -A:preprocess:fig:build . it's a single command for our engineers to enter rather than multiple . easier to document/ transfer. and it still allows clj to be the top-level entry point rather than a bash or other script which would have to be maintained if our deps.edn setup changes. this way, if deps.edn setup changes, it's merely the addition of another "-A"#2020-05-1221:02johanatanit's a "turnkey" solution. one command startup type situation#2020-05-1221:05johanatanalthough there seems to be a catch, the repl 's :main-opts is apparently overriding the preprocess one (which effectively turns preprocess into a no-op).#2020-05-1221:09Alex Miller (Clojure team)yes, that's my point above about an invocation of clj running only one program - you need to do two things here#2020-05-1221:05borkdude@johanatan Btw, deps.clj has -Scommand which may also do what you want here: https://github.com/borkdude/deps.clj#-scommand#2020-05-1221:06johanatan@borkdude yea, i saw that and explored it. but it requires two additional installs (at least): deps.clj and babashka.#2020-05-1221:06johanatanand since this will be running in GitHub Actions, i wanted to minimize installs#2020-05-1221:06borkdude@johanatan deps.clj doesn't require babashka, it's also available as a standalone executable#2020-05-1221:06borkdudebut point taken#2020-05-1221:07johanatanit would be nice if there could be a self-install of these tools#2020-05-1221:07johanatanlike say cljfmt-runner.check#2020-05-1221:07johanatandirectly within the deps.edn#2020-05-1221:07johanatanlike this i mean:
:lint {:extra-deps {com.jameslaverack/cljfmt-runner
{:git/url ""
:sha "97960e9a6464935534b5a6bab529e063d0027128"}}
:main-opts ["-m" "cljfmt-runner.check"]}
#2020-05-1221:08borkdude@johanatan deps.clj is available as a standalone executable. in CI you would only have to install that instead of clojure, so one thing instead of another thing, not an extra thing#2020-05-1221:08borkdudeand "installing" is just downloading that executable to the path, that's it#2020-05-1221:10johanatanyea but there is a github action for tools.deps:
uses: vouch-opensource/
unfortunately only one uses per step. but yea i could do a manual install#2020-05-1221:11borkdudejust giving you the option, I'm not saying you should 🙂 - running npm using a clojure command, feels like a detour to me#2020-05-1221:11johanatanhehe, true. i'll think about it. thanks!#2020-05-1221:12johanatando you think something like this could be possible in future?:
https://clojurians.slack.com/archives/C6QH853H8/p1589317666016100#2020-05-1221:12johanatan(for those tools)#2020-05-1221:12borkdudenpm also supports invoking commands. maybe the other way around would be more natural. see https://github.com/dundalek/closh/blob/master/package.json#L26-L56 for an example of this#2020-05-1221:13borkdudethat link doesn't work for me#2020-05-1221:13johanatan:lint {:extra-deps {com.jameslaverack/cljfmt-runner
{:git/url ""
:sha "97960e9a6464935534b5a6bab529e063d0027128"}}
:main-opts ["-m" "cljfmt-runner.check"]}
#2020-05-1221:13johanatan^^ including babashka as a git-based "extra-dep". i guess it would be impossible for deps.clj to be used that way#2020-05-1221:15borkdudedeps.clj doesn't install anything, it's a drop-in replacement for clojure that has some benefits for some people (like Windows users that want to invoke it in cmd.exe). And some custom options. Nothing more than that.#2020-05-1221:16borkdudeof course you can script an installer in some Clojure file and put that behind a main-opts#2020-05-1221:16johanatanah right. good point i guess it wouldn't require babashka but for this simple script, babashka would perhaps be a better option than fullblown clojure#2020-05-1221:17johanatanor.. i guess you're saying no need for either since i could invoke npm directly#2020-05-1221:17borkdudeyeah#2020-05-1221:17borkdudeit has a nice scripts option: https://github.com/dundalek/closh/blob/master/package.json#L26-L56#2020-05-1221:18johanatanah i see#2020-05-1221:20johanatanalright you've convinced me. deps.clj it is#2020-05-1221:20johanatandevelopers will just install it instead of clojure#2020-05-1221:20johanatan(or both if they so like)#2020-05-1221:22johanatanbut can the argument to -S be placed inside deps.edn ?#2020-05-1221:23johanatan(as an alias)#2020-05-1221:23borkdudeno, deps.clj doesn't change anything to deps.edn#2020-05-1221:24johanatanhmm, yea, not sure that helps in this case as it would mean two separate invocations of deps.clj#2020-05-1221:25johanatanprobably the "script" feature of package.json itself is the best bet since those can be strung together / combined#2020-05-1221:25borkdudeyeah, if you're already using npm I'd use that#2020-05-1221:25johanatan👍#2020-05-1221:33ghadivouch-opensource/tools.deps-build
I do not recommend that @johanatan. I have found a lot of success with DeLaGuardo/ in Github actions#2020-05-1221:34ghadiit allows you to run clj as many times as you need#2020-05-1221:34ghadivouch-opensource/tools.deps-build makes you cram it all into one invocation#2020-05-1221:39ghadiI added it to the wiki https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-05-1221:43johanatanOh yes, haha, that cram all into one was part of the motivation here as well. I’ll look at making the change to setup-clojure. Thx!#2020-05-1221:44ghadi- name: Setup Java
uses: actions/#2020-05-1221:44ghadithen I go buck wild#2020-05-1221:46johanatan👍:skin-tone-2:#2020-05-1221:46johanatanIf you needed NPM, you’d just install it in a prior step?#2020-05-1221:46ghadinpm is installed by default in Actions, IIRC#2020-05-1221:47ghadialready available#2020-05-1221:47johanatanAh cool. Yea I thought so too #2020-05-1400:18fabraohello all, is there any way to exclude dependency of a lib in deps.edn ?#2020-05-1400:20Alex Miller (Clojure team)you can use :exclusions [lib/name] in a coordinate#2020-05-1400:21Alex Miller (Clojure team)I'm assuming you're talking about a transitive dependency#2020-05-1400:21Alex Miller (Clojure team)there is no global exclusions right now#2020-05-1400:22fabraoHello Alex, I´m doing this
{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
mount {:mvn/version "0.1.16"}
com.sikulix/sikulixapi {:mvn/version "2.0.4"
:exclusions [org.bouncycastle/bctsp-jdk14]}}}
and still getting
clj -J'-Dclojure.server.repl={:port,5555,:accept,clojure.core.server/repl}'
Error building classpath. Could not find artifact bouncycastle:bctsp-jdk14:jar:138 in central ()#2020-05-1400:23fabraowhat is maybe wrong?#2020-05-1400:24Alex Miller (Clojure team)not sure, that seems like good deps syntax#2020-05-1400:25Alex Miller (Clojure team)I can repro locally#2020-05-1400:25Alex Miller (Clojure team)seems like its in maven central to me https://repo1.maven.org/maven2/org/bouncycastle/bctsp-jdk14/#2020-05-1400:26Alex Miller (Clojure team)that 138 looks suspicious to me#2020-05-1400:26Alex Miller (Clojure team)maybe should be 1.38#2020-05-1400:26fabraoso, excluding is not an option for this?#2020-05-1400:26Alex Miller (Clojure team)which is an actual version - I assume that's in com.sikulix/sikulixapi deps#2020-05-1400:27Alex Miller (Clojure team)might be an edge case where it's failing while getting the metadata for something that it's going to exclude#2020-05-1400:27Alex Miller (Clojure team)I am getting on a call, but I can look at it tomorrow#2020-05-1400:27fabraooh, ok, thanks#2020-05-1400:28fabraoI´ll try with lein#2020-05-1400:31seancorfield@fabrao apparently what you want to exclude is bouncycastle/bctsp-jdk14 (without the org.)#2020-05-1400:32seancorfieldI noticed in the initial set of download messages:
Downloading: org/jboss/logging/jboss-logging/3.1.4.GA/jboss-logging-3.1.4.GA.pom from central
Downloading: org/jboss/jboss-parent/9/jboss-parent-9.pom from central
Downloading: bouncycastle/bctsp-jdk14/138/bctsp-jdk14-138.pom from central <======== LOOK AT THIS LINE!
Downloading: org/bouncycastle/bctsp-jdk14/1.38/bctsp-jdk14-1.38.pom from central
Downloading: org/ghost4j/ghost4j/1.0.1/ghost4j-1.0.1.jar from central
#2020-05-1400:32fabrao@seancorfield You are the BOSS !!! That´s it !!! Thank you#2020-05-1400:32seancorfieldIt's this: https://search.maven.org/artifact/bouncycastle/bctsp-jdk14#2020-05-1400:33seancorfield(which looks ancient and maybe even bogus?)#2020-05-1400:55Alex Miller (Clojure team)yeah, that must be old or something#2020-05-1400:59Alex Miller (Clojure team)in case you're wondering the path to badness is
com.sikulix/sikulixapi 2.0.4
net.sourceforge.tess4j/tess4j 4.4.1
org.ghost4j/ghost4j 1.0.1
com.lowagie/itext 2.1.7
bouncycastle/bcprov-jdk14 138#2020-05-1400:59Alex Miller (Clojure team)there are newer versions of stuff lower down, didn't investigate in detail but probably could update some stuff to newer there to fix#2020-05-1401:17fabraoI could use with excludes like Sean said, thanks#2020-05-1407:37Andreas EdvardssonI have added a plain java class alongside my clojure code. Is it possible to have it compiled as a part of the ordinary clojure compilation? In lein there seems to be a :java-source-paths option, is there something corresponding in tools.deps?#2020-05-1419:32just.sultanovI use this solution. If you want you can use something like this:
https://gist.github.com/just-sultanov/e2f61734e22e2ed3981216ed9b880bc0#2020-05-1407:46plexusnot in tools.deps. Some of the build tools based on tools.deps can do it though. Or you may be able to get away with compiling them manually, e.g.
for x in **/*.java; do javac -cp `clojure -Spath` $x ; done
#2020-05-1408:35Andreas EdvardssonThanks for the answer! The class is expected to be more or less static, so just a manual javac invocation indeed seems to be the simplest solution. :)#2020-05-1422:04bringeHi, I'm a bit new to the clojure command line tools. I'm wondering if there is an equivalent to lein deps and this is because I'm using it with docker. It is recommended with docker to download dependencies in a separate step so that they aren't re-downloaded if none changed, such as:
FROM clojure
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY project.clj /usr/src/app/
RUN lein deps
COPY . /usr/src/app
RUN mv "$(lein uberjar | sed -n 's/^Created \(.*standalone\.jar\)/\1/p')" app-standalone.jar
CMD ["java", "-jar", "app-standalone.jar"]
#2020-05-1422:04Alex Miller (Clojure team)you can do clj -e nil to trigger deps to resolve and download#2020-05-1422:04bringeAhhh thanks!#2020-05-1422:11deactivateduserYeah I had a need for this recently (in a TravisCI build), and ended up doing clojure -Srepro -e '(println "Dependencies downloaded")'. Felt a bit hackish, but glad to see that’s basically how it’s done.#2020-05-1422:12deactivateduserI stand corrected - it was a GitHub action/workflow thingy - but basically the same idea.#2020-05-1615:01gfrederickscan a deps.edn provide jvm-opts for default calls, without having to specify an alias?#2020-05-1615:01Alex Miller (Clojure team)No#2020-05-1615:02gfredericksis that intentional, or a maybe-future-feature?#2020-05-1615:02Alex Miller (Clojure team)Maybe future #2020-05-1615:02Alex Miller (Clojure team)But it seems weird that all clj invocations in a project would share those #2020-05-1615:03Alex Miller (Clojure team)Which is why I haven’t added it yet#2020-05-1615:03gfrederickswhen imagining it was intentional I started thinking that maybe for a nontrivial project clojure isn't meant to be called directly, and you'd setup bin/ scripts that call clojure with appropriate aliases#2020-05-1615:04gfredericksat least for running specific code; maybe it's fine for launching a repl#2020-05-1615:05Alex Miller (Clojure team)Like maybe it would make more sense if it was scoped only to running a repl#2020-05-1615:05Alex Miller (Clojure team)Or running a specific main #2020-05-1615:06gfredericksyeah, specific main would definitely be adequate for anything I want to do#2020-05-1615:12Alex Miller (Clojure team)But if you’re doing that, don’t you already have an alias where you can put it?#2020-05-1615:13Alex Miller (Clojure team)Or is it that you don’t want to repeat it in aliases#2020-05-1615:17gfrederickshmm, you're right, I guess a human typing in a main ns might not be very common
so my particular situation, which is maybe not common enough to be worth targeting, is that my "project" mostly consists of a monolithic clj file at the root, and I just run clojure code.clj a lot
but I'd like to have it with a restricted -Xmx by default, and can't do that; I just have to remember to run clojure -Afoo code.clj instead#2020-05-1615:17gfredericksI think "stop being weird" is a perfectly good answer here, I was just curious#2020-05-1619:11seancorfieldOr even just clojure -A:foo since you could add code.clj to the alias (as a :main-opts element), right?#2020-05-1619:16seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "2a594f4b44496a6e6f79617e657a077b7f187f601b64"}, :content ("[email protected]")}#2020-05-1621:30dominicmIf tools.deps exposed a :init-opts option in deps.edn, that would remove the need for multiple mains. Most uses (I've had) involve side-effects like launching an nrepl for personal dev purposes. If I use a main for that, I can't use a project-specific main.#2020-05-1621:56seancorfield:main-opts can contain -e and -i -- and you can combine multiple :main-opts from multiple aliases @dominicm#2020-05-1621:56seancorfieldThat's partly why I separate out :test and :runner for example.#2020-05-1621:58seancorfieldI admit, I do miss the ability of combining aliases in other aliases -- that's probably the thing that would add the most ergonomic utility for me. That's really why we have a shell script wrapped around clojure: to do some "alias expansion"#2020-05-1621:58dominicm@seancorfield you can only have one :main-opts active from an alias at a time.#2020-05-1621:59dominicmCompare with jvm-opts which does concatenate.#2020-05-1622:00seancorfieldUgh! Yeah, you're right. I'd convinced myself they combine...#2020-05-1622:06seancorfieldWhich means I change I made recently to my dot-clojure file doesn't actually work.#2020-05-1622:09seancorfield(well, it does work, but not quite how I thought and it explains why I still need a manual step at the end)#2020-05-1622:10seancorfieldSo, yeah, you've now convinced me that having an :init-opts for just "init opts" that actually combined like :jvm-opts would be a good thing 🙂#2020-05-1622:10seancorfield(or, frankly, even if :main-opts itself combined and it was just "caveat programmer" to use it that way)#2020-05-1622:14dominicmThe problem with main opts composing is that there's an order problem. Not so with init-opts.
I suppose people may also be relying on the fact that main-opts conflict.#2020-05-1622:15seancorfieldIf you have multi -e options you might want them to execute in a particular order 🙂#2020-05-1622:16seancorfieldI added -e to my :deps alias to set up a DCL, so that if you do -A:deps:socket then you get a DCL in place before the Socket REPL is created so you able to use add-lib via Atom.#2020-05-1622:16seancorfieldBut if you do -A:deps:socket:rebl-11 you get a REBL window without DCL 😞#2020-05-1622:18seancorfieldSo you have to do -A:deps:socket -R:rebl-11 -r (omitting the :main-opts from REBL) and that sets up a DCL and starts a Socket REPL and finally starts a command-line REPL... and in there you can do ((requiring-resolve 'cognitect.rebl/ui)) to start the REBL window and then it does have a DCL.#2020-05-1622:18seancorfield(this is all a workaround because Socket REPLs don't use DCLs by default)#2020-05-1622:23dominicm@seancorfield my implementation of this in user space might interest you. You set properties using jvm opts, then have a main which reads those properties and runs evaluations from them.#2020-05-1623:41Alex Miller (Clojure team)Early versions of clj did this :)#2020-05-1622:23dominicmImplementation is a little strong perhaps. Idea :).#2020-05-1623:01Alex Miller (Clojure team)Re init opts, at some point you’re programming, just write a clj and call it#2020-05-1623:46seancorfieldWell, that was why, in the end, we wrote our little dev/repl.clj script at work -- it looks at what's on the classpath and starts a Socket REPL, an nREPL Server, REBL, or a regular REPL.#2020-05-1700:19Alex Miller (Clojure team)There are a couple generic ideas we have in this area that we will be working on, not sure yet if they are clj or clojure.main#2020-05-1706:37dominicmThe composition is important I think. I don't want to maintain a clj for every project I work on.#2020-05-1708:22didibusI find writing a little clj file for custom stuff pretty convenient. I wouldn't say no to more systematic approaches if clj adds them. Just saying, it's not that bad to have a .clj file#2020-05-1708:23didibusIt means everyone who knows Clojure can understand the build script, since it's just a clj file. As opposed to having to learn some new configuration system.#2020-05-1708:24didibusThe only downside is paying the startup price on every script execution.#2020-05-1709:12dominicmDifferent contexts I think. A project can have a single clj file, but a user might have many they compose in different ways.#2020-05-1709:12dominicmFor example, in one project I might want to run a clj file which launches postgres in the background if it's not already started, but I want that to work alongside the project's clj file.
clj -i pg.clj -i theproject.clj
#2020-05-1713:05Alex Miller (Clojure team)clj is for running a (single) Clojure program. It sounds like you’re trying to run multiple programs. At that point, use bash or babashka or write a generic clj tool that launches multiple programs. #2020-05-1713:44dominicmI want to have side effects on the world before a program runs. Usually within the same jvm.#2020-05-1717:26wilkerluciohello, I'm having some version conflicts problem in my project, and on trying to debug I remember on lein when we ask for the tree it suggests some exclusions to avoid dependency conflicts, is there any such tool like this for use with deps.edn dependencies?#2020-05-1717:31Alex Miller (Clojure team)depot can help with some similar things#2020-05-1718:42dominicmTo just clarify what I was saying above ^ I'm mostly interested in dev tooling. Things like scope-capture, nrepl, etc. So I want to effect the running JVM.#2020-05-2006:15dharriganis it necessary to include clojure as a dep in the deps.edn? I've seen a few projects with it included.#2020-05-2010:34dominicmThere was a news-and-articles by daniel janus recently about this#2020-05-2011:22dharriganooh. ta.#2020-05-2012:43Alex Miller (Clojure team)Yes, but it is included by default#2020-05-2012:48Alex Miller (Clojure team)So I guess no :)#2020-05-2012:49dominicmMy personal thoughts are that you should mark your minimum required version. Because if you don't update your computer (for some reason) and then rock up on a project using Clojure 1.22 features, and you've got 1.15 you'll be scratching your head for a while. One less thing to manage externally.#2020-05-2013:47dharriganIt's more to do with docker containers. If I can remove the dependency (as it's already there as part of openjdk-11-tools-deps-slim-buster), then I can remove it from the deps.edn file, thus making the binary ever so slightly smaller.#2020-05-2013:56dominicmHmm, I think you'll only get one copy either way...#2020-05-2014:01Alex Miller (Clojure team)you need to have it, and you will via the install deps.edn, it's just a question of version#2020-05-2014:01dominicmMaybe by removing it, you're just forcing to match the version in your docker.#2020-05-2016:12dharriganProbably safer to leave it in.#2020-05-2016:12dharriganAfter all, storage is cheap 🙂#2020-05-2007:41idkfaHello, I've seen examples of installing dependencies using clj -Sdeps '{:deps {bidi {:mvn/version "2.1.6"}}}' but I was wondering if it's possible to instruct clj to install dependencies listed in deps.edn without doing anything else? Similar to lein deps I suppose.#2020-05-2007:52delaguardoclj -Spath as an option. it will download dependencies and print out calculated class-path#2020-05-2011:38seancorfieldI think I've seen clj -e nil given as a quieter example. #2020-05-2012:24teodorluI've used clj -e :deps-download-complete when building Docker images#2020-05-2021:45idkfaThanks for the suggestions, that's really helpful. My use case is also for building Docker images.#2020-05-2017:42dvingois there a way via clj to see all available aliases?#2020-05-2017:46Alex Miller (Clojure team)Not currently. I’ve been thinking about adding thAt#2020-05-2017:50dvingonp, thanks#2020-05-2018:46aviI apologize for the probably FAQ, but I have searched for this… when I specify a gitlib as a dependency, is there a way to specify the path within the gitlib to the source code? e.g. there’s a dep I’d like to use which has its Clojure code in src/clj/#2020-05-2019:07dominicm@aviflax 👋 That would be :deps/root :)#2020-05-2019:07dominicmhttps://github.com/roman01la/uix see an example in UIx#2020-05-2019:10avi👋 Hi Dominic, thanks!
I saw :deps/root in the docs, but the description therein says:
> Specifies the relative path within the root to search for the manifest file.
In the case of this particular dep, the manifest file doesn’t even really exist… I’m faking it by manually setting :deps/manifest to :deps, to finagle tools.deps into treating this dep as if it has an empty deps.edn, which works in this case because the dep has no dependencies itself.
Anyway, that’s why I didn’t try it, but it makes sense that it would also change how the lib is added to the classpath. I’ll try it. Thanks!#2020-05-2019:12aviHmm that’s not quite working. I set :deps/root to src/clj but the path added to the classpath is /src/clj/src 😞#2020-05-2019:15dominicmOh I see "to the source code". Yeah, that makes sense. I'm not sure you can get away with that.#2020-05-2019:16aviI figured. I kinda wish there was a way to override that src segment that tools.deps adds to the end of the root. Something like :deps/code-path or something.#2020-05-2019:16dominicmhttps://github.com/clojure/tools.deps.alpha/blob/f94815dd55bdf5eb30ac8fa075c39e757cbbcca5/src/main/clojure/clojure/tools/deps/alpha/extensions/deps.clj#L29-L34#2020-05-2019:16dominicmIt would probably be :deps/paths or something :).#2020-05-2019:18aviThanks for the link, but I’m not really understanding that function. I think you’re pointing out that there isn’t currently a way to do this?#2020-05-2019:18aviI was thinking maybe :deps/paths but I’m not sure about the plural part. I’m not sure it makes sense to add multiple entries to the classpath for a dep.#2020-05-2019:24hiredman:deps/root#2020-05-2019:24aviTried that already.#2020-05-2019:24aviDoesn’t work in this case, see above.#2020-05-2019:26dominicm@aviflax I am saying there's no way to do this currently :)#2020-05-2019:26aviYup, got it. Thanks though!#2020-05-2019:26aviI guess I’ll open a ticket/issue as a feature request, couldn’t hurt.#2020-05-2019:28Alex Miller (Clojure team)I think this already does what you want#2020-05-2019:29aviOh! What am I missing?#2020-05-2019:29Alex Miller (Clojure team)or rather let me back up...#2020-05-2019:29Alex Miller (Clojure team)what kind of project is the git dep?#2020-05-2019:29Alex Miller (Clojure team)what kind of manifest file (pom, lein, deps)#2020-05-2019:29aviproject.clj#2020-05-2019:29avibut it has no deps#2020-05-2019:29aviso I’m specifying :deps/manifest as :deps#2020-05-2019:30Alex Miller (Clojure team)that should just work - what doesn't work?#2020-05-2019:30aviIn the dep’s dir structure, the Clojure code is in `src/clj/`#2020-05-2019:30Alex Miller (Clojure team)ah, ok#2020-05-2019:31aviI tried setting `:deps/root` to `src/clj` but the path added to the classpath was then `/src/clj/src`#2020-05-2019:31Alex Miller (Clojure team)yeah, there is no way right now to externally set a git dep's paths#2020-05-2019:31aviah, ok#2020-05-2019:31aviwould a ticket be helpful?#2020-05-2019:31Alex Miller (Clojure team)sure#2020-05-2019:31avi👌#2020-05-2019:31Alex Miller (Clojure team)you coudl fork it and add a deps.edn though#2020-05-2019:32aviah, good point. Maybe I will. Thanks!#2020-05-2019:32Alex Miller (Clojure team)or clone and do the same with a local project#2020-05-2019:32aviI was thinking maybe I’ll open a PR to contribute the deps.edn to the main project; I’d rather not use a fork indefinitely#2020-05-2019:35Alex Miller (Clojure team)or that#2020-05-2019:52aviI posted the request as a question to ask as that seems to be the preferred way to suggest an issue these days: https://ask.clojure.org/index.php/9314/is-it-possible-to-override-the-paths-of-a-git-dep#2020-05-2019:54Alex Miller (Clojure team)I think the question presupposes the need to do this, but maybe the broader request is about supporting lein projects#2020-05-2019:54aviyeah#2020-05-2019:54aviagreed!#2020-05-2019:55Alex Miller (Clojure team)or even more broader, supporting extensible project types, one of which could be lein#2020-05-2019:55Alex Miller (Clojure team)tools.deps is actually written to support this but clj does not expose that capability currently (b/c there are a lot of questions about how to do it well)#2020-05-2019:56aviI can see it being a kind of support nightmare#2020-05-2019:56aviI’m sure a million edge cases would pop up#2020-05-2019:57avimaybe once tools.deps drops the “alpha” — then there could be a subsequent alpha/beta/whatever release that would enable the feature, with the caveat that it’s unstable, only for testing, etc?#2020-05-2120:37dominicmSilly question, what does override-deps do that extra-deps doesn't? I realized I've been using extra-deps in my deps.edn when I want to override a dep.#2020-05-2120:46Alex Miller (Clojure team)override replaces, extra adds#2020-05-2120:46Alex Miller (Clojure team)override does not add#2020-05-2120:47Alex Miller (Clojure team)that is, it is ignored if you don't already have the lib#2020-05-2120:49Alex Miller (Clojure team)I guess it would be more accurate to say override replaces and extra adds or replaces#2020-05-2120:51Alex Miller (Clojure team)further refining...
• extra adds new top-level deps (which may effectively replace existing top or transitive deps)
• override replaces the version of an existing top or transitive dep #2020-05-2120:52Alex Miller (Clojure team)in both cases, you should be guaranteed to get the version you've said - in extra, because it's at the top level, and in override because it replaces at all levels#2020-05-2121:29seancorfield@dominicm For a concrete use case, we use :override-deps heavily at work to "pin" versions of libraries across our entire repo. If a particular app doesn't use a given dependency, the :override-deps does nothing. For deps we expect to be overridden, we specify the coords as just {} (since they'll be supplied via the override). Does that help?#2020-05-2121:31seancorfieldIn a community context: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn#L4-L17 -- those aliases let you pick a specific version of Clojure by overriding whatever would otherwise be pulled in.#2020-05-2121:35dominicmCool, thanks for both your input. 😁#2020-05-2612:49AudriusHi, I have
Error building classpath.
java.lang.NullPointerException
in one project when running what ever. I believe there could be a bug int the deps tool
Here is source: https://github.com/audriu/fake#2020-05-2612:55dpsuttonYour figwheel main has “mvn/version” as a symbol rather than a keyword#2020-05-2613:07Alex Miller (Clojure team)I’ll take a look at why you’re seeing that error message, I would expect better#2020-05-2613:08AudriusI am very happy to help at least by contributing this 😎#2020-05-2613:40dominicmFull stack:
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
at java.util.regex.Matcher.reset(Matcher.java:309)
at java.util.regex.Matcher.<init>(Matcher.java:229)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at clojure.core$re_matcher.invokeStatic(core.clj:4856)
at clojure.core$re_find.invokeStatic(core.clj:4898)
at clojure.core$re_find.invoke(core.clj:4898)
at clojure.tools.deps.alpha.util.maven$version_range_QMARK_.invokeStatic(maven.clj:250)
at clojure.tools.deps.alpha.util.maven$version_range_QMARK_.invoke(maven.clj:248)
at clojure.tools.deps.alpha.extensions.maven$eval703$fn__706.invoke(maven.clj:46)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.alpha$canonicalize_deps$fn__1136.invoke(alpha.clj:69)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
at clojure.core.protocols$fn__8140.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8140.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8088$G__8083__8101.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6828)
at clojure.core$reduce.invoke(core.clj:6810)
at clojure.tools.deps.alpha$canonicalize_deps.invokeStatic(alpha.clj:68)
at clojure.tools.deps.alpha$canonicalize_deps.invoke(alpha.clj:66)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:236)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:217)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invokeStatic(make_classpath2.clj:55)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invoke(make_classpath2.clj:47)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:81)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:72)
at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:101)
at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:95)
at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:146)
at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:118)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
#2020-05-2613:41Alex Miller (Clojure team)Thx#2020-05-2613:41dominicmOh, I see the problem#2020-05-2613:41dominicmhttps://github.com/audriu/fake/blob/master/deps.edn#L4 it's a symbol, not a keyword. So there's no :version defined.#2020-05-2613:42dominicmI'm guessing the ns code doesn't care about keyword vs symbol though#2020-05-2613:43dominicmYeah, it gets through coord-type because of the way it's written. But then version can't be read at any point, because it's not a keyword.#2020-05-2613:44dominicmThere's also the fact it's figheel-main instead of fig*w*heel-main :)#2020-05-2717:05Alex Miller (Clojure team)added some error checking for this for next release, now you'll see:
Error building classpath. No :mvn/version specified for com.bhauman/figheel-main
#2020-05-2613:44dominicm@stebokas also, you've typo'd figwheel-main :)
- com.bhauman/figheel-main {mvn/version "0.1.9"}
+ com.bhauman/figwheel-main {:mvn/version "0.1.9"}
#2020-05-2613:46bhaumanthe most common typo ever :face_with_rolling_eyes:#2020-05-2713:17respatializedIs there a way using deps.edn to add code that gets eval'd on startup for an alias or for a repl launched using clj? The problem I'm specifically trying to solve is requiring jansi-clj for any terminal clojure session so it picks up my system terminal colors, but I was wondering if there was a generally recommended approach to the problem of "default startup code" using tools-deps.#2020-05-2713:18Alex Miller (Clojure team)you can use clojure.main's -i or -e sometimes for things likee this#2020-05-2713:19Alex Miller (Clojure team)clj -h will tell you more#2020-05-2713:19Alex Miller (Clojure team)-i loads an arbitrary clj file, -e runs an expression#2020-05-2713:22respatializedgot it, so it's the simple matter of combining -e (require 'jansi-clj.auto) -r to launch a repl with some short startup code, and -i "/path/to/file" -r for anything longer. thanks!#2020-05-2713:23Alex Miller (Clojure team)yep, and you can put that in an alias with :main-opts#2020-05-2713:23Alex Miller (Clojure team)so clj -A:repl or whatever#2020-05-2915:17jjttjjI'm trying to use andare, a replacement for core async with the same namespace hierarchy, in a project that also depends on things which depend on core.async (shadow-cljs specifically). However just adding core.async to the :exclusions for shadow-cljs seems to prevent it's namespaces which use core.async from working. Is there something I'm missing?#2020-05-2915:21seancorfield@jjttjj Maybe shadow-cljs is AOT'd? That would be my guess. (your problem doesn't sound like it would be specific to Clojure CLI / deps.edn / tools.deps.alpha?)#2020-05-2915:24thhellerit is only AOT if you request the AOT variant#2020-05-2915:24thhellerI'm guessing that andare just isn't fully compatible? how does it fail?#2020-05-2915:28jjttjjFile: jar:file:C:\Users\justin\.m2\repository\thheller\shadow-client\1.3.2\shadow-client-1.3.2.jar!/shadow/util.cljs
failed to require macro-ns "shadow.util", it was required by "shadow.util"
Error in phase :compile-syntax-check
RuntimeException: No such var: ioc/state-machine
#2020-05-2915:31thhellerguess that doesn't exist in andare? it is an internal cljs.core var#2020-05-2915:31jjttjjYup, I see it doesn't now, thanks#2020-05-2915:32jjttjjIt looks like shadow.util in shadow-client uses some core.async internals#2020-05-2915:35thhellerit does. I can probably remove it but I'm not sure trying to replace the entire core.async impl is sustainable#2020-05-2915:35thhelleryou are doing this for self-hosted CLJS right?#2020-05-2915:37jjttjjyes. I wouldn't necessarily request that either, just messing around right now 🙂#2020-05-2915:39jjttjjalthough it looks possible that your go! macro that requires it might not be used much at all in shadow#2020-05-2915:43ghadioh look, an incompatible thing with the same name#2020-05-2915:45seancorfield^ That was my first thought too. Any "replacement" really should use at least a different root namespace, precisely to avoid these sorts of conflicts 😐#2020-05-2915:47dpsuttoni think its so self-host can use it. it needs to be named the same#2020-05-2915:47jjttjjmakes sense. I'll prob just fork andare and rename things#2020-05-2916:09dpsuttonrename to what?#2020-05-2916:09dpsutton> Andare preserves the namespaces present in core.async. Thus, bootstrap-compatible ClojureScript code that makes use of core.async can operate in self-hosted environments if you make the Andare artifact available for loading in lieu of the official core.async artifact.#2020-05-2916:11seancorfieldAlso, none of this seems relevant to this channel's topic -- am I missing something there @jjttjj?#2020-05-2916:13jjttjjNope, sorry about that. Thought it might be tools.deps specific at first (relating to exclusions) but realize now that's not the case#2020-05-2921:39FlavaDaveI’m trying to file a ticket at https://clojure.atlassian.net/projects/TDEPS/issues/?filter=allissues but it is saying I’m not authorized to create issues. What is the best way for me to report an issue?#2020-05-2921:39seancorfieldhttp://ask.clojure.org#2020-05-2921:39seancorfieldTag it with the tools.deps tag...#2020-05-2921:40seancorfieldhttps://ask.clojure.org/index.php/questions/contrib-libs/tools-deps-alpha are existing issues/questions.#2020-05-2921:40lvhah, I thought that was just for discussion, not for bug reports#2020-05-2921:41seancorfield@dgonsalves22 The About page explains why that exists https://ask.clojure.org/index.php/about and the official site's contributing page links to that forum https://clojure.org/community/contributing#_reporting_problems_and_requesting_enhancements /cc @lvh#2020-05-3109:39p-himikLet me know if this is not the right place for this question.
I'm trying to build an uberjar with depstar. It's my (almost) first attempt to do something like that.
Going by this snippet from the README:
# generate pom.xml (or create it manually)
clojure -Spom
# build the uberjar with AOT compilation
clojure -A:depstar -m hf.depstar.uberjar MyProject.jar -C -m project.core
# Main-Class: project.core
java -jar MyProject.jar
Did all that in a very simple "hello world" project.
Running that gives me
Error: Could not find or load main class project.core
Caused by: java.lang.ClassNotFoundException: project.core
I opened the jar with jd-gui and I see that there's indeed no project.core class even though the manifest mentions it.
Running a non-AOT variant with java -jar MyProject.jar -m project.core works just fine.
Are there some obvious steps that I might have missed? Or can it be an issue with depstar?#2020-05-3109:45seancorfieldIs your project on GitHub so I can take a look @p-himik?#2020-05-3109:46seancorfieldDoes your project.core namespace have (:gen-class) in the ns form? Does it have a -main function?#2020-05-3109:47seancorfield(if you create an app project with clj-new, it'll have all the right bits in it that you need for an uberjar -- it'll even have the :aliases already set up in deps.edn)#2020-05-3110:22p-himik(:gen-class) seems to be it, thanks! I had no idea it was needed for -main with AOT. I'll try clj-new as well.#2020-06-0105:18didibusWould it make sense to add a npm resolver?#2020-06-0105:20didibusRight now, it seems a ClojureScript lib can't depend on another ClojureScript lib that depends on a JS lib unless someone repackages the JS lib into Maven. But since tools.deps has custom resolvers, if it could resolve from npm, it seems it would allow this scenario to work#2020-06-0105:23seancorfieldWhen you say "Maven", does that include the cljsjs infrastructure or whatever it is?#2020-06-0105:25didibusYa it does#2020-06-0105:26didibuscljsjs just repackages NPM packages into Clojars#2020-06-0105:27didibusAs it does so, it might also go ahead and perform some minification on it, and it'll add the required extern files.#2020-06-0105:29didibusSo cljsjs is convenient when you're too lazy to minify or write extern files yourself. But right now, it is also mandatory just to be able to resolve deps and deps conflicts. At least this is my understanding.#2020-06-0105:32didibusI guess the challenge of a NPM resolver would be transitive NPM deps. Tools.deps would need to understand the npm package.json (the equivalent of pom.xml) and all that#2020-06-0105:36seancorfieldI would imagine the #clojurescript channel would be the best place to petition for such a thing? How does David Nolen feel about cljs dependencies from the NPM/JS world? I saw the recent changes to make it easier to publish cljs stuff to NPM (as I understood the news)...#2020-06-0105:49didibusHum, ya not sure. From my understanding, the new bundle target isn't really about publishing to NPM.
Basically, when you want to deploy a ClojureScript app, you need to do a sort of uberjar, but in JS world it's called a bundle.
The ClojureScript compiler has the capability to bundle your ClojureScript code and all its dependencies together. Since all users will need to download your bundle when viewing your site, bundle size is a sensitive topic.
ClojureScript code and ClojureScript dependencies as well as JS dependencies that are designed with Google Closure support can be bundled by the Google Closure compiler in a super optimized way (in terms of having a very small size). It renamed all vars to super short names, removed all whitespace, and does dead code elimination at the function and variable level.
That said, JS dependencies not designed for Google Closure can't go through it. So ClojureScript compiler lets you specify those as "foreign-libs". In which case, it will still bundle them up, but it won't optimize them at all.
For a project that depends on a lot of such library, that isn't very good, because it still results in a large bundle size.
There exist other bundlers for JS, which can take those type of libraries and bundle them with a reduced size. But until now, if you tried to used them to bundle the already bundled output of the ClojureScript compiler, it would fail.
The new bundle target for the ClojureScript compiler fixes this. It makes sure to output a bundle that can be further bundled by JS bundlers.
Nothing about it really has anything to do with package management. At least this is my current understanding.#2020-06-0117:17seancorfield@U0K064KQV Thanks for the explanation. Some of that I already knew but I was hazy on the relationship between cljs and bundles. One day I hope to find the time/bandwidth to dig back into cljs and maybe build something interesting so I can learn about the tooling available now (2014 was the last time I tried to build anything serious in cljs and a lot has changed since then! 🙂 )#2020-06-0121:45didibusMy pleasure, sometimes I write detailed explanations but it's really for myself that I do it. Forces me to put some order in my understanding and tests also whether I'm still missing some bits.#2020-06-0107:55thhellerFWIW I don't think it make sense to move this into deps.edn. I doubt Alex wants to maintain and interact with the insanity that is npm. We already have the mechanism to declare npm dependencies and installing them in CLJS directly, just a bit basic maybe. can always add more options to that when needed.#2020-06-0108:23didibusIs that through shadow-cljs? Or without?#2020-06-0108:28thhellerCLJS has --install-deps via cljs.main and shadow-cljs just does it automatically on startup#2020-06-0108:35didibusHum, so there's something I don't understand then. Install-deps is unaware of versions. And it does no transitive deps handling. Or I haven't figured out how#2020-06-0108:36didibusSo for example, if lib A wants to declare some npm-deps, and app X wants to use lib A, app X won't automatically pull down lib A's deps#2020-06-0108:39thhellerin CLJS automatically no, you have to run --install-deps in app X#2020-06-0108:39thhellerin shadow-cljs it is automatic#2020-06-0108:40didibusWell, I'll definitly be trying shadow-cljs eventually. But for now, --install-deps, even run manually doesn't pull transitive dependencies for me#2020-06-0108:40thhellerboth look for deps.cljs files in the libraries and use :npm-deps {"some-lib" "some-version"} to install#2020-06-0108:40didibusOh, what is a deps.cljs file 😛#2020-06-0108:40thhellerwell do this libs you are testing contain the proper deps.cljs? many that still just depend on cljsjs don't#2020-06-0108:41didibusI am writing the lib. First time I hear about a deps.cljs file#2020-06-0108:41thhellereg. https://github.com/reagent-project/reagent/blob/master/src/deps.cljs#2020-06-0108:43didibusOk, official doc seems here: https://clojurescript.org/reference/packaging-foreign-deps#2020-06-0108:45thhellerhmm that doesn't even mention npm-deps#2020-06-0108:45thhellermaybe there are no docs for this? I always thought there were#2020-06-0108:46didibusThere doesn't seem to be. And I felt like I had gone over most of the docs looking for something like this. Maybe it just needs some better doc#2020-06-0108:47didibusSo in shadow-cljs you just leverage the compiler install-deps feature under the hood? Or you make use of npm directly? Or you implemented a commonJS repository resolver of your own?#2020-06-0108:48thhellerno I don't use the CLJS install-deps ... mostly because it didn't exist when I wrote the feature for shadow-cljs#2020-06-0108:48thhellerI don't handle any package installing whatsoever, it just runs npm install nothing more.#2020-06-0108:48thheller(same as CLJS does nowadays for --install-deps)#2020-06-0108:49didibusOk, and you grab the-package and the-version from the deps.cljs file on the resource path?#2020-06-0108:54didibusJust trying to figure out how to best package my lib. So if I make use of a npm deps, I just declare it in the deps.cljs along with foreign-libs and externs. And now anyone would be able to just add a dependency to my lib and unless they have a newer version of the npm deps themselves, everything would work as is and they'd get my lib and my transitive npm deps (assuming shadow or needing to run install-deps manually otherwise)#2020-06-0108:56thhellergenerally it is either/or. either you depends on :npm-deps OR you use foreign-libs. don't mix them.#2020-06-0108:57didibusAnd where am I supposed to put the deps.cljs ? That weird doc mentions inside a src folder? But I assume just anywhere on a resource path of my deps.edn? Or should it be project root?#2020-06-0108:57thhellerthe file needs to be in the published .jar file at the root#2020-06-0108:58thhellerso generally any source-path will do#2020-06-0108:58didibusOk, and if I'm temporarily using a local dep? Same logic applies?#2020-06-0108:59thhellerfor CLJS don't know. for shadow-cljs yes, it just loads any deps.cljs files on the classpath#2020-06-0109:00didibus👍 alright, will try it out. Thanks.#2020-06-0109:11didibusThis seems to be the spec in case there is interest: http://wiki.commonjs.org/wiki/Packages/Registry#2020-06-0118:09Alex Miller (Clojure team)the idea of using npm to find transitive cljs deps has come up here several times and it's interesting but needs better thinking starting from the goals. tools.deps really has several parts and it seems like the relevant part here is using the tools.deps dependency expander to trace the transitive set of dependencies, starting from a cljs project. tools.deps is designed to be extended for this by implementing various multimethods in the ctda.extensions ns. I've looked at the npm repo and it has everything you would need to do so and would be pretty simple to plug in in that way.#2020-06-0118:10Alex Miller (Clojure team)however, it is then at odds with the rest of tools.deps (and clj) in that the other parts are about making a jvm classpath and I don't think that's what you need or want for cljs.#2020-06-0118:12Alex Miller (Clojure team)so yes, you could use tools.deps to get a transitive dependency expansion from a heterogenous set of sources including maven, local, git, and npm. I think you want to do that from some cljs tool that then knew what to do with all that stuff to appropriately run the cljs compiler (or whatever).#2020-06-0118:14Alex Miller (Clojure team)if that big picture made sense, I'd be happy to lay out what needs to be done to plug in an npm procurer into the tda extensions. That part is not hard, but it's not useful unless this in service of some greater vision.#2020-06-0119:04dominicmI think npm is useful outside of cljs. It would make a good alternative to webjars.#2020-06-0119:24Alex Miller (Clojure team)useful to do what#2020-06-0119:50dominicmI use webjars to serve assets (js, css) for libraries like codemirror#2020-06-0121:59didibusWoa, webjars already mirrors all of npm and Bower into Jars?#2020-06-0121:59didibusThat means I'm theory, we can already use tools.deps.to.depend on those.#2020-06-0122:00didibusThe only missing piece is from the cljs compiler. But you could easily write a tools.deps alias that extracted the content from the jar and put them in a node_modulrs folder and then created the deps.cljs file. So you could run that, and then run the compilation#2020-06-0122:00didibusHave you found webjars is up-to-date with npm?#2020-06-0122:00didibusAnd mirrors absolutely everything?#2020-06-0122:18dominicmIt's an automated process to request more#2020-06-0120:05hiredmanthose are different things though, one is providing dependencies to the cljs compiler, the other is providing runtime resources#2020-06-0120:11hiredmanit seems like tools.deps is mostly about runtime resources (building a classpath), it just happens that those things almost always overflap significantly in clojure, and people have built tools on top of tools.deps that reinterpret it as build time dependencies (building uberjars out of the stuff it puts on the classpath) because of that overlap#2020-06-0120:12hiredmanthere is, from my very limited perspective on cljs, a lot less overlap between compile time and runtime dependencies there because of the split between the compiler and runtime, and the way js is usually delivered#2020-06-0120:26hiredmanthinking about it, it also strikes me as very unlikely that packages right from npm are suitable to be served up ala webjars, my guess is you need some kind of intervening build step, so really both cases are inputs to some build/packaging process#2020-06-0121:51didibusTo me, this is the same. We want to provide the ClojureScript compiler (which is a Clojure application) the necessary resources that it needs. Because it relied on npm currently, it makes an assumption to not look for them on the classpath, but instead look for them in a node_modules folder. But I feel if it could leverage tools.deps, it's not unrealistic for it to look for them on the classpath instead. With target :bundle it could create a node_modules folder of symlinks to the deps pulled down by tools.deps and thus allow JS bundlers to "find" the tools.deps managed JS dependencies.
Or maybe, tools.deps could just pull the npm deps into that folder directly. Not sure if it's a core part of tools.deps to put all the deps in some central place? Or if it would be fine for it to put them in different places?#2020-06-0122:26Alex Miller (Clojure team)It‘a not - it’s up to the procurer to do that, and then tell tools.deps where it is#2020-06-0200:22didibusThat makes it even better then. Anyways, I'm just plenting the seed of an idea. See where things go.#2020-06-0121:54didibusAlso, I'm not super familiar with JS bundlers, do they expect to bundle node_modules stuff? Or could the compiler just copy over all the necessary JS on the output folder and the JS bundler can just bundle it up.#2020-06-0121:54hiredmanthey are not dependencies of the clojurescript compiler, they are dependencies of the code it is compiling#2020-06-0121:54didibusRight, I see them as resources needed by the ClojureScript compiler. I was using dependencies loosely here.#2020-06-0121:59hiredmansticking things on the class path because we have a tool for building class paths does not seem like a considered approach#2020-06-0122:04hiredmanit sounds like the problem you want to solve is transitive js dependencies#2020-06-0122:05didibusWell, personally, I find an explicit list of paths to resources is more considered then an implicit one. But NPM already made that choice. Though here it seems we could make a different one. The only difference is where to look for the resources, do you just scan the classpath, or do you also look for a specific relative folder.
That said, it seems @alexmiller was saying that it would be possible that tools.deps doesn't do classpath generation for those, but just feeds the resolution to something else. So that could be an option as well. So maybe that something else can generate the node_modules folder instead.
That said, I think going with the classpath is best. Because cljs already uses the classpath for ClojureScript deps, and for cljsjs deps. So reconciling those two seem like a nice outcome.#2020-06-0122:10didibusI'm trying to solve JS dependency resolution. So pulling down all required direct and transitive JS dependencies, detecting conflicts between them and letting the user resolve those. That's the first part. Feel tools.deps could do this. Right now the ClojureScript compiler itself is slowly building this functionality inside itself (though it delegates to npm mostly).
The second issue is setting up the environment for running the ClojureScript compiler where it can find your dependencies. It currently relies on a mixture of the classpath + looking for an implicit node_modules folder. I'm not as adamant here, but I still feel if we moved the nodes deps to the classpath as well, it seems to simplify things. Also, it gives us a single place for ordering in terms of which lib to grab first if the JS class exists in more than one place.#2020-06-0122:11didibusBut probably we should discuss these two things separately.#2020-06-0318:48vlaaadnot sure it's important, but I was reminded of a small caching problem I had with tools-deps: it seems to cache the classpath per list of aliases enabled or something similar that excludes the java version, while maven allows selecting different dependencies based on java version right inside the pom#2020-06-0318:49vlaaadso the same deps.edn might produce different classpaths based on JDK used#2020-06-0318:51ghadia repro would be helpful. By design tools.deps doesn't attempt to mimic Maven#2020-06-0318:51vlaaadhttps://github.com/cljfx/cljfx/ is a repro#2020-06-0318:52ghadithat is not a repro, that's a repo#2020-06-0318:52vlaaadgood point#2020-06-0318:52ghadi"`clojure -A:foo -Spath` -> expected this, but received that" is a repro#2020-06-0318:52vlaaadah, sorry, I'll make a repro#2020-06-0319:02vlaaadrepro#2020-06-0319:07Alex Miller (Clojure team)"switch java version to 8 so it should trigger different profile according to activation rules" is not something clj does#2020-06-0319:07Alex Miller (Clojure team)that's a maven thing#2020-06-0319:09Alex Miller (Clojure team)you can make Clojure aliases that include the desired classifier variant of cljfx if you like#2020-06-0319:11vlaaadbut it's what it does?#2020-06-0319:11Alex Miller (Clojure team)it's what Maven does#2020-06-0319:11Alex Miller (Clojure team)clj is not Maven and I have no plans to add that#2020-06-0319:12vlaaadsure, I understand that is abusing maven features#2020-06-0319:13Alex Miller (Clojure team)it's very much using Maven features as they were intended#2020-06-0319:13Alex Miller (Clojure team)just don't expect clj to support all of that (this is just the tip of the iceberg)#2020-06-0319:17vlaaadOkay, thanks#2020-06-0322:08Drew VerleeI have deps that specifies a private github repo via SSH (e.g /cdn-cgi/l/email-protection) This works finally locally. But i'm trying to know run my build scripts as part of a deployment pipline (specifically google steps). I'm running into some complications concerning deployment keys, ssh, etc... by any chance has anyone been through this before and have any advice?#2020-06-0323:44hiredmanIt may be easier to use the git build step to fetcch the repo and then treat it as a local dep#2020-06-0323:46hiredmanOr the docs for private git repos might help directly https://cloud.google.com/cloud-build/docs/access-private-github-repos#2020-06-0400:52Drew VerleeThe downside is that you have to have a GitHub deploy key per repo.#2020-06-0323:52kennyThere's no actual fix mentioned in https://ask.clojure.org/index.php/8725/deps-support-newer-private-file-formats-types-such-ed25519. Anyone know what the fix is? We're onboarding new developers and they all hit this issue. Guessing the fix has something to do with generating an older ssh key?#2020-06-0323:56kennyPerhaps https://ask.clojure.org/index.php/7818/authentication-issues-through-jsch?show=7879#a7879?#2020-06-0400:09kennyGit deps won't work at all unless they generate an older key. I think anyone with a new Mac will hit this issue 😞#2020-06-0400:39ghadied25519 keys work fine if you use an ssh-agent#2020-06-0400:39ghadiIf you rely on .ssh/config you will have a bad time#2020-06-0414:09Drew Verleecan you point to something that explains why using a config will lead to a bad time?#2020-06-0414:09Drew VerleeThis seems relevent to some of the waters i'm having to navigate.#2020-06-0414:09Drew VerleeAs i removed the config and things "just worked"#2020-06-0414:49Drew VerleeHow can i get more information about why the connector is available?
Cloning:
In the same context i can run git clone on a repo and it succeeds.#2020-06-0414:53Drew Verleemaybe the docker image i'm running in doesn't have something git deps expects to be installed at a system level?#2020-06-0414:54Alex Miller (Clojure team)are you running ssh-agent?#2020-06-0414:55Drew Verleechecking...#2020-06-0414:59ghadiecho $SSH_AUTH_SOCK will tell you definitively#2020-06-0414:59ghadiif you're running in a docker container, probably not, unless you started it explicitly#2020-06-0415:01Drew Verleethanks ghadi.#2020-06-0415:15Drew Verleeit doesn't appear to output anything. argh, ill have to somehow start it and save it between google cloud steps.#2020-06-0415:41Drew Verleeyea. i just jump started the ssh agent first and then called clojure#2020-06-1000:32Bobbi TowersI can use the CLI to launch the Liquid Clojure editor from Windows Powershell, but only if I escape the double-quotes:
PS C:\Users\admin> clj -Sdeps '{:deps {mogenslund/liquid {:mvn/version \"RELEASE\"}}}' -m liq.core
It fails however if I try to use the alias:
PS C:\Users\admin> clj -A:liquid
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate dk/salza/liq/core__init.class, dk/salza/liq/core.clj or dk/salza/liq/core.cljc on classpath.
Full report at:
C:\Users\admin\AppData\Local\Temp\clojure-4978726344790284382.edn
My deps.edn entry:
:liquid
{:extra-deps {mogenslund/liquid {:mvn/version "RELEASE"}}
:main-opts ["-m" "dk.salza.liq.core"]}
All the other aliases seem to work fine.#2020-06-1000:41seancorfield@porkostomus Looks like the project changed it's main namespace (from dk.salza.liq.core to just liq.core)#2020-06-1000:42seancorfieldI'll update my dot clojure file#2020-06-1000:45seancorfield@porkostomus I fixed the alias in my repo -- if you cloned that, you can just git pull to get the latest#2020-06-1000:56Bobbi Towers@seancorfield Great, thanks!#2020-06-1021:34Aaron CummingsI have just upgraded from 1.10.0.442 to 1.10.1.536, and I've found a change in behavior. In .442 I had changed my $INSTALL/lib/clojure/deps.edn to point the central and clojars repos to a local partial mirror. In .536, the same change has no apparent effect. Here is an example of my lib/clojure/deps.edn:
{
:paths ["src"]
:deps {
org.clojure/clojure {:mvn/version "1.10.1"}
}
:aliases {
:deps {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.8.677"}}}
:test {:extra-paths ["test"]}
}
:mvn/repos {
;; "central" {:url ""}
;; "clojars" {:url ""}
"central" {:url "file:///home/aaron/cljtest/central/"}
"clojars" {:url "file:///home/aaron/cljtest/clojars/"}
}
}
The reason I'm doing this is our development machines at work do not have internet access, so I determine in advance all of our central and clojars dependencies and cache those locally in NFS.
Is there a different way that I should override central and clojars default locations?#2020-06-1021:37Alex Miller (Clojure team)that file is no longer used - it's embedded in a resource in the uber jar#2020-06-1021:37Alex Miller (Clojure team)you can set those in your ~/.clojure/deps.edn though and should have the same effect#2020-06-1021:42Aaron CummingsOkay, I'll give that a try.#2020-06-1021:43Alex Miller (Clojure team)(that would have worked on older version too)#2020-06-1021:44Derekon that note, when I run clj -Sverbose on 1.10.1.536, the config_paths still lists the system install deps.edn file#2020-06-1022:01Alex Miller (Clojure team)yeah, it's hard-coded in there and hasn't been updated#2020-06-1022:02Alex Miller (Clojure team)there are actually some tools that rely on this output and do (now incorrectly) still use the file so I have been delaying on removing it#2020-06-1022:02Derekmakes sense#2020-06-1021:51Aaron CummingsSuper, that did work. You're right @dpassen1 - the -Sverbose flag output did lead me astray.
I did like the file in the installation - that let me setup for all of my users in one place. But I'll switch us over to the ~/.clojure/deps.edn instead.#2020-06-1021:57seancorfieldGiven that the system deps is meant to be stable and immutable, I think it's good for it to no longer be an actual file.#2020-06-1021:58DerekAgreed Sean#2020-06-1021:58DerekThere’s a reason for ~/.clojure/deps.edn and CLJ_CONFIG env var#2020-06-1021:58Derekat least i think it’s CLJ_CONFIG#2020-06-1021:59seancorfieldIt is. We use that at work to point to a repo-wide deps file, and then each subproject in the repo has its own deps file.#2020-06-1022:00DerekI probably learned it from your mention somewhere 🙂#2020-06-1022:00seancorfieldCLJ_CONFIG=../versions clojure -A:defaults ... -- the :defaults alias pulls in all the :override-deps from versions/deps.edn#2020-06-1023:31Aaron CummingsI had forgotten about CLJ_CONFIG - that seems like the better solution to what I'm doing. Thanks for the pointers everybody.#2020-06-1119:27nhaI am looking for something like a maven api that I can call from a Clojure app to make a .jar from a java file + it’s main class name.
I’ve never used maven directly (at least not in anger). Any opinions here?
For context: I am making a .jar and uploading it to a lambda-like service.
Not about tool deps so I hope this is OK - seems like the place where I am most likely to get answers 🙂
Please reply in the thread#2020-06-1119:27nhaPlease reply here to keep the channel tidy 🙂#2020-06-1119:28dominicm@U0ALP2929 What service are you using? I created an AWS lambda-specific output for pack and would be interested in targetting others.#2020-06-1119:29nhait’s actually pulsar Functions https://pulsar.apache.org/docs/en/functions-overview/#2020-06-1119:30nhaSo hence lambda-like, it’s not exactly the same. However they accept a
--jar target/my-jar-with-dependencies.jar \
--classname org.example.functions.WordCountFunction \
#2020-06-1119:31dominicmOK, looks like you need an uberjar of some kind in that case. I would recommend the library @U04V70XH6 maintains for that.#2020-06-1119:31dominicmThere's downsides to uberjars (which is why pack doesn't produce them), I'm curious to know if they accept anything else, so I'm reading the docs now :)#2020-06-1119:34nhais this the lib? https://github.com/seancorfield/depstar#2020-06-1119:37dominicmThat's the one :)#2020-06-1119:37dominicmDoesn't look positive for alternative forms. Unfortunate. Depstar is what I'd recommend :)#2020-06-1119:41nhadepstar looks like it is supposed to be used from the CLI, and not sure it supports .java files?
I am looking for something to embed. I could always shell it / reach into the namespaces though#2020-06-1119:43nhawell the api looks simple enough simple_smile https://github.com/seancorfield/depstar/blob/master/src/hf/depstar/jar.clj#L9#2020-06-1119:43dominicmOh, sorry, I thought you wanted to make a jar for clojure#2020-06-1119:43dominicmapologies#2020-06-1119:43dominicmRight, you don't need much at all#2020-06-1119:43dominicmjavac MyFile.java then zip jar.jar MyFile.java#2020-06-1119:44dominicmthat's it, you're done!#2020-06-1119:44dominicmthere might be an official java tool for making a jar, but zip works OK#2020-06-1119:44seancorfield@U0ALP2929 To build uberjars with depstar, you should be able to depend on the library and call hf.depstar.uberjar/run with the appropriate options -- see how the -main function sets those up. It assumes there's a pom.xml file available because it uses the presence of that to trigger some of the AOT and manifest generation stuff. It assumes, at a minimum, you've run clojure -Spom in a project, but you could probably generate a stub pom.xml as a temp file and pass that in.#2020-06-1119:45seancorfieldAnd depstar works from the classpath -- so it will build a JAR from whatever is on your classpath which is very likely not what you're looking for 🙂#2020-06-1119:45seancorfieldjar is the "official Java tool for making a jar" but it's basically just zip 🙂#2020-06-1119:46nhaAhah I see 🙂 not what I am looking for indeed. Shelling to javac or even mvn looks like it could be the way then#2020-06-1119:46nhaI found this https://maven.apache.org/ref/3.1.0/maven-embedder/index.html but I fear it may be a can of worms 😄#2020-06-1216:57TravisI am currently attempting to use the new Google Artifact Maven Registry in Google cloud which has a custom maven wagon. Is this possible to use with tools-deps ?#2020-06-1217:02Alex Miller (Clojure team)if it needs a custom wagon, then no#2020-06-1217:02Alex Miller (Clojure team)why does it need a wagon?#2020-06-1217:02Alex Miller (Clojure team)got a link?#2020-06-1217:11TravisI believe its to handle the security mostly to connect to it. Uses the underlying gcloud ( cli ) creds#2020-06-1217:12Travishttps://cloud.google.com/artifact-registry/docs/java/quickstart#2020-06-1217:13Alex Miller (Clojure team)yeah, that was my guess, might be be some storage access optimizations over just http too#2020-06-1217:13Alex Miller (Clojure team)well, feel free to log this as a request at https://ask.clojure.org so people can vote on it#2020-06-1217:13Travisok, thanks#2020-06-1217:13Alex Miller (Clojure team)if it's a wide enough need, will consider#2020-06-1305:54sveriHi, Is there a way to make sure I download needed dependencies for a givem library? Like: clojure -Sdeps '{:deps {seancorfield/depstar {:mvn/version "1.0.94"}}}' but without opening a repl?#2020-06-1306:06seancorfield@sveri Add -e nil to the end of that command.#2020-06-1306:07seancorfieldIt will perform all the downloads etc and then evaluate nil (and not print it because it only prints non-`nil` values)#2020-06-1306:08seancorfield-Stree and -Spath will also circumvent a REPL and will show you the dependency tree or class path respectively, if you want some confirmation of the resources being accessed.#2020-06-1306:09sveriThank you, exactly what I am looking for. I don't need the extra output for now.#2020-06-1306:09sveri@seancorfield#2020-06-1308:20sveri@seancorfield I hope it's fine if I ask you directly here. I have been using depstar successfully so far and am experimenting with graalvm right now. It seems like native-image analyzes faster if direct linking is enabled. Is there a way to pass the :jvm-opts ["-Dclojure.compiler.direct-linking=true"] for aliases, if I read that right.#2020-06-1318:30seancorfield@sveri Or you can pass it on the command-line as -J-Dclojure.compiler.direct-linking=true#2020-06-1319:05sveri@seancorfield Does that -J-D... work for clj calls too? In only saw it in reference to the native-image tool#2020-06-1319:06seancorfieldYes, -J is a clojure/`clj` option to pass JVM options.#2020-06-1320:22sveriOkay, great, thank you.#2020-06-1522:44Aleedtools.deps doesn’t load :local/repo aliases, right? how do you handle conditionally loading dev/test/etc aliases on per subproject basis?#2020-06-1522:44Alex Miller (Clojure team)Right, and you don’t #2020-06-1522:45Alex Miller (Clojure team)Something we’ve thought about a little #2020-06-1522:45Alex Miller (Clojure team)But needs more thinking#2020-06-1522:46Aleedyeah, I found some archived slack convos about it. kind of hard to appropriately work with monorepos without such a feature, everything has to configured in the root deps.edn (?)#2020-06-1523:03Aleedwhat if it aliases were specifically included, just like dependencies are?
{my-subproject {:local/root "./my-subproject"
:aliases [:dev :test]}}#2020-06-1523:19Alex Miller (Clojure team)I’m not going to start from what if - this needs a design process#2020-06-1523:25seancorfieldWe have a monorepo at work and we specifically use CLJ_CONFIG to set the "user" deps.edn to come from a specific directory in the monorepo so that we have a single, central place for all repo-wide aliases anyone needs. This also helps maintain repeatability because each developer's "user" deps.edn is excluded and we can control exactly what each alias does.#2020-06-1523:31Aleedif subprojects have paths that are loaded based on alias, do you configure this all in the central deps.edn instead?
e.g {:extra-paths ["subproject1/test" "subproject2/test"]}#2020-06-1523:52seancorfieldIt depends on whether the alias (and the optional subproject) are going to be needed for "all" other subprojects or just specific one(s). We scope things appropriately.#2020-06-1523:53seancorfieldAll our subprojects are at the same level so ../subproject/path always works.#2020-06-1609:18robert-stuttafordapologies if an FAQ (i did try to google this) - should :git/url work if the target sha is in a feature branch? so tools.gitlibs appears to be failing silently - situation: i see the gitlibs' "Checking out: " message, but no errors, and then when tools.deps tries to use the checkout, it fails because it's not there. i checked ~/.gitlibs manually and found the folder to be missing.#2020-06-1609:58borkdude@robert-stuttaford yes. have you tried -Sforce? I find myself needing that occasionally with git deps#2020-06-1611:49robert-stuttaford-Sforce doesn't seem to make a difference, unfortunately, thanks @borkdude#2020-06-1611:51Alex Miller (Clojure team)Might be something wrong with object dir - you could try rm’ing that under gitlibs#2020-06-1611:53Alex Miller (Clojure team)Can’t say I’ve seen this kind of problem before#2020-06-1611:55vlaaadmaybe feature branch isn’t pushed? 🙂#2020-06-1611:56Alex Miller (Clojure team)should fail if sha isn’t known#2020-06-1612:19robert-stuttafordi have one other hash in the folder, so it's defo succeeded at least once before. i'll scrub and retest! (feature branch defo pushed haha)#2020-06-1612:22robert-stuttafordthe commit hash that succeeded is on the default github branch (`develop` in our case), whereas the one that failed is in a feature branch. could it be that the git checkout it's doing isn't able to find the feature branch's commit?#2020-06-1612:24Alex Miller (Clojure team)Shouldn’t matter at all, I do this all the time#2020-06-1612:24robert-stuttafordscrubbing and retesting yields the same results#2020-06-1612:25Alex Miller (Clojure team)You sure you have a commit sha and not like a tree sha or something#2020-06-1612:26robert-stuttafordyes - i can visit the sha on https://github.com/org/repo/commit/SHA and i have it checked out locally#2020-06-1612:26robert-stuttafordChecking out:
the first line is one that works (it's there after i run), the second line is the one that's meant to work but doesn't (not there after i run), and the third line is the downstream failure due to the missing folder.#2020-06-1612:27robert-stuttafordi guess the next step is to dig with tools.gitlibs at the repl!#2020-06-1612:27Alex Miller (Clojure team)Yeah that definitely drops down a layer#2020-06-1612:28robert-stuttafordglad to know that this should work, thank you#2020-06-1612:28Alex Miller (Clojure team)Let me know how you make out#2020-06-1612:28robert-stuttafordinterestingly this resolves fine on our CI instance, but not locally :thinking_face: on the CI, i see Skipping coordinate: with the same sha, presumably because it's already got it. of course, good old :local/root works locally.#2020-06-1715:24mokrHi, I’m not sure where to start explaining to get the right context, but for short I’m trying to convert a somewhat large, working leiningen project to tools.deps. I’ve got the tools.deps + shadow-cljs dev experience working, but I’m struggling with the uberjar part.
The project is based on a Luminus template from 2017 so I haven’t done this part myself and I’m not an experienced Java developer either.
Right now I’m using this tool and guide (https://github.com/tonsky/uberdeps), but clj -Aclj -C:assets -e "(compile 'mtools.core)" (“clj” and “assets” are actual aliases) seems to just hang and nothing is created in the “classes” dir. At the terminal I see some output from a (println) in a middleware ns and some WARNING: An illegal reflective access ... and similar.
Any tips to help me along the way? E.g. how to get more verbose output from that compilation stage?#2020-06-1715:40hiredmanHanging when running compile usually indicates you have a top level form doing something that blocks/takes a long time#2020-06-1715:42mokrI’m quite confused here. Does it run the code when it compiles? I’m surprised to see that println output as well to be honest.#2020-06-1715:43hiredmanIt does#2020-06-1715:43hiredmanAot compilation is the same as regular code loading, it just saves the bytecode to disk as it generates it#2020-06-1715:45hiredmanLikely your lein project was never aot compiled, because you would have experienced the same thing there#2020-06-1715:46mokrI’m quite confident that it was. Lein profile has :aot :all and after extracting the jar I see lots of .class files#2020-06-1715:47mokrSo when it hangs it might be the Immutant web server doing its job, waiting for connections?#2020-06-1715:48hiredmanThen you've made other changes than just switching from lein to deps.edn#2020-06-1715:48hiredmanIt could be, it depends how you are starting it#2020-06-1715:49hiredmanI believe luminus uses mount to manage how and when things are started#2020-06-1715:50mokrI have certainly made quite a few changes, but not “that much” to that part of the code.#2020-06-1715:50mokrYes, it uses mount.#2020-06-1715:50hiredmanPast tense?#2020-06-1715:50mokrtypo#2020-06-1715:52hiredmanIf you have something like (startmount) as a top level form, it will start mount while running compile#2020-06-1715:52mokrDue to shadow-cljs liking to have all it’s source paths in one listing, I’m having to move away from the “env” approach used in Luminus. That is: Luminus uses source path to include different versions of a namespace in dev vs prod.#2020-06-1715:52mokrI might have messed something up in that ongoing process.#2020-06-1715:52hiredmanIf you have a user.clj that starts mount when loaded#2020-06-1716:00mokrNo user.clj, but mount is in core.clj that I call as main.
Mount has actually bitten me quite a few times.#2020-06-1716:02mokrWould have been way easier if my project was open source.
So, one way to get further is to start adding some printlns to my code, maybe?#2020-06-1716:05mokrThanks, @hiredman at least I did learn something here. It’s time for dinner so I’ll be away for a while.#2020-06-1814:33mokrJust for information in regards to my question above:
I got the uberjar generation with AOT working after switching to https://github.com/seancorfield/depstar and doing AOT by passing the -C option.
It was a lot of trial and error so maybe I could have gotten https://github.com/tonsky/uberdeps to work as well.
On top of of this I surfaced an issue that has been in my code for a long time: I run my prod server on port 6000, but remap it in docker-compose. Due to that I have never experienced that browsers (Chrome, FF, …) https://superuser.com/questions/188058/which-ports-are-considered-unsafe-by-chrome, before I now attempted to run the jar locally and couldn’t connect.#2020-06-1819:06markbastianIs there deps.edn analogue of the lein-environ plugin? I.e. a facility for setting an env map in a sidecar file (or your global deps.edn file) that is merged with the global env?#2020-06-1906:37rickmoynihanYou could try using: https://direnv.net/#2020-06-1819:12Alex Miller (Clojure team)not to my knowledge#2020-06-1819:13markbastianIs it possible to set env vars in deps.edn or is that not really what it's meant for?#2020-06-1819:13Alex Miller (Clojure team)it's not possible right now#2020-06-1819:13Alex Miller (Clojure team)you can of course just set them on the call to clj using normal shell methods#2020-06-1819:14Alex Miller (Clojure team)I wouldn't rule out the possibility of adding something like that in the future#2020-06-1819:14markbastianYeah, just trying to figure out the "right" way to set them when launching a REPL from Cursive. Using the Cursive config works fine for now.#2020-06-1819:15Alex Miller (Clojure team)in clj, one option is to create an alias that adds an :extra-paths and then load config via a resource#2020-06-1819:15Alex Miller (Clojure team)an alias can also set Java system properties#2020-06-1819:16Alex Miller (Clojure team)so those are two paths to have an alias add extra config#2020-06-1819:17markbastianThat makes sense. "an alias can also set Java system properties" sounds promising. Thanks!#2020-06-1819:17Alex Miller (Clojure team)but be careful that you're not putting sensitive values in your deps.edn#2020-06-1819:18Alex Miller (Clojure team)you'd want to have a system property that point to a file (or env var) to read a sensitive value#2020-06-2315:31deactivateduserG’day! Does anyone happen to know of an equivalent of lein-licenses (https://github.com/technomancy/lein-licenses) for tools.deps? Either my google-fu is weak, or such a thing doesn’t exist yet.#2020-06-2317:14seancorfieldI'd never heard of such a thing, even for Leiningen before. Looking at the code, I don't think it would be too hard to write a deps.edn-based tool that did the same thing if someone felt sufficient need...#2020-06-2320:23deactivateduserYeah that was my thought too, and it helps a lot with license compliance (which has been a big deal most of the places I’ve worked).#2020-06-2320:25deactivateduserIs there a standard place in a deps.edn to put license information? Maven offers such a thing, though it’s not mandatory so not universally adopted (which is a shame).#2020-06-2320:37seancorfieldThere is nowhere in deps.edn to put such a thing. The repo might have pom.xml and you'd look in there, or look for a LICENSE file of some sort in the project root perhaps.#2020-06-2320:39seancorfieldFor example: https://clojars.org/seancorfield/next.jdbc -- display the license type because it's in the pom.xml file: https://github.com/seancorfield/next-jdbc/blob/develop/pom.xml#L10-L15#2020-06-2320:41seancorfieldBe aware that clojure -Spom does not add <licenses> into the generated pom.xml -- I have it because I start my projects off with clj-new and the templates for pom.xml in app, lib, and template all have a default pom.xml file with an EPL 1.0 license listed in them.#2020-06-2322:19deactivateduserRight, and most of the tools that do this kind of thing “for real” rely on source repository analysis to determine licensing, rather than (potentially incorrect) declarations in build files like pom.xml.
The challenge in the Java ecosystem is that for dependencies often all you can reliably obtain is a pom.xml (and associated JAR(s)). That’s all lein-licenses uses, for example.#2020-06-2322:22deactivateduserThis is (part of) the problem that the https://spdx.dev/ is trying to solve, and I’m a bit of an advocate for their efforts. They would get a huge boost if language ecosystems (such as Clojure’s) were to adopt SPDX for licensing information.#2020-06-2322:24deactivateduserWhile they’re better than some wild west ecosystems (cough Javascript cough), the JVM-based ecosystems are still a bit too cavalier in their approach to stating and disseminating licensing information, imho…#2020-06-2321:10Drew Verlee{}
Is a valid deps.edn right?#2020-06-2321:11seancorfieldYup.#2020-06-2321:11seancorfieldExactly the same as not having one.#2020-06-2321:12Drew VerleeWhat could cause two invocations of a the same clj command to result in different outcomes? i get a error building my claspath "dont know how to create an iseq from symbol" and everyone else gets the desired result.#2020-06-2321:13Drew Verleesomething in my local maven repo?#2020-06-2321:15Drew Verleehttps://gist.github.com/drewverlee/6a65af3d3b586b7bc272c3a4c2d90944 command and ST for those interested 🙂#2020-06-2321:16Drew Verleeclojure 1.10.1#2020-06-2321:18Drew Verleei blew away my m2 repo and got the same result.#2020-06-2321:21Drew Verleethat command as a missing curly bracket#2020-06-2321:21Drew Verleenevermind, its fine#2020-06-2321:24Drew Verleemy argument order was wrong#2020-06-2321:24Drew Verleei needed to pass -Srepro first then Sdeps#2020-06-2321:38seancorfield-Sdeps takes an argument, which is the {:deps ...} EDN.#2020-06-2321:39seancorfieldSo it's not so much that you need -Srepro first as that you need the EDN immediately following -Sdeps#2020-06-2323:53colliderwriterI have a repo with multiple related projects in it which i'm moving to tools.deps. Is there a way to include an external edn file so i can keep my aliases and common paths DRY?#2020-06-2400:04seancorfield@colliderwriter What we have done at work is to take advantage of the (undocumented?) CLJ_CONFIG environment variable to point the "user deps" path at a directory in our monorepo and have all the common aliases, overrides, etc in a deps.edn file in there -- including aliases for all the tooling needed for working across all the projects in our monorepo.#2020-06-2400:07seancorfieldSo our repo structure is:
clojure
|- versions (the common deps folder)
|- project-a
|- project-b
|- ...
and then we cd into a project to work and run clojure like this:
$ CLJ_CONFIG=../versions clojure -A:some:aliases and args
and it uses versions/deps.edn instead of a user-level deps.edn file plus the deps.edn from the project-x folder that we are running in.#2020-06-2400:20colliderwriterthanks for that. it will get the gears to mesh so i can move on today, but i foresee a problem with devops once i try to hook that up because bazel/rules_clojure won't know how to do that. This seems like a missing feature in tools.deps, especially because this is possible via lein-parent in old money. I guess i should go look at the I wish there were an issue list to see if anyone else has asked for this.#2020-06-2400:34seancorfield@colliderwriter You mean this https://clojure.atlassian.net/projects/TDEPS/issues ?#2020-06-2400:35colliderwriterI sure do. Thanks again.#2020-06-2400:35seancorfieldGiven that it is driven via an environment variable, and that could be automated via a simple "build" shell script in the repo, I don't see why it would cause problems for devops.#2020-06-2400:36seancorfieldIt has the benefit that you can control all tools directly within the repo since that "main" deps.edn file is under version control with everything else.#2020-06-2400:41colliderwriterI absolutely agree that the shared files are in the right place. I'm already using Bazel to do my builds and it is pretty opinionated. I'll have to look into my options there.#2020-06-2401:43hiredmanI don't know bazel, but I suspect slot of those 'srcs' instead of being lists of source files should be directories containing source files#2020-06-2401:47hiredmanclj, lein, boot, most clojure tooling has you specify what ends up being classpath roots (directories of classes or source code, and jar files via deps) not individual source files#2020-06-2400:43colliderwriterin case anyone's curious https://github.com/simuons/rules_clojure#2020-06-2400:47seancorfieldWow, that looks horribly restrictive. I'm sorry 😐#2020-06-2402:26hiredmanIt is kind of a shame to have to explicitly list intra mono repo dependencies, given a closed mono repo world you can derive a project dependency graph from the namespace dependency graph#2020-06-2509:19danielcomptonDoes tools-deps support specifying version ranges? I couldn’t see anything about it. It looks like it supports it when reading a POM as a transitive dependency#2020-06-2511:33dominicmI'm pretty sure it doesn't#2020-06-2511:33dominicmBy design#2020-06-2511:37dominicmI'm lying https://github.com/clojure/tools.deps.alpha/blob/0ceb075c24ce28d4ec22027aa141cb19baad1b69/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj apparently support was added#2020-06-2511:37dominicmIt just gets the highest version in the range#2020-06-2514:05murtaza52I have put my aliases in the ~/.clojure/deps.edn file (I am suing the dot file from seancorfield), but when I try to call clj -A:new app test/one , I get the following error -
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
app (No such file or directory)
Full report at:
/tmp/clojure-3392454598929432067.edn
It seems it is not able to find the alias, because when I inline the alias and call the command it works. Why is it not recognizing the ~/.clojure/deps.edn file ?#2020-06-2515:34seancorfield@murtaza52 Did you also get this WARNINg about that error message?
WARNING: Specified aliases are undeclared: [:new]
What does clojure -Sdescribe show for you?#2020-06-2515:36murtaza52~ >>> clojure -Sdescribe [1]
{:version "1.10.1.536"
:config-files ["/usr/local/lib/clojure/deps.edn" "/home/murtaza52/.config/clojure/deps.edn" ]
:config-user "/home/murtaza52/.config/clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/murtaza52/.config/clojure"
:cache-dir "/home/murtaza52/.config/clojure/.cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}#2020-06-2515:37murtaza52@seancorfield No I did not get that warning#2020-06-2515:39murtaza52I moved the file /.clojure/deps.edn to /.config/clojure/deps.edn, but I am still getting the same output as above.#2020-06-2515:46seancorfieldHow did you install the Clojure CLI tools? I've never seen it try to use ~/.config/clojure as the user-level location...#2020-06-2515:47seancorfieldWhat O/S is this on? What is your HOME environment variable? (`echo $HOME`)#2020-06-2515:48murtaza52~ >>> echo $HOME
/home/murtaza52#2020-06-2515:48borkdudethis is caused by xdg-config stuff on linux right#2020-06-2515:48murtaza52This is manjaro linux#2020-06-2515:48borkdude@seancorfield https://github.com/borkdude/deps.clj/blob/5d51b80176bdcb96927ce5e6aedb25581db0e2e0/src/borkdude/deps.clj#L341#2020-06-2515:51murtaza52sorry I m still lost, how should I fix it ? where does that deps.clj go ?#2020-06-2515:51seancorfieldIt's a separate tool. He's just showing that the XDG thing is a problem.#2020-06-2515:51murtaza52this is a new install of manjaro without a DE, I had installed it before with DE and it worked fine.#2020-06-2515:51borkdudeyeah, I should point to the bash script, this is a clojure port of it#2020-06-2515:52borkdudecan you do echo $XDG_CONFIG_HOME?#2020-06-2515:52murtaza52this one just bspwm as the tiling manager#2020-06-2515:52murtaza52~ >>> echo $XDG_CONFIG_HOME
/home/murtaza52/.config#2020-06-2515:52borkdudeand then try with the -Sforce with your changes in place#2020-06-2515:53borkdudeyes, so it will look for /home/.../.config/clojure/deps.edn#2020-06-2515:54murtaza52but in already has a dir called /.config/clojure , should I replace it with /.config/.clojure ?#2020-06-2515:55borkdudeno sorry, without the dot, it was a typo on my behalf#2020-06-2515:55borkdudeand then re-try your action with -Sforce#2020-06-2515:56murtaza52what does sforce do ? yes it worked !#2020-06-2515:56borkdudeit ignores a cached file#2020-06-2515:56murtaza52cool thanks !#2020-06-2515:58seancorfieldNew topic: tools.deps.alpha ignores :scope in a coordinate, yes? That's a Maven thing that Lein/Boot support but is not relevant for deps.edn...?#2020-06-2516:01seancorfieldHahaha... well, someone opened a bug against depstar about it putting :scope "provided" dependencies into the JAR and when I created a quick test to check the classpath clojure -Spath in a new project with a :scope "provided" dependency, yes, it's on the classpath so this is t.d.a behavior.#2020-06-2517:23souenzzohello. I'm trying to use git deps with aws codecommit, inside a ec2 instance but I get
Error building classpath. : Authentication is required but no CredentialsProvider has been registered
but when I (inside the ec2 instance) git clone it works
Anyone else using tools_deps + codecommit?
(stacktrace on comment)#2020-06-2517:23souenzzoCloning:
Error building classpath. : Authentication is required but no CredentialsProvider has been registered
org.eclipse.jgit.api.errors.TransportException: : Authentication is required but no CredentialsProvider has been registered
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:248)
at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:306)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:89)
at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:50)
at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:41)
at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:71)
at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:68)
at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:110)
at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:100)
at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
at clojure.tools.deps.alpha.extensions.git$eval999$fn__1001.invoke(git.clj:41)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:189)
at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:169)
at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:237)
at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:217)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invokeStatic(make_classpath2.clj:55)
at clojure.tools.deps.alpha.script.make_classpath2$create_classpath.invoke(make_classpath2.clj:47)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:81)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:72)
at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:101)
at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:95)
at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:146)
at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:118)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:665)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
Caused by: org.eclipse.jgit.errors.TransportException: : Authentication is required but no CredentialsProvider has been registered
at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:530)
at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:345)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:137)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1269)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:237)
... 37 more#2020-06-2517:32hiredmanjgit may not support whatever authentication method git is using there#2020-06-2517:44ghadiyou need to use ssh auth, not https#2020-06-2518:17souenzzohttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_ssh-keys.html
To use ssh, i need to create a key and autorize it. That's the path @U050ECB92?#2020-06-2518:17ghadiyes#2020-06-2518:17ghadiand put the key in an ssh-agent#2020-06-2520:19souenzzo@U050ECB92 that't not aplicable once I'm trying to use LaunchConfig#2020-06-2519:26murtaza52If I am using a template to generate a project, and the template has changed, will just using -Sforce force the new template to be retrieved ?#2020-06-2519:28seancorfieldNo. The template is only used when the project is first created.#2020-06-2519:29seancorfieldAfter that, it's just "your project" and no longer related to the template.#2020-06-2519:53murtaza52nope want to create a new project with the template, and the template was recently updated, how do I ensure that the updated template from clojars will be pulled for the new project ?#2020-06-2519:54seancorfieldBy default, clj-new always uses the most recent stable version of a template.#2020-06-2519:55seancorfieldPer the clj-new documentation (README), you can override that behavior:
-S or --snapshot -- look for -SNAPSHOT version of the template (not just a release version)
-v or --verbose -- enable debugging -- be verbose! -vv and -vvv are increasingly verbose
-V or --version, followed by a version -- use this specific version of the template
#2020-06-2519:58murtaza52cool thanks#2020-06-2522:00dvingo@seancorfield I'm trying to debug the template @murtaza52 is using - is there a way to determine what clj-new will interpret as the "most recent version" I read through this: https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855
and this answer (https://stackoverflow.com/questions/13004443/how-does-maven-sort-version-numbers) led me to org.apache.maven.artifact.versioning.ComparableVersion
I tried deploying new versions that I confirmed ComparableVersion should sort as "new" but I'm not seeing them when I try using the template#2020-06-2522:05seancorfield@danvingo It will use whatever version t.d.a. would select if it were a dependency with a "RELEASE" version.#2020-06-2522:05seancorfieldWhat is the template you're trying to work with?#2020-06-2522:05dvingohttps://clojars.org/dv.fulcro-template/clj-template#2020-06-2522:06dvingoI wanted to use dates as the versions... I'm just learning about the wonderful world of maven version semantics#2020-06-2522:07seancorfieldI posted the options above that would let you do that.#2020-06-2522:08seancorfieldWhen I tried it -- having never looked at that template before -- here's what I got:
(! 709)-> clj -A:new dv.fulcro-template dvingo/example -vvv
Downloading: dv/fulcro-template/clj-template/maven-metadata.xml from clojars
Output from locating template:
Downloading: dv/fulcro-template/clj-template/2020-06-25-4/clj-template-2020-06-25-4.pom from clojars
Downloading: dv/fulcro-template/clj-template/2020-06-25-4/clj-template-2020-06-25-4.jar from clojars
Generating fresh 'clj new' dv.fulcro-template project.
#2020-06-2522:10dvingoah thank you! sorry I overlooked that#2020-06-2522:10seancorfieldAnd here's how to pull down specific versions:
(! 714)-> clj -A:new dv.fulcro-template dvingo/example -V 2020-06-25-4
Generating fresh 'clj new' dv.fulcro-template project.
Thu Jun 25 15:09:24
(sean)-(jobs:0)-(~/clojure)
(! 715)-> clj -A:new dv.fulcro-template dvingo/example2 -V 2020.06.3
Generating fresh 'clj new' dv.fulcro-template project.
Thu Jun 25 15:09:55
(sean)-(jobs:0)-(~/clojure)
(! 716)-> clj -A:new dv.fulcro-template dvingo/example3 -V 2020-06-25-3
Generating fresh 'clj new' dv.fulcro-template project.
Thu Jun 25 15:10:29
(sean)-(jobs:0)-(~/clojure)
(! 717)-> clj -A:new dv.fulcro-template dvingo/example1 -V 1.0.0
Generating fresh 'clj new' dv.fulcro-template project.
Thu Jun 25 15:10:41
(sean)-(jobs:0)-(~/clojure)
(! 718)->
#2020-06-2522:13dvingohmm, I'm on a machine that has prior versions of the template in .m2 and I'm not seeing what you're seeing:
$ clj -A:new dv.fulcro-template dvingo/example -vvv
Output from locating template:
Generating fresh 'clj new' dv.fulcro-template project.
#2020-06-2522:14seancorfieldWell, you won't see any output if it doesn't have to download anything.#2020-06-2522:15dvingoIt at some point stopped downloading the newest versions:
Thu Jun 25, 2020 18:05:15:~/.m2/repository/dv/fulcro-template/clj-template ()
$ l
total 24
drwxr-xr-x 9 default staff 288B Jun 25 18:12 ./
drwxr-xr-x 5 default staff 160B Jun 3 13:01 ../
drwxr-xr-x 7 default staff 224B Jun 23 21:47 2020-06-23-2/
drwxr-xr-x 7 default staff 224B Jun 25 13:31 2020-06-24/
drwxr-xr-x 7 default staff 224B Jun 25 13:56 2020-06-25/
drwxr-xr-x 7 default staff 224B Jun 25 14:05 2020-06-25-2/
-rw-r--r-- 1 default staff 918B Jun 25 14:05 maven-metadata-clojars.xml
-rw-r--r-- 1 default staff 40B Jun 25 14:05 maven-metadata-clojars.xml.sha1
-rw-r--r-- 1 default staff 280B Jun 25 18:12 resolver-status.properties
#2020-06-2522:16seancorfieldI think t.d.a. only checks once per day for releases.#2020-06-2522:16dvingoahhh, ok#2020-06-2522:17seancorfieldIt's very unusual to have multiple releases on the same day. Even with snapshots.#2020-06-2522:18seancorfieldIf you blow away ~/.m2/repository/dv/fulcro-template/clj-template then it should download whatever is the latest version.#2020-06-2522:18dvingoyea that makes sense to save time for the usual case. I wasn't aware of how the interaction with maven occurred - thanks for the help!
deleting the dir from the .m2 repo made it pull the latest#2020-06-2522:18seancorfieldYup, that 🙂#2020-06-2522:19seancorfieldLike I said above, you can always use the -V / --version option to download a specific version.#2020-06-2522:19dvingoright -thanks again!#2020-06-2522:22seancorfieldYou may be able to use a version range like this -V '[0,)' but, again, Aether (the Maven-stuff underneath t.d.a.) may only check once a day.#2020-06-2522:24seancorfield(FWIW, I downloaded 2020-06-25-2 per your directory listing above, then used [0,) and it did pull down 2020-06-25-4... @danvingo#2020-06-2522:25seancorfieldSo it may be that the range checks every time? (maybe a Maven expert can confirm/deny that?)#2020-06-2522:26dvingogotcha - my main use case is template users and not wanting to have them specify versions
i'll just use ComparableVersion at the repl before sending a new version 🙂#2020-06-2522:26seancorfield(hard to tell -- having pulled down the -2 version, running it without a version still pulled down -4 for me, based on RELEASE instead)#2020-06-2522:27seancorfieldIf you're not doing more than one release a day, I suspect you won't have a problem 🙂#2020-06-2522:28dvingoperfect, thanks!#2020-06-2522:36seancorfieldOne final thing @danvingo if you use the -S option, it uses a version range of (0.0.0,)#2020-06-2522:38dvingoahh good to know.
I think the convention I wanted to use is working - 2020-06-25 compares -1 to 2020-06-25-1 and 2020-06-25-1 comes before 2020-06-25-2
so looks like the issue was maven not being checked by t.d.a. and not version sorting#2020-06-2522:40seancorfieldThis is partly why I've moved to MAJOR.MINOR.COMMITS so I don't have to worry about that. COMMITS is always increasing.#2020-06-2522:43dvingomakes sense - I just want to do my part to help kill semantic(less) versioning (I know you're not following that, per the note in the readme, but it still follows the syntax)#2020-06-2522:48seancorfieldAgreed. There are plenty of ways to do it. YYYY-MM-DD.SEQ is as good as any -- and there are plenty of libraries out there, even in the Java world, that use date/time versions instead of semantic(less) versioning (I like that term... Imma gonna stealz it!)#2020-06-2601:08dvingoawesome, that's good to hear, and yes pls do!#2020-06-2601:24dpsuttonand if you are wondering, the comparison seems to be https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj#L88-L97#2020-06-2617:06deactivateduserSlightly OT, but does anyone have any good links to further reading on why semantic versioning is problematic? (and yes, I’ve watched the video of Mr Hickey’s presentation on the topic)#2020-06-2617:13hiredmanhttps://gist.github.com/jashkenas/cbd2b088e20279ae2c8e#2020-06-2617:38seancorfieldThe linked issue is definitely worth a read too https://github.com/jashkenas/underscore/issues/1805 (it's so long that GH "helpfully" hid 50+ comments that I had to click to expand!).#2020-06-2617:51deactivateduserThanks!#2020-06-2617:58deactivateduserSo criticism is great, but what’s the alternative?#2020-06-2617:59deactivateduser(let me move this to #off-topic)#2020-06-2720:33practicalli-johnI published a page on using the very unofficial tools.deps hot-loading of dependencies. Be aware that this is very unofficial and no guarantee it will be included in anything official. Credit to Sean Corfield for the alias configuration. I've shown an example of using hot loading with the rebel REPL. https://practicalli.github.io/clojure/alternative-tools/clojure-tools/hot-load-dependencies.html
Again, this is very unofficial, so use with caution and dont rely on it being available.#2020-06-2720:45Alex Miller (Clojure team)Given that this is an unreleased branch of an alpha lib and we’ve already determined the api is going to change, can you make it real obvious on that page that this is not “official” stuff?#2020-06-2720:46Alex Miller (Clojure team)There is a wip add-lib2 branch out there but it is actually a branch off another wip branch (which has now moved forward and not been merged back)#2020-06-2720:48Alex Miller (Clojure team)It’s entirely possible for that existing add-lib branch to become broken at any time (as master drifts), and I actually expect that to happen soon#2020-06-2720:54seancorfield@alexmiller Understood. I rely on it quite a bit but I'm happy to switch to any other working version, if I can just prevail on you to point me at such a thing if the SHA I'm depending on right now breaks (and I'm happy to use Clojure master for dev, if that's what I have to switch to)#2020-06-2721:02Alex Miller (Clojure team)There may be a period of time where there is no working add-lib branch#2020-06-2721:03seancorfieldI haz a sad 😢#2020-06-2721:03seancorfieldFair enough. I can survive for a "period of time" without it I guess.#2020-06-2721:05Alex Miller (Clojure team)I will try to get it remerged, just won’t be my priority so may take a few days as the tools.build stuff moves forward#2020-06-2721:05seancorfieldCool. And I'm happy to test that out too, as you know 🙂#2020-06-2720:55seancorfield^ @jr0cket I would have tried to dissuade you from writing this up, if you had asked me ahead of time. Even tho' it's in my public dot-clojure file and I use it day-in, day-out, I wouldn't want to see that baked into the public documentation base around Clojure...#2020-06-2720:58practicalli-johnI did add a caveat to that effect in the deps.edn file, however, I should have put that same warning on the page too. I will update asap#2020-06-2721:04seancorfield@jr0cket This is why I've held off doing a YouTube video showing how I use add-lib on that t.d.a branch -- I don't think people should be encouraged to use it when it's likely to go away/break/change.#2020-06-2721:09practicalli-john@alexmiller I've updated the page with a large up front warning. If you prefer I remove the page entirely, I am happy to do so.
I appreciate there are other approaches and I should probably work on documentation for them in hindsight, its just someone was asking me how to specifically do this hot reloading.#2020-06-2721:16Alex Miller (Clojure team)It’s a public thing, so I don’t care if people are aware of it, just want to be sure it’s experimental nature is clear#2020-06-2812:33practicalli-johnI've moved the page on add-lib to a reference section of the book, as an alternative and very much experimental approach. Apologies for not thinking about this better.#2020-06-2810:32borkdudeWhat does this mean?
Exception in thread "main" clojure.lang.ExceptionInfo: Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"} {:lib babashka.curl/babashka.curl, :coord {:git/url "", :sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"}}, compiling:(/tmp/prog.clj:3:1)#2020-06-2810:32borkdudeas a result of:
(require '[clojure.tools.deps.alpha :as deps])
(-> {:deps '{babashka.curl
{:git/url ""
:sha "2dc7f53271de3c2edc3e1474a000b9dfa7324eaf"}}
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}}
(deps/resolve-deps nil)
(deps/make-classpath nil nil))#2020-06-2810:40borkdudeI get the same result when using this coordinate with -Sdeps from the command line#2020-06-2810:59lreaddue to empty :deps? https://github.com/borkdude/babashka.curl/blob/2dc7f53271de3c2edc3e1474a000b9dfa7324eaf/deps.edn#L1#2020-06-2812:42Alex Miller (Clojure team)this should not affect anything#2020-06-2813:47lreadoh, looks like I misdiagnosed! thanks for the info!#2020-06-2813:48Alex Miller (Clojure team)it's just map merges and stuff in the code, so I would not expect it to violate any Clojure-ish expectations here#2020-06-2811:02borkdudeThat appears to be it. Weird.#2020-06-2811:03borkdudeThanks!#2020-06-2811:05lreadYou are most welcome, by chance I hit that one yesterday.#2020-06-2812:43Alex Miller (Clojure team)do you have a repro of that?#2020-06-2811:07borkdudehttps://clojure.atlassian.net/browse/TDEPS-156#2020-06-2811:26borkdude@lee Hmm, now it's coming back:
$ clj -Sdeps "{:deps {babashka.curl
{:git/url \"\"
:sha \"cbf8429addac1d2f71ac8ddfb847657223fdf0f7\"}}}" -Spath
Error building classpath. Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "cbf8429addac1d2f71ac8ddfb847657223fdf0f7"}
#2020-06-2811:28borkdudeit did work for:
$ clj -Sdeps "{:deps {babashka.curl
{:git/url \"\"
:sha \"1dc0f61d7baabee1f009543d60bfb79ee4a7101c\"}}}" -Spath#2020-06-2811:29borkdudewhile the commit cbf8 didn't do anything relevant compared to 1dc0with the deps.edn file....#2020-06-2811:31borkdudeAdding :deps/manifest :deps seems to fix it.#2020-06-2811:33borkdudeAdded this to the issue#2020-06-2811:53borkdudeI'm also getting this for a local/root dep:
$ clj -Sdeps "{:deps {org.clojure/tda {:local/root \"/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha\" }}}" -Spath
Error building classpath. Manifest type not detected when finding deps for org.clojure/tda in coordinate #:local{:root "/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha"}#2020-06-2812:03Alex Miller (Clojure team)Are they reproducible with -Sforce#2020-06-2812:04borkdudesame result#2020-06-2812:05borkdude/tmp $ clj -Sdeps "{:deps {babashka.curl
{:git/url \"\"
:sha \"cbf8429addac1d2f71ac8ddfb847657223fdf0f7\"}}}" -Spath -Sforce -Srepro
Error building classpath. Manifest type not detected when finding deps for babashka.curl/babashka.curl in coordinate {:git/url "", :sha "cbf8429addac1d2f71ac8ddfb847657223fdf0f7"}#2020-06-2812:38Alex Miller (Clojure team)@borkdude I can't repro any of your failing cases#2020-06-2812:39borkdudeHah. Worked after a rm -rf ~/.gitlibs#2020-06-2812:39borkdudeNot for
clj -Sdeps "{:deps {org.clojure/tda {:local/root \"/Users/borkdude/Dropbox/dev/temp/tools.deps.alpha\" }}}" -Spath
though#2020-06-2812:41borkdudeThat's probably another problem. It does work with a vanilla checkout of tools.deps:
clj -Sdeps "{:deps {org.clojure/tda {:local/root \"tools.deps.alpha\" }}}" -Spath#2020-06-2812:41borkdudeI'll close the issue.#2020-06-2812:44Alex Miller (Clojure team)tools.deps.alpha has a couple build steps (java compilation, resource replacement) so it's not a good example lib for use as local/git dep#2020-06-2812:50Alex Miller (Clojure team)btw, in the near future, using unqualified lib symbols is going to be deprecated and will complain, so "babashka.curl" should be changed to "babashka.curl/babashka.curl" etc#2020-06-2812:51Alex Miller (Clojure team)(and this is a step towards actually removing support for that)#2020-06-2813:24borkdude@alexmiller I know. I had already compiled the java and added target/classes to the paths#2020-06-2813:24borkdudeBut still didn’t work probably due to the resource replacement #2020-06-2813:26borkdudeI’m trying to get the native tools deps working using Ghadi’s git shell branch of the git libs library, but this stuff got me distracted for now #2020-06-2813:30borkdudeShould probably push the compiled assets to Clojars and work with that instead of tools deps deps #2020-06-2813:36borkdudeBtw, the url of a git dep is already a unique identifier. Why should a user have to make up some fully symbol keyword in the first place?#2020-06-2813:41Alex Miller (Clojure team)well you can just mvn clean install and use that local snapshot artifact too - sometimes that's eaesiest for maven built things#2020-06-2813:47Alex Miller (Clojure team)re unique id, for the exact same reasons namespaces are useful for global differentiation, group-ids are a good idea for qualified names. using same symbol group+artifact was always a limited idea. lein/clojars made it too easy and common. we're not going to perpetuate making it implicit.#2020-06-2813:52borkdudeBut why have a name for a git dep at all? #2020-06-2813:54Alex Miller (Clojure team)because names are the identity of the lib, which may exist at different coordinates and you want to avoid putting the "same" lib on the classpath from multiple locations#2020-06-2813:55Alex Miller (Clojure team)(admittedly still some work to do on cross-coordinate type version selection, but the hooks are there)#2020-06-2813:57borkdudeGood point. The only use case for doing that would be when you would mount different dirs from the same git dep at different SHAs, which seems far fetched. Still if you would use the URL as the identifier you would be able to detect conflicts more easily since different people are going to use different qualified syms for the same lib?#2020-06-2908:07rickmoynihanNot that far fetched as we actually do this in one or two places.
One of our apps supports different customer configurations via t.d.a aliases, and there is essentially a :customer/all alias that pulls together a combined classpath for all customers, for working at the REPL (which lets you switch between customers).
One of the deps in that alias contains fixture data for all customers with a :deps/root for each customer, and it’s possible for the sha’s for customers to drift slightly from each other (even though they’re all in the same repo). So this t.d.a feature is very handy, as otherwise we’d need a fixture data repo per customer, which would make updating fixture harder across customers harder.
Essentially it just means the artifact id in this becomes git url + deps-root. It makes a lot of sense when you use :deps/root.
Normally they all reference the same SHA, but sometimes it’s pragmatic to apply changes to one customer first, before rolling out wider. This can also happen when you want a branch to built repeatably by C.I. (before you’ve applied the change across all customers).#2020-06-2908:17rickmoynihanWe then coin a local dep-id of something like: swirrl/fixtures.customer-name#2020-06-2908:19borkdudeYeah, that does make sense.#2020-06-2908:21rickmoynihanIt’s essentially just a lighter weight way to manage separate dependencies that you often make cross-cutting changes over.#2020-06-2908:24rickmoynihanThough I do suspect t.d.a might have some bugs resolving these sometimes — I’ve seen a few cases where when fetching these deps it seems to either blow up or mix up the commits, e.g. it would resolve customer-b to customer-a’s sha.
I suspect it might be related to the parallel resolution stuff. cc @alexmiller#2020-06-2908:24rickmoynihanIt’s quite hard to reproduce though#2020-06-2913:09Alex Miller (Clojure team)yes, I am almost certain that it's parallel resolution of git deps#2020-06-2813:58Alex Miller (Clojure team)we do not currently do that level of detection but it is something I would like to do#2020-06-2813:59Alex Miller (Clojure team)ultimately the library has to determine the identifier by which it should be known#2020-06-2814:01Alex Miller (Clojure team)the organization should be something you own or register, like domain name, trademark. third party sites like github can also confer identity - we recommend in https://clojure.org/guides/deps_and_cli to combine them by creating a group-id like github-borkdude#2020-06-2814:01borkdudeWhich the URL is already an instance of #2020-06-2814:04Alex Miller (Clojure team)url is really a combination of site (github), org, and repo. github-org/repo combines those#2020-06-2814:04Alex Miller (Clojure team)but it does not account for forks#2020-06-2814:04borkdudeInteresting :thinking_face: #2020-06-2814:05Alex Miller (Clojure team)to use a fork, you need to use the same lib, but different url, and then they diverge#2020-06-2814:05Alex Miller (Clojure team)your goal with using a fork really is to replace (identity) the lib#2020-06-2814:06Alex Miller (Clojure team)but then you have interesting questions of ownership over the lib identity#2020-06-2814:06borkdudeIf I want to maintain a fork as a real fork, and not just for PRs, I would not use the original identifier anyway? #2020-06-2814:06Alex Miller (Clojure team)well that's the question#2020-06-2814:06borkdudeSo then the URL still is the most identifying thing for it #2020-06-2814:07Alex Miller (Clojure team)if you didn't use the original, then dep resolution will give you both libs if you have a transitive dep using the original#2020-06-2814:07Alex Miller (Clojure team)and in that case, it would be good to have deps tell you that you've done this#2020-06-2814:07borkdudeIt’s the same in maven world right now #2020-06-2814:07Alex Miller (Clojure team)true#2020-06-2814:08Alex Miller (Clojure team)although probably much harder to automatically detect#2020-06-2814:09Alex Miller (Clojure team)(note that git urls are also not unique - there are multiple url formats to refer to the same repo, but I think that's a solvable problem)#2020-06-2816:27borkdudeI have got something compiling now with tda + /cdn-cgi/l/email-protection branch
One issue I stumbled upon:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: org.apache.maven.project.ReactorModelPool. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
at parsing clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(pom.clj:51)
Call path from entry point to clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(Object):
at clojure.tools.deps.alpha.extensions.pom$model_resolver.invokeStatic(pom.clj:41)
at clojure.tools.deps.alpha.extensions.pom$model_resolver.invoke(pom.clj:41)
at clojure.core.proxy$clojure.lang.APersistentMap$ff19274a.hashCode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
at java.util.Properties.getProperty(Properties.java:1125)
at com.oracle.svm.core.jdk.SystemPropertiesSupport.getProperty(SystemPropertiesSupport.java:144)
at com.oracle.svm.core.jdk.Target_java_lang_System.getProperty(JavaLangSubstitutions.java:345)
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_lang_System_2_0002egetProperty_00028Ljava_lang_String_2_00029Ljava_lang_String_2(generated:0)
I now use --allow-incomplete-classpath just to see how far it gets.
I now get a NPE in util/maven.clj on the line local-repo-mgr (.newLocalRepositoryManager system session local-repo) where system appears to be null. Might be related to the --allow-incomplete-classpath but I'll keep digging.#2020-06-2816:29Alex Miller (Clojure team)It is almost certainly related#2020-06-2816:30Alex Miller (Clojure team)There is a similar thing like this logged where an error occurs while setting up the service locator#2020-06-2816:38borkdudeSo ditching jgit for shell git works. When are you going to ditch org.apache.maven libs? 😉#2020-06-2816:47borkdudeI wonder what's the issue with this class#2020-06-2817:19borkdudePosted current findings + code here: https://github.com/borkdude/tools-deps-native-experiment#2020-06-2818:49ghadibefore using a solution like shelling out to git, I think it’s wise to re-examine the problem#2020-06-2818:49ghadiI know I sound like a broken record#2020-06-2818:56borkdudeAlex pointed me at that shell-git branch when I told him jgit didn't work with GraalVM. Just examining + listing all the problems. Shelling out to git allows me to explore the next problem without being stuck on it. There might not be only one "the problem", there might be many.#2020-06-2819:07ghadiwhat part of jgit didn't work?#2020-06-2819:08borkdudeI think the output I got was similar to this one: https://bugs.eclipse.org/bugs/show_bug.cgi?id=546175#2020-06-2819:09borkdudeI think that problem should be solvable using the approach quarkus takes: https://quarkus.io/guides/jgit#2020-06-2819:09borkdudeso if there's need, it can be made to work, I'm pretty sure#2020-06-2819:11borkdudeI added this as a note to the README here now https://github.com/borkdude/tools-deps-native-experiment/blob/master/README.md#2020-06-2908:49Jakub Holý (HolyJak)Hello! Could somebody be so kind and explain to me why clj is trying to download a dependency from Maven Central when I told it to you our company repository (I assume M.C. is always used as a back up; but it doesn't seem it has even tried?!:
$ cat deps.edn
{:mvn/repos {"telia-aws" {:url ""}}
:deps {com.oracle.jdbc/ojdbc8 {:mvn/version "12.2.0.1"}}}
$ deps.exe -Sverbose
deps.clj version = 0.0.8
...
Refreshing classpath
Error building classpath. Could not find artifact com.oracle.jdbc:ojdbc8:jar:12.2.0.1 in central ()
🙏 Thank you!#2020-06-2908:51borkdudeI think that error message just picks a repo and doesn't list all of them#2020-06-2910:45Jakub Holý (HolyJak)It seems you were right! The problem turned out to be that my URL was wrong, it was missing a part of the path. The -Sverbose option was quite useless for troubleshooting this... :-(#2020-06-2911:13Jakub Holý (HolyJak)Any idea why is clj trying to download a snapshot .jar from Central (where it isn't) when I have it in the local .m2 repo? Do I need to tell it to use the local repo??
> Error building classpath. Could not find artifact no.telia:clj-common:jar:0.1.0-20200507.122109-9 in central (https://repo1.maven.org/maven2/)
ls ~/.m2/repository/no/telia/clj-common/0.1.0-SNAPSHOT/clj-common-0.1.0-20200507.122109-9.jar
/Users/me/.m2/repository/no/telia/clj-common/0.1.0-SNAPSHOT/clj-common-0.1.0-20200507.122109-9.jar
The dependency is declared as:
no.telia/clj-common {:mvn/version "0.1.0-20200507.122109-9"} ;; a particular SNAPSHOT
#2020-06-2911:16borkdudeSNAPSHOT versions are refreshed from external repositories at least once a day (by default, you can configure this in .m2 settings I believe)#2020-06-2911:16borkdudeit's maven behavior#2020-06-2911:20Jakub Holý (HolyJak)Ah, ok. And if it fails to refresh, it just crashes without using the old version?#2020-06-2911:21borkdudehttps://stackoverflow.com/questions/1348603/how-can-i-get-maven-to-stop-attempting-to-check-for-updates-for-artifacts-from-a#2020-06-2911:21Jakub Holý (HolyJak)I found https://maven.apache.org/ref/3.6.2/maven-settings/settings.html#class_snapshots, I can try to set updatePolicy to never...#2020-06-2913:25Alex Miller (Clojure team)deps doesn't currently expose this setting#2020-06-2915:51Jakub Holý (HolyJak)So it doesn't take all of settings.xml into account, only parts of it (ie auth)?#2020-06-2915:52Alex Miller (Clojure team)it does not consider that setting, it does take more than just auth though#2020-06-2915:52Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli describes supported stuff#2020-06-2915:53Alex Miller (Clojure team)specifically, also supports mirrors and proxies#2020-06-2915:53Alex Miller (Clojure team)and http properties#2020-06-2911:32Jakub Holý (HolyJak)Hm, it seems that clj simply ignores my local .m2/repository. Now it is trying to download com.oracle.jdbc:ucp:pom:12.2.0.1 even though ~/.m2/repository/com/oracle/jdbc/ucp/12.2.0.1/ucp-12.2.0.1.jar exists...:
$ cat deps.edn
{:deps {com.oracle.jdbc/ucp {:mvn/version "12.2.0.1"}}}
$ clj
Error building classpath. Could not find artifact com.oracle.jdbc:ucp:jar:12.2.0.1 in central ()
$ ls -lh ~/.m2/repository/com/oracle/jdbc/ucp/12.2.0.1/ucp-12.2.0.1.jar
-rw-r--r-- 1 holyjak staff 1.0M Apr 2 13:55 /Users/holyjak/.m2/repository/com/oracle/jdbc/ucp/12.2.0.1/ucp-12.2.0.1.jar
Why?!#2020-06-2913:23Alex Miller (Clojure team)did you put that jar manually in your .m2?#2020-06-2913:24Alex Miller (Clojure team)if so, you may be missing some metadata in your local cache that's triggering a lookup#2020-06-2915:52Jakub Holý (HolyJak)No, via mvn install-file . And it used to work...#2020-06-2915:56Alex Miller (Clojure team)based on what I see here, I would expect it to find and use the jar in your local cache#2020-06-2915:56Alex Miller (Clojure team)you might want to clj -Sforce to make sure you don't have a cached classpath or something#2020-06-2915:57Alex Miller (Clojure team)I don't think that will matter since it seems to be building the classpath when it encounters the error, but just a good habit when diagnosing clj stuff#2020-06-2920:43Jakub Holý (HolyJak)Ok, thank you!#2020-06-3011:22Jakub Holý (HolyJak)FYI my problem is fixed now. I believe something was wrong with my .m2/repository#2020-06-3012:01Alex Miller (Clojure team)Cool#2020-06-3015:46Lone RangerHeya @alexmiller I'm reading through tools.deps and you're using some pretty advanced kung fu in there. I don't suppose in all your ample free time you've given any talks or write-ups on a deep dive there have you? I'm trying to write some clojurescript CLI tools for working with deps and trying to port some of the stuff. Also, I tried running with CIDER but nREPL on the deps ran into the predictable errors for including java classpaths -- I'm curious what your REPL technique is for developing tools.deps.alpha?#2020-06-3016:44Alex Miller (Clojure team)The last conj talk is probably the most detailed info about the impl. What part are you having trouble with? Re repl, I just open it in Cursive as a Maven project#2020-06-3015:47borkdude@goomba There are a couple of tools.deps talks, they are listed in the repo#2020-06-3015:48borkdudeHmm, only the one from EuroClojure is https://www.youtube.com/watch?v=sStlTye-Kjk. But there's another one from the latest Conj.#2020-06-3015:48Lone RangerI was there for the last one. I think I'm looking for more "how to contribute" style info#2020-06-3015:49Lone Ranger😄 reading seems like a good start#2020-06-3015:49borkdude@goomba ClojureScript already has cljs.main which works great in tandem with tools.deps?#2020-06-3015:50Lone Rangeryeah of course. I'm not trying to replicate that functionality -- I'm working on something more like "pip for deps"#2020-06-3015:50Lone RangerI'm lazy and I don't like editing my deps file by hand#2020-06-3015:50Lone Rangeralso I can never remember version numbers#2020-06-3015:50Lone RangerI've got it working for clojars but there's a bunch of stuff on git and maven and I don't know how to work with those yet#2020-06-3015:51borkdudeinteresting, anything public yet?#2020-06-3015:51Lone Rangernot yet because it's still a mess but I'll send you some stuff on what I've got#2020-06-3015:53dpsuttonthere is a project out there that will search and add things to your deps files#2020-06-3015:54dpsuttonhttps://github.com/hagmonk/find-deps#2020-06-3015:57Lone RangerAhh that's great. Reason I'm going the cljs route is just b/c of startup time etc#2020-06-3015:58Lone Rangerand for personal education, I suppose#2020-06-3015:58borkdude@goomba if startup time is important, you can also use GraalVM and/or babashka#2020-06-3015:58Lone RangerI need to learn to use both of those, for sure#2020-06-3015:58borkdude(all good options, including what you're doing now of course!)#2020-06-3015:58dpsuttondidn't mean to derail your exercise. but you can peruse the good if you like 🙂#2020-06-3015:58Lone RangerI'm hear here to learn in all forms!#2020-06-3015:59Lone RangerHas GraalVM gotten to the point where we can write arbitrary CLI stuff in Clojure now?#2020-06-3015:59Lone Ranger(out of the loop)#2020-06-3015:59borkdudeyeah, but you can do a lot with babashka as well, without compiling.#2020-06-3016:00Lone RangerI need to sneak that one in at work 😈#2020-06-3016:00borkdude@goomba meta-example: https://github.com/borkdude/deps.clj
that is the clojure bash script ported to both babashka and GraalVM, so you can either run it as script or binary#2020-06-3016:01Lone Rangeryou can even babashka in powershell!? @borkdude your level of masochism is very impressive#2020-06-3016:01borkdudeyeah, Windows supported too#2020-06-3016:02Lone Rangersome heroes don't wear capes#2020-06-3016:02borkdudehow do you know I'm not wearing a cape? (haha, just kidding...)#2020-06-3016:03Lone Rangerwell with covid I'm willing to wager most aren't even wearing pants. So I was just playing the odds#2020-06-3016:04Lone RangerIt looks like find-deps , from a quick glance, is dealing with the same limitation I have#2020-06-3016:05Lone Rangerwhich is finding reagent is easy but org.clojure/core.match or org.clojure/core.async is not as easy#2020-06-3016:05Lone RangerI'm not sure how to differentiate in Maven what is Clojure and what is Java, for instance, conceptually#2020-06-3016:06Lone Rangeror... maybe it doesn't matter -- as I say it out loud. That's kind of the point of Clojure, isn't it? 😆#2020-06-3016:17dpsuttonthere are some nice todos and vision in that find-deps lib that you could tackle#2020-06-3016:17dpsuttondealing with rankings and preferences if i recall correclty#2020-06-3016:47Alex Miller (Clojure team)If you’re talking about searching/finding deps, we’ve actually done some thinking about this in tandem with add-lib and I’ve even prototyped some of it in the add-lib2 branch of tools.deps but it’s not in a usable state right now#2020-06-3016:48Alex Miller (Clojure team)I guess really more in the context of repl dev not updating deps.edn#2020-06-3016:49Alex Miller (Clojure team)But could potentially be used for either I guess#2020-06-3022:44flefikIs there a good commit on the add-lib or add-lib2 branch that can be checked out and played around with?
I'm thinking of adding a filewatcher on deps.edn and just automatically add any new libraries that are added there without having to restart the repl.#2020-06-3023:29seancorfieldI have an example in my dot-clojure repo that I use all the time -- but @alexmiller has cautioned that add-lib will probably break as code moves around#2020-06-3023:29seancorfield^ @UAEFFG05B#2020-07-0107:47flefikthanks!#2020-06-3022:49lreadhey @aviflax did you ever get a tools dep answer for tar.gz files ending up on your classpath? https://github.com/oracle/graal/issues/2481#2020-06-3022:50aviNo#2020-06-3022:50aviI meant to ask Alex about it here… thanks for the reminder#2020-06-3022:51aviThe comment on that issue from olpaw to the effect that the tar.gz file shouldn’t have been included in the classpath in the first place made sense to me.#2020-06-3022:51aviSo I thought maybe this might be one of those places wherein tools.deps is deviating from commonly assumed Maven behavior/convention#2020-06-3022:52aviI don’t really know ¯\(ツ)/¯#2020-06-3022:54lreadYeah, I bet @alexmiller knows! Here’s the dep that brings in tar.gz files: https://github.com/FundingCircle/repro-graalvm-mac-issue/blob/83abc805467808221cf48108f735527e3e3de9fa/deps.edn#L3#2020-06-3022:56lreadI guess this does not happen with leiningen, as @borkdude references it from babashka project.clj https://github.com/borkdude/babashka/blob/e667bb5d2f344fdfd12ce12639c768d80e252138/project.clj#L57#2020-06-3022:58aviRight#2020-06-3023:00aviAs I mentioned in that graal issue, the tar.gz file comes from https://search.maven.org/artifact/org.graalvm.truffle/truffle-nfi/20.1.0/jar for org.graalvm.truffle/truffle-nfi 20.1.0:
<dependency>
<groupId>org.graalvm.truffle</groupId>
<artifactId>truffle-nfi-native-linux-amd64</artifactId>
<version>20.1.0</version>
<type>tar.gz</type>
</dependency>#2020-06-3023:01aviI assume must be some use case for including such a file in a pom, although I have no idea what it is. And I assume that Maven, and whatever mechanism Leiningen uses to work with Maven, somehow filters these files out when/where they’re not needed/useful.#2020-06-3023:23Alex Miller (Clojure team)That’s a native dep and there is a facility in Maven to support loading those for the jvm#2020-06-3023:24Alex Miller (Clojure team)That Linux and amd64 there are actually parsed and used to select the right native dep for the os/arch#2020-06-3023:25Alex Miller (Clojure team)I think lein supports at least some of this too, but deps does not#2020-06-3023:26lreadThanks @alexmiller!#2020-06-3023:29seancorfieldI have an example in my dot-clojure repo that I use all the time -- but @alexmiller has cautioned that add-lib will probably break as code moves around#2020-06-3023:34lread@aviflax I have something working for my project, I’ll head over to #graalvm to followup#2020-06-3023:34aviGreat!#2020-07-0111:38practicalli-johnDoes the clojure or clj command from Clojure CLI tools read a user.clj file if it contains (ns user) expression when starting the REPL?
With Leiningen projets using a user or dev/user.clj file was one way to start component services.
For a Clojure CLI project, if the dev directory was added as an extra path under a :dev alias and clj -A:dev used to start the REPL, should the dev/user.clj file be automatically evaluate on REPL start?
I though I had this working the other week, but not sure if I imagined this now 🙂#2020-07-0111:47borkdude@jr0cket I think the answer is yes, it's not tooling dependent: https://github.com/clojure/clojure/blob/d07ef175c700329f7afbef8770332b6247a34a49/src/jvm/clojure/lang/RT.java#L486#2020-07-0111:54practicalli-johnAh, I have it working in terms of requiring a namespace and evaluating a funtion call. However, using (in-ns main-project-namespace) does not seem to switch to the namespace (or if it does, then maybe it been switched back to user afterward).#2020-07-0111:57borkdude@jr0cket load-file also switches back to the original namespace afterwards, because it uses binding + *ns*#2020-07-0111:57borkdudeI imagine something similar is happening here#2020-07-0112:13practicalli-johnOkay, I dont have to change the namespace, I can require the relevant libraries and use fully qualified namespace function calls.
https://practicalli.github.io/clojure/clojure-tools/configure-repl-startup.html
If there is a way in Clojure CLI tools to change the namespace using a user.clj file, I would be interested to know, but not essential. I can use the --eval option to run the relevant expressions.#2020-07-0112:15borkdude@jr0cket This seems to work:
$ clj -e "(ns foo.bar) (alter-var-root #'*ns* (constantly 'foo.bar))" -r
#2020-07-0112:17borkdude@jr0cket or even:
$ clj -e "(ns foo.bar) (alter-var-root #'*ns* (constantly 'foo.bar)) (clojure.main/repl)"
foo.bar
foo.bar=>
#2020-07-0112:18practicalli-johnI can set the namespace easily using --eval option on the command line. Unfortunately the above do not work from within user.clj. Ideally I'd like use it with rebel readline, which is the REPL UI I use if I am just using the REPL directly rather than an editor. --eval doesnt work when using the :rebel alias#2020-07-0112:22borkdude@jr0cket Very well.
$ clj -R:rebel -e "(ns foo.bar) (alter-var-root #'*ns* (constantly (find-ns 'foo.bar)))" -m rebel-readline.main
#object[clojure.lang.Namespace 0x46cf05f7 "foo.bar"]
[Rebel readline] Type :repl/help for online help info
foo.bar=>
#2020-07-0112:23practicalli-johnSuccess 🙂 I have learnt many things today, thank you.#2020-07-0119:52Drew VerleeIs there a way to do a universal exlucions? i see you can attach exluded to individual deps, but what if i wont want this lib from any dep?#2020-07-0119:55borkdude@drewverlee you can use :classpath-overrides:
{:aliases {:babashka {:classpath-overrides {org.clojure/clojure nil
org.clojure/spec.alpha nil
org.clojure/core.specs.alpha nil}}}}#2020-07-0119:57borkdudeThis only works in an alias#2020-07-0119:58Drew Verleegotcha.#2020-07-0213:22arohneris there a library that provides lein deps :pedantic :abort? i.e. looks for and warns on conflicting versions?#2020-07-0213:23arohneror documentation on how conflicting versions are handled?#2020-07-0213:27delaguardoyou could use -Strace to create edn file with all that information. but I’m not aware of any tooling to make reading of that file easy#2020-07-0213:28Alex Miller (Clojure team)there are a few tools like depot#2020-07-0213:28Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-07-0213:29Alex Miller (Clojure team)or https://github.com/slipset/deps-ancient#2020-07-0213:30arohnerI’m looking to handle a jarhell situation where I declare dependencies on A and B, and they each declare a dependency on C, but with conflicting versions#2020-07-0213:31sveriYou could refernce the conflicting dependency as a top level dependency and decide for a version yourself#2020-07-0213:32arohnerRight, but I need to discover conflicts before resolving them. I’d like a tool to detect them and warn before I find out the hard way#2020-07-0213:30arohner:pedantic :abort warns about that, in (some) situations#2020-07-0213:31arohnerbut I’d also need to understand how tools.deps resolves that tree. lein’s dependencies are a vector, so deps are ordered sequentially. With a map, how does that work?#2020-07-0213:33Alex Miller (Clojure team)well I did whole conj talk about it. it's complicated#2020-07-0213:34arohnerIs the talk uploaded?#2020-07-0213:36Alex Miller (Clojure team)yeah - https://www.youtube.com/watch?v=7CM7Ef-dPWQ#2020-07-0213:33Alex Miller (Clojure team)but generally the idea is to pick the newest version#2020-07-0213:34Alex Miller (Clojure team)but it also picks consistent subtrees (so no child lib should be included if it's parent isn't)#2020-07-0213:35Alex Miller (Clojure team)clj -Strace is the best info about what it did and why - the log details each lib/version it looked at (breadth first), what it decided, and why#2020-07-0213:36Alex Miller (Clojure team)tools.deps.graph has a tool for using trace.edn files to generate a series of pictures from that that can be somewhat helpful as well#2020-07-0216:38dominicmMy argument is that pedantic abort doesn't make sense in a deps world.#2020-07-0216:40dominicmYou'll never get an older version than what is required, which eliminates 90% of problems you get with maven. In maven, it's assumed everything is conflicting. Deps assumes that newer things are compatible with older things (so the conflict is picking the older dependency?).#2020-07-0216:42dominicmHowever, the equivalent tool I think would be useful is detecting a top level dependency on an older artefact, when you've bumped the version and it transitively depends on something newer.#2020-07-0219:41markbastianIf I have a dependency specified in my main deps section and another version of that dependency under an alias, does the alias take precedence when it is active?#2020-07-0219:43markbastianAh, think I found it at https://clojure.org/guides/deps_and_cli. Just use override-deps.#2020-07-0220:15Alex Miller (Clojure team)yes#2020-07-0221:04markbastianIs there a way to specify a subdirectory when providing git coordinates for a dependency? For example, the git project is foo and within that you have the folders foo-clj and foo-py. deps.edn is in foo-clj. I want to point deps to the foo-clj folder.#2020-07-0221:06borkdude@markbastian yes, :deps/root#2020-07-0315:40arohneris there a way to exclude a dep from the tree, regardless of how it got there?#2020-07-0315:40arohner:exclusions appears to be per-dep only, but I want to assert e.g. logback should never be present#2020-07-0315:49borkdude@arohner https://clojurians.slack.com/archives/C6QH853H8/p1593633340186100#2020-07-0319:31neumannIs there a way to replace or remove paths in an alias? I want to exclude on of my source paths in just one alias.#2020-07-0319:32neumannMy current coping mechanism is to set -Sdeps '{:paths ["some" "paths" ...]}' on the command line.#2020-07-0319:45dominicm@neumann why not move the optional paths to an alias?#2020-07-0319:50neumann@dominicm I'd like to avoid that because I don't want to have to specify another -A: in all of the cases except one. The extra -A: is just another thing to forget to do.#2020-07-0319:54borkdude@neumann major hack, this removes the src path from the classpath:
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "21434e534a455445446157515210181019"}, :content ("[email protected]")}#2020-07-0319:56neumann@borkdude Nice! Yes, I was thinking of maybe using babashka. (Which is amazing! Thank you so much for creating it!)#2020-07-0320:01neumann@borkdude Something like this, using "src" as the example:
cat deps.edn | bb -I -e '(->> *input* first :paths (remove #{"src"}) vec)'#2020-07-0320:02borkdude@neumann if you need only one EDN input value:
$ cat deps.edn | bb -e '(->> *input* :paths (remove #{"src"}) vec)'
["parser" "resources" "inlined"]#2020-07-0320:02borkdude-I is for collecting a lazy seq of multiple EDN input values#2020-07-0320:03neumannOh! Very nice! Thanks!#2020-07-0418:28p-himikIf A depends on B, and my project depends on A and B but doesn't really care about the version of B as long as it's compatible with A, how should I specify this dependency on B in deps.edn?
Or should I just omit the dependency on B from deps.edn?#2020-07-0419:10seancorfieldI would figure out what version of B your chosen version of A depends on and explicitly specify that in deps.edn -- and add a comment about the version selection reason. Assuming you mean that your project has an explicit dependency on B.#2020-07-0419:23p-himikThat was exactly my thought initially. But now that I've given it some thought - wouldn't it be more practical to just omit it altogether?
If A starts to depend on a different version of B, I should be unaffected, and I won't need to change anything.
If A drops the dependency, the code will just not run because there's an explicit :require.
And specifying B in my deps.edn seems like just giving myself more work when I update my dependencies.#2020-07-0419:33seancorfieldIf A changes the version of B it depends, it also changes its own version -- so unless you are changing the version of A you depend on, you wouldn't be affected 🙂#2020-07-0419:34seancorfieldThat's my point: if you choose to depend on a newer version of A, you should also verify the compatibility of the newer version of B that it now depends on as well. Having an explicit dependency on B lets you force the earlier version of B if you need that.#2020-07-0419:35seancorfieldIf A stops depending on B, that also means a new version of A and you could decide to stay with the old version or move to the new version -- and because you have an explicit dependency on B, you code won't break (at least, not because A's transitive dependency on B went away).#2020-07-0419:36seancorfieldI really don't like relying on transitive dependencies. I much prefer to have all my dependencies listed explicitly. I've learned that over the years after a decade of Clojure in production 🙂#2020-07-0419:43p-himikHmm, I think I didn't give enough context.
I definitely agree with your reasoning about the generic case of relying on a transitive dependency. But in my particular case, the code should either break completely if B is changed/removed, or it should continue working just as I expected it to. So the compatibility with my project is not a concern.
At the same time, if I manage to forget to update B after updating A, I may introduce some incompatibilities between A and B that are hard to test.#2020-07-0419:44p-himikTo be more concrete - I'm just using ring.util.mime-type/ext-mime-type function without having an explicit dependency on ring/ring-core.#2020-07-0419:46p-himikOf course even such a dead simple function can potentially be broken with an update. But I consider it so improbable that it's not worth taking into account. Akin to how I don't expect clojure.core/map to suddenly start giving me different results.#2020-07-0419:53seancorfieldBut you have code in your project that depends on Ring namespaces?#2020-07-0419:54seancorfieldTo be concrete, what are A and B specifically in your project?#2020-07-0420:02p-himikA - yada/core
B - ring/ring-core
I need only ring.util.mime-type/ext-mime-type from B, which I consider stable in a sense that any changes around it either break the app startup or don't affect the app at all.#2020-07-0420:02p-himikI don't depend on anything ring-related but that small function.#2020-07-0420:08seancorfieldAh, OK. If that is the only Ring namespace you're touching then maybe I would agree and just depend on Yada. It still feels a bit sketchy to me. We depend on Ring and we don't use that var -- we have our own MIME type structures (because we want a specific set of supported values, rather than a generic list).#2020-07-0420:13p-himikI wish I would have a specific set. :)
Yeah, I can see how it can feel sketchy. I'm still not sure about the way I work with my dependencies at all right now, given that there's also CLJS and a build process that involves shadow-cljs. Maybe this all could be made nice and simple with aliases. An unrelated to the rest of the thread question - aliases cannot be grouped together under a new alias, right? So I will probably have to write a script to call clj -A:al1:al2:al3.#2020-07-0420:20seancorfieldNope. Grouping of aliases is a common request.#2020-07-0420:21p-himikI see.
Anyway, thanks for the input Sean. Much appreciated. :)#2020-07-0420:22seancorfieldIt's why at work we have a build shell script that wraps the clojure script so we can have "higher-level" aliases.#2020-07-0420:24p-himikRight, I've seen this pattern before. Sometimes even with -Sdeps, to avoid cluttering the main deps.edn I guess or to make the script more portable.#2020-07-0420:38seancorfieldOur build script serves two purposes: to allow us to run a series of clojure commands from a single invocation, and to provide various shorthands for common combinations of aliases (our aliases tend to be very granular). But we're all Clojure / backend. I suspect if we were managing ClojureScript as well, we'd be even more reliant on our wrapper script!#2020-07-0420:41seancorfieldI'm looking forward to tools.build -- I suspect it will help simplify our tool chain quite a bit.#2020-07-0420:49p-himikI don't think I've heard of it and I can't find anything that seems relevant. Is there an announcement or something like that?#2020-07-0420:59seancorfieldAlex mentioned it in an Inside Clojure blog post and has mentioned it several times here.#2020-07-0421:00p-himikHeh, found it: "tools.build - shh, top secret"#2020-07-0421:06seancorfieldGiving Cognitect's solid track record of producing simple, well-considered, composable tooling that addresses core Clojure developer needs, I'm looking forward to whatever this turns out to be 🙂#2020-07-0508:36dominicm@p-himik I've been considering writing a tool which ensures your top level dependencies are at least newer than those required by your dependencies#2020-07-0508:38p-himikThat may be useful. 👍 Although not for my problem in particular because I ended up just dropping the explicit dependency. :)#2020-07-0610:36dominicmI think that's wrong. You depend on that library.#2020-07-0610:50p-himikWe've discussed that at length with Sean just a bit above. :) It's not that wrong in my particular context, I think.#2020-07-0610:52p-himiktl;dr: I depend on the library, but I don't depend on the version of the library. And if something breaks, the app just won't start. So by not using an explicit dependency, I cannot really run time runtime errors.
But with an explicit dependency, I definitely can if I forget to update it along with other libraries that depend on it as well.#2020-07-0513:06Alex Miller (Clojure team)it would probably be pretty easy to read a trace file (or use the api and the trace data returned) to see that#2020-07-0610:36dominicmOh really? Interesting. I'd been resolving each dependency independently and comparing to the top level. Your idea is better.#2020-07-0610:37dominicmI wonder if you could build a whole lint suite out of the trace, catching other common errors (hello Jackson and jetty).#2020-07-0611:40fabraoHello all, how to listen in all interfaces for this
:aliases {:dev {:jvm-opts ["-Dclojure.server.repl={:port,5555,:accept,clojure.core.server/repl} -Dfile.encoding=UTF-8"]}}
?#2020-07-0611:42fabraoI tried :address,10.10.20.whatever and it didn´t work#2020-07-0611:45fabrao-> Exception in thread "main" java.lang.NumberFormatException: Invalid number: 10.10.20.82#2020-07-0612:33borkdude@fabrao try escaped double quotes around the IP#2020-07-0613:09fabrao@borkdude :aliases {:dev {:jvm-opts ["-Dclojure.server.repl={:port,5555,:address,\"10.10.20.82\",:accept,clojure.core.server/repl} -Dfile.encoding=UTF-8"]}} -> same error#2020-07-0613:09fabraoI tried with ' too, not working#2020-07-0614:02Alex Miller (Clojure team)that seems weird to me. can you share the full output?#2020-07-0614:16fabraoException in thread "main" java.lang.NumberFormatException: Invalid number: 10.10.20.82
at clojure.lang.EdnReader.readNumber(EdnReader.java:224)
at clojure.lang.EdnReader.read(EdnReader.java:136)
at clojure.lang.EdnReader.readDelimitedList(EdnReader.java:766)
at clojure.lang.EdnReader$MapReader.invoke(EdnReader.java:680)
at clojure.lang.EdnReader.read(EdnReader.java:145)
at clojure.lang.EdnReader.read(EdnReader.java:111)
at clojure.lang.EdnReader.readString(EdnReader.java:67)
at clojure.edn$read_string.invokeStatic(edn.clj:46)
at clojure.edn$read_string.invokeStatic(edn.clj:37)
at clojure.core.server$parse_props$fn__8907.invoke(server.clj:152)
at clojure.core.protocols$naive_seq_reduce.invokeStatic(protocols.clj:62)
at clojure.core.protocols$interface_or_naive_reduce.invokeStatic(protocols.clj:72)
at clojure.core.protocols$fn__8159.invokeStatic(protocols.clj:169)
at clojure.core.protocols$fn__8159.invoke(protocols.clj:124)
at clojure.core.protocols$fn__8114$G__8109__8123.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
at clojure.core.protocols$fn__8136.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8136.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8088$G__8083__8101.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6828)
at clojure.core.server$parse_props.invokeStatic(server.clj:146)
at clojure.core.server$start_servers.invokeStatic(server.clj:157)
at clojure.core.server$start_servers.invoke(server.clj:157)
at clojure.lang.Var.invoke(Var.java:384)
at clojure.lang.RT.doInit(RT.java:493)
at clojure.lang.RT.init(RT.java:467)
at clojure.main.main(main.java:38)#2020-07-0614:20Alex Miller (Clojure team)you need to put that second -D in a separate string#2020-07-0614:21Alex Miller (Clojure team){:aliases {:dev {:jvm-opts ["-Dclojure.server.repl={:port,5555,:address,\"10.10.20.82\",:accept,clojure.core.server/repl}" "-Dfile.encoding=UTF-8"]}}}#2020-07-0614:21fabraolet me try#2020-07-0614:22Alex Miller (Clojure team)I can't repro what you're getting#2020-07-0614:22Alex Miller (Clojure team)I get an error binding that address, but that's to be expected#2020-07-0614:26Alex Miller (Clojure team)I would expect that error w/o the quotes around the address#2020-07-0614:27borkdude@fabrao Are you doing this from a deps.edn file or from a cmd line with -Sdeps?#2020-07-0614:31fabraodeps.edn#2020-07-0614:34borkdudelinux, macOS, Windows?#2020-07-0614:39markbastianFrom what I've read, it sounds like a fairly standard approach to publishing library artifacts with deps is to create a pom with clj -S:pom and then use mvn deploy. When you do this model, are there any best practices regarding how to keep things in sync? For example, do you:
1. Regenerate the pom every time you want to cut a release? This seems wrong as you lose all of your version and other info.
2. Generate the pom once and then maintain it? This seems like a good route, but the main downside I see is keeping dependencies in sync.
I am assuming #2 is the best option. Any tips for syncing dependencies?#2020-07-0615:35seancorfield@markbastian clojure -Spom does not overwrite anything in your pom.xml except <dependencies>#2020-07-0615:48Alex Miller (Clojure team)And repositories and source dir#2020-07-0615:35seancorfieldAnd for publishing to Clojars, you really need a lot more in your pom.xml file than clojure -Spom provides.#2020-07-0615:38seancorfieldIf you create a new project with clj-new (typically clj -A:new lib myname/myproject) it will have a suitable pom.xml file and then for each release you just run clojure -Spom, update the version and tag in the pom for the upcoming release, commit and push those pom changes, add a release on GitHub and pull, clojure -A:jar (because clj-new makes that alias available), and then clojure -A:deploy (again, from clj-new's template)#2020-07-0615:39markbastianAh, I hadn't realized what the pom command did. I thought it generated a pom (replacing the existing). I just did an experiment based on your comment and suddenly things make a lot more sense. Very cool! Thanks for the info.#2020-07-0615:42seancorfieldPer clojure -h:
-Spom Generate (or update an existing) pom.xml with deps and paths
#2020-07-0615:40seancorfieldThe :jar alias is for depstar and it gets the group, artifact, and version from pom.xml and builds everything you need inside the .jar incl. manifest, multi-release, etc.#2020-07-0615:40fabrao@borkdude It´s Windows#2020-07-0615:40fabrao:aliases {:dev {:jvm-opts ["-Dclojure.server.repl={:port,5555,:address,\"10.10.20.82\",:accept,clojure.core.server/repl}" "-Dfile.encoding=UTF-8"]}} didn´t work#2020-07-0615:40borkdudeaaah#2020-07-0615:41seancorfieldThe :deploy alias is for deps-deploy and it expects
CLOJARS_PASSWORD=<your secret>
CLOJARS_USERNAME=<your username>
to be set in your env vars#2020-07-0615:41borkdude@fabrao windows is a bit different here maybe. can you try "" instead of \"?#2020-07-0615:41borkdudeoh no, that makes the EDN file invalid#2020-07-0615:41borkdudehmm#2020-07-0615:42borkdudetry two \"'s#2020-07-0615:43borkdudeor just try WSL2 🙂#2020-07-0615:44borkdudebut I suspect doubling the double quotes works#2020-07-0615:44fabrao\"\"10.10.20.82\"\" and ""10.10.20.82"" -> Invalid Number#2020-07-0615:45seancorfieldOr perhaps just read the t.d.a. wiki about Windows? https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#escaping-quotes#2020-07-0615:45borkdude@seancorfield he isn't using -Sdeps from the command line, this is in an EDN file#2020-07-0615:45seancorfieldI am pretty sure the same recommendations apply...#2020-07-0615:46fabraosorry ->
Error: Could not find or load main class 10.10.20.82
#2020-07-0615:46fabraoCaused by: java.lang.ClassNotFoundException: 10.10.20.82
#2020-07-0615:47seancorfield@fabrao There's a #clj-on-windows channel that has real Windows users that have probably solved this problem. You're just kind of flailing right now...#2020-07-0615:48fabraook Sean#2020-07-0615:48seancorfield(I would probably try \\" in the string in deps.edn tho')#2020-07-0615:51fabraowith \\" didn´t work too#2020-07-0615:52seancorfieldI'm firing up a Windows VM. I'll try a few years and report back in #clj-on-windows#2020-07-0616:01borkdude@fabrao, this worked for me on Windows:
{:aliases {:dev {:jvm-opts ["-Dclojure.server.repl={:port,5555,:address,\"\"\"127.0.0.1\"\"\",:accept,clojure.core.server/repl}" "-Dfile.encoding=UTF-8"]}}}
so doubling the escaped quotes, just like I suspected 😎#2020-07-0620:08seancorfieldTechnically that is tripling the escaped quotes#2020-07-0620:09borkdudeoh wait, I didn't even know I did that.#2020-07-0620:09seancorfieldAnd weird that it seems to work across all platforms... 😐#2020-07-0620:11borkdudehaha#2020-07-0616:01borkdudeand I verified with netstat that the server was indeed running with clojure -A:dev#2020-07-0616:19seancorfield(sorry, got distracted by a phone call with my mum in England, but I was getting there I hope...)#2020-07-0616:39seancorfieldFWIW, that format with \"\"\" around the host IP seems to work on Linux and macOS as well. Which surprised me a bit. I was able to start a REPL with that alias on Powershell, CMD (via powershell clj), and WSL -- as well as on macOS (Terminal) -- and connect to it via telnet to verify the socket REPL connection.#2020-07-0618:18cap10morganIs there a way to just install the deps w/ tools-deps? I'm looking for an equivalent to lein deps (for caching them in a Docker image build).#2020-07-0618:21borkdude@cap10morgan you can just do -Spath and will print the classpath, downloading the deps as a side effect#2020-07-0618:27seancorfield-Spath, -Stree, and -e nil seem to be the most commonly mentioned ways to have the CLI just download deps, optionally print some information about them, and just exit.#2020-07-0618:28seancorfieldI like -Stree myself since it's easier to read than -Spath if I want to verify the actual dependencies being used.#2020-07-0618:28seancorfieldOr just -e nil if I don't care.#2020-07-0618:29cap10morganGreat. Thanks!#2020-07-0619:36fabrao@borkdude It worked !!! Strange things#2020-07-0619:47borkdude@fabrao Might be good to add to the wiki.#2020-07-0623:47Spencer AppleWith the monorepo root CLJ_CONFIG deps.edn
{:aliases {:dev {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]}}}
And a local deps.edn
{:aliases {:dev {:extra-deps {...}}
I can pull configuration from the monorepo root CLJ_CONFIG
CLJ_CONFIG=../.clojure clojure -A:dev
However, this doesn’t pass down the :jvm-opts from the CLJ_CONFIG deps.edn. Is this working as intended? Is there a better way to do this? (I could create another alias with this jvm-opt or perhaps use environment variables.)#2020-07-0623:48seancorfieldAliases do not compose.#2020-07-0623:49Spencer Applethere’s a line in the guide:
> If multiple maps with these keys are activated, `:jvm-opts` concatenate#2020-07-0623:49Spencer Applethat confused me#2020-07-0623:50seancorfieldRight, that means across all the aliases you specified.#2020-07-0623:50Spencer Appleah gotcha, thanks for the help!#2020-07-0623:51seancorfieldBut they have to be different aliases. I answered in the main channel how we handle this at work.#2020-07-0623:50seancorfieldYou can either have an alias in the user-level file (via CLJ_CONFIG) or in the project-level file. What we did is to have something like :dev-defaults in the user-level (monorepo) file and then :dev as an empty alias there and overridden in each project -- and then do CLJ_CONFIG=../defaults clojure -A:dev-defaults:dev#2020-07-0623:51Spencer Applegreat, this seems like a good solution!
And thanks for the quick answer 😄#2020-07-0623:51Spencer Applegreat, this seems like a good solution!
And thanks for the quick answer 😄#2020-07-0623:52seancorfieldWe have about thirty subprojects in our monorepo (and over 100,000 lines of code overall).#2020-07-0623:53Spencer AppleThank you for sharing your deps.edn btw, I should have studied that a bit more closely…#2020-07-0623:55seancorfieldLMK if you have any Qs -- feel free to DM me any time about it.#2020-07-0715:44deactivateduser@seancorfield might be time to consider creating a #seans-deps-edn channel here? 😉#2020-07-0723:13deactivateduserIs there a preference or recommendation between these two (equivalent) styles of specifying a version:
{:deps {some/library {:mvn/version "2.7.42"}}}
and
{:deps {some/library #:mvn {:version "2.7.42"}}}
?
(yes I realise this is a stylistic question not a technical one - I’m wondering if there’s a preferred / recommended / idiomatic style for deps.edn content)#2020-07-0723:32hiredmanI'd say namespaced maps are not mentioned in https://github.com/edn-format/edn, so if deps.edn is intended to live up to the .edn on the end, leave out the namespaced maps#2020-07-0723:34deactivateduserOoooh that’s very interesting! Thanks!#2020-07-0800:30Alex Miller (Clojure team)the Clojure edn reader does read namespace maps but hiredman is correct that this is not currently part of the spec#2020-07-0800:07Alex Miller (Clojure team)the first#2020-07-0809:07Jakub Holý (HolyJak)Hello! How do I run clj to only download dependencies for a profile and exit at once? Like this: clojure -R:nrepl -e "(println :DOWNLOADED)" ? Thanks!#2020-07-0809:39dominicm@holyjak clojure -R:nrepl -Spath should do it :)#2020-07-0810:51practicalli-johnUsing -Spath also gives you more data should you need to debug anything (a Sean Corfield tip). I am using this form of command in my continuous integration yml configuration (in the step where I would have use lein deps with Leingingen projects)
- run: clojure -R:test:runner -Spath
https://practicalli.github.io/clojure/testing/integration-testing/circle-ci/random-clojure-function.html#2020-07-0811:00Jakub Holý (HolyJak)thanks!!!#2020-07-0819:22fabraohello all, what is the magic about starting so fast in comparing with lein and boot ?#2020-07-0819:25souenzzo- less deps, clean design
- do less things (lein/boot do A LOT more stuff)
- cache classpath "in bash"
...#2020-07-0819:25borkdudefire up only one java process (when classpath is cached)#2020-07-0819:27souenzzodesign choise example:
lein help will build a classpath, start a JVM, load all plugins, ask to every plugin if it implements "help" and eventually show the help message
clj --help is handled without start any JVM#2020-07-0819:51fabraothis saved me time and memory, many thanks who made it#2020-07-0911:16ejelome...
Hello, I have a deps.edn question, I tried to replicate what I use in the terminal as config options with deps.edn, but it seems that the generated .js file logs some errors,
Here's the CLI:
clj -m cljs.main -o public/js/main.js -O advanced -c hello-world.core
Here's what I expected as equivalent deps.edn version:
{:deps {org.clojure/clojurescript {:mvn/version "1.10.764"}
reagent {:mvn/version "1.0.0-alpha2"}}
:main "hello-world.core"
:optimizations :advanced
:output-dir "public/js/out"
:output-to "public/js/main.js"}
Then I run it with:
clj -m cljs.main -co deps.edn
The generated .js file from CLI works fine, but the generated version from deps.edn options gives an error:
main.js:314 Uncaught TypeError: Cannot read property 'mg' of undefined
at Mb (main.js:314)
at Tn (main.js:699)
at Rn (main.js:698)
at Nn.h.Me (main.js:718)
at Function.Wn.A (main.js:710)
at Nn.h.Le (main.js:718)
at Function.Wn.j (main.js:710)
at ep (main.js:761)
at main.js:761
... and it doesn't seem to exit from the terminal (the terminal becomes unusable)#2020-07-0911:26euccastroyou may want to try asking in #clojurescript?#2020-07-0911:27ejelomehey, thanks, that's probably the right place#2020-07-0919:13Wesley MatsonAre there any default aliases or good ways to get some extra resource paths added to the classpath for all local repls started with the clojure command? (some key in ~/.clojure/deps.edn perhaps?)
I’d prefer not to need to add a -A flag to all invocations across the several IDEs I use#2020-07-0919:14Alex Miller (Clojure team)if you mean an alias that is automatically supplied, no#2020-07-0919:14Alex Miller (Clojure team):paths in ~/.clojure/deps.edn will be used everywhere#2020-07-0919:15Wesley MatsonI thought :paths got merged across the candidate deps.edns with a “last one wins” strategy#2020-07-0919:16Alex Miller (Clojure team)oh yes, sorry#2020-07-0919:21Wesley MatsonDoes :extra-paths only exist for aliases? My use case is that I’ve got a local directory for various properties files that’s global to all projects (shared DB/service locations, and whatnot for local development, shared across several languages).#2020-07-0919:33Alex Miller (Clojure team)yes, only for aliases#2020-07-0919:48Wesley MatsonOkay, thanks for your responsiveness and time 🙂#2020-07-0922:12souenzzo@alexmiller I see that :git/url "file:///foo" isn't supported by gitlibs
I already find that it's a issue with clojure.tools.gitlibs.impl/clean-url
A PR/Patch is welcome about this problem?#2020-07-0922:13Alex Miller (Clojure team)I'm unclear why this is better than using :local/root#2020-07-0922:14Alex Miller (Clojure team)git deps let you get the dep from a remote git repo. if you already have it locally, just use it?#2020-07-0922:14Alex Miller (Clojure team)I guess a git dep would let you use multiple shas#2020-07-0922:14Alex Miller (Clojure team)can you walk back and say how you're using it?#2020-07-0922:15Alex Miller (Clojure team)stepping away for evening but will check later#2020-07-0922:19souenzzoLet's go:
I'm trying to deploy my application with clojure -Sdeps "{:deps {my/app {:git/url \"...\" :sha \"...\"}}}" -m my.app.start
But I'm on AWS/CodeCommit
CodeCommit on ephemeral EC2 recommends use https with credential provider
jgit do not support credential provider
so I'm trying to use git-cli to clone, then use local file
I can run directly but I want to keep my SHA control in the "start command line"#2020-07-0923:29souenzzoTL; DR: How to use git deps with CodeCommit?#2020-07-1011:27jeroenvandijk@U2J4FRT2T You have to put some magic in your .gitconfig . Something like:
[credential ""]
helper = !aws --profile dev1 codecommit credential-helper
And maybe enfore https (not sure, don’t remember):
[url ""]
insteadOf =
#2020-07-1011:28jeroenvandijkIf I remember correctly this did work with tools.deps#2020-07-1016:13kennyDoes anyone know of an up-to-date guide on deploying a jar to a private maven s3 repo? I've tried several different guides and they all reference deprecated or archived maven build plugins. It seems like such a simple task but there seems to be dozens of different ways to get it done.#2020-07-1016:31kennyOh interesting... it appears all I need to do is
aws s3api put-object --bucket my-bucket --key release/my-group-id/my-artifact-id/my-version/my.jar
#2020-07-1018:06kennyActually, tools-deps does not seem completely satisfied with the above approach, printing Download corrupted: Checksum validation failed, no checksums available#2020-07-1018:06kennyGuessing it expects some additional files in s3 that are not placed via the simple put-object action.#2020-07-1019:18Drew VerleeIf i have a lib calling a method that doesn't exist and i suspect its because two version of the lib exist and the one without it won, how can i verify this?#2020-07-1019:19Drew Verleethe stacktrace lists the class package and class name and i compared this against the output of Stree and their isn't a direct overlap#2020-07-1019:20Drew Verleelikley because stree is output a higher level of information#2020-07-1019:24seancorfield@drewverlee Which lib? Which class? Which method?#2020-07-1019:27Drew Verleemethod: ""com.google.common.collect.ImmutableList.toImmutableList()Ljava/util/stream/Collector;"
lib that calls it: pack/pack.alpha https://github.com/juxt/pack.alpha.git @ 0c20aa5#2020-07-1019:28Drew Verleewell more accurately:
[{:type java.lang.NoSuchMethodError,
:message
"com.google.common.collect.ImmutableList.toImmutableList()Ljava/util/stream/Collector;",
:at
[com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep
makeList
"BuildAndCacheApplicationLayerStep.java"
67]}]
#2020-07-1019:29Drew Verleeso google.cloud.tools.jib.builder calls com.google.common.collect.ImmutableList.toImmutableList and it doesn't exist. I assume that error isn't saying the arguments were bad in someway, its saying that method isn't there at all.#2020-07-1019:38seancorfieldAnd what Google libs show up in clojure -Stree for you?#2020-07-1019:38seancorfieldYou may need to run clojure -Strace and see what other versions of the various Google libs were considered.#2020-07-1019:40Drew Verlee> And what Google libs show up in clojure -Stree for you?
I see guava listed under one dep i just added to my project. Is that because Stree only shows the winner of the dep merge? I suppose i'm looking for everyone that tried to add it.
I'll look at Strace more carefully.#2020-07-1019:45Drew Verleei see Strace contains anyone that wanted to add the dep. isee it all over the place here.#2020-07-1019:50seancorfieldThe output of -Strace should show you every version of every library that it considered and why it made the decision it did to pick a particular version.#2020-07-1019:51seancorfieldhttps://guava.dev/releases/21.0/api/docs/com/google/common/collect/ImmutableList.html#toImmutableList-- shows that method was added in Guava 21.0#2020-07-1019:52seancorfieldSo you could just add a top-level dep on Guava to force a recent version, but I'm surprised tools.deps would pick a lower version, since I would expect the Google Cloud libs you're using to pull in at least that version or higher...#2020-07-1019:54seancorfieldAccording to Guava's docs, it's up to 28.0 now. 👀#2020-07-1019:54seancorfieldHmm, odd, we're using 29.0 at work...#2020-07-1019:55seancorfield21.0 is about 3 1/2 years old at this point. 29.0 came out in April this year @drewverlee#2020-07-1019:56seancorfieldWe've had to pin it as a top-level dependency because other things were bringing in older versions. Java's equivalent of "DLL hell" 🙂#2020-07-1101:05tolitiuswhat is a tools.depsy way of starting (--interactive) REPL-y connected to nrepl?
:repl {:extra-deps {nrepl {:mvn/version "0.7.0"}
reply {:mvn/version "0.4.4"}}
:main-opts ["-e" "(require,'[reply.main]),(defn,reply,[_,port,_],(reply.main/launch-nrepl,{:attach,(str,\"127.0.0.1:\",port)})),(ns,user)"
"-m" "nrepl.cmdline" "-i" "-f" "user/reply"]}}
this works (ish), but feels complex + does not exit nrepl once REPL-y is done (ctrl c / ctrl d)#2020-07-1110:57euccastrodo you want to start your nREPL server with the same command?#2020-07-1110:57euccastroI have this:
{:extra-deps {reply {:mvn/version "0.4.4"}}
:main-opts ["-m" "reply.main" "--attach" "1337"]}#2020-07-1110:58euccastrobut that's to attach to an already running nREPL#2020-07-1114:39tolitiusright nrepl + REPL-y connected to it. I do most of the work in a vi + terminal REPL#2020-07-1114:39tolitius*right => "start your nREPL server with the same command"#2020-07-1114:41tolitiusnrepl.cmdline optionally takes a repl-fn / -f for the interactive repling#2020-07-1114:56seancorfieldI think, for simplicity's sake, I'd be inclined to use two separate aliases: one to start an nREPL server (and leave it running in a window, or run it in the background), and a second alias to start REPL-y connected to that background process.#2020-07-1114:58seancorfieldI can understand wanting a single command, but I'd probably write a small shell script that just ran two clojure commands if it proved too annoying to run those two commands manually.#2020-07-1117:18tolitius@seancorfield definitely a possibility, but if I work with several (different) apps that's a twofold repl things to run and exit from
I am just looking for "lein repl" / "boot repl" repl-y like experience with deps.edn#2020-07-1117:21seancorfieldHmm, I never type into a REPL so I only care what my editor integration does. Why do you need REPL-y, specifically @tolitius?#2020-07-1117:21tolitiusI always type into repl )#2020-07-1117:22tolitiusit's connected to vi where I "finalize" my thoughts after I am done thinking in repl#2020-07-1117:23tolitiusand vice versa compile/reload from vi back to repl tp keep thinking#2020-07-1117:23seancorfieldI "think" in a "Rich Comment Form" in my editor -- like Stu Halloway talks about in some of his presos#2020-07-1117:30tolitiusyea, a lot of Clojure devs use an integrated playground to "think" that is just connected to nrepl that runs in the background. I like terminals, terminal repl works best for me since I am focused on small fragments of code at a time.
with REPL-y this experience is simple, so I am just looking to get there with deps.edn#2020-07-1117:30tolitiuswhich I did get with an above alias but it rather feels off + does not exit the nrepl on ctrl c/d#2020-07-1117:31seancorfieldI think you're stuck with the complex/almost-works invocation for now. I'll have a think about it on and off and come back if I figure out anything better.#2020-07-1117:31tolitiusyea, nothing urgent, I was just wondering whether I am missing something obvious, i.e. a flag, an option, etc..#2020-07-1117:31seancorfield(I don't use nREPL at all so it'll be a fun experiment/diversion)#2020-07-1117:31tolitiusgood luck and thank you )#2020-07-1119:28richiardiandrea@tolitius Hey 😉
I am using https://github.com/bhauman/rebel-readline
For your exact use case and I found some problems with multi-line expressions sometimes, I am pretty happy with it.
I have two aliases I can share (the second is for cljs) not in front of the keyboard now though. Will sent them soon.#2020-07-1216:33tolitius@richiardiandrea niice:
:repl {:extra-deps {nrepl {:mvn/version "0.7.0"}
cider/cider-nrepl {:mvn/version "0.22.4"}
com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
:main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]" "-i" "-f" "rebel-readline.main/-main"]}
this works: connects to an editor, reloads recompilations in repl from editor, and also I can now even see some color as a positive side effect )
any differences you found between rebel-readline and REPL-y (that REPL-y does better)?#2020-07-1217:27richiardiandreaNot much, as I mentioned sometimes rebel's multi-line support has glitches but it was never bothering so much to even open an issue (I should).
Nice thanks for sharing I do something similar myself.#2020-07-1217:34tolitiusthat's great, I am still mostly on boot due to little things like this in "deps.edn". but I am slowly moving over )
thanks a lot for a "rebel-readline" idea!#2020-07-1223:58richiardiandreamy pleasure 😄#2020-07-1316:55Space GuyWhen I used rebel-readline with a background process (Figwheel {:mode :serve} , http-kit, etc), the default rebel-readline/-main won't automatically end those processes if you quit with 'ctrl+d'.
Looks like REPL-y does (shutdown-agents) when done, I haven't checked what nrepl.cmdline does with that specific case
It's not a bug, just a difference (https://github.com/bhauman/rebel-readline/issues/141)
In my case I decided to start both then just exit when done:
(figwheel.main.api/start {:mode :serve} "dev")
(rebel-readline.main/-main)
(System/exit 0)
I currently run this in a clj dev.clj, otherwise a namespace with a main method is the same#2020-07-1321:35Ian Fernandezhello folks, some help on depstar pls (edited)
clj -C:build -A:build -m hf.depstar.uberjar target/abc.jar -C -m abc.core (edited)
this is giving me
Could not locate abc/core__init.class, abc/core.clj or abc/core.cljc on classpath.
but I have a src/abc/core.clj with (:gen-class)
some clue?#2020-07-1321:48hiredmanwhen you say it is giving you that, do you mean when you get that when building the uberjar or trying to run it?#2020-07-1321:57seancorfieldSince you're specifying -A:build you don't need to also specify -C:build -- -A means "all" so it already includes the effect of the -C option.#2020-07-1321:58seancorfieldWhat is your :build alias in your deps.edn @d.ian.b?#2020-07-1321:59Ian Fernandez:build {:extra-deps {seancorfield/depstar {:mvn/version "1.0.94"}}}#2020-07-1322:00seancorfieldAnd what exact output are you getting from that clj -A:build command? (I'm assuming the Could not locate... error is from trying to run the JAR?)#2020-07-1322:01Ian FernandezCompiling abc.core ...
Execution error (FileNotFoundException) at hf.depstar.uberjar/run$fn (uberjar.clj:331).
Could not locate abc/core__init.class, abc/core.clj or abc/core.cljc on classpath.
Full report at:
/tmp/clojure-5041631337400948497.edn
#2020-07-1322:02Ian Fernandezclj -A:build -m hf.depstar.uberjar target/abc.jar -C -m abc.core#2020-07-1322:04seancorfieldSo that says it's compiling norach.core -- which means depstar thinks that is the main class/namespace you told it to compile.#2020-07-1322:05Ian Fernandezops#2020-07-1322:05Ian Fernandezsrr#2020-07-1322:07Ian Fernandezops#2020-07-1322:07Ian Fernandezit was abc.core#2020-07-1322:07Ian FernandezI've tried to anonimize =(#2020-07-1322:07Ian Fernandezthe names#2020-07-1322:07Ian Fernandezand did it wrong#2020-07-1322:08seancorfieldIt really helps if you paste actual commands and actual error messages so you avoid transcription problems.#2020-07-1322:09seancorfieldI just did this:
$ clj -A:new app norach.core
... creates norach.core project ...
$ cd norach.core
$ clj -Sdeps '{:deps {seancorfield/depstar {:mvn/version "RELEASE"}}}' -m hf.depstar.uberjar target/abc.jar -C -m norach.core
Compiling norach.core ...
Building uber jar: target/abc.jar
Processing pom.xml for {norach/norach.core {:mvn/version "0.1.0-SNAPSHOT"}}
$ java -jar target/abc.jar
Hello, World!#2020-07-1322:09Ian Fernandezclj -A:build -m hf.depstar.uberjar target/norach.jar -C -m norach.core
Compiling norach.core ...
Execution error (FileNotFoundException) at hf.depstar.uberjar/run$fn (uberjar.clj:331).
Could not locate norach/core__init.class, norach/core.clj or norach/core.cljc on classpath.
Full report at:
/tmp/clojure-5874121970680475144.edn
#2020-07-1322:10seancorfieldSo that indicates that your deps.edn file is not setup correctly -- so norach/core.clj is not on your classpath.#2020-07-1322:10seancorfieldCan you run this: clj -m norach.core#2020-07-1322:12Ian FernandezExecution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate norach/core__init.class, norach/core.clj or norach/core.cljc on classpath.
Full report at:
/tmp/clojure-13768805695698659229.edn#2020-07-1322:12Ian Fernandez=(#2020-07-1322:12Ian Fernandeznope#2020-07-1322:12seancorfieldThat confirms it's not a depstar problem -- it's a problem with your project.#2020-07-1322:12Ian Fernandezthanks#2020-07-1322:13seancorfieldYou're sure you have src/norach/core.clj and that file starts with (ns norach.core ...) ?#2020-07-1322:13Ian Fernandezdammit, misstypo#2020-07-1322:13Ian Fernandezns norahc.core#2020-07-1322:13Ian Fernandezdammit#2020-07-1322:13Ian Fernandezthanksss#2020-07-1322:18Ian FernandezThanks @seancorfield 😃#2020-07-1411:54eval2020Any chance that tda will support providing multiple config files? e.g. by allowing CLJ_CONFIG to be ~/.clojure:/considered/first
This would be great to share a deps.edn via dotfiles or in a monorepos.
context: I'd like to curate a deps.edn for the team I'm working with. This file would contain aliases and overrides that are now duplicated across repositories.
This file should exist along a user's ~/.clojure/deps.edn .
E.g. CLJ_CONFIG=~/.clojure:~/.clojure/work clojure -Sdescribe would result in ...:config-files ["<global>/deps.edn" "~/.clojure/deps.edn" "~/.clojure/work/deps.edn" ?"$PWD/deps.edn"] ensuring that within the team we share aliases while allowing for user-specific aliases and versions (e.g. for editor setup).
In a monorepos I can see this being helpful as well. E.g. with the following project setup and a global export CLJ_CONFIG=~/.clojure :
.
├── shared/deps.edn
└── subproject1/deps.edn
doing export CLJ_CONFIG=$CLJ_CONFIG:/path/to/project/shared as part of a project setup (or in scripts) would allow one to move through the project and have a config-files collection of ["<global>/deps.edn" "~/.clojure/deps.edn" "/path/to/project/shared/deps.edn" ?"$PWD/deps.edn"]#2020-07-1412:15dominicmWrite a tool to update the deps edn file in each repo#2020-07-1412:43eval2020@U09LZR36F thought about going that way but seems hard to maintain across a lot of repositories...#2020-07-1413:43dominicmWhy's that#2020-07-1514:05eval2020In our case it would be the amount of repositories (50-ish). Also the contents of the deps.edn: some contain a linter alias, others a clj-kondo, any docs for existing alias. And then doing it for every update to the shared stuff.
Have you successfully applied this approach?#2020-07-1515:23dominicmNot to 50 or so repos. But if you want people to actually update, then you probably want to push it into their repo, instead of hoping that they will update it on their local machine.#2020-07-1412:25Alex Miller (Clojure team)Not planning to make CLJ_CONFIG into a path but we may at some point add some way to have additional deps files#2020-07-1412:40eval2020good to hear it's being considered, thanks#2020-07-1415:33seancorfieldI would stop relying on CLJ_CONFIG if I could specify one or more additional deps.edn files that went into this slot: <system> <user> <additional file(s) here> <project>.#2020-07-1416:37Alex Miller (Clojure team)I prefer that way of thinking about it to path idea above#2020-07-1416:38Alex Miller (Clojure team)we had lots of questions the last time we talked through this. Some of those have been resolved since, some have not.#2020-07-1416:38Alex Miller (Clojure team)we = Rich and I, that is#2020-07-1419:42eval2020The scenarios I sketched could then be solved by specifying an alias or wrapper like clj --deps project-deps.edn ?#2020-07-1420:06Alex Miller (Clojure team)well that is the question - how would you want to work with this in practice#2020-07-1502:16salamis it possible to automatically install clojure cli on jenkins as a custom tool where the jenkins user doesn't have enough privileges (i.e., neither root nor on the sudo list) to do so?#2020-07-1502:18salamthis is what we are getting right now:
[clojure_cli] $ sh -e /var/lib/jenkins/tools/com.cloudbees.jenkins.plugins.customtools.CustomTool/clojure_cli/hudson7631100789357617495.sh
[...]
You need to be root to perform this command.
[...]
Installing libs into /usr/local/lib/clojure
install: cannot create directory /usr/local/lib/clojure: Permission denied
#2020-07-1502:28seancorfield@abdusalam I believe you can run the script with the --prefix option and a directory to which you have write access.#2020-07-1502:32seancorfield(yes, I just tested this works)#2020-07-1502:32salamah, indeed! (https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/linux-install.sh)
thanks, @seancorfield#2020-07-1619:51markbastianIf I want to publish a tools deps project as a jar to a repo (e.g. clojars) and I have an up-to-date pom.xml, what is the "right" artifact to publish if you are using git coordinates for your dependencies? Do I need to create an uberjar and publish that? Is there a way to package the git deps src in a regular jar target? This is assuming a library project with git dependencies.#2020-07-1620:01Alex Miller (Clojure team)there is no way to do that#2020-07-1620:01Alex Miller (Clojure team)git deps are only usable from source deps#2020-07-1620:02Alex Miller (Clojure team)as soon as you say "artifact", you're talking about maven and you need to depend only on other artifacts#2020-07-1620:03markbastianMakes sense. Seems like an uberjar as lib or jar + git deps as lib would be a mess.#2020-07-1620:17kennyIs there a Maven command or Clojure lib someone made that uploads a jar file to a S3 bucket? All the tools I've found seem like they expect to build the jar for you.#2020-07-1620:18Alex Miller (Clojure team)well you can just do an aws s3 cp if that's sufficient#2020-07-1620:20kennyI tried that and receive this message from tools-deps "Download corrupted: Checksum validation failed, no checksums available." It actually all works (i.e., jvm starts up with the jar on the cp) but it'd be nice to do it the correct way.#2020-07-1620:20kennyOr maybe that's a bug in tools-deps?#2020-07-1620:24Alex Miller (Clojure team)well if you just upload the jar without the checksum files, you will see this message#2020-07-1620:26kennyI figured. Is there something out there that can do this the correct way to avoid this warning?#2020-07-1620:28Alex Miller (Clojure team)the correct way is to compute an md5 or sha1 checksum for the jar and upload those (and ideally to gpg sign all of those files as well). I do not know of a tool that does that (although I have started working on one)#2020-07-1620:29Alex Miller (Clojure team)an example of what goes into maven central is something like this: https://repo1.maven.org/maven2/org/clojure/tools.deps.alpha/0.8.677/#2020-07-1620:29Alex Miller (Clojure team)how that all gets created and uploaded is quite complicated, and clojars does much less (and even less is really required)#2020-07-1620:30Alex Miller (Clojure team)really, just the jar and the md5 or sha1 file (and ideally the pom) are probably sufficient#2020-07-1620:31Alex Miller (Clojure team)you can use a tool like sha1sum or md5sum to make that#2020-07-1620:32Alex Miller (Clojure team)there is also a version metadata file that should be updated too (like https://repo1.maven.org/maven2/org/clojure/tools.deps.alpha/maven-metadata.xml) if you want something like RELEASE version to work.#2020-07-1620:34kennyWowza. Did not realize what I was getting myself in to. Thank you for all this info. I assume the work you started on is not public yet?#2020-07-1620:35Alex Miller (Clojure team)no, nothing usable#2020-07-1620:37Alex Miller (Clojure team)I would like to handle a) clojars b) s3 and c) maven central in that order which I think handles both the most common and easiest items first#2020-07-1620:37Alex Miller (Clojure team)there are multiple options on each and it is way harder than it should be for sure#2020-07-1620:42kennyWill try the jar + hash suggestion. And try to restrain myself from building something complete. Thanks again 🙂#2020-07-1621:12dominicmMaybe I should build a tool on top of pack. It would force me to finish the api.#2020-07-1622:11Drew VerleeFor deps that need to be kept in sync, or even in times where its likely two things should udpate together, what strategy have people used?
My first thought would be that deps wouldn't be responsible for this and this more when you need a tool like lein. But if i only have this issue maybe soemthing a tad more lightweight.#2020-07-1622:31seancorfield@drewverlee I put them next to each other in deps.edn with a comment explaining the relationship.#2020-07-1622:32seancorfield(it's a project/application concern, not a tooling concern really -- so it's more about making sure people know how/when to change stuff)#2020-07-1622:34seancorfieldBTW, that's also why I'm generally against tools that update my configuration files automatically -- those tools cannot know about such constraints, e.g., I may have very valid reasons for keeping two particular dependencies on older versions together, even if newer versions of one or both exist.#2020-07-1623:10Drew VerleeSeems reasonable, I guess I should start with the comment and add more as needed. Thanks @seancorfield#2020-07-1708:51danieroux@kenny This is how I solved it:
mvn org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install-file -Dfile=dev-local-0.9.172.jar -DlocalRepositoryPath=s3_m2_repository
aws s3 sync s3_m2_repository
#2020-07-1715:29kennyOh, nice! This seems like it'd upload all the necessary files. Sweet.#2020-07-2102:30kennyThis approach still results in the Download corrupted: Checksum validation failed, no checksums available warning message though 😞#2020-07-1709:06danieroux@drewverlee I have entries like these:
org.slf4j/slf4j-api {:mvn/version "1.7.14" :required-by :datomic-cloud}#2020-07-1721:03Alex Miller (Clojure team)hey clj super fans, there is a new version of clj (and tools.deps) out and I'd love to have some people try it and verify no dep tree differences. you can install the latest dev release with:
brew uninstall clojure
brew install --devel clojure/tools/clojure
#2020-07-1721:24seancorfield@U064X3EF3 I expect you know about this
Warning: Calling 'devel' blocks in formulae is deprecated! Use 'head' blocks or @-versioned formulae instead.
Please report this issue to the clojure/tools tap (not Homebrew/brew or Homebrew/core), or even better, submit a PR to fix it:
/usr/local/Homebrew/Library/Taps/clojure/homebrew-tools/Formula/clojure.rb:7
#2020-07-1721:26Alex Miller (Clojure team)yeah#2020-07-1721:26Alex Miller (Clojure team)brew breaks things faster than I can fix them#2020-07-1721:27seancorfieldThen I got
==> Downloading
######################################################################## 100.0%
Error: clojure 1.10.1.536 is already installed
To upgrade to 1.10.1.561, run `brew upgrade clojure/tools/clojure`.
==> `brew cleanup` has not been run in 30 days, running now...
and it did not actually upgrade clojure 😐#2020-07-1721:28seancorfield(! 1406)-> clojure -Sdescribe
{:version "1.10.1.547"
:config-files ["/usr/local/Cellar/clojure/1.10.1.547/deps.edn" "/Users/sean/.clojure/deps.edn" ]#2020-07-1721:28seancorfieldWhich is not 561#2020-07-1721:29Alex Miller (Clojure team)oh, sorry#2020-07-1721:29Alex Miller (Clojure team)brew upgrade --devel clojure/tools/clojure#2020-07-1721:29seancorfieldNope, that doesn't work either.#2020-07-1721:29seancorfieldError: invalid option: --devel
#2020-07-1721:29Alex Miller (Clojure team)well then brew uninstall clojure first#2020-07-1721:30Alex Miller (Clojure team)then install --devel#2020-07-1721:30seancorfield(! 1413)-> clojure -Sdescribe
{:version "1.10.1.561"
:config-files ["/usr/local/Cellar/clojure/1.10.1.561/deps.edn" "/Users/sean/.clojure/deps.edn" ]
Had to uninstall then install --devel#2020-07-1721:31Alex Miller (Clojure team)yeah, that's the correct one#2020-07-1721:06Alex Miller (Clojure team)the change of note here is I have rewritten how exclusions are handled if the same dep shows up more than once in the dep tree with different exclusion sets. this is uncommon, but possible and most commonly happens by having the dep with exclusions in one place and with no exclusions in another. in the past, what happened was somewhat arbitrary - depending which you first encountered you may get the dep or not.#2020-07-1721:08Alex Miller (Clojure team)the updated behavior is that you will get the intersection of exclusion sets from all occurrences, so in the case above you will always get the dep#2020-07-1721:09Alex Miller (Clojure team)in the process of implementing this, I've also substantially refactored the deps expansion code to make it easier to work on and comprehend. while those changes should be a no op, I did change a lot of code and well, you know.#2020-07-1721:10Alex Miller (Clojure team)so if anyone has time to do a sanity check, that would be cool#2020-07-1721:12Alex Miller (Clojure team)also, in the process of checking these changes, it's become clearer to me that -Stree is likely to lie to you about what happened in exclusion cases like the above, because it's based on the lib map, which is an inverted tree that loses exclusion path info. I'm going to rewrite it in terms of the trace data and then it will be both correct but can also include substantially more useful information (akin to tools.deps.graph trace mode). but going to do other work first.#2020-07-1721:32seancorfieldA quick test shows that it doesn't attempt to download any new dependencies for our monorepo so that looks sane.#2020-07-1721:33Alex Miller (Clojure team):)#2020-07-1721:34Alex Miller (Clojure team)having a new clj should make everything stale (so effectively -Sforce), but maybe -Sforce too?#2020-07-1721:35seancorfieldI'm running our script that checks for updated versions of deps -- it builds new projects with our deps and then runs -Stree in those so there no cache. I'll let you know if that turns up any differences.#2020-07-1721:36seancorfieldLooks good. It detected a new version of org.apache.axis/axis and all the other dependencies remained unchanged.#2020-07-1721:36seancorfield(that script hammers clojure -Stree 🙂 )#2020-07-1721:43seancorfieldJust for good measure, I blew away the cache folder in every subproject in our monorepo and rebuilt things. No changes in deps as far as I can tell.#2020-07-1721:45Alex Miller (Clojure team)Thanks, good to have someone else try it :)#2020-07-1721:45seancorfieldYou know me -- always happy to be on the bleeding edge!#2020-07-1722:06kennyI can try this. @seancorfield how did you check for differences? Set diff of classpath?#2020-07-1722:16seancorfieldJust compared -Stree before and after.#2020-07-1722:17seancorfieldBut also ran our test suite to see if any new dependencies got downloaded (none).#2020-07-1802:24kennyNo new deps were printed when running clojure -Spath -Sforce . Also compared the classpath of the old and the new and they are identical.#2020-07-1802:47Alex Miller (Clojure team)Thanks, as expected,#2020-07-2102:31kennyAny idea why clojure -Spom would sometimes result in duplicate entries? e.g., something like this:
...
<build>
<build>
<sourceDirectory>src</sourceDirectory>
</build>
<build>
<sourceDirectory>src</sourceDirectory>
</build>
</build>
...#2020-07-2102:32Alex Miller (Clojure team)are you making a new one or syncing?#2020-07-2102:33Alex Miller (Clojure team)and which clj version?#2020-07-2102:33kennyThis resulted from this:
rm -rf pom.xml
clojure -Spom
clojure -Spom
#2020-07-2102:33Alex Miller (Clojure team)the answer to your question is certainly that it's a bug :)#2020-07-2102:33kenny1.10.1.561#2020-07-2102:34kennyThis is my deps.edn
{:paths ["src"]
:aliases {:jar {:extra-deps {seancorfield/depstar {:mvn/version "1.0.94"}}
:main-opts ["-m" "hf.depstar.jar" "s3-mvn-upload.jar"]}}}#2020-07-2102:34Alex Miller (Clojure team)there was a change to this in the prior version, which I actually have found was bad and a pretty subtle bug. I have a fix for it but haven't gotten it into master yet.#2020-07-2102:35Alex Miller (Clojure team)is it ok after the first clojure -Spom ?#2020-07-2102:35kennyNo. It creates this:
<build>
<build>
<sourceDirectory>src</sourceDirectory>
</build>
</build>#2020-07-2102:35kennySecond run:
<build>
<build>
<sourceDirectory>src</sourceDirectory>
</build>
<build>
<sourceDirectory>src</sourceDirectory>
</build>
</build>#2020-07-2102:36Alex Miller (Clojure team)yeah, I know why it's doing this#2020-07-2102:36Alex Miller (Clojure team)well I will try to get a fix out for it soon#2020-07-2102:36kennyAny idea which version doesn’t have this? 2 versions back?#2020-07-2102:37Alex Miller (Clojure team)check the changelog#2020-07-2102:38Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md#2020-07-2102:39Alex Miller (Clojure team)prior was tools.deps 0.8.677 which corresponds to 1.10.1.510#2020-07-2102:39Alex Miller (Clojure team)use https://github.com/clojure/homebrew-tools#version-archive-tool-releases to fall back on mac#2020-07-2102:39kennyYeah was looking for the mapping. Is there a place for tdeps -> brew version?#2020-07-2102:39Alex Miller (Clojure team)at the top of that changelog is a link to the other changelog#2020-07-2102:40Alex Miller (Clojure team)and vice versa#2020-07-2102:40kennyPerfect. Thank you!#2020-07-2102:41Alex Miller (Clojure team)there was a bug in initial gen in the prior version too though so you may not be any happier#2020-07-2102:42kennyHaha alright. At some point it worked, at least for my simple use case, since I’ve been using -Spom to deploy libs to clojars. Or perhaps I’ve just never noticed it generating something incorrect 😬#2020-07-2102:52seancorfieldLuckily http://clojars.org doesn't care I guess 🙂#2020-07-2102:52kennyFor those wanting to simply upload a jar file to a S3 Maven repo, I created a small Clojure lib to do (some of) the right things. You can paste it into a bb script if you want it to go a bit faster. https://github.com/kennyjwilli/s3-mvn-upload#2020-07-2102:54kennyFolks using Datomic’s dev-local may find that useful ^#2020-07-2216:50practicalli-johnI'm using https://github.com/seancorfield/clj-new to add a new file/namespace called handlers to an existing project, practicalli/banking. I assume this is the right command given an alias called :new in my user level deps.edn file. It seems to work okay.
clojure -R:new -m clj-new.generate ns=practicalli.handlers
cc @seancorfield#2020-07-2216:54seancorfieldThat's a reasonable approach. I'd probably add a :gen alias so I didn't have to type -m clj-new.generate each time.#2020-07-2216:54seancorfield(caution: the generators are very primitive and not particularly well thought out -- I don't actually find them valuable and I would prefer they not be promoted as a way to do anything!)#2020-07-2216:55seancorfieldAt some point, they'll probably be completely rewritten (in a new ns, naturally).#2020-07-2217:21practicalli-johnOk, thanks. I only just used it today for the first time successfully. I tried previously but was getting an error as I didnt include any kind of clj-new alias, so content in just getting it working for myself 🙂#2020-07-2217:22practicalli-johnI am more interested in writing some templates to use with clj-new, so will focus on that instead, which I assume is quite stable.#2020-07-2217:31seancorfield@jr0cket Yup, if you run into any problems, feel free to DM me about it.#2020-07-2217:31seancorfieldThe built-in app/lib templates are probably a good place to get started. Also let me know if you have suggestions for improving the README around writing templates.#2020-07-2221:21markbastianWhat's the best way to run a socket or nrepl server locally? The scenario is that we have a windows user with a deps.edn project that is having trouble getting a repl started in IntelliJ. I was thinking he could just run a socket REPL and jack in. I see aliases for nrepl https://nrepl.org/nrepl/usage/server.html as well as socket repl directions https://clojure.org/reference/repl_and_main#_launching_a_socket_server. Any recommendations besides having an accident and getting a Mac instead?#2020-07-2221:43markbastianShould this be the right invocation? clj -Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"#2020-07-2221:48markbastianGot it! From https://oli.me.uk/exploring-repl-tooling-with-prepl/:
clj -J-Dclojure.server.jvm="{:port 5555 :accept clojure.core.server/repl}"#2020-07-2303:06salama better approach would be to create an alias in the project deps.edn: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L66
i'm a cursive user myself and this is how i start a socket repl server in the terminal ( clojure -O:socket ) and connect cursive (a socket repl client) to it by creating a remote socket repl run configuration in intellij idea.#2020-07-2315:26markbastianNice. Thanks! I think 90% of the answers are in Sean Corfield's deps.edn file.#2020-07-2315:27salamyup, i personally found a lot of gems in there. 🙂#2020-07-2305:48tengstrandI can set up my deps.edn at the root so that it includes a "module" (that has its own deps-edn) like this:
:aliases {:dev {:extra-deps {env/dev {:local/root "environments/dev"}}}
:test {:main-opts ["-m" "polylith.clj.core.cli.poly" "test"]}}
Then I can run:
clj -A:test:dev
...and it will run my tests (that is what the 'test' alias does for me).
But the deps.edn under environments/dev contains its own aliases with some :extra-paths that I want to include also. Is that possible? I imagine that I need to do something like this (which doesn't work):
clj -A:test:dev:dev/local-alias
...where local-alias is defined as an alias in environments/dev/deps.edn:
:aliases {:local-alias {:extra-paths ["../../components/help/src"]}
#2020-07-2314:34seancorfieldNo, aliases in dependencies are not relevant. That would be kind of dangerous as you'd suddenly become vulnerable to whatever aliases developers added to their own projects, which could interfere with your own user-level aliases. And you'd also be affected if a dependency switched from Leiningen to Clojure CLI or vice versa -- which runs completely counter to any notion of reproducibility.#2020-07-2406:04tengstrandOkay, but here I suggest to use existing aliases that lives in environments/dev/deps.edn, that we "import" from deps.edn at the root. Then we don't "mess up things from the outside" just selecting aliases within that "imported" deps.edn. If I "cd environents/dev" first, then I have access to all those aliases which have the purpose to add :extra-paths which I need. But here I have several "sub modules" (not just the one in this example) and wants to do the same from the root deps.edn without having to manually merge all those deps.edn files into one huge deps.edn at the root.#2020-07-2406:06seancorfieldAs I said @U1G0HH87L "That would be kind of dangerous".#2020-07-2406:08seancorfield:local/root isn't special -- it's just another way to identify a dependency. You're asking for dependencies that include deps.edn to have their aliases combined with the current project. That's not a practical way to deal with reproducibility in project builds.#2020-07-2406:11tengstrandThen I guess I need to solve it with code generation or create my own REPL or similar.#2020-07-2406:15seancorfieldWhat we do at work, with our 100k+ line monorepo containing 30+ subprojects, is that we have a special subproject (called versions) that contains the "master" deps.edn that has all the aliases needed by all projects, including :defaults which brings in all the :override-deps to pin versions etc, and then all the other subprojects really just have :deps and an alias for :test-deps to add anything needed for testing that subproject (`versions/deps.edn` has a :test alias for all the common testing infrastructure). Then we use CLJ_CONFIG=../versions clojure -...aliases in each subproject.#2020-07-2406:16seancorfieldCLJ_CONFIG lets you treat a specific folder as if it contains your user-level deps.edn (and so any actual ~/.clojure/deps.edn file is ignored -- which is good: the monorepo then control all dependencies and aliases!)#2020-07-2915:24tengstrandThanks for the information, can be very useful!#2020-07-2308:54p-himikThere's no way to prevent clj from reading deps.edn in the current directory, is there?#2020-07-2309:56borkdude@p-himik don't think so.#2020-07-2309:59p-himikThanks. That's seems a bit inconvenient because if I want to run some tool with -Sdeps (like Capsule in my particular case) within my project, I cannot really do it without mixing two different classpaths.#2020-07-2310:02plexusit's fairly annoying, I've had scripts that temporarily move deps.edn out of the way because of this...#2020-07-2310:04borkdudedeps.clj, an unofficial re-implementation of the clj script does have an option to choose a different deps.edn file than the one in the current directory#2020-07-2310:05vlaaadI think setting :paths in aliases to replace source path from deps.edn is as close as it gets to what you want#2020-07-2310:06vlaaadand :deps#2020-07-2310:06borkdudeit's also possible to remove deps from the classpath in an alias:
https://github.com/borkdude/babashka/issues/152#2020-07-2310:06vlaaadsee https://clojure.org/reference/deps_and_cli#_aliases (“-A” section)#2020-07-2310:07p-himikBut then I'd have to restructure my whole deps.edn just to make some thirdparty tool happy. That's not good.#2020-07-2310:08vlaaadnot sure I’m following, can you put that third-party tool as a user-level alias?#2020-07-2310:09p-himikThe tool itself - yes. The thing is, it must be run in the presence of deps.edn. But it would be best to not let clj (and I run the tool using clj) also interpret deps.edn.#2020-07-2310:10p-himikThe tool itself is used via -Sdeps, so it's not in the deps.edn.#2020-07-2310:08borkdudeyou can also manually construct the classpath (using a script) and pass it to -Scp#2020-07-2310:08vlaaadthat way you don’t touch your project’s deps.edn at all#2020-07-2310:14p-himikOh, it's now possible to use :deps in aliases. There's hope!#2020-07-2310:15borkdudeThat overrides the base deps?#2020-07-2310:15p-himikYep, seems like it.#2020-07-2310:16p-himikSo basically I should be able to mimic an empty deps.edn. Will experiment later today.#2020-07-2310:16borkdudenote that you will still have clojure and spec on the cp by default#2020-07-2310:17p-himikHmm, right. But that should've really interfere with tools that are already meant to be run with clj.#2020-07-2310:27borkdudeI also use deps.edn for babashka projects, the extra deps don't hurt, but they can make searching the classpath a tad slower (like a few ms maybe)#2020-07-2310:29p-himikIn my case, they don't anything really bad as well, but I really want to get rid of those slf4j warnings.#2020-07-2312:27Alex Miller (Clojure team)yes, using :deps / :paths in an alias is what we call "tooling" and will replace the project :deps/:paths (for the purpose of using a tool that shouldn't use the project classpath)#2020-07-2312:28Alex Miller (Clojure team)until yesterday, you could only use those via -A, but as of the latest dev release yesterday, tooling -T is now a thing#2020-07-2312:38p-himikAwesome, thanks!#2020-07-2312:40Alex Miller (Clojure team)more to come on that and other changes in latest, just been kind of busy with ... other things :)#2020-07-2313:06Tamizhvendan SHi, I am getting FileAlreadyExistsException while trying to build an uberjar
Here is my deps.edn file
{:paths ["src" "resources"]
:deps {io.appium/java-client {:mvn/version "7.3.0"}
org.clojure/data.xml {:mvn/version "0.0.8"}
metosin/reitit-ring {:mvn/version "0.5.1"}
metosin/reitit-middleware {:mvn/version "0.5.1"}
metosin/reitit-spec {:mvn/version "0.5.1"}
mount {:mvn/version "0.1.16"}
ring/ring-jetty-adapter {:mvn/version "1.7.1"}
org.slf4j/slf4j-simple {:mvn/version "1.7.30"}}
:aliases {:dev {:extra-paths ["dev"]
:extra-deps {org.clojure/tools.namespace {:mvn/version "1.0.0"}}}
:uberjar {:extra-deps {seancorfield/depstar {:mvn/version "1.0.94"}}
:main-opts ["-m" "hf.depstar.uberjar"]}}}#2020-07-2313:07Tamizhvendan SHere is the command that I am running clj -A:uberjar my-app.jar#2020-07-2313:08Tamizhvendan SBuilding uber jar: my-app.jar
Execution error (FileAlreadyExistsException) at com.sun.nio.zipfs.ZipFileSystem/createDirectory (ZipFileSystem.java:422).
META-INF/versions/9/
#2020-07-2313:09Tamizhvendan SAm I missing anything obvious here?#2020-07-2314:34seancorfieldNo, aliases in dependencies are not relevant. That would be kind of dangerous as you'd suddenly become vulnerable to whatever aliases developers added to their own projects, which could interfere with your own user-level aliases. And you'd also be affected if a dependency switched from Leiningen to Clojure CLI or vice versa -- which runs completely counter to any notion of reproducibility.#2020-07-2317:19cap10morganWhat are folks using to store / set their project's version number so it makes it into the generated pom.xml (in an automated way; think CD pipeline, not hand-editing the pom.xml)?#2020-07-2317:31seancorfieldI'm treating the pom.xml as the "system of record" for the version number and deriving it elsewhere from the POM file.#2020-07-2317:33seancorfield(and that's the workflow that I use for all my deps.edn-based projects that get deployed to clojars -- note that Contrib libs such as core.cache, core.memoize, tools.cli that I maintain already follow this workflow with "Maven Release" process in CI editing the pom.xml to change the version number explicitly based on the version you enter in that Jenkins form)#2020-07-2319:53cap10morgan@seancorfield OK, thanks. What tools do you use to read / bump it in the POM file? Curious if you have anything resembling the lein release workflow going there.#2020-07-2322:35slimslenderslackshey @cap10morgan, we're treating git tags as the source of truth for versions. When we deploy, the version in the generated pom.xml is made to be in sync with "git describe". Since the pom.xml is only ever needed for the deploy action, the pom.xml doesn't actually have to be checked in.#2020-07-2322:42seancorfieldNote that for http://clojars.org and also for http://cljdoc.org there's an expectation that pom.xml contains a lot more than just the dependencies and the version, so having it as a "loose" file outside git would be... ill-advised if the generation process doesn't include all that information (which lein does put there, but clojure -Spom does not, so I'm talking about deps.edn projects -- note that clj -A:new would generate initial pom.xml files with all the appropriate fields for http://clojars.org and http://cljdoc.org)#2020-07-2322:51slimslenderslacksYa, our process is still using lein for the pom.xml generation. That's super interesting @seancorfield - I was probably going to hit this in the near term as I had been planning on migrating to use clojure -Spom. Thanks!#2020-07-2322:53seancorfieldclojure -Spom is fine for updating <dependencies> in an existing pom.xml file but it only generates a very bare-bones version if no file exists. That's why I added a full-featured pom.xml to the templates that clj-new uses for app, lib, and template.#2020-07-2322:57slimslenderslacksWhen git describe is the source of truth for version maybe we'd compose clojure -Spom to update the dependencies, with a git-describe->pom-version , and keep a clj-new generated pom.xml checked in.#2020-07-2323:01slimslenderslacksIt sounds like a checked in pom.xml might currently be the best place to version the extra metadata that clj-new adds.#2020-07-2323:05seancorfieldclj-new creates a baseline pom.xml. It never updates it after that.#2020-07-2323:06seancorfieldclojure -Spom updates an existing pom.xml with the latest dependencies (from deps.edn) so as long as you always do it prior to a build/deploy cycle, I guess those changes don't really matter.#2020-07-2323:07seancorfieldYour git describe update also seems to be "do it before build/deploy" but don't really care about the changes.#2020-07-2323:08seancorfieldHowever, a lot of people would be very confused by a pom.xml file in a repo that didn't match the dependencies in deps.edn and/or didn't seem to reflect the current released version 🙂#2020-07-2323:08seancorfieldSo whether you check it in or not after those changes probably depends on whether it's going into a public repo I suppose 🙂#2020-07-2323:12slimslenderslacksYa, I agree. It could be confusing to see a pom.xml with a weird version. You'd almost want the checked in copy to have a version of :project-uses-git-tags , and maybe have an empty dependencies, so it appears to be there just to define the extra maven stuff that you can't derive from elsewhere.#2020-07-2323:13slimslenderslacksI don't mind having a file with the correct version, I was more just trying to avoid the commits that go along with those workflows.#2020-07-2323:14slimslenderslacksOur unit tests and commit messages are the signals the produce Tags, and Tags are the signals that create new publishes to maven repos.#2020-07-2323:15slimslenderslacks(from a CI/CD perspective)#2020-07-2323:23seancorfieldYup. I think internal CI/CD processes are under different constraints and, unless you are publishing every successful build to (public) Maven Central / Clojars and also publishing every version to a (public) GitHub repo or similar, for internal releases you can definitely just keep a baseline pom.xml under version control and update that as needed for each build to your repo (and throw it away afterward).#2020-07-2320:00Alex Miller (Clojure team)there are maven plugins for that although it's all pretty tedious#2020-07-2320:01Alex Miller (Clojure team)the mvn release / deploy infrastructure does a lot for you but you have to buy pretty far into it all the way for it to be worth doing#2020-07-2320:01Alex Miller (Clojure team)(but it's a lot less fragile than the lein release stuff)#2020-07-2320:02cap10morganThanks Alex. lein release can indeed be fragile. 🙂#2020-07-2320:12Alex Miller (Clojure team)the idea to put the version inside a file was probably bad from the beginning#2020-07-2320:13seancorfield@cap10morgan For my clojars-deployed stuff, I just update it as part of my general release process where I'm updating the README and the CHANGELOG and I update both the <version> and the <tag> fields in the pom.xml file all at the same time.#2020-07-2320:14seancorfield(so I grep for <old-version>|<new-version> so I can see all the places that refer to the current release and what will be the next one, and then I can visually verify that all references get changed)#2020-07-2320:15seancorfieldIn most of my libs, that includes several places in the README and usually in the documentation files as well that refer to installation/getting started and when new features were introduced.#2020-07-2320:19Alex Miller (Clojure team)I've automated that step on some libs :)#2020-07-2320:21seancorfieldI still have artisanal documentation 🙂#2020-07-2415:07Drew VerleeGiven a deps in project A that has a :local/root to another deps in project B. If i update a dep in project B and start my repl in poject A i would expect to have the new dep from project B be used. This doesn't seem to be the case. Any ideas what could cause this?#2020-07-2415:11Alex Miller (Clojure team)yes, you need to -Sforce to recompute the classpath#2020-07-2415:12Alex Miller (Clojure team)right now, changes in transitive local deps are not automatically discovered#2020-07-2415:12Drew Verleegotcha. thanks alex#2020-07-2415:15borkdudeI saw a mention of -T in this channel. Where is this documented?#2020-07-2415:23Alex Miller (Clojure team)in not yet published docs :)#2020-07-2415:24Alex Miller (Clojure team)only exists in the newest dev release of clj#2020-07-2415:24Alex Miller (Clojure team)so I wouldn't tell anyone to use it yet#2020-07-2415:24Alex Miller (Clojure team)but it's exactly what's supported now via -A for :deps and :paths in aliases, it just has an independent name now#2020-07-2415:58borkdudeI think I'll just wait for the docs 🙂#2020-07-2416:02Alex Miller (Clojure team)it's not really giving you anything new that you didn't already have :)#2020-07-2416:05Alex Miller (Clojure team)there are other new and interesting things in this release though :)#2020-07-2416:06vlaaadNow I'll have to go through the changes...#2020-07-2416:07Alex Miller (Clojure team)writeup is coming#2020-07-2616:37kennySomething strange is going on with s3 deps and tools-deps. I’m on 1.10.1.561. Given this deps.edn,
{:deps {com.datomic/dev-local {:mvn/version "0.9.184"}
com.datomic/client-cloud {:mvn/version "0.8.102"}}
:mvn/repos {"kwill-mvn" {:url ""}}}
the output of clojure -Stree -Sforce is
org.clojure/clojure 1.10.1
org.clojure/core.specs.alpha 0.2.44
org.clojure/spec.alpha 0.2.176
com.datomic/dev-local 0.9.184
com.datomic/client-cloud 0.8.102
com.cognitect/anomalies 0.1.12
com.datomic/client 0.8.111
...
It is not picking up the dev-local deps specified in the jar’s pom.xml. If I change the dev-local dep to be a local/root pointing to the Maven jar, I get the correct tree.
{:deps {com.datomic/dev-local {:local/root "/Users/kenny/.m2/repository/com/datomic/dev-local/0.9.184/dev-local-0.9.184.jar"}
com.datomic/client-cloud {:mvn/version "0.8.102"}}
:mvn/repos {"kwill-mvn" {:url ""}}}
Output of clojure -Stree -Sforce
org.clojure/clojure 1.10.1
org.clojure/core.specs.alpha 0.2.44
org.clojure/spec.alpha 0.2.176
com.datomic/dev-local /Users/kenny/.m2/repository/com/datomic/dev-local/0.9.184/dev-local-0.9.184.jar
org.clojure/tools.analyzer 0.6.9
com.google.errorprone/error_prone_annotations 2.3.4
...
Note how dev-local is now bringing in deps and previously it was not.#2020-07-2617:32hiredmanIf I recall, maven doesn't use the pom out of a jar, it keeps the pom alongside the jar, so it looks like you are deploying your jar to your maven repo with a different pom then you are building into it#2020-07-2617:44kennyAh, that does make sense. From a previous discussion with Alex, I was under the impression that I only needed to upload a jar & a hash to s3 to get it to work.#2020-07-2618:43hiredmanThat may be the minimal required for maven to download the jar, but more may be required to communicate dependencies#2020-07-2622:45Alex Miller (Clojure team)I think you may actually just be seeing bad output from -Stree#2020-07-2622:47Alex Miller (Clojure team)in general, -Stree will only list a dep once (even when it's included under multiple branches) - and it's pretty arbitrary where you will see it. for these two libs there are many shared deps.#2020-07-2622:49Alex Miller (Clojure team)these two deps particularly share deps but with different exclusion sets, a case where this version of clj newly makes better, more intentional and reproducible choices. Unfortunately, -Stree is built from the lib map, which does not retain the original path structure and it's quite likely to see deps incorrectly under exclude points (because they are included at other places). -Stree is actively misleading in these cases. (the actual set of libs is right though!)#2020-07-2622:51Alex Miller (Clojure team)what I'm trying to say is you may have seen two different things from -Stree, and yet they are actually both right in terms of the transitive lib set, but wrong in how they are showing you the tree#2020-07-2715:20kenny@alexmiller While that may be happening, the S3 Maven dep approach would result in a classnotfound exception when loading dev-local in the REPL (something with fressian). Switching to local/root with the jar does not result in the classnotfound. I also uploaded a pom to S3 and that approach did not result in a classnotfound.#2020-07-2716:01Alex Miller (Clojure team)what jar are you talking about wrt s3?#2020-07-2716:02Alex Miller (Clojure team)dev-local?#2020-07-2716:02Alex Miller (Clojure team)if so, what you're saying makes sense#2020-07-2716:03Alex Miller (Clojure team)afaik, maven will use the pom to report deps (which is a layer that clj is going through for that)#2020-07-2716:04kennyYes - dev-local.#2020-07-2716:05kennyTo be clear, here's what works and doesn't work.
S3 jar - classnotfound
S3 jar, jar md5 - classnotfound
S3 jar, jar md5, pom - works
local/root jar - works#2020-07-2717:23Alex Miller (Clojure team)yeah, that makes sense to me. sorry I didn't understand what you were doing earlier#2020-07-2717:23Alex Miller (Clojure team)local/root jar will use the pom inside the jar#2020-07-2803:12Oliver GeorgeI'm curious how RELEASE resolves in this maven dep...
{:extra-deps {clj-kondo {:mvn/version "RELEASE"}}
:main-opts ["-m" "clj-kondo.main"]}
Perhaps it's a normal maven feature. Hard to google for "release"!#2020-07-2803:26Alex Miller (Clojure team)RELEASE (and LATEST) are special "virtual" version numbers that tell Maven to look for the newest released version (non snapshot)#2020-07-2803:27Alex Miller (Clojure team)they are officially unsupported by clj (but do kind of work with some big caveats)#2020-07-2803:28Alex Miller (Clojure team)big caveat #1 - they break caching and you will never see a new release after the one that it first resolves to unless you -Sforce (or otherwise cause a cache recompute)#2020-07-2803:29Alex Miller (Clojure team)at some point, we may actually prevent caching at all if you use these "versions"#2020-07-2803:30Alex Miller (Clojure team)subtle caveat #2 - the answer you get is only as good as the metadata index in the repository. for maven or clojars, no worries. if you're using a bespoke s3 repo, then you probably aren't updating the right metadata files and you usually won't see the newest version#2020-07-2803:32Alex Miller (Clojure team)generally, I think it's harmless to do this in -Sdeps to grab "latest". it's mildly bad to do it for tools in your deps.edn aliases (like the example here). it's very bad to do it for your project deps.#2020-07-2805:21deactivateduserDoes there happen to be an equivalent for git/sha coordinates? That could be quite handy…#2020-07-2806:01Alex Miller (Clojure team)Intentionally not, for the same reasons re caching#2020-07-2806:04deactivateduserI guess I don’t understand the caching issue. If you resolve “RELEASE” to a concrete version, and that version happens to exist in the cache already, wouldn’t tools.deps use it?#2020-07-2806:05deactivateduser(and conversely, if it doesn’t, wouldn’t that version be added to the cache, and thereby eligible for future “cache hits”)#2020-07-2806:06Alex Miller (Clojure team)The caching is based on time stamps of deps,edn files. If the file never changes (set to RELEASE), the class path is never recomputed, even if newer versions come out#2020-07-2806:06Alex Miller (Clojure team)Caching of classpath (not lib version)#2020-07-2806:08deactivateduserAh ok. I assume there’s a good reason for making the deps.edn file the cache key, rather than just recomputing dependencies the classpath every time and having those in the cache?#2020-07-2806:09Alex Miller (Clojure team)Recomputing deps starts a second JVM so it’s at least a second, usually more depending on io #2020-07-2806:09Alex Miller (Clojure team)This is why clj starts faster than lein usually#2020-07-2806:10Alex Miller (Clojure team)It also goes to build repeatability#2020-07-2806:11Alex Miller (Clojure team)Or “clj repeatability”#2020-07-2806:17deactivateduserOk. I’ll assume there’s a good reason for needing a second JVM to do that (not requesting an explanation, btw, unless you’d like to provide one), and understood full well the performance implications of starting a JVM.#2020-07-2806:41Alex Miller (Clojure team)it's running a Clojure program with a different classpath the program you're executing#2020-07-2806:43Alex Miller (Clojure team)specifically, it's running tools.deps to compute the classpath based on your deps.edn (talking to Maven, maybe S3, maybe Git, etc). in the end it spits out the results in cache. Your actual program is then started with the classpath you intended. Once we've cached that work, we can spit the first part and just run the program.#2020-07-2806:52deactivateduserSo the advantage of caching based on the deps.edn file, rather than the classpath it contains, is all the I/O to talk to Maven, S3, git, etc.?#2020-07-2807:09Alex Miller (Clojure team)well, that's all of the work. not much point to caching otherwise#2020-07-2807:11Alex Miller (Clojure team)expanding and downloading a full deps tree can easily mean dozens of jars and many megabytes of network traffic (if your m2 and/or git cache is cold)#2020-07-2807:16Alex Miller (Clojure team)even if you have all the jars, expanding the tree is more than you'd want to wait, example in tools.deps.alpha repo (~80 jars in the full tree), all jars in cache so no downloads:#2020-07-2807:16Alex Miller (Clojure team)amac:tools.deps.alpha alex$ time clj -Sforce -e nil
real 0m3.028s
user 0m12.567s
sys 0m0.756s
amac:tools.deps.alpha alex$ time clj -e nil
real 0m0.745s
user 0m1.647s
sys 0m0.133s
#2020-07-2807:17Alex Miller (Clojure team)(one caveat here is that the clj uber jar is not currently aot compiled - that's coming soon and will improve that first time, which includes a lot of loading)#2020-07-2807:18Alex Miller (Clojure team)but it's still not time anyone wants to wait#2020-07-2807:20Alex Miller (Clojure team)with a cold m2 cache, time is about 7.5 s (aot may knock a couple seconds off that)#2020-07-2815:00deactivateduserI can see why that approach will be faster, though it breaks down (cache becomes stale) in the presence of dependencies that are expressed using a “floating” version reference (RELEASE, LATEST, SNAPSHOT, etc.).
Those were deprecated in Maven3, and perhaps tools.deps should follow suit and emit a warning when it encounters such a thing?#2020-07-2815:00Alex Miller (Clojure team)perhaps#2020-07-2805:24seancorfield@deactivateduser10790 My dot-clojure deps.edn shows how you can resolve the head of the default branch -- in the comments.#2020-07-2805:25seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L171-L175#2020-07-2805:26seancorfieldAnd then (load-master 'clojure/tools.trace)#2020-07-2805:26seancorfield(following the https://github.com URL pattern, not the org.clojure/tools.trace Maven artifact pattern)#2020-07-2805:41deactivateduserPotentially dumb question, but is it expected that code inside an EDN file would be executed? Or does that rely on a coincidence of how the Clojure EDN reader functions?#2020-07-2805:53seancorfield@deactivateduser10790 Can you give a specific example?#2020-07-2805:54seancorfieldI mean, EDN can contain tagged literals and you can provide reader functions that process those... but that's not exactly "code inside an EDN file".#2020-07-2805:58Alex Miller (Clojure team)edn files are data, not code, and are not evaluated#2020-07-2806:00Alex Miller (Clojure team)They are read but that will only construct tagged literals, and clj does not provide a way to install readers for tagged literals#2020-07-2806:03deactivateduser@seancorfield your example includes code (`(require '[clojure.tools.gitlib :as gitlabs]) …`) - how is that code evaluated?#2020-07-2806:06seancorfieldThose are comments showing what you would execute in your REPL...#2020-07-2806:07seancorfieldThe files contains lots of comments explaining how to use the aliases, with links to repos.#2020-07-2806:09deactivateduserOh ok. So I would not put those in a deps.edn then.#2020-07-2806:09seancorfieldHuh? Those are comments and they are in a deps.edn file.#2020-07-2806:10deactivateduserYes they are, and as a result I would expect to uncomment them and have them Just Work:tm: .#2020-07-2806:10seancorfieldI mean, that repo literally is my ~/.clojure/deps.edn file.#2020-07-2806:10deactivateduserBut if I’m following you, you’re saying that’s not correct - that those lines need to be copypasta’ed into a REPL.#2020-07-2806:11seancorfieldForgive me but I'm a bit shocked you would think code in EDN files would be evaluated...#2020-07-2806:11deactivateduserI didn’t - that’s why I asked the question.#2020-07-2806:11deactivateduserI was quite surprised to see commented out code in an EDN file.#2020-07-2806:12deactivateduser(and by “quite”, I mean “very”…)#2020-07-2806:13deactivateduserRegardless, @alexmiller separately explained that by design you can’t do an equivalent of RELEASE for git/sha coords in a deps.edn file.#2020-07-2806:13seancorfieldhttps://github.com/practicalli/clojure-deps-edn/blob/master/deps.edn contains comments with lots of shell commands, and even some elisp -- you wouldn't expect any of that to be executed just by reading the EDN file, would you?#2020-07-2806:14deactivateduserWhy would you assume I’ve read some random deps.edn file? Sadly I don’t have that much disposable time…#2020-07-2806:17seancorfieldWell, you read my file and came to a ... strange conclusion ... so I offered another example deps.edn and wondered what conclusion you would come to from that...#2020-07-2806:18deactivateduserIf you re-read my original question (https://clojurians.slack.com/archives/C6QH853H8/p1595913713054800), you might see why I was assuming a deps.edn-only solution, and therefore didn’t understand your non-`deps.edn` answer.#2020-07-2806:18deactivateduserIn general, answering questions people didn’t ask is a wonderful source of confusion. 😉#2020-07-2806:20seancorfieldWith the Clojure CLI/`deps.edn` the "equivalent" to Maven/Leiningen/etc is often "some code that you run" -- which seemed like an acceptable answer to your question.#2020-07-2806:21seancorfieldThat's why I was so puzzled by your follow-up question, as it seemed unrelated to my answer (I was thinking "No, of course reading EDN won't execute code!").#2020-07-2806:22seancorfieldI guess I should add "Here's some code you can run in your REPL" to my deps.edn file where there are code examples 🙂#2020-07-2806:23deactivateduserThat would have avoided any confusion, no doubt. 😉#2020-07-2806:24seancorfieldIt is sometimes so hard to predict your audience's state of mind 😆 (when writing documentation, I should add).#2020-07-2806:27deactivateduserProfessional technical writing is such an under-appreciated skill, I’ve found.#2020-07-2807:20spfHey guys. If i have a setup as described in https://clojure.org/guides/deps_and_cli#_using_local_libraries. is there a way to fire up a repl in the hello project which reloads the time-lib deps if they are changed?#2020-07-2807:22Alex Miller (Clojure team)no, that's not a built-in feature of Clojure (or Java). there are ways to do some variants of this but it's not common#2020-07-2807:22Alex Miller (Clojure team)well, do you mean local dir or local jar?#2020-07-2807:23Alex Miller (Clojure team)if local dir, you can just use Clojure's normal :reload or :reload-all mechanism built into require#2020-07-2807:23spfi meant local dir#2020-07-2807:23Alex Miller (Clojure team)yeah, so that's just normal reloading#2020-07-2807:23spfok. cool. thx#2020-07-2813:06rickmoynihanany idea why running brew upgrade clojure/tools/clojure installs 1.10.1.561 not 1.10.1.590 ?#2020-07-2813:28Alex Miller (Clojure team)561 is the newest stable release#2020-07-2813:28Alex Miller (Clojure team)590 is a dev release#2020-07-2816:10seancorfieldFollowing up from my comment in the announce thread:
{:paths [:clj-paths :resource-paths]
:aliases
{:clj-paths ["src/clj" "src/cljc"]
:resource-paths ["resources"]}}
Is that correct, or should :clj-paths and :resource-paths be underneath an alias?#2020-07-2816:11seancorfieldIf that is correct, I don't see the value since they are fixed labels and values -- am I missing something @alexmiller?#2020-07-2816:12Alex Miller (Clojure team)oh, that is right - they are aliases#2020-07-2816:12Alex Miller (Clojure team)an alias is just a name for some data#2020-07-2816:13Alex Miller (Clojure team)this was the original meaning here but we have kind of gotten off talking about things that way by tying it into clj's usage of alias data to affect classpath construction#2020-07-2816:14seancorfieldOK, so how is that useful since they are just fixed values?#2020-07-2816:14Alex Miller (Clojure team)at the moment, it's not very useful#2020-07-2816:14seancorfieldAlso, you have this paragraph twice "Replace project environment ("tool")" -- one with plenty of detail, one with very little.#2020-07-2816:16Alex Miller (Clojure team)that's actually intentional - the first part is kind of an overview, and the second is more how the tool works with more detail#2020-07-2816:16Alex Miller (Clojure team)doing a lot of heavy editing on this page lately but will likely change more over time#2020-07-2816:17seancorfieldOK, I think it's just really confusing at the moment with two identical headers...#2020-07-2816:17Alex Miller (Clojure team)I'm ok with that for the moment :)#2020-07-2816:17Alex Miller (Clojure team)the toc at the top side bar may make the structure more evident#2020-07-2816:18Alex Miller (Clojure team)I've gone through about 5 different structures for this page recently, and I don't think I'm done#2020-07-2816:23seancorfieldOn another wordsmithing issue: now that page refers to "tool" as something that might want to override :deps and/or :paths, this section name might be confusing (a different meaning of tools) https://clojure.org/reference/deps_and_cli#_clojure_tools_usage#2020-07-2816:24Alex Miller (Clojure team)yeah, I've been fighting that, haven't decided what to do about it yet#2020-07-2816:24Alex Miller (Clojure team)not enough words :)#2020-07-2816:24Alex Miller (Clojure team)it's like naming things is hard#2020-07-2816:28seancorfieldTools are just so useful 🙂#2020-07-2816:14Alex Miller (Clojure team)but if you had say, a build tool that wanted to also use a subset of paths it might be useful#2020-07-2816:14Alex Miller (Clojure team);)#2020-07-2816:17seancorfieldThe Maven install example is missing an alias tho' (even if the paths one isn't), yes?
{:aliases
{:fn clojure.tools.deps.alpha.install/install
;; :args map could be provided but can pass on command line instead
}}#2020-07-2816:18seancorfieldThat's meant to be executed with -X:some-alias (most people will pick -X:install I suspect)#2020-07-2816:18Alex Miller (Clojure team)ah, yes that is#2020-07-2816:18Alex Miller (Clojure team)I actually saw that late last night and forgot to fix#2020-07-2816:20seancorfieldI'm curious as to why that ability surfaced as a built-in option, given that JAR files can already be used as :local/root deps?#2020-07-2816:21Alex Miller (Clojure team)it's not a built-in option, it's a built-in program#2020-07-2816:22Alex Miller (Clojure team)didn't seem big enough to make into its own thing, depended heavily on all the code in tools.deps, wanted it for dev-local, etc#2020-07-2816:22seancorfieldInteresting...#2020-07-2816:22Alex Miller (Clojure team)serves as a good example :)#2020-07-2816:22Alex Miller (Clojure team)the plan is to shift some of the magic switches in clj over to just be programs you can -X#2020-07-2816:23Alex Miller (Clojure team)some of those might become built-in to the root deps.edn, not sure yet#2020-07-2816:30seancorfieldThe install line you gave in the announcement doesn't seem to work
Error: No available formula with the name "clojure/tools/
#2020-07-2816:30seancorfieldThat's from brew install clojure/tools/#2020-07-2816:30seancorfield(after a brew uninstall clojure)#2020-07-2816:33seancorfieldShould be 1.10.1.590#2020-07-2816:33Alex Miller (Clojure team)Ah, shoot#2020-07-2816:34Alex Miller (Clojure team)Edited#2020-07-2816:54rickmoynihanwow need to get my head around these new tools.deps features; they might make some stuff I’ve been planning to do completely redundant :thinking_face:#2020-07-2816:58rickmoynihanhow is the -T alias different to having an alias with a :deps key?#2020-07-2816:58Alex Miller (Clojure team)It’s not#2020-07-2816:58seancorfieldI need to figure out how to get the clojure script updated on a couple of our older servers since this would help eliminate some of the extra script stuff we've written (but I think there's a blocker for installing clojure on a couple of our old servers).#2020-07-2816:59Alex Miller (Clojure team)It is exactly what was already there via -A, just has its own switch now#2020-07-2817:00rickmoynihanok so it just makes the intent clearer on the command line?#2020-07-2817:01Alex Miller (Clojure team)Yep#2020-07-2817:01Alex Miller (Clojure team)A is meant to be an “all” kind of thing but this didn’t have its own name, so now it does#2020-07-2817:02rickmoynihanso what would -T do with :extra-deps instead of :deps?#2020-07-2817:03Alex Miller (Clojure team)(Which also helped further clarify some things in the code which has gone through some substantial changes)#2020-07-2817:03Alex Miller (Clojure team)It would be ignored#2020-07-2817:04rickmoynihanwow I had no idea how many deps had unqualified names. I thought it was just a handful, but it’s way more than I expected#2020-07-2817:04Alex Miller (Clojure team)Each of the sub processes takes what it expects and ignores the rest, like all good Clojure programs :)#2020-07-2817:06seancorfieldShould this check include tools_aliases now?
-h|--help|"-?")
if [[ ${#main_aliases[@]} -gt 0 ]] || [[ ${#all_aliases[@]} -gt 0 ]]; then
break
else
help=true
shift
fi
;;#2020-07-2817:08Alex Miller (Clojure team)No?#2020-07-2817:09seancorfieldMaybe I'm just missing the actual flow of the script then, nm.#2020-07-2817:10Alex Miller (Clojure team)Yeah this has to do with falsely interpreting a -h appended to main aliases as a help opt#2020-07-2817:10seancorfield(I was just grepping for (tool|all).alias through the new script out of curiosity)#2020-07-2817:11Alex Miller (Clojure team)Would be curious if you try the win install :)#2020-07-2817:13seancorfield@alexmiller re: -h -- that means that clojure -A:test:runner -h and clojure -T:test:runner -h behave differently.#2020-07-2817:14seancorfield(! 1537)-> clojure -T:test:runner -h|more
Version: 1.10.1.590
...
(! 1538)-> clojure -A:test:runner -h|more
Unknown option: "-h"
Wasn't sure whether that was intentional or not since you said -T and -A were "the same"?#2020-07-2817:15seancorfield(the first case displays help -- -h is not passed to the main program; the second case passes -h to the main program in the test runner here)#2020-07-2817:27Alex Miller (Clojure team)this is kind of a corner case - the condition is guarding against having -A/-M supplying the first part of the main args and then -h on the command line where it is not actually the main args to pass to clojure.main. I'm not worried about it.#2020-07-2817:40seancorfieldFair enough. Just wanted to point out that it is a difference between -A and -T 🙂#2020-07-2817:57Alex Miller (Clojure team)there's no difference in how :deps and :paths work, which is what I meant#2020-07-2817:24rickmoynihanJust tried doing -X:myfn in one of my projects and I get the error:#2020-07-2817:24rickmoynihanclojure -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.
#2020-07-2817:25Alex Miller (Clojure team)are you on clj 1.10.1.590?#2020-07-2817:26rickmoynihanI thought I was#2020-07-2817:26rickmoynihanyep#2020-07-2817:26rickmoynihan$ clojure --help
Version: 1.10.1.590
...
#2020-07-2817:30seancorfieldI get that too, also on 1.10.1.590:
(! 974)-> clj -Sdescribe
{:version "1.10.1.590"
:config-files ["/usr/local/Cellar/#2020-07-2817:30Alex Miller (Clojure team)if you clj -Sdescribe and then jar tf <the-install-dir/clojure-tools-1.10.1.590.jar | grep exec.clj#2020-07-2817:30Alex Miller (Clojure team)do you see it?#2020-07-2817:32Alex Miller (Clojure team)what installer are you using?#2020-07-2817:32rickmoynihanthe jar is further down than that in /usr/local/Cellar/#2020-07-2817:33Alex Miller (Clojure team)sorry, bad multi copy/paste on my part, that's right#2020-07-2817:33rickmoynihanexec.clj is in there yes#2020-07-2817:33Alex Miller (Clojure team)this is mac? linux?#2020-07-2817:33rickmoynihan$ jar tf /usr/local/Cellar/
#2020-07-2817:33rickmoynihanmac#2020-07-2817:33seancorfield(yup, same here)#2020-07-2817:33rickmoynihanbrew was the installer#2020-07-2817:34Alex Miller (Clojure team)clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn is a standalone example#2020-07-2817:34rickmoynihanedn is broken#2020-07-2817:34seancorfieldFails for me#2020-07-2817:35Alex Miller (Clojure team)edn is broken means?#2020-07-2817:35rickmoynihanah wait sorry was in a project with a broken edn file#2020-07-2817:35seancorfield(! 982)-> clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.
Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-12219293780956477884.edn#2020-07-2817:35Alex Miller (Clojure team)ah, I was able to repro#2020-07-2817:35rickmoynihan$ clj -Sdeps '{:aliases {:myfn {:fn clojure.core/pr}}}' -X:myfn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.
Full report at:
/var/folders/ln/536xqskd3_g80n5pdbsjkrjw0000gn/T/clojure-8308868078358895829.edn#2020-07-2817:35rickmoynihantoo#2020-07-2817:35Alex Miller (Clojure team)it matters whether you're in a directory with a deps.edn#2020-07-2817:36Alex Miller (Clojure team)well, that's definitely a bug :)#2020-07-2817:36Alex Miller (Clojure team)will fix#2020-07-2817:36Alex Miller (Clojure team)thx#2020-07-2817:36seancorfieldCrashes for me whether there's a deps.edn file or not.#2020-07-2817:36rickmoynihanyeah same#2020-07-2817:36Alex Miller (Clojure team)hm, maybe it's a cached classpath for me or something#2020-07-2817:37rickmoynihanthere are 3 hard things in computer science…#2020-07-2817:37seancorfieldI tried it via an alias in a deps.edn file (shown above), then tried via the command line with just {} in deps.edn, and then tried in a directory with no deps.edn -- all of them crash the same way. Even tried -Sforce and it still failed.#2020-07-2817:38rickmoynihanI tried two of those; just tried with -Sforce too — and same#2020-07-2817:38Alex Miller (Clojure team)well, that's weird. :) I've been using this for a while and others have tested it as well, somehow never saw an error#2020-07-2817:39Alex Miller (Clojure team)I'm sure it will make sense when I understand why it's doing it ... this is why it's a dev release!#2020-07-2817:40Alex Miller (Clojure team)-X has to go at the end, in case that affects anything you tried#2020-07-2817:41rickmoynihannope, I had -X as my only alias#2020-07-2817:42rickmoynihanWell I’d be happy to try and help narrow debug it but I’m afraid I need to call it an evening 😞
I’ll check back here tomorrow.
Thanks for your help, and also thanks for the feature… it looks like it’ll be really useful.#2020-07-2817:43Alex Miller (Clojure team)ah, this makes sense to me now, and why it worked for me#2020-07-2817:44Alex Miller (Clojure team)and why things others tried worked. such a combination of coincidences... anyhow will fix#2020-07-2817:45seancorfieldNow I'm curious, what's the bug?#2020-07-2817:46Alex Miller (Clojure team)it's not putting that class on the classpath, obviously :)#2020-07-2817:46Alex Miller (Clojure team)I was testing in the tools.deps.alpha repository itself#2020-07-2817:46Alex Miller (Clojure team)where it is#2020-07-2817:47Alex Miller (Clojure team)and others were testing things that used the install program, which requires you to pull in tools.deps.alpha#2020-07-2817:47Alex Miller (Clojure team)so those all worked#2020-07-2817:47Alex Miller (Clojure team)I may actually have flubbed a merge somewhere#2020-07-2819:00Alex Miller (Clojure team)@seancorfield @rickmoynihan ok, new clj 1.10.1.596 is available, should fix this#2020-07-2822:39rickmoynihanworks now! Thanks 🙂#2020-07-2823:10Alex Miller (Clojure team)thx#2020-07-2819:05seancorfieldLooks fixed! Thank you!#2020-07-2819:08Alex Miller (Clojure team)if you try it on windows (or linux), would be happy to hear there too#2020-07-2819:09seancorfield@vlaaad If you want to get strings from the command line in your -X-executed function, you can always use *command-line-arg* (although you'd have to discard leading arguments passed to the CLI itself I guess)#2020-07-2819:11seancorfield(! 1010)-> clj -A:test -X:example :test value
("-X:example" ":test" "value") ({:test value})
Looks like you'd only have to skip over the first arg?#2020-07-2819:12vlaaadI don't want to make every arg a string 🙂#2020-07-2819:14seancorfieldHow would you want the CLI to distinguish between "I want this as a string" vs "I want this as an EDN value"?#2020-07-2819:17seancorfieldIt seems like you'd have to try to read as EDN and if you got an error or a symbol then use the whole argument as a string, else whatever the EDN reader produced? Which is open to all sorts of surprises I would expect...?#2020-07-2819:20vlaaadI guess it's impossible to have it both simple and easy 😞#2020-07-2819:24vlaaadI don't want to deal with escaping because I use both powershell and bash that have different escaping rules. With those, whenever I have a need to escape I wish I just used a file. This might be a bit too limiting for convenience, but I'd be fine with this:
everything that can be read as number/boolean/keyword/nil -> number/boolean/keyword/nil
everything else, including things that can be read as collections -> string#2020-07-2819:27vlaaadit doesn't work with repeated args though#2020-07-2819:32vlaaadI've been thinking a bit more about simple vs easy, and currently I think I'm overreacting, I think escaping might be an okay price to pay for the limitations different approaches provide#2020-07-2819:33Alex Miller (Clojure team)when you have static data, the best thing to do is put it in deps.edn and reference via alias#2020-07-2819:34seancorfield> including things that can be read as collections -> string
I think that's a bad idea since it's common to want to pass a map of stuff when calling Clojure function (in general) so that choice would force EDN parsing on everyone#2020-07-2819:35Alex Miller (Clojure team)with -X you are always passing a map - the command line is overrides to the data in deps.edn#2020-07-2819:35Alex Miller (Clojure team)and the overrides support nested paths#2020-07-2819:35seancorfieldRight, I meant as one of the arguments (inside that map)#2020-07-2819:35Alex Miller (Clojure team)nested paths gets you out of another set of cases#2020-07-2819:35seancorfieldi.e., to produce a nested map#2020-07-2819:36vlaaadlike -X:deploy :envs '#{"prod" "qa"}'#2020-07-2819:36seancorfieldYeah, having :test 1234 and :test [1234 5678] produce a number and a string respectively would be very strange.#2020-07-2819:36vlaaadI agree 🙂#2020-07-2819:36Alex Miller (Clojure team)it turns out vectors are actually ok bash if you use the comma#2020-07-2819:36vlaaadnot ok pwsh#2020-07-2819:37Alex Miller (Clojure team):test [1234,5678] is actually ok#2020-07-2819:38seancorfieldI sympathize with the additional pain on PS (I too use it from time to time with the Clojure CLI) but since the vast majority of Clojure CLI users are on macOS/Linux I'd rather not make all their lives more painful, just to make PS less painful -- and it's already a bit painful to use for the Clojure CLI...#2020-07-2819:40Alex Miller (Clojure team)maybe this is a topic for #clj-on-windows but what's the story on the new Windows terminal?#2020-07-2819:40vlaaadhmm, I might be tripping, [1,2,3] worked fine in pwsh#2020-07-2819:48seancorfieldand clj -X:example [:test,:value] 1234 works on macOS so, yay, portability! 🙂#2020-07-2819:45seancorfield@alexmiller The new Windows terminal is just a fancy tabbed shell around existing PS, cmd, and Linux terminals.#2020-07-2819:45Alex Miller (Clojure team)so nothing new#2020-07-2819:47nateout of curiosity, why are unqualified lib names now deprecated? was there a deeper issue that needed addressing or is it just because qualified names are unambiguous and more correct?#2020-07-2819:47Darin Douglasshttps://clojurians.slack.com/archives/C06MAR553/p1595944089258600?thread_ts=1595943934.258100&cid=C06MAR553#2020-07-2819:48nateah#2020-07-2819:48nateI saw it in the blog post that he posted https://insideclojure.org/2020/07/28/clj-exec/#2020-07-2819:48natewhich mentioned the deprecation again, but I was curious if there was a deeper reason for it#2020-07-2820:25dominicmTo encourage use of groupid#2020-07-2820:26dominicmAnd discourage hiccup/hiccup in favor of com.weavejester/hiccup. But in a general sense, not the global sense.#2020-07-2820:29sveriFinally 🙂 That was one thing that always made me wonder why one would have unqualified library names when clojure is so heavily rooted in java.#2020-07-2819:52Alex Miller (Clojure team)not sure what I can add beyond what's in the blog#2020-07-2819:52Alex Miller (Clojure team)unqualified names are bad so we're going to stop encouraging them in this area#2020-07-2819:53natethat makes sense, thank you#2020-07-2820:17vlaaadPS C:\Users\Vlaaad\Projects\tdeps> cat .\deps.edn
{:aliases {:prn {:fn clojure.core/prn}}}
PS C:\Users\Vlaaad\Projects\tdeps> clj -Sdescribe
{:version "1.10.1.596"
..."}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/tools/deps/alpha/exec__init.class, clojure/tools/deps/alpha/exec.clj or clojure/tools/deps/alpha/exec.cljc on classpath.#2020-07-2820:21Alex Miller (Clojure team)hrm, let me check on that, shouldn't be using that class anymore#2020-07-2820:21vlaaad-Spath is
C:\Users\Vlaaad\.m2\repository\org\clojure\clojure\1.10.1\clojure-1.10.1.jar;C:\Users\Vlaaad\.m2\repository\org\clojure\spec.alpha\0.2.176\spec.alpha-0.2.176.jar;C:\Users\Vlaaad\.m2\repository\org\clojure\core.specs.alpha\0.2.44\core.specs.alpha-0.2.44.jar;src
`#2020-07-2820:22Alex Miller (Clojure team)path is fine, it's the class, I missed one change there#2020-07-2820:25Alex Miller (Clojure team)windows 1.10.1.600 has the fix#2020-07-2820:26dominicmDoes -X compose?#2020-07-2820:28borkdudeWhat do you mean by that?#2020-07-2820:28Alex Miller (Clojure team)depends what you want to compose with :)#2020-07-2820:28Alex Miller (Clojure team)with other -X, no#2020-07-2820:28Alex Miller (Clojure team)with all the classpath stuff, yes#2020-07-2820:29Alex Miller (Clojure team)but if you want to run multiple things, write a program that does so and run it with -X :)#2020-07-2820:31dominicmI don't want to write lots of programs! I want to change my programs on a whim at the cli. Start a socket server, start a whatever, run the program in the local project that does setup, run my custom dev function.#2020-07-2820:31dominicm(We've discussed before though, not really trying to convince, but I hoped that exec would provide this)#2020-07-2820:32dominicmRight now I'm (ab)using the fact you can supply an init and a main to load a dev.clj in my home directory, which will run things based on the directory the jvm started in...#2020-07-2820:34Alex Miller (Clojure team)there's this thing called a repl that lets you do like ... anything#2020-07-2820:35dominicmYeah, but I have to like type all these weird ( characters.#2020-07-2820:35vlaaadPS C:\Users\Vlaaad\Projects\tdeps> clj -Sdescribe
{:version "1.10.1.600" ...}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
Execution error (ExceptionInfo) at clj-exec/check-first (clj_exec.clj:32).
Invalid first arg to exec: prn
#2020-07-2820:36vlaaadcould it be PS escaping away the :? :thinking_face: other args like -A:alias work fine thogh#2020-07-2820:47Alex Miller (Clojure team)will take a look, not sure#2020-07-2820:50Alex Miller (Clojure team)maybe I'm missing the first arg I match in the pwsh for this, but seems like we'd get nothing here, not part of the arg#2020-07-2820:53Alex Miller (Clojure team)how hard is it for you to just modify the ClojureTools.psm1 ?#2020-07-2820:54Alex Miller (Clojure team)I think line 84 just needs to be $ExecAlias += $arg, $params#2020-07-2820:54Alex Miller (Clojure team)it's losing the matching $arg right now#2020-07-2918:32vlaaadI tried what you suggested, now clj "-X:prn" :a 1 succeeds, while clj -X:prn :a 1 still fails with error Invalid token: :#2020-07-2918:42Alex Miller (Clojure team)hrm#2020-07-2918:48vlaaadhacked it to work with this:
$sym, $params = $params
$ExecAlias += "$arg$sym", $params
now it doesn't work with "-X:prn" though#2020-07-2918:51vlaaadseems to be PS parsing issue:
> echo -X:prn
-X:
prn
#2020-07-2918:51vlaaad> echo -A:prn
-A:
prn
#2020-07-2918:52vlaaadhmm, that works:
s> clj -A: beep:boop:bap
WARNING: Specified aliases are undeclared: [:beep :boop :bap]
Clojure 1.10.1
user=>
#2020-07-2918:54Alex Miller (Clojure team)is there a difference in powershell parsing a single thing (into leading char / remaining char) or many things?#2020-07-2918:55vlaaadyou mean -XXX:yyy vs -X:y ?#2020-07-2918:58Alex Miller (Clojure team)no, I mean are we getting different behavior on clj -X:foo vs clj -X:foo :a 1 ?#2020-07-2919:01vlaaadno:
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn :a 1
{:a 1}
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
nil#2020-07-2919:02vlaaadbut this wont work: clj "-X:prn", because I made the script to expect args -X: and prn separately#2020-07-2919:06vlaaadso what we really need to do is accept both -X:alias and -X: , alias for powershell to make it predictable#2020-07-2919:06vlaaadI'll try to write some powershell..#2020-07-2919:11vlaaadif ($arg -eq "-X:") {
$sym, $params = $params
$ExecAlias += "$arg$sym", $params
} else {
$ExecAlias += $arg, $params
}#2020-07-2919:12vlaaadPS C:\Users\Vlaaad\Projects\tdeps> clj "-X:prn" :a 1
{:a 1}
PS C:\Users\Vlaaad\Projects\tdeps> clj "-X:prn"
nil
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn
nil
PS C:\Users\Vlaaad\Projects\tdeps> clj -X:prn :a 1
{:a 1}#2020-07-2919:24Alex Miller (Clojure team)the code for other aliases in tools.deps is actually tolerant of the split here, maybe that's necessary for the -X stuff too. in other words, we've been masking this problem#2020-07-2820:37dominicmIn earnest, there's a little bit of a desire to hide implementation behind flags. Telling someone to run clojure -A:nrepl:piggieback is easier than explaining that they need to run a set of functions once waiting for the repl to connect, and then press buttons on their editor, in an exact sequence to make it work.
Right now I use jvm options.#2020-07-2820:56seancorfieldDang, we use a lot of single-segment lib names at work! 😞#2020-07-2820:57seancorfield(updating 27 deps.edn files in our monorepo!)#2020-07-2821:00Alex Miller (Clojure team)feedback on the warning message? I tried to make it useful#2020-07-2821:03seancorfieldIt's perfect.#2020-07-2821:03seancorfieldAnnoying, but perfect.#2020-07-2821:04seancorfieldThe only slightly weird thing was that I seemed to get all of the warnings twice?#2020-07-2821:12Alex Miller (Clojure team)that dimly rings a bell. I wrote this like a month ago. I think because it's encountered during canonicalization and also during expansion or something?#2020-07-2820:58Alex Miller (Clojure team)I have https://www.youtube.com/watch?v=jsW9MlYu31g running through my head ...#2020-07-2821:00seancorfieldAm I right in thinking the new "alias data" feature is going to allow people to add all sorts of configuration stuff into deps.edn as data under an alias?#2020-07-2821:01Alex Miller (Clojure team)if they like#2020-07-2821:01seancorfieldFolks have been asking for that for quite a while 🙂#2020-07-2821:01Alex Miller (Clojure team)like most things, taste is helpful#2020-07-2821:01seancorfield(and have been told in the past, "don't do that" 🙂 )#2020-07-2821:01Alex Miller (Clojure team)it might be a good place to put like, build configuration#2020-07-2821:02seancorfieldHmm, and because aliases don't compose, it's "just" fixed data so I guess that's a nice place to draw the line.#2020-07-2821:03seancorfieldI suspect shadow-cljs and other tools might leverage it for build config to avoid having a separate file for that...#2020-07-2821:07dominicmAliases can compose, but it's down to their interpreter to decide how.#2020-07-2821:08Alex Miller (Clojure team)btw, I don't know if add-libs works with latest (haven't tried). I am still injecting the lib map atm which is what add-libs works starts from, but not sure if it also uses anything else in tools.deps that's changed. the branch I have has the stuff to switch to basis but also other stuff too so some interpolation will need to be done, not sure when I'll get to that#2020-07-2821:09seancorfieldNP. I just updated my dot-clojure deps.edn to rename the alias from :deps to :add-lib and add notes that it should be considered experimental and likely to break or go away 🙂#2020-07-2821:10seancorfield(I had forgotten that the installed deps.edn contains a :deps alias when I originally added it)#2020-07-2821:11seancorfieldI also just cleaned up the :rebl* aliases to reflect the latest REBL having its own deps.edn files for java8 and openjfx15ea (plus some reorganization and updating several versions).#2020-07-2821:11Alex Miller (Clojure team)cool#2020-07-2821:13borkdudeMy work is now finally using deps.edn/tools.deps/clojure CLI (how do you even call this stuff?). I moved out our front-end build to it (using figwheel.main / cljs.main) last week while we still do the uberjar and REPL stuff with boot. Also now we can finally run Java 11 everywhere, since we were stuck, the boot frontend stuff didn't work with it (at least not for us).#2020-07-2821:15ghadithat's exciting @borkdude#2020-07-2821:15Alex Miller (Clojure team)nice!#2020-07-2821:17borkdudeAlso we use this dev script to start 3 processes simultaneously: clojure -A:cljs/dev, clojure -A:less/dev and ./boot dev, while having only one terminal tab open:
https://gist.github.com/borkdude/8f5dff7c2330ca520403eb44c9013a83, just as a convenience#2020-07-2821:20dominicm3 jvms! I bet that doesn't integrate well with cider jack in. Might I suggest starting 3 threads instead?#2020-07-2821:21borkdudeI've always avoided cider-jack-in, I always use cider-connect. Even more so now I'm developing on a remote machine but editing files in my local emacs using tramp.#2020-07-2821:25borkdudeThreads may work but it's not a huge deal right now. We need at least two JVMs given we're using two different build/classpath/dependency tools right now.#2020-07-2821:26seancorfield"I've always avoided cider" -- ftfy 🙂#2020-07-2821:27seancorfieldAs part of this monorepo deps cleanup I'm doing (to prevent all those DEPRECATED warnings!) I'm also removing the last few references to nREPL in our setup since neither I nor my team mate use that now.#2020-07-2821:28dominicmseems unfair to people who use jack in. ¯\(ツ)/¯#2020-07-2821:28borkdudeWell, I asked my teammates and this was all OK for them, so I guess we're good 🙂#2020-07-2822:36rickmoynihanI’m curious why the new tools.deps adds a tool to do local maven installs. Is this something people frequently need?
I’ve probably only ever needed to install free form jars into my local maven repo a few times in the past ten years. I mean sure if I’ve built a jar myself I will frequently want to install it, but it seems strange to supply a method to install jars, but not build them.
I’m just curious what problem it’s solving for people.#2020-07-2822:39rickmoynihanAm I missing something in my workflow; or is it just that this happens to have been baked before other build tasks in tools.build?#2020-07-2822:56kennyPerhaps related to Datomic's dev-local needing to be installed into your local Maven repo to be used (it's not distributable).#2020-07-2823:03seancorfield^ That is what Alex said when asked earlier @rickmoynihan#2020-07-2823:06rickmoynihanahh ok makes sense in that use case. Thanks @seancorfield#2020-07-2823:10andy.fingerhut@alexmiller Robot Chicken has some pretty darned funny riffs on Star Wars scenes, including the one you linked above: https://www.youtube.com/watch?v=WpE_xMRiCLE#2020-07-2918:32vlaaadI tried what you suggested, now clj "-X:prn" :a 1 succeeds, while clj -X:prn :a 1 still fails with error Invalid token: :#2020-07-2901:15seancorfield(OK, running 1.10.1.596 on all our servers at work now 🙂 )#2020-07-2901:50salamhow would one represent this in deps.edn?
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc-bom</artifactId>
<version>19.7.0.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc10</artifactId>
</dependency>
</dependencies>#2020-07-2901:51salamif possible, of course.#2020-07-2902:00salamfor the context: https://www.oracle.com/database/technologies/maven-central-guide.html#DIY#2020-07-2902:03seancorfield@abdusalam Pretty sure you'll need to figure out what that Bill Of Materials expands to and add each of those dependencies directly to deps.edn#2020-07-2902:06salamthanks, Sean. does it mean there's no direct, built-in support in tools.deps for this kind of use case?#2020-07-2902:25seancorfieldNot as far as I'm aware.#2020-07-2902:35Alex Miller (Clojure team)no, deps does not support the BOM / import scope stuff#2020-07-2902:38Alex Miller (Clojure team)it seems radical, but deps just expects you to list the specific dependencies you're using :)#2020-07-2903:03salami see. oracle's jdbc driver (com.oracle.database.jdbc/ojdbc10) used to list other "extra goodies" as dependencies in its pom.xml. i was looking for an elegant way to exclude the extra stuff in order to avoid doing the "exclusion dance" but just found out that it's dropped all of the dependencies since version 19.7.0.0. all we need to do now is to add com.oracle.database.jdbc/ojdbc10 {:mvn/version "19.7.0.0"} to get just the driver itself. 🙂#2020-07-2903:03salamall good now#2020-07-2903:23Alex Miller (Clojure team)yeah, that's all most people need. the whole bom thing seems like something an enterprise java developer thought up while they were high#2020-07-2906:54rickmoynihanSo I really appreciate the expanded deps guide and reference, but there are two things from the previous iteration of the docs that I found useful that appear to have disappeared now:
One was the old deps data-flow/arg/alias mapping diagram: https://github.com/clojure/clojure-site/blob/master/assets/images/content/guides/deps/deps.png which I appreciate might now either need to be two diagrams, or reworked into a more complex one. Whilst it’s not something I used often, I thought it was effective in the guide at showing what aliases influenced each phase of execution etc.
And the other perhaps more importantly was the “Aliases” section in the old docs here: https://github.com/clojure/clojure-site/blob/285fda270445a70af9b1ec7bdb069309d2d224d7/content/reference/deps_and_cli.adoc#aliases which clearly described how keys in a deps file were used by the command line alias flags.
Are there any plans to bring updated versions of these sections back?#2020-07-2914:05Alex Miller (Clojure team)the docs are (always) a work in progress. I pulled the diagram b/c it was way out of date. I have updated it a number of times during the process of the basis changes but I have concerns about whether it actually provides that much value as it has grown increasingly complicated. All of the info from the old Aliases section is still in the doc.#2020-07-2919:50onetomi found that diagram quite helpful, when i was initially learning the clojure cli.
in practice however, i never really had to use anything else then -A so far...#2020-07-2907:02rickmoynihanSecond question — how stable are the new features in tools.deps? In particular the -X feature and the args map data? Are they likely to experience breaking changes between now and the next stable release? I get the feeling not.
Sometimes it’s not clear to me what is blessed as stable and isn’t… i.e. the installer, the command line args, or the tools.deps api. (Though I guess it’s all technically alpha because it’s t.d.a not t.d)#2020-07-2913:09Alex Miller (Clojure team)should be stable unless we discover some reason to change. not expecting to discover anything but that's why we're letting people try it. I'm expecting the gap to stable to be like days, not a long time.#2020-07-2915:55seancorfieldI was surprised that 1.10.1.590 seems to be the latest stable version on brew already, given the flurry of bug fix releases yesterday (1.10.1.596 and 1.10.1.600).#2020-07-2916:40Alex Miller (Clojure team)It shouldn’t be?#2020-07-2916:41Alex Miller (Clojure team)https://github.com/clojure/homebrew-tools/blob/master/Formula/clojure.rb has 561#2020-07-2916:43Alex Miller (Clojure team)So I’m curious why you said that#2020-07-2916:43seancorfieldI just did brew upgrade clojure on WSL1 on my Windows laptop last night and it updated me to 590. Hence my surprise.#2020-07-2916:43seancorfieldNote: brew on Linux.#2020-07-2916:43Alex Miller (Clojure team)That’s homebrew core, which I no longer update#2020-07-2916:44Alex Miller (Clojure team)clojure/tools/clojure is official#2020-07-2916:44seancorfieldI know. Hence my surprise 🙂#2020-07-2916:48Alex Miller (Clojure team)Well I released 590 a week ago, just took me a while to doc and announce#2020-07-2916:48Alex Miller (Clojure team)But I guess somebody updated it#2020-07-2910:46thheller@alexmiller how come you are dumping all the new features into :aliases? kinda confusing to give aliases multiple meanings IMHO. maybe I'm just not seeing the bigger picture though.#2020-07-2910:53thhellerlike why is the exec stuff in :aliases and not a new top-level :exec key or so?#2020-07-2911:05borkdudeas far as I have understood it: :aliases was always intended for naming a chunk of EDN data that could be re-used elsewhere, but it's only now that we can see that being used in more places#2020-07-2913:07Alex Miller (Clojure team)correct - I would flip it around to say :aliases is a generic facility, and we are making more use of it#2020-07-2911:06borkdudeI wonder if you can combine -A and -X to include extra deps in an -X invocation#2020-07-2913:06Alex Miller (Clojure team)you can, absolutely#2020-07-2913:06Alex Miller (Clojure team)that's kind of the idea#2020-07-2913:07Alex Miller (Clojure team)correct - I would flip it around to say :aliases is a generic facility, and we are making more use of it#2020-07-2913:51rickmoynihan:thinking_face: so you can change the function that executed with the default data (by changing the classpath via :extra-deps); but you can’t change the data for a function alias#2020-07-2914:00Alex Miller (Clojure team)I'm not sure that's "changing the function" as much as supplying the function#2020-07-2914:00Alex Miller (Clojure team)and you can override the data on the command line#2020-07-2914:01Alex Miller (Clojure team)maybe I'm not getting what you're describing#2020-07-2914:02rickmoynihanis that intended usage?
It feels a bit weird to have the same fully qualified function name, but with two different definitions (in two different :paths for example).
Would it not be more natural for -X :args to compose?#2020-07-2914:02Alex Miller (Clojure team)I don't understand why you have that#2020-07-2914:02Alex Miller (Clojure team)can you more fully describe what you're talking about, because I don't get it#2020-07-2914:10vlaaad@rickmoynihan you mean something like that?
{:aliases {:deploy-to {:fn my.ns/deploy}}
:prod {:args {:url ""}}
:qa {:args {:url ""}}}
clj -X:deploy-to:prod#2020-07-2914:14rickmoynihanyes#2020-07-2914:26vlaaadI thought it would work, but it doesn’t..#2020-07-2914:37Alex Miller (Clojure team)yeah, that's not a thing#2020-07-2914:40Alex Miller (Clojure team)I would probably do this:
{:aliases
{:deploy-prod
{:fn my.ns/deploy
:args {:url ""}}
:deploy-qa
{:fn my.ns/deploy
:args {:url ""}}}#2020-07-2914:40Alex Miller (Clojure team)but there may be other variants depending on shared config, command-line overrides etc#2020-07-2914:40Alex Miller (Clojure team)also my.ns/deploy could interpret data in :args as something to look up in :aliases via the basis#2020-07-2914:41Alex Miller (Clojure team)you don't get that for free, but it's like 2 lines of code#2020-07-2914:41Alex Miller (Clojure team)these 2 lines https://insideclojure.org/2020/07/28/clj-exec/#runtime-basis#2020-07-2914:55vlaaadYeah, I get what you mean. Basis FTW!#2020-07-2915:59seancorfieldThe basis stuff looks very interesting, but you lose access to it if your build your artifacts as uberjars and run them via java -jar (although I guess you could still do -Dclojure.basis='{...}' 👀 )#2020-07-2916:00seancorfieldIt almost makes me want to go back to running code in production via clojure instead 🙂#2020-07-2914:11rickmoynihanYeah sorry… I think I misinterpreted the motivation behind what @borkdude was saying as a way of effectively doing what @dominicm was asking earlier about composing :args data.
But I misread :extra-deps as being :deps.
Either way it seems :deps would kind of allow the kind of thing @dominicm was asking for; but in a nasty way… i.e. you could swap the function rather than the data. e.g.
- clj -A:myfn -X:args
- clj -A:myfn2 -X:args
Where myfn and myfn2 aliases defined the same ns but with different function definitions.#2020-07-2914:18dominicmI want to run fn2 and fn1 in this example. Tbh, I assumed args and fn had to co exist.#2020-07-2914:20Alex Miller (Clojure team)I'm unclear if you have a question, and if so, what it is...#2020-07-2914:21dominicmNot a question, I already asked mine!#2020-07-2914:21dominicmJust clarifying that I don't think this would allow for multiple functions like on after :)#2020-07-2914:24Alex Miller (Clojure team)no#2020-07-2914:11rickmoynihanNot saying this is a good idea btw!!!#2020-07-2914:12Alex Miller (Clojure team)one helper here is that :args can also refer to an alias instead of a map#2020-07-2914:13vlaaadthat’s something new!#2020-07-2914:12rickmoynihanoh right… that is interesting!#2020-07-2914:12rickmoynihanseems more useful than the :paths one#2020-07-2914:13Alex Miller (Clojure team)well the :paths one is foreshadowing :)#2020-07-2914:13vlaaadthat’s something new!#2020-07-2914:14Alex Miller (Clojure team)I did forget to mention that in my blog writeup, but I think it's in the docs#2020-07-2914:14Alex Miller (Clojure team)or maybe it isn't!#2020-07-2914:15rickmoynihanI don’t recall seeing it#2020-07-2914:15vlaaadyeah, me neither#2020-07-2914:15Alex Miller (Clojure team)it was in some version of the docs but looks like it got lost in the N reworks I did#2020-07-2914:15rickmoynihan😆 easily done#2020-07-2914:15Alex Miller (Clojure team)anyhow, that's a thing#2020-07-2914:16rickmoynihanwow — I was silently disappointed that something like this wasn’t there#2020-07-2914:16rickmoynihanbut it’s great that it is#2020-07-2914:16rickmoynihanis there an example of how it works?#2020-07-2914:23Alex Miller (Clojure team){:aliases
{:f1 {:fn my/fn1
:args :data}
:f2 {:fn my/fn2
:args :data}
:data {:something :here}}}
and then
clj -X:f1 or clj -X:f2#2020-07-2914:17vlaaadwhere else can I put aliases?!?!?!#2020-07-2914:17Alex Miller (Clojure team)data#2020-07-2914:17Alex Miller (Clojure team)it's just edn#2020-07-2914:18Alex Miller (Clojure team)oh sorry, misread your question#2020-07-2914:18dominicmI want to run fn2 and fn1 in this example. Tbh, I assumed args and fn had to co exist.#2020-07-2914:18Alex Miller (Clojure team)if you read https://insideclojure.org/2020/07/28/clj-exec/ check out the basis injection stuff there#2020-07-2914:26vlaaadI thought it would work, but it doesn’t..#2020-07-2914:27dominicmNow we have runtime basis, does that mean you can go back and interpret the exec aliases however you like?#2020-07-2914:36Alex Miller (Clojure team)you can write a program that uses alias data#2020-07-2914:36Alex Miller (Clojure team)which is broader than what you said#2020-07-2916:03seancorfieldI asked in a thread, but I'll ask in the main channel for everyone to see: given that clojure.basis is a property added by the clojure command, what is the thinking behind leveraging this in production code if you normally build uberjars and run those with java -jar? All of that lovely combined deps.edn structure has gone by that point...#2020-07-3009:11rickmoynihanI also had a similar thought when I stumbled across the basis property etc.
Though I figured you could spit out the basis to a clojure.basis file inside the uberjar, and slurp it as an io/resource in your -main.
However the big disadvantage to this is that the basis would be generated at build time rather than run time, so it would be more like a manifest, than what the basis is really supposed to be.
On reflection I think your idea is better — though I’m not entirely sure what you’d use the basis for in that case, other than as a means of using the clojure tool to provide config to your app.#2020-07-3009:21rickmoynihanI guess the only real advantage is using the clojure tool and a deps.edn to manage the classpath in production code, rather than the underlying java command line.#2020-07-3009:25rickmoynihanI have in the past done things like java -cp:myapp-uberjar.jar:some/server/resources/* myapp.main
to for example add assets to an apps resource path as a production overlay… so I guess using the clojure tool itself to help manage these kind of things with aliases might be useful; though I’m not convinced the benefits are huge#2020-07-3009:26rickmoynihanthough I guess it might sometimes be useful to for example provide profiles in production for enabling socket servers etc :thinking_face:#2020-07-3009:30rickmoynihanAlso another usecase is you could provide extra tooling for prod systems in this manner.
e.g. a set of aliases for various production tasks, e.g. triggering a -A:backup alias over cron etc.#2020-07-3009:31rickmoynihanand it would mean you don’t need to waste time inventing new command line parsers and config formats etc#2020-07-2916:04seancorfieldIt almost feels like it's worth switching to using clojure to run uberjars via an alias in the deps.edn so that all that stuff remains available in a "production" style context...#2020-07-2916:04ghadimy guess is that an uberjar tool could convey the basis into the jar#2020-07-2916:05ghadithe more I think about uberjars, the more disadvantages I see vs. sync'ing deps#2020-07-2916:05seancorfieldThe basis differs depending on the aliases used to run clojure tho', right?#2020-07-2916:06seancorfield(because the paths and lib map etc can be different from invocation to invocation -- although the set of aliases etc remains the same)#2020-07-2916:06ghadiI'm shooting from the hip#2020-07-2916:07ghadii'd mostly want the alias data#2020-07-2916:07ghadifor stuff like config#2020-07-2916:08seancorfieldI'd add it to depstar but I think at this point I want to wait and see what tools.build ends up being 🙂#2020-07-2916:11seancorfieldI might experiment with using clojure to run uberjars tho' later today puts on mad scientist hat...#2020-07-2916:11ghadiwdym?#2020-07-2916:12ghadiusing clojure instead of uberjars?#2020-07-2916:12seancorfieldHaving an alias that treats the uberjar as a :local/root dependency and runs the expected -main function inside it, so you get easy access to the t.d.a. / Clojure CLI infrastructure#2020-07-2916:19seancorfieldThis:
:aliases
{:run {:deps {worldsingles/worldsingles {:local/root "../../build/uberjars/worldsingles-1.0.0.jar"}}
:fn worldsingles.publisher/-main}}
...
And then clojure -X:run#2020-07-2916:20seancorfieldSo you get the "benefit" of an AOT'd uberjar with the benefits of the Clojure CLI. And the program can read the basis 🙂#2020-07-2916:23seancorfield(of course, now you need the Clojure CLI installed on a target server, not just the JVM, and you need a deps.edn file in the folder where you start the app)#2020-07-2916:35vlaaadAnd internet access#2020-07-2916:35vlaaadAnd credentials for private repos#2020-07-2916:37seancorfield@vlaaad Not if the entire :deps is just the uberjar, right?#2020-07-2916:37seancorfieldThere are no other deps to fetch at that point.#2020-07-2916:40vlaaadAh, sorry, misread the situation#2020-07-2916:41vlaaadI was thinking about using clj without uberjar#2020-07-2916:42seancorfieldRight, which we used to do on production (and, on one server, for "reasons" we still do that).#2020-07-2918:31borkdudeNot that I really care, just wanted to report that if you call a private function with -X, it also works:
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "6c0e031e07081908092c212e3c5e5c5d55"}, :content ("[email protected]")}#2020-07-2918:44Alex Miller (Clojure team)that's not intentional but interesting :)#2020-07-2918:44Alex Miller (Clojure team)it does make sense given the impl#2020-07-2918:46seancorfieldBecause requiring-resolve can get at private Vars and there's no access check on that dynamic call path, right?#2020-07-2918:48Alex Miller (Clojure team)right, just like calling through a var#2020-07-3121:09borkdudeThe private var call still works with 619 🙂#2020-07-3121:16Alex Miller (Clojure team)I'm ok with that#2020-07-3121:16Alex Miller (Clojure team)I don't consider it a bug#2020-07-3121:16Alex Miller (Clojure team)private functions are only in your mind#2020-07-3121:17borkdudeI'm going to quote you on that when somebody will ask questions about this 😂
$ bb '(ns foo) (defn- foo [] :foo!) (ns bar (:require foo)) (foo/foo)'
:foo!
#2020-07-2918:40p14nANybody else having issues with the recent tools-deps docker?#2020-07-2918:42p14nSpecifically
Caused by: Syntax error compiling . at (aleph/http/core.clj:251:3).
Caused by: java.lang.IllegalArgumentException: No matching method setTransferEncodingChunked found taking 2 args for class io.netty.handler.codec.http.HttpUtil
#2020-07-2918:42p14n(I'm making assumptions here because nothing else has changed)#2020-07-2918:44seancorfield@p14n change in JDK version perhaps? Last night I was helping some beginners debugging a weird reflection-based problem that changed between JDK 11 and JDK 14 (it was broken on 8 and 11 but worked on 14).#2020-07-2918:46p14nThat's what I thought. I switched to openjdk-8-tools-deps (tools-deps now 11) but no dice yet#2020-07-2919:12p14nFixed by using clojure:openjdk-8-tools-deps-1.10.1.502 🤷#2020-07-2919:13seancorfieldA change in the default Clojure version perhaps, with stricter syntax checking?#2020-07-2919:14ghadii don't use any of the clojure:* images#2020-07-2919:15ghadithey are not made by clojure team, and are not necessary IMHO#2020-07-2919:15ghadiI use an adoptopenjdk image and then add the clojure installer#2020-07-2919:19p14nDo you host your own?#2020-07-2919:15cap10morganHow is the groupId derived in clojure -Spom? mine keeps getting reset to the artifactId when I run it.#2020-07-2919:16seancorfieldThat's a bug. That should not happen.#2020-07-2919:16seancorfield(and it's a new bug ^ @alexmiller)#2020-07-2919:17cap10morganah, interesting. this is with 1.10.1.590.#2020-07-2919:17Alex Miller (Clojure team)I don't think anything has changed there#2020-07-2919:17seancorfield(! 1133)-> head pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
<modelVersion>4.0.0</modelVersion>
<groupId>seancorfield</groupId>
<artifactId>next.jdbc</artifactId>
<version>1.1.569</version>
<name>next.jdbc</name>
<description>The next generation of clojure.java.jdbc: a new low-level Clojure wrapper for JDBC-based access to databases.</description>
<url></url>
<licenses>
Wed Jul 29 12:17:19
(sean)-(jobs:0)-(/Developer/workspace/next.jdbc)
(! 1134)-> clojure -Spom
Wed Jul 29 12:17:30
(sean)-(jobs:0)-(/Developer/workspace/next.jdbc)
(! 1135)-> git diff
diff --git a/pom.xml b/pom.xml
index 3dce875..5632254 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="" xmlns:xsi="" xsi:schemaLocation=" ">
<modelVersion>4.0.0</modelVersion>
- <groupId>seancorfield</groupId>
+ <groupId>next.jdbc</groupId>
<artifactId>next.jdbc</artifactId>
<version>1.1.569</version>
<name>next.jdbc</name>
#2020-07-2919:18seancorfieldThe only change it makes is to overwrite the groupId#2020-07-2919:17Alex Miller (Clojure team)well, nothing intentional :)#2020-07-2919:21Alex Miller (Clojure team)can you file a jira, I'm juggling many conversations right now#2020-07-2919:22cap10morganWill do#2020-07-2919:25cap10morgan@seancorfield The expected behavior was just that it should leave that value alone when the pom.xml already exists?#2020-07-2919:28cap10morganHopefully that captures it: https://clojure.atlassian.net/browse/TDEPS-157#2020-07-2919:31souenzzoThe function that turns (merge-cli-ops {:default :opts} "[:extra :opts]" "true") => {:extra {:opts true} :default :opts} is public?#2020-07-2919:42Alex Miller (Clojure team)sorry, I don't understand the question#2020-07-2919:46Alex Miller (Clojure team)or how it relates to tools.deps#2020-07-2919:50souenzzoI think that is this one
https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/clj_exec.clj#L59#2020-07-2919:51souenzzoI liked the "DSL" to "assoc-in" values.#2020-07-2920:47Alex Miller (Clojure team)Oh, yes that’s it. Treat anything about this code as implementation detail though#2020-07-2920:47Alex Miller (Clojure team)It may migrate into Clojure itself#2020-07-2919:39seancorfield@cap10morgan Yup, it should only touch the dependencies in an existing pom.xml file.#2020-07-2919:43Alex Miller (Clojure team)well, it's broader than that - deps, repos, directories#2020-07-2919:44Alex Miller (Clojure team)there has been some work done internally here to make this more configurable and that will eventually make its way into user configurability#2020-07-3002:14cflemingIs there anything in the new tools.deps stuff that would make removing the Clojure coming from the system deps file easier? My main use cases are using a fork of Clojure, and also using Deps to manage non-Clojure projects (i.e. parts of Cursive).#2020-07-3002:16seancorfield@cfleming Can't you just use :override-deps and an alias?#2020-07-3002:16cflemingI don’t think that works.#2020-07-3002:17cflemingI’ll play around with that and see, but IIRC it didn’t work last I tried.#2020-07-3002:17seancorfieldAs for non-Clojure stuff, nothing yet but maybe tools.build might fingers crossed ?#2020-07-3002:17cflemingtools.build?#2020-07-3002:18seancorfieldI know you can select any released Clojure version via :override-deps so I assume you could point it to a forked artifact...?#2020-07-3002:18seancorfieldtools.build was mentioned in one of Alex's blog posts with a "shhh, secret" comment and it's popped up occasionally here#2020-07-3002:19seancorfieldI assume it's something Cognitect have developed internally, probably to help with Datomic (like the recent -X option in the Clojure CLI with the install option for Datomic dev-local).#2020-07-3002:19cflemingMaybe I will finally be able to get off Ant 🙂#2020-07-3002:21cfleming@seancorfield Looking at the docs, :override-deps seems to only be about overriding versions, not changing the artifact.#2020-07-3002:22seancorfieldI assumed your "artifact" would have the same group/artifact ID but the "version" would point to :local/root (a JAR)?#2020-07-3002:27cflemingWow, what do you know - it works!
~/d/tools-deps-test> cat deps.edn
{:override-deps {org.clojure/clojure {:local/root "/Users/colin/.m2/repository/com/cursive-ide/clojure/1.10.1/clojure-1.10.1.jar"}}}⏎ ~/d/tools-deps-test> clj -Spath
/Users/colin/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/colin/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/colin/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:src#2020-07-3002:28cflemingThanks @seancorfield, that may make my life much easier.#2020-07-3002:29cflemingFor modules where I don’t want Clojure, I should be able to use that trick to point Clojure to an empty jar, and then just filter that out when building.#2020-07-3002:42seancorfieldYou just made my week! 🙂#2020-07-3002:43cflemingAnd mine 🙂#2020-07-3002:44cflemingOops, wait a minute - I didn’t look closely enough. The classpath still refers to the original Clojure 😞#2020-07-3002:44seancorfieldAw, sorry...#2020-07-3002:46cflemingActually, no, the root cause is that I’m an idiot. Here’s the working version, with override-deps in an alias as it should be:
~/d/tools-deps-test> cat deps.edn
{:aliases {:fork {:override-deps {org.clojure/clojure {:local/root "/Users/colin/.m2/repository/com/cursive-ide/clojure/1.10.1/clojure-1.10.1.jar"}}}}}
~/d/tools-deps-test> clj -Spath -A:fork
/Users/colin/.m2/repository/com/cursive-ide/clojure/1.10.1/clojure-1.10.1.jar:/Users/colin/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/Users/colin/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:src#2020-07-3002:47seancorfieldOh, thank goodness!!#2020-07-3016:36dominicmOne risk of the new runtime basis is that programs are hard-coded against them, e.g. a repeat of the lein-figwheel problem. So the clj tools become mandatory for tooling. Setting a system property is easier than whatever assumptions lein plugins make about the environment. We should perhaps dissuade ourselves from doing this 🙂#2020-07-3016:37dominicmOn another note, if I do store information inside of deps.edn (e.g. a figwheel build info), how could I reload it? Is that part of tools.build?#2020-07-3016:41hiredmanthe add-lib branch distinguishes between the initial basis (static set at launch) and the runtime basis mutable as you add libraries#2020-07-3016:41hiredmanso I imagine you don't reload the basis#2020-07-3016:41hiredmanyou take it as an initial value and stick it in an atom and mutate that#2020-07-3016:47seancorfieldThere are definitely tradeoffs involved. When we got started with Clojure a decade ago, we used lein run in production and continued to use that approach when we switched to Boot (2015) and later to the Clojure CLI. It's only been since that switch that we started building uberjars instead I think, and for a long while, we did not AOT code for those. In the end, we added the AOT step purely to improve startup time (it made a big difference for a couple of our processes, so they could come back into the cluster faster after a deployment). We've always generally had our app configuration in external files -- separate from the build/run tooling -- and/or via system properties (or environment variables). But I don't have a concern about requiring our programs to be run via clojure if there are enough benefits to be worth switching from java -jar or java -cp.#2020-07-3016:47hiredman(add-lib2 I guess, add-lib still calls it the lib-map instead of the basis)#2020-07-3016:49seancorfieldAs for "clj tools become mandatory" -- we're sort of already in that place: to work with a given project pretty much requires you use whatever tooling that project has adopted, be it lein, boot, or clj.#2020-07-3016:57Alex Miller (Clojure team)the basis is just a jvm property that points to a file that is an edn map. right now, clj makes that and injects it but nothing there is "special". lein could make a basis map and pass it too. or a user could make it and set it in jvm opts in lein.#2020-07-3016:59dominicm> we're sort of already in that place:
That's true for projects, but not tools. To clarify, I mean that your project makes clojure cli mandatory (although it could support lein alongside clj too, some projects do this). I'm worried about tooling like lein-figwheel or lein-sass, etc. which is tied to a particular JVM start tool needlessly.#2020-07-3016:59Alex Miller (Clojure team)@dominicm rather than expecting to reload the basis, you could have the basis include a reference to a different file that you watch and reload (or whatever)#2020-07-3017:00Alex Miller (Clojure team)tooling that is tied to a particular tool (ie a lein plugin) was always not a good factoring of the problem. it is far better to write a tool that is independent and just code and then adapt it to tooling as needed (and to prefer tools that levy minimal requirements such that the adapter may be nothing)#2020-07-3017:01Alex Miller (Clojure team)some lein plugins are written well for this, some aren't#2020-07-3017:02dominicmThe clojure tools have inherently encouraged a lot of this through encouraging adoption of clojure main, so there's an inherent reduction of coupling to only be Clojure itself for many tools. (Not mine, mine all read deps.edn files!)#2020-07-3017:02Alex Miller (Clojure team)I don't know enough about lein-figwheel or lein-sass or whatever to say on those (but having "lein" in the name seems like a flag)#2020-07-3017:04dominicmOne consideration will be what is standard/non-standard in the runtime basis. Lein can't tell you what aliases were loaded, it's a semantic mismatch. It could tell you some information about the deps though.#2020-07-3017:15Alex Miller (Clojure team)https://clojure.github.io/tools.deps.alpha/#clojure.tools.deps.alpha/calc-basis#2020-07-3017:16Alex Miller (Clojure team)docs the basis map return (more detail is needed, a bit of it is intentionally a little vague right now, but it's mostly pretty set). I have not yet updated the specs for tools.deps but I will get to it eventually#2020-07-3017:17dominicmTo me, those seem like they should be namespaced to tools deps if you think it's OK for other tools to generate. With the exception of lib-map, which I could see being generated by other tools (boot/lein)#2020-07-3017:29seancorfieldThe overall map is the combined deps.edn hash map, with those four extra keys added, so you're already tied to the deps.edn structure in the clojure.basis property. Leiningen isn't going to be able to create that master-edn map anyway, unless it adopts t.d.a. and deps.edn as an option (alongside project.clj) and changes its dependency resolution approach. So it seems you're already tied to clojure.basis and those simple keys at this point -- no need to qualify them, IMO.#2020-07-3017:30seancorfieldPerhaps Leiningen could/should produce a lein.basis file/property when running code, that reflects the hash map structure of the project.clj file? But I don't think you can expect portability between tools (and I don't think you should).#2020-07-3017:33seancorfield"Lein can't tell you what aliases were loaded, it's a semantic mismatch." -- clojure.basis doesn't tell you what aliases were used to run the program, only the resulting "basis" of those aliases, but it includes the whole merged master-edn so you get "all" aliases available -- and lein could provide the :aliases structure from project.clj (although it's a completely different beast).#2020-07-3022:25deactivateduserSo interesting finding just now that might be useful to others. I operate a tiny little chat bot, that runs on a cheap / low-spec PAAS. Earlier today I switched from running it via the Clojure CLI (i.e. via something like clojure -m mybot.main) to constructing an uberjar and running that (i.e. via something like java -jar mybot.jar), mostly in order to reduce startup times (the PAAS doesn’t provide blue/green deployment at the subscription level I’m on, and the brief outage during scheduled restarts had been noticed and commented on by some users).
As expected, startup time basically went to zero. What I didn’t expect was how much of a memory saving there’d be too. Previously, my bot had stabilised at around 340MB total (heap & off-heap) memory usage, which was quite close to the 512MB total provided by the PAAS, and which has to cover all processes’ memory use (not just that of my bot’s JVM).
Since the switch to an uberjar, total memory usage has stabilised at around 210MB. While I didn’t confirm this in detail, my preliminary conclusion is that the extra ~130MB is allocated by the tools.deps machinery (after all that’s the only thing that changed), and is not released after tools.deps has done its thing and is no longer in use.#2020-07-3022:37Alex Miller (Clojure team)Those are separate jvms so I don’t think I buy that if that was your theory#2020-07-3022:37deactivateduserWhat is your theory?#2020-07-3022:38deactivateduserNote: the uberjar is not AOT compiled (except for the ns with the -main function).#2020-07-3023:09hiredmanaot compilation is transitive, so unless you work to avoid it, aot compiling a single namespace will also aot all namespaces it depends on as well#2020-07-3023:14deactivateduserIs that true of (gen-class) ?#2020-07-3023:15deactivateduserThat’s the only thing I’m using (since it’s required in order to generate an “executable JAR” that java -jar can run directly).#2020-07-3023:15deactivateduserThat said, this is my first use of depstar, so perhaps it’s doing something like that…#2020-07-3023:20hiredmanit is complicated, but if you are using gen-class in the ns form, then yes, everything is being aot compiled#2020-07-3023:22deactivateduserI think you’re onto something. I cracked open the uberjar and sure enough my code is in there twice in .clj and .class form.#2020-07-3022:39Alex Miller (Clojure team)I would buy it for a container around the clj call, not sure what the PAAS is or how it works#2020-07-3022:40deactivateduserHeroku dyno (“hobby” subscription level). It’s pretty simple - a VM, as best I can tell. I’m not using Docker or any other container-like deployment mechanism.#2020-07-3022:40Alex Miller (Clojure team)I guess I'm saying "is allocated by the `tools.deps` machinery" is not in your final process, but may consume resources in the container itself#2020-07-3022:40deactivateduserDoes that JVM stick around though? Once the “app” JVM is started?#2020-07-3022:40Alex Miller (Clojure team)no#2020-07-3022:41Alex Miller (Clojure team)the script waits for it to exit#2020-07-3022:41Alex Miller (Clojure team)before running the app#2020-07-3022:48Alex Miller (Clojure team)the clojure script forks a child process to run the first jvm, waits for it to exit, then exec's the app which replaces the script with the final jvm process#2020-07-3022:48deactivateduserI’m out of ideas. I did look into whether the local “ephemeral” filesystem might be a RAM drive that counts towards the VM’s RAM total, but that’s not the case (the ephemeral filesystem is 6GB).#2020-07-3022:49deactivateduser(since my app’s dependencies total around 90MB - somewhat close to the 130MB delta)#2020-07-3022:49Alex Miller (Clojure team)I can easily envision a container expanding its memory footprint and thus taking more time in the first case or something like that#2020-07-3022:50Alex Miller (Clojure team)but certainly the time savings makes sense#2020-07-3022:50deactivateduserRight, but the dashboard I’m looking at shows total memory used within the VM (and doesn’t give me any idea of the total RAM of the physical hardware that VM is running on).#2020-07-3022:51deactivateduserYeah, and startup time savings was the impetus for making this change. I’m just surprised (and wish I could explain) the additional savings in RAM usage. Especially given how substantial those savings were (memory use dropped by ~40%).#2020-07-3022:55deactivateduserHere’s what I see, fwiw, spanning the time of the deployment where I switched to an uberjar:#2020-07-3022:56deactivateduserThat graph is basically flat, going back a month or two (to when I first deployed the bot).#2020-07-3022:56Alex Miller (Clojure team)well, that could be reserved memory based on the max used (affected by the first jvm)#2020-07-3022:57deactivateduserIt isn’t though - that dashboard further breaks down the JVM’s heap and off-heap memory usage, and both of those have dropped by approximately the same %ages.#2020-07-3022:57Alex Miller (Clojure team)I don't really have any way to tell#2020-07-3022:57deactivateduserI wouldn’t expect Heroku to “keep counting” a JVM that’s been terminated, indefinitely.#2020-07-3022:58deactivateduserLet me see if I can repro locally…#2020-07-3023:05deactivateduserDifferent numbers obvs, but I think I see a similar pattern locally (using visualvm to look at just heap & metaspace usage, so definitely not an apples-to-apples comparison with Heroku).#2020-07-3023:05deactivateduserHere’s clojure -m mybot.main:#2020-07-3023:05deactivateduserAnd here’s java -jar mybot.jar:#2020-07-3023:06deactivateduser(look at “used”, specifically - the JVM has allocated different total amounts from the OS in these runs, which throws off the vertical scales between the two runs)#2020-07-3023:08deactivateduserFor easier reading, here are the key numbers:
| clj | jar |
------------+--------+--------+
Heap | 83MB | 39MB |
Metaspace | 71MB | 60MB |
------------+--------+--------+
#2020-07-3023:12deactivateduser(similar pattern, albeit different numbers, on successive runs, and with a GC requested before taking measurements)#2020-07-3023:13deactivateduserNo clue what’s going, tbh. I’m not one to look a gift horse in the mouth, but then unexpected changes of this magnitude also make me suspicious. 😉#2020-07-3023:15Alex Miller (Clojure team)one difference is that the jvm memory maps the jars - and here you've got many jars vs 1#2020-07-3023:15Alex Miller (Clojure team)might be some per-jar overhead you're avoiding?#2020-07-3023:16Alex Miller (Clojure team)another thing to try would be to use -Scp#2020-07-3023:16Alex Miller (Clojure team)clojure -Scp the.jar -m your.namespace#2020-07-3023:17Alex Miller (Clojure team)that would be still using the script, not computing a classpath but using 1 jar#2020-07-3023:18Alex Miller (Clojure team)guessing that's still the lower number#2020-07-3023:18Alex Miller (Clojure team)and the next test would be to get the script out of the way. grab the computed classpath with -Spath, then java -jar vs java -cp with that#2020-07-3023:37deactivateduserYeah I was wondering about total # of open file handles, but it’s hard to come up with 130MB that way. 😉#2020-07-3023:38deactivateduser@hiredman found something though - it looks like all my code is being AOT compiled (the uberjar contains both .clj and .class versions of all of my namespaces). I don’t know why that’s happening (this is my first time using depstar, so I may have messed something up there), though apparently (gen-class) (which is used on my “-main” ns) can cause transitive AOT compilation too.#2020-07-3023:41deactivateduserWould runtime compilation of Clojure generate a lot of long-lived memory usage like that though? That’s the bit I can’t quite wrap my head around - this extra memory isn’t being GCed.#2020-07-3023:42deactivateduser(as part of trying to keep memory usage down on this restricted runtime environment, the bot calls (System/gc) every hour - I know that doesn’t guarantee that the GC will kick in but I figure it can’t hurt, especially as this particular bot is 99% batch/offline functions so it doesn’t matter if it’s not highly responsive to user activity)#2020-07-3023:49hiredmanhave you set a max memory on the jvm?#2020-07-3023:49deactivateduserWhen running locally, yes: -Xmx300m (I keep it low to try to emulate the memory-limited Heroku environment).#2020-07-3023:50deactivateduserOn Heroku it’s container managed: -XX:+UseContainerSupport#2020-07-3023:50deactivateduser(locally my JVM is 11.0.2+9, while Heroku is JVM 11.0.8, btw)#2020-07-3100:23ghadi(isn't container support the default now in jdk11?)#2020-07-3100:31seancorfield@deactivateduser10790 "I don’t know why that’s happening (this is my first time using depstar, so I may have messed something up there)" -- no, that's just how AOT works. In order to compile your main namespace, it has to load (compile) all the namespaces it depends on so you get pretty much your whole program compiled to .class files. That's just normal. What (:gen-class) does is flag that particular namespace to be generated as a class with methods -- functions beginning with - (by default) will become named methods on that class, instead of (what Clojure usually does which is) to compile each function to its own class with an .invoke() method. So your -main function becomes a public static method of the class (which is what Java expects).#2020-07-3100:33deactivateduserI didn’t realise AOT-compiled-Clojure couldn’t “thunk” into on-demand-compiled-Clojure.#2020-07-3100:33seancorfieldLeiningen tries to do some fancy post-AOT clean up I think, to delete a lot of .class files that aren't directly part of your project so you don't get "everything" compiled? Can't remember whether that is the default or an option.#2020-07-3100:33deactivateduser(I almost never use AOT, so have minimal experience with it)#2020-07-3100:33deactivateduserYeah and all of my previous experience has been with leiningen.#2020-07-3100:34seancorfieldIf you don't want other nses compiled, don't :require them in your main ns -- instead use runtime require/`resolve` (or the nice new requiring-resolve in 1.10).#2020-07-3100:34deactivateduserThanks - looking now…#2020-07-3100:34deactivateduserI don’t particularly care either way, tbh. Have just heard folks here recommend against AOTing much, if any, of a codebase (if possible).#2020-07-3100:35seancorfieldOne good test to run here would be to build your uberjar without -C and -m on depstar -- which would not AOT anything -- and then run it with java -cp path/to/the.jar clojure.main -m mybot.main#2020-07-3100:36seancorfieldThat will do compile-on-demand and if it behaves the same as the original clj memory usage, then compile-on-demand might be responsible for the larger memory usage.#2020-07-3100:38deactivateduserOk now that my 5pm US-PT (actually midnight UTC) batch job is done, I’ll give that a try. 😉#2020-07-3100:42deactivateduserSo locally, removing AOT seems to have resumed the “higher memory usage” scenario.#2020-07-3100:42deactivateduserNext up, heroku env.#2020-07-3100:45seancorfieldSo it sounds like there's some overhead from loading source and compiling on demand that doesn't go away? I'm a bit surprised it doesn't all get GC'd eventually.#2020-07-3100:47deactivateduserYeah me too. Though I only have up to 24 hours of continuous operation available to look at, since Heroku forcibly restarts the dyno (VM) at least once per day. (as mentioned above, that includes at least 24 calls to (System/gc)).#2020-07-3100:49deactivateduserDeployed. Now we wait for a bit for it to show up on the Heroku dashboard.#2020-07-3100:51seancorfieldI can't say I noticed a difference at work when we switched from compile-on-demand to AOT, but then our smallest process runs with 1G heap so a few 10s of MBs wouldn't really be noticeable...#2020-07-3100:53deactivateduserMakes sense. I can definitely see the CPU cost of on-demand compilation at startup showing up. 😉 Memory graph still a bit inconclusive…#2020-07-3100:54hiredmanCalling System/gc is silly. Predicting what effect (if any)calling that method has is hard, so calling it, and then trying to reason from the basis of calling it is not helpful#2020-07-3100:54deactivateduserSure, but it can’t hurt in a non-interactive app.#2020-07-3100:54hiredmanYou should look at the gc logs, and look at the sizes of the generationd#2020-07-3100:55deactivateduserAnd even a multi-second pause in this bot is preferable to exceeding memory (which puts me in Heroku “R14” jail…).#2020-07-3100:55ghadihow will you be in jail if the memory is bounded?#2020-07-3100:55hiredmanCalling System/gc will never stop you from running out of memory#2020-07-3100:56hiredmanThat is how automatic garbage collection works#2020-07-3100:56deactivateduserBy exceeding the 512MB quota for everything inside the VM (JVM heap + JVM metaspace + memory required by any other processes + OS memory requirements (caches, etc.) + …).#2020-07-3100:56hiredmanIt can only monkey with the heuristics in the gc#2020-07-3100:57deactivateduserRegardless, I tested it first (obvs) and it improved memory utilisation over longer periods of time.#2020-07-3100:57hiredmanYou should set a lower max heap for the jvm then#2020-07-3100:57deactivateduserI don’t set the heap - the container does.#2020-07-3100:57deactivateduserRemember I’m running with -XX:+UseContainerSupport.#2020-07-3100:57hiredmanThen have the container do a better job#2020-07-3100:57deactivateduserNot to mention that controlling the heap doesn’t help with metaspace anyway.#2020-07-3100:57hiredmanIf you are exceeding the limits#2020-07-3100:57deactivateduser(which is where on-demand compiled classes etc. go)#2020-07-3100:58deactivateduserKnow anyone who works at Heroku to improve their VM impl.? Cause I don’t. 😉#2020-07-3100:58deactivateduserIt’s not a container.#2020-07-3100:58deactivateduserIt’s a “dyno”.#2020-07-3100:58deactivateduserI don’t provide anything but the stuff that gets fed to the JVM.#2020-07-3100:59deactivateduser(and while Heroku also supports Docker, I have negative interest in switching to that model of app construction)#2020-07-3100:59deactivateduserSo it’s still a bit early to be sure, but it looks like it was AOT that was saving much of the memory:#2020-07-3101:00deactivateduser(that second “hump” is when I deployed with AOT disabled)#2020-07-3101:01hiredmanYou should check the different generations, compilation may result in extra old generation garbage, that is never collected because it never needs to be#2020-07-3101:03deactivateduserFrom what I’m seeing most of the savings are in the metaspace, not the heap.#2020-07-3101:03deactivateduser(recalling that old generation is on-heap)#2020-07-3101:05deactivateduserWhich makes sense if on-demand compiled Clojure code is somehow more off-heap-expensive (e.g. metaspace, code-cache) than AOT compiled Clojure code.#2020-07-3101:05seancorfieldIn your original numbers, you only showed 11MB difference in metaspace, but 44MB in heap.#2020-07-3101:05deactivateduserYeah that’s local.#2020-07-3101:05ghadiif you can AOT, AOT#2020-07-3101:06seancorfieldAye, AOT as the last step of an application, prior to production deployment seems like a reasonable step to me.#2020-07-3101:06deactivateduser@ghadi I probably will, given the memory restrictions in this environment. That doesn’t help me understand why this is happening though.#2020-07-3101:06deactivateduserTo date I’ve never used AOT, and if there is a substantial (~40% memory saving, for this one bot) saving to be had, that could be a discriminating factor in choosing AOT vs on-demand compilation for other apps in future.#2020-07-3101:06seancorfieldYou'll need VisualVM or something similar to see what's actually going on inside the heap/metaspace...#2020-07-3101:07deactivateduser@seancorfield no easy way to hook that up to Heroku, though it runs an agent that does break down on-heap vs off-heap memory usage for their dashboard.#2020-07-3101:07seancorfieldRight, I meant locally, with a low heap size to vaguely mirror what's happening on Heroku.#2020-07-3101:09deactivateduserYeah - I’m doing that, but the absolute numbers are pretty different than what I see on Heroku. Possibly because it’s hard to limit the JVM’s off-heap memory usage (-Xmx only affects the heap), so it’s happily slurping up the (copious, compared to Heroku) memory on my laptop.#2020-07-3101:11deactivateduserBTW, here’s a more complete view of heap vs off-heap usage reported by heroku. The two deployments (the first to AOT uberjar, then back to non-AOT uberjar) are pretty obvious:#2020-07-3101:11deactivateduserStand by - I accidentally chopped off “total memory usage”.#2020-07-3101:11ghadiwe're in the wrong channel, btw#2020-07-3101:12deactivateduserWe are now, yes. Originally this looked like tools.deps issue.#2020-07-3101:12ghadiI concur with @hiredman’s suggestion to get better details about heap regions.#2020-07-3101:13ghadithere are several interesting metrics depending on the GC used (G1 by default on 11)#2020-07-3101:13deactivateduserHere’s a better picture:#2020-07-3101:15deactivateduserEven at this granularity, it’s pretty clear that the delta in the non-heap memory usage is greater than that on-heap.#2020-07-3101:15deactivateduseri.e. looking at the heap regions likely won’t help#2020-07-3101:16deactivateduser(and yes the default Heroku dashboard sucks - it’s deliberately setup to encourage one to pay for a better one)#2020-07-3101:18deactivateduserSadly I have to step away from this now, but it looks like AOT is causing the difference, even if it’s unclear why. Thanks for indulging me on this!#2020-07-3101:31ghadinot a mystery to me. compiling is more work than not compiling#2020-07-3102:06deactivateduserMore work == CPU load, sure. But that work, once complete, shouldn’t leave long-lived garbage hanging around.#2020-07-3102:08deactivateduserWell, unless the JVM does tricks for AOT-compiled classes that it can’t use for dynamically generated (non-disk-resident) classes (e.g. mmap’ing pre-compiled .class files rather than keeping them resident in memory at all times, unloading disk-resident .class files once they’ve been JITted, etc.).#2020-07-3102:28deactivateduserOr if the Clojure compiler has caches that don’t get primed if it’s not called upon to dynamically compile some Clojure code.#2020-07-3120:57Alex Miller (Clojure team)A new clj release candidate (what we were formerly calling "dev" releases) is now available - 1.10.1.619 (see https://github.com/clojure/homebrew-tools#version-archive-tool-releases for installation info)
• Fixes -Spom regression in overwriting groupId in existing pom.xml files
• Improvements in error handling for -X
• New: -F execution specifier to invoke an arbitrary function that takes a map at the command line: clj -Fclojure.core/pr :a 1 :b 2 => {:a 1 :b 2}#2020-07-3121:03borkdudeAwesome, I already wondered why one had to specify a function in deps.edn instead of just being able to call any function#2020-07-3121:05seancorfield(! 1038)-> clojure -Sdeps '{:aliases {:foo {:fn clojure.core/prn :args "test"}}}' -X:foo
Invalid :args for exec, must be map or alias keyword: "test"
(! 1039)-> clojure -Sdeps '{:aliases {:foo {:fn clojure.core/prn :args :bar} :bar "test"}}' -X:foo
"test"
🙂#2020-07-3121:08seancorfieldThanks for the quick fix to pom.xml groupId!#2020-07-3121:13borkdudeWhy not just pass the entire map as one cmd line arg?
$ clojure -Fclojure.core/pr '{:a 1 :b 2}'
Key is missing value: {:a 1 :b 2}
or support both?#2020-07-3121:14borkdudenot a big deal, but if the map is coming from a file, this could get a bit weird#2020-07-3121:14Alex Miller (Clojure team)because quoting sucks and it's more of a match to the -X#2020-07-3121:15borkdudequoting sucks, but I bet you will soon be quoting the individual args as well, there is no escape 😉#2020-07-3121:15Alex Miller (Clojure team)but I implemented both at different points :)#2020-07-3121:15Alex Miller (Clojure team)no pun intended :)#2020-07-3121:45vlaaadpowershell's -X:alias still broken 😞#2020-07-3121:46vlaaadhave you missed my suggested fix that handles splits to -X: and alias ?
if ($arg -eq "-X:") {
$sym, $params = $params
$ExecAlias += "$arg$sym", $params
} else {
$ExecAlias += $arg, $params
}#2020-07-3121:57Alex Miller (Clojure team)yeah, I haven't had a chance to look at that yet#2020-07-3121:57Alex Miller (Clojure team)it's in the queue!#2020-07-3122:38seancorfield@alexmiller I'm a bit surprised this first version fails (in 1.10.1.619):
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "106375717e73505455435b445f403d414522455a215e"}, :content ("[email protected]")}#2020-07-3122:42seancorfieldIt's not a big deal but it worked before and I wasn't sure if it was a deliberate change or just an artifact of the implementation.#2020-07-3122:49Alex Miller (Clojure team)Yeah, that should probably work #2020-08-0304:08cflemingWith the new :paths alias data feature, I’m assuming that those aliases are just fished directly out of the :aliases map in the (combined system/user/project) deps.edn map? i.e. there’s no alias selection done based on which aliases are passed by the user, or anything like that.#2020-08-0304:44Alex Miller (Clojure team)Right#2020-08-0304:50cflemingOne further question: Cursive also shows aliases that the user can select in a tool window, which allows the user to decide which source paths etc should be synced to their project. I assume it doesn’t make sense to show these aliases there, i.e. it only makes sense to show those aliases whose value is a map.#2020-08-0304:50cfleminghttps://cursive-ide.com/userguide/deps.html#working-with-aliases#2020-08-0312:07Alex Miller (Clojure team)Seems right#2020-08-0304:09seancorfield@cfleming That's my understanding based on playing with it a bit.#2020-08-0304:10cflemingThanks @seancorfield. It gets a little confusing in the case of :extra-paths - those are calculated using the provided aliases, but then their alias values are taken from the raw map (I think).#2020-08-0304:46Alex Miller (Clojure team)In the latter question, aliases should be resolved from the merged edn map, same as paths. But maybe I’m not getting some nuance of your question#2020-08-0304:46cflemingYep, that was what I had assumed.#2020-08-0304:47cflemingSo using this map:
{:paths ["src" :clj-paths :resource-paths]
:aliases {:clj-paths ["src2"]
:resource-paths ["resources"]
:moar {:extra-paths ["src3" :moar-paths]}
:moar-paths ["src4"]}}#2020-08-0304:48cflemingsrc, src2 and resources will always be selected. src3 and src4 will be selected if the user has chosen to use the moar alias.#2020-08-0304:51seancorfield@U064X3EF3 It feels like this is part of a bigger picture that we don't have all the pieces of -- since the alias references in ;paths etc is all static at the moment, so you can't actually override any of it yet?#2020-08-0312:03Alex Miller (Clojure team)@cfleming right#2020-08-0312:03Alex Miller (Clojure team)@seancorfield not sure what you mean about overriding#2020-08-0312:04Alex Miller (Clojure team)This is about giving names to sunsets of paths#2020-08-0312:04Alex Miller (Clojure team)Subsets#2020-08-0312:05Alex Miller (Clojure team)One important place these names show up is in the classpath data in the basis#2020-08-0316:54seancorfieldBy "override" I meant that project aliases override user-level, and you can further override those on the command-line -- so I was thinking that you could provide a new :clj-paths alias on the command-line to "override" a :clj-paths alias in the project deps.edn.#2020-08-0316:54seancorfieldLike I say, I'm still not seeing much benefit in that feature yet, so I assume there's a "bigger picture" and it will make more sense "soon" 🙂#2020-08-0317:22Alex Miller (Clojure team)you could do all that but I do not have any use case in mind where that would be a good idea#2020-08-0317:22Alex Miller (Clojure team)better to manipulate with a -C alias modifying :extra-paths or overrides#2020-08-0317:46seancorfieldAye, and that's what I would tend to do -- so, again, I'm not seeing much benefit in being able to specify parts of paths etc as aliases right now.#2020-08-0317:54Alex Miller (Clojure team)then you should not do it :)#2020-08-0304:11seancorfieldI guess you can supply new values via the command-line (`-Sdeps`) but I don't see much value in it as it stands.#2020-08-0304:12seancorfieldI assume it's part of a bigger picture that we don't have all the pieces of yet...#2020-08-0307:40borkdudeWhat does -S stand for? Just a random name?#2020-08-0311:58Janne SauvalaBased on all the comments I guess it stands for “something” 😁#2020-08-0312:07Alex Miller (Clojure team)Setting#2020-08-0312:10borkdudeRiiight#2020-08-0312:30Alex Miller (Clojure team)but "something" or "stuff" is pretty good too :)#2020-08-0312:54borkdude🙂#2020-08-0312:54borkdudeBtw, I have wondered how you prevent :) turning into an emoticon. Is that a setting?#2020-08-0312:55borkdudeNm, I found it :)#2020-08-0308:08vlaaadI just went through deps guide and reference searching for words starting with "s". My guess is it means "Some option" or "Specify"#2020-08-0311:08dominicmI think it was script#2020-08-0312:08Alex Miller (Clojure team)Setting#2020-08-0619:15dharriganI have a question, related to this extra-deps, using a couple of :local/root declarations, i.e., one that loads in rebel readline (with little customisation) and another which launches a nrepl (with some middleware injected). Can I still benefit from what the guide says? Can I compile the extra-deps includes, so that my clj startup is reduced?#2020-08-0619:16Alex Miller (Clojure team)sure#2020-08-0619:16Alex Miller (Clojure team)it's all on the same classpath#2020-08-0619:17dharriganOkay, but where would the classes directory live, i.e., if I have something like this...#2020-08-0619:17dharrigan:rebel {:extra-deps {local-rebel {:local/root "/home/david/.clojure/libs/local.rebel" :deps/manifest :deps}
local-nrepl {:local/root "/home/david/.clojure/libs/local.nrepl" :deps/manifest :deps}}
:main-opts ["-m" "local.rebel.main"]}#2020-08-0619:17Alex Miller (Clojure team)I'd put it in its own alias and then use multiple aliases#2020-08-0619:17Alex Miller (Clojure team)ie, it's actually the same as what's in the doc, you just might have extra tool aliases in play#2020-08-0619:18dharriganokay, let me have a play around. My clj startup at the moment is taking about 8 seconds to launch...#2020-08-0619:29souenzzoOh....
Mine is ~2min on good machines
and we really don't care about that#2020-08-0619:36dharrigan:shocked_face_with_exploding_head:#2020-08-0619:39souenzzoI open the repl less then once a day.
(it remains open)#2020-08-0619:18Alex Miller (Clojure team)so clj -A:dev:rebel or whatever#2020-08-0619:18Alex Miller (Clojure team)or actually you've got main args in there#2020-08-0619:18Alex Miller (Clojure team)so you might want to do clj -R:dev:rebel when doing the compilation#2020-08-0619:19Alex Miller (Clojure team)so you get just the deps but not the main stuff#2020-08-0619:19dharriganokay, will have a play 🙂 Thank you Alex! 🙂#2020-08-0619:19Alex Miller (Clojure team)🍺#2020-08-0619:19dharriganI'd love one thanks 🙂#2020-08-0619:38seancorfieldI only restart my REPL about once a week so even a few minutes startup wouldn't really bother me. I'm curious about workflows where startup times of eight seconds is problematic @dharrigan?#2020-08-0619:40dharriganI have a need, a need for speed 🙂 I guess I'm just learning about if by applying some of the recommendations in that web page I can shave off some time! No harm in trying to optimise if I can 🙂#2020-08-0619:45dharriganInteresting, got it down to 3 seconds 🙂#2020-08-0619:59Alex Miller (Clojure team)I'll send you my bill#2020-08-0619:59dharriganErrrr....would an I.O.U suffice?#2020-08-0620:00dharriganI.O.U one beer, fresh from the tap, if when you get to Ireland/UK 🙂#2020-08-0620:01Alex Miller (Clojure team)I hope I can leave my house again someday#2020-08-0620:23lread@dharrigan’s avatar is a duck… he already has a bill!#2020-08-0620:26dharriganHe's the Egyptian Household God of Frustration.#2020-08-0620:30lreadTIL!: https://www.youtube.com/watch?v=im29S6ZWRDI#2020-08-0620:33dharriganBest Show Evar!#2020-08-0620:25seancorfield@alexmiller So I went and tried that dev startup time stuff (for the first time). I compiled a bunch of big libraries that my code relies on that rarely change, but did not compile my own code, then a ran a "load file" on my app's main file and it threw an exception about a clojure.core.cache.CacheProtocol not found... so I'm guessing there's a possibility for this pre-compile step to mess up in some way?#2020-08-0620:26Alex Miller (Clojure team)depends how you compiled everything#2020-08-0620:26Alex Miller (Clojure team)if you're trying to compile independent libs then you might not be compiling in the same order you actually load them in via your app#2020-08-0620:27Alex Miller (Clojure team)and then you can get into the case of recompiling a protocol, which can definitely go wrong#2020-08-0620:27seancorfieldWhen I tried to repro with a (require 'api.main) instead of the load-file that the editor does, that seemed to work, so maybe it's something specific to load-file?#2020-08-0620:28Alex Miller (Clojure team)require will load too under the hood, but maybe#2020-08-0620:29seancorfieldAh, no difference. I started a fresh REPL, with those libraries precompiled and require failed this time, just as load-file had before.#2020-08-0620:30seancorfieldSo, you have to transitively compile "everything" to make this work, so that compile order doesn't matter?#2020-08-0620:30Alex Miller (Clojure team)transitively compiling everything should work (b/c it's exactly what your app does when it loads)#2020-08-0620:30seancorfieldWhat about a project where you have multiple entry point nses, that represent separate applications?#2020-08-0620:31Alex Miller (Clojure team)other things might work :)#2020-08-0620:31Alex Miller (Clojure team)you could use more than one alias/classes dir if you find that to be a problem#2020-08-0620:32Alex Miller (Clojure team)or you could still use just compile with one and use from the other entry point. you might miss some stuff but presumably it's mostly the same in deps.#2020-08-0620:32seancorfieldCurrently, I start a REPL once a week (roughly) and just load code into it as needed so I usually end up with our entire code base loaded into the REPL eventually.#2020-08-0620:33Alex Miller (Clojure team)you are a bit outside the target user of this page :)#2020-08-0620:33seancorfieldCompiling two different entry points that overlap might fall into the "recompiling a protocol" area?#2020-08-0620:33Alex Miller (Clojure team)depends#2020-08-0620:34Alex Miller (Clojure team)I am shorthanding the actual problem (recompilation itself is not the actual problem)#2020-08-0620:35seancorfieldSo the problematic piece (that surprises me a bit) is that (compile 'clojure.core.memoize) does not seem to transitively compile clojure.core.cache which I would have expected... I think that's the root of my earlier problem perhaps?#2020-08-0620:36Alex Miller (Clojure team)https://clojure.atlassian.net/browse/CLJ-1544#2020-08-0620:36Alex Miller (Clojure team)it should transitively compile anything loaded by clojure.core.memoize#2020-08-0620:36Alex Miller (Clojure team)but not everything in core.cache is used by memoize#2020-08-0620:39seancorfieldWhen I look in the classes tree after (compile 'clojure.core.memoize) there are no core.cache classes at all.#2020-08-0620:41hiredmanwhat about if you compile clojure.core.cache? my kind of guess would be it is already compiled/loaded somewhere else in your repl#2020-08-0620:41Alex Miller (Clojure team)any chance you've already loaded something that has loaded core.cache prior?#2020-08-0620:41Alex Miller (Clojure team)compile piggybacks load so if it's not being loaded (b/c it already is), it won't be compiled#2020-08-0620:43seancorfieldAh, that is a possibility, because something in my dev tooling might well be using that... If I explicitly (compile 'clojure.core.cache) I do see the .class files produced for it which would suggest it isn't loaded previously?#2020-08-0620:43seancorfield(or am I misunderstanding your point?)#2020-08-0620:44hiredmanthat is interesting, I wouldn't expect to see those if clojure.core.cache was previously loaded#2020-08-0620:45Alex Miller (Clojure team)I suspect you're getting the equivalent of :reload at the top level namespace you compile#2020-08-0620:45Alex Miller (Clojure team)but not on the ones below#2020-08-0620:45seancorfieldYup, confirmed. A "bare" REPL with just my dev tooling loaded has clojure.core.cache clojure.core.protocols clojure.core.server clojure.core.specs.alpha clojure.data.priority-map ... (non-`clojure.*` stuff omitted).#2020-08-0620:46hiredmanah, right, compile calls load-one directly, bypassing the already loaded check#2020-08-0620:47seancorfieldSo, yeah, I'd need to explicitly (compile 'clojure.core.cache) to make all this work, right?#2020-08-0620:48Alex Miller (Clojure team)you can do what's in the bottom of that guide (kind of), which is to address a similar problem (when user.clj has already loaded stuff)#2020-08-0620:48Alex Miller (Clojure team)(binding [*compile-files* true] (require 'clojure.core.memoize :reload-all))
#2020-08-0620:49Alex Miller (Clojure team)the only thing that makes load do compile is that dynvar#2020-08-0620:49Alex Miller (Clojure team)and the :reload-all will force a reload#2020-08-0620:56seancorfieldI'm going to stick a list of (compile 'x.y.z) forms in a compile.clj file so I can do load-file on it to compile "everything" that I want pre-compiled and I'll add to it as/when I run into problems. FWIW, just with the set of libs I've put in that list so far, a cold (require 'api.main) has dropped from 18 seconds to 10 seconds. Like I say, since I only restart my REPL occasionally, that's really not a concern but now I'm curious about this idea of pre-compiling dependencies that rarely change...#2020-08-0620:56seancorfield...and if a dependency does change, presumably you'd have to force a recompile of the newer version (and everything that depended on it) in order to pick those changes up?#2020-08-0620:57Alex Miller (Clojure team)you don't have to#2020-08-0620:57Alex Miller (Clojure team)a newer dependency should have newer .clj files and be preferred#2020-08-0620:57Alex Miller (Clojure team)it will just be slower till you recompile#2020-08-0620:57Alex Miller (Clojure team)clojure takes the newer .class or .clj#2020-08-0620:58Alex Miller (Clojure team)I guess if you changed deps to a newer version of a lib that was still older than your local .class files that wouldn't have that effect#2020-08-0620:59Alex Miller (Clojure team)I spent a half day trying to build this into clj so it could do it automatically (since it knows when your deps go stale). it was tricky for a bunch of reasons but maybe a future feature#2020-08-0621:00Alex Miller (Clojure team)the point where your .cpcache is stale is also the point when you want to recompile#2020-08-0621:00Alex Miller (Clojure team)so was trying to hook that#2020-08-0621:00seancorfieldYeah, discussions about it being a possible future feature of clj was what made me even curious to try it, once it was brought up here.#2020-08-0621:03Alex Miller (Clojure team)relevant for tools that replace the classpath too#2020-08-0920:11borkdudeI get a ton of "DEPRECATED libs must be qualified" warnings not from my own deps.edn, but for transitive deps. Can I somehow turn that off? It's not something I can fix in my own codebase.#2020-08-0920:12borkdudeAlso one thought: would it make sense to only deprecate it for git deps and not for mvn deps, since mvn already has the foo = foo/foo implicit convention and there's not much tools.deps is going to change about that?#2020-08-0920:23vlaaadThis is not about foo=foo/foo convention, this about a way of thinking about dependencies. Shorthands incentivize names squatting, full group/artifact names make you think who you depend on. I'd rather depend on borkdude/clj-kondo than on just clj-kondo. I'd rather released vlaaad/cljfx than just cljfx... I think it makes sense to state group/artifact for every procurer.#2020-08-0920:23dominicmI don't think maven does have that implicit thing, does it?#2020-08-0920:23dominicmThey're very big on groups#2020-08-0920:34borkdudeIf you push a library foobar with lein to clojars, it's really foobar/foobar, so the group is foobar and the library is also called foobar. That's what I mean with a mvn library. The mechanism for pushing/retrieving is based on mvn. I know avoiding conflicts is important, but this convention has been there since ages in leiningen. Is this a lein convention or a mvn convention, I'm not sure. I haven't used pom.xml directly that much.#2020-08-0920:36borkdudeSo is this a design mistake of lein that deps now tries to solve? I totally understand the need for git deps to have a unique and predictable identifier.#2020-08-0920:36dominicmI suspect it's a lein and clojars thing. Maven central won't let you create an artifact like that, only clojars does (to controversy)#2020-08-0920:40borkdudeAnyway, I hope that I turn off the warning for transitive deps, since I'm getting a ton of them:
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "0a686578616e7f6e6f4a47485a383a3b33"}, :content ("[email protected]")}#2020-08-0920:44seancorfield@borkdude You're only getting those for git deps -- not for all transitive deps.#2020-08-0920:44seancorfieldIf you were depending on released versions, you wouldn't get those warnings. I've only had to update my current project deps to get rid of warnings -- but I don't generally depend on git deps.#2020-08-0920:46borkdudeI'm getting warnings for 1) my own git deps, 2) my own mvn deps, 3) mvn deps of transitive git deps. I'm not sure if a git dep depending on another git dep would also yield a warning, but I suspect this situation isn't that prevalent yet (let's call this 4).
I understand the need for 1. 2 is disputable, but from a "let's fix what lein did wrong" perspective, I can live with it. 3 and 4 I wouldn't find acceptable.#2020-08-0920:47borkdude1 and 2 aren't shown in the output, I already fixed those.#2020-08-0920:49Alex Miller (Clojure team)You are getting these any time clj reads any deps manifest#2020-08-0920:50Alex Miller (Clojure team)So would for 4 as well#2020-08-0920:52Alex Miller (Clojure team)I could tailor these more, but this is a step towards removing support entirely#2020-08-0920:56Alex Miller (Clojure team)The thing I’m really trying to get rid of here is the need for canonicalization of libs when reading deps files#2020-08-0920:56seancorfieldI have run across a handful of "single" named Java libs (`wsdl4j` plus a few others), but it's pretty rare in non-Clojure world. I agree that we've been very sloppy about that practice in the past (and I'm guilty a few times -- but I'm working to fix that 🙂 )#2020-08-0920:56borkdudeWell, leaving it as it is now will probably force people to update their deps.edns quicker ;)#2020-08-0920:57Alex Miller (Clojure team)Well that’s the idea :)#2020-08-0921:01dominicmI'm glad there's more of a push on this. I've seen a few conflicts in libraries (I've seen at least two libraries using roll.core).
I've gotten friction in the past because it's "verbose", so it's nice this will become non optional.#2020-08-0921:02borkdudeI think that example just shows how .core doesn't contribute to avoiding conflicts while still having a multi-segmented name#2020-08-0921:04seancorfieldI think .core is a horrible default 😐#2020-08-0921:06seancorfieldThat's why clj-new doesn't do that. You must provide either a group-id/artifact-id project name (and the README talks about using reverse domain names or other identity-based groups) or you can provide a multi-segment namespace name. But I may downplay that option a bit more, now that t.d.a. is "encouraging" proper group/artifact lib names 🙂#2020-08-0923:28cfleming@alexmiller -Stree seems to be broken since 1.10.1.624. It’s because previously root deps used to return nil in their :dependents set, and now they return an empty collection. This broke Cursive, I’ve fixed it there by inlining make-tree and changing:
(let [{roots false, nonroots true} (group-by #(-> % val :dependents boolean) lib-map)]
to:
(let [{roots false, nonroots true} (group-by #(-> % val :dependents empty? not boolean) lib-map)]#2020-08-0923:30cflemingI’m not sure what the cause of that change is, I couldn’t see anything immediately suspicious in the changelog but I only had a fairly cursory look.#2020-08-0923:32cflemingOn a related note, is the :dependents/`:children` info returned from resolve-deps considered to be part of the API and thus something I can rely on? It’s not mentioned in the docstring.#2020-08-0923:43seancorfieldThe output of -Stree has certainly changed -- I just updated from .619 to .641 and my outdated deps script broke (so that'll be a bit of fun for me to debug and fix tomorrow morning when I get to work!).#2020-08-0923:43cflemingWell, it’s changed from appearing to not appearing at all.#2020-08-0923:44seancorfieldAh, yes, that would definitely break my script. I hadn't actually tested basic output yet.#2020-08-0923:44cfleming~/d/c/issues (master)> clj -Stree
~/d/c/issues (master)>
#2020-08-0923:45seancorfieldOops.#2020-08-0923:45seancorfieldWell... I guess I don't need to debug my script 🙂#2020-08-0923:46cfleminghttps://clojure.atlassian.net/browse/TDEPS-161#2020-08-1001:47Alex Miller (Clojure team)I actually rewrote most of the tree stuff this morning, still working on it though#2020-08-1002:24Alex Miller (Clojure team)@cfleming ah, I see where I broke it, will fix. I don't know who's updating the homebrew core tap, but it's just random versions now#2020-08-1002:26cfleming@alexmiller Ok, I basically copied your make-tree function and fixed my copy in Cursive. As long as I can rely on the :dependents/`:children` bit coming back from resolve-deps that’s probably a better idea anyway, since I won’t have to fix anything if you change make-tree to do something different.#2020-08-1002:30Alex Miller (Clojure team)the bug was in the last refactoring of lib-paths#2020-08-1002:31Alex Miller (Clojure team)so basically in the guts of resolve-deps#2020-08-1002:33Alex Miller (Clojure team)you can't rely on :children - that's implementation detail leaking. :dependents you can rely on as that's part of the lib-map data structure and is spec'ed https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/specs.clj#L60-L64#2020-08-1002:34cflemingIs dependents enough to build the tree without children?#2020-08-1002:34Alex Miller (Clojure team)depends what you're trying to do with it#2020-08-1002:35Alex Miller (Clojure team)make-tree is going to be moving and changing quite a bit - are you using that directly?#2020-08-1002:35cflemingBasically, show the user a dependency tree in Cursive.#2020-08-1002:35cflemingI was, but after working around this bug this morning I’m now using a fixed copy of the old version.#2020-08-1002:36Alex Miller (Clojure team)this whole thing with you depending on whatever arbitrary version the user happens to have is crazy talk man#2020-08-1002:37cflemingThat’s why I want data I can depend on in the resolve-deps results (or anywhere else, don’t care where I get it from) that I can use to build a tree to show the user.#2020-08-1002:37Alex Miller (Clojure team)the reason I'm rewriting make-tree is I suspect the same reason you're relying on that stuff - to better form the tree and the lib-map not being sufficient#2020-08-1002:38cflemingBasically, until there is a defined API that gives me what I need I have no choice but to hack the guts.#2020-08-1002:38cflemingWhich is intrinsically tied to concrete versions.#2020-08-1002:38Alex Miller (Clojure team)well, depending on specific concrete versions is fine :) it's having no control over what you're using that seems like crazy talk#2020-08-1002:39cflemingI can never control what the user has installed.#2020-08-1002:39cflemingI can let them choose the version that they want to use, but I can’t mandate versions.#2020-08-1002:39seancorfield@cfleming What are you actually using this information for?#2020-08-1002:39Alex Miller (Clojure team)I guess I don't see why you can't use a specific version of tools.deps independent from whatever clj they have installed#2020-08-1002:40cflemingBecause when you introduce new stuff in deps, people aren’t going to like having to update Cursive to get it.#2020-08-1002:40Alex Miller (Clojure team)seems better than having it break every time I update clj#2020-08-1002:40cflemingAnd as long as there’s a stable API, it shouldn’t matter.#2020-08-1002:40Alex Miller (Clojure team)there's not a stable API#2020-08-1002:40Alex Miller (Clojure team)that's the alpha part#2020-08-1002:41cflemingThen it will break.#2020-08-1002:42cfleming@seancorfield In this particular case, to show the user the deps tree in the IDE: https://cursive-ide.com/userguide/deps.html#viewing-your-project-dependencies#2020-08-1002:45cflemingIf the user finds a particular version of deps breaks with a particular version of Cursive, they can choose the version of deps they want to use within Cursive, or they can downgrade their CLI until either Cursive or deps gets fixed.#2020-08-1002:46seancorfieldI'm continually impressed at the work you do with Cursive but at the same time it just reinforces why I like a simple editor and simple command line tools 🙂#2020-08-1002:46cfleminged will always be there for you.#2020-08-1002:46cfleming🙂#2020-08-1002:48seancorfieldYou can pry Atom/Chlorine and the Clojure CLI from my cold, dead hands mate!#2020-08-1002:49seancorfield(although I've recently taken to using Reveal instead of REBL, just as an experiment with my workflow... but I'm missing several of the REBL features)#2020-08-1002:50cflemingTo be fair, I think you have a broken script until the CLI is fixed.#2020-08-1002:50Alex Miller (Clojure team)fixed in clj 1.10.1.645#2020-08-1002:50Alex Miller (Clojure team)which is uploading right now#2020-08-1002:52seancorfieldlooks fixed -- thanks @alexmiller#2020-08-1002:55seancorfieldAnd my outdated deps script works again -- yay!#2020-08-1003:00Alex Miller (Clojure team)@cfleming I'll give you a heads up before I publish the tree stuff - still working on it.#2020-08-1003:01cflemingOk, thanks.#2020-08-1003:01Alex Miller (Clojure team)just as background, deps expansion does not actually ever build an explicit tree during expansion and in fact it's not a tree, it's really a graph.#2020-08-1003:02Alex Miller (Clojure team)the make-tree/print-tree stuff is kind of retroactively trying to build a tree from what's in the lib map, but that's really also not an explicit tree and does not contain sufficient path info to respect the exclusions the user originally stated#2020-08-1003:03Alex Miller (Clojure team)I've rebuilt the tree stuff to work from the trace (which is actually returned in the metadata on the lib map if it was requested), but it's surprisingly complicated to rebuild a tree that matches the user's original intent#2020-08-1003:04cflemingI can believe that.#2020-08-1003:04cflemingMostly I’d just like a stable way to replicate what you show the user on the command line in Cursive.#2020-08-1003:05Alex Miller (Clojure team)yeah, and there will be a function somewhere that can work from the trace (which you can actually emit from clj with -Strace) to give you that tree data#2020-08-1003:05cflemingI mean, I can actually show them a graph if required.#2020-08-1003:05Alex Miller (Clojure team)that's kind of the stuff I'm wrestling with - there are multiple things you might want to show#2020-08-1003:06Alex Miller (Clojure team)what most of the deps tools show is basically a tree with every lib listed once (even though it's really possibly a dep at many points)#2020-08-1003:07Alex Miller (Clojure team)but you in particular might really want the "full" tree - full expansions at every node (could still stop at cycles in a path)#2020-08-1003:08Alex Miller (Clojure team)and then for debugging you really want to know - what version was requested, what version was actually selected, and what libs were excluded at which different points#2020-08-1003:08cflemingYup, and then I could collapse duplicates to mirror what users are used to seeing in other tools, while still allowing them to expand them if required.#2020-08-1003:09Alex Miller (Clojure team)all of that info is in the version map included in the trace but it requires a fair amount of trawling#2020-08-1003:10cflemingWhen you talk about the trace, do you mean something included in the new basis stuff?#2020-08-1003:11Alex Miller (Clojure team)no, pre-dates that actually#2020-08-1003:11Alex Miller (Clojure team)clj -Strace will dump a trace.edn file#2020-08-1003:12cflemingIs that documented anywhere? It doesn’t seem to be in the reference doc.#2020-08-1003:12Alex Miller (Clojure team)no, I wasn't sure how stable it was going to be. I added that when I made tools.deps.graph so basically prior to my conj talk#2020-08-1003:13Alex Miller (Clojure team)but it hasn't actually changed since then I don't think and this tree work will add another "user" of it#2020-08-1003:14cflemingOk, looks like I could use that if the generation of it were available in t.d.a and I could call that.#2020-08-1003:15Alex Miller (Clojure team)it has been available in t.d.a since since last November in the return of resolve-deps#2020-08-1003:15Alex Miller (Clojure team)if you pass the :trace option, it will be included in the metadata of the lib map that's returned#2020-08-1003:16Alex Miller (Clojure team)that part is doc'ed at https://clojure.github.io/tools.deps.alpha/#clojure.tools.deps.alpha/resolve-deps#2020-08-1003:16cflemingOk. If that’ll be a more stable thing for me to use I’ll start using that.#2020-08-1003:16Alex Miller (Clojure team)no plans to change it ;)#2020-08-1003:17cflemingWhat’s missing from the doc is descriptions of what’s actually in the trace log, or the runtime basis for that matter. Are they specced or something?#2020-08-1003:17Alex Miller (Clojure team)no, I wasn't quite ready to commit to them yet and haven't really caught up on that#2020-08-1003:17Alex Miller (Clojure team)the basis is doc'ed in calc-basis#2020-08-1003:18Alex Miller (Clojure team)https://clojure.github.io/tools.deps.alpha/#clojure.tools.deps.alpha/calc-basis#2020-08-1003:18Alex Miller (Clojure team)it is all a bit strewn around atm and specs are missing some of those#2020-08-1003:19Alex Miller (Clojure team)so, definitely some doc debt to pay back. my time is kind of split right now with joining nubank :)#2020-08-1003:20cflemingOk, when I get some time I’ll investigate switching to that, and falling back to what I do now if the metadata isn’t available. I’ll be interested to see your tree code when it’s done.#2020-08-1011:22pezAnyone know of a docker image containing both leiningen and clojure? I’m migrating a project piece by piece and am using the lein-tools-deps plugin. Currently using jdk 8, as well, unsure how hard that dependency is though.#2020-08-1418:33cap10morganThe clojure:latest image now has both lein and CLI tools installed if that helps.#2020-08-1011:37borkdude@pez FWIW I usually just install clojure in a Docker container myself#2020-08-1011:37borkdudeSee any of my latest projects#2020-08-1418:32cap10morganIs this a bug? When I update my pom.xml file with clojure -Spom using http://1.10.1.itit is nesting another <build> element under the existing one. So my diff looks like this:
<build>
- <sourceDirectory>src</sourceDirectory>
+ <build>
+ <sourceDirectory>src</sourceDirectory>
+ </build>
</build>
#2020-08-1418:33Alex Miller (Clojure team)it is a bug#2020-08-1418:33Alex Miller (Clojure team)has been fixed in a subsequent prerelease but not yet in a stable one#2020-08-1418:34cap10morganah ok. cool, thanks!#2020-08-1418:34Alex Miller (Clojure team)it's fixed as of 1.10.1.590 (latest prerelease is 1.10.1.645)#2020-08-1418:35Alex Miller (Clojure team)or you could fall back to older stable#2020-08-1418:35seancorfieldWhich prerelease introduced the bug where it overwrote the groupId? (since fixed)#2020-08-1418:36cap10morganexactly what I was wondering 🙂#2020-08-1418:37Alex Miller (Clojure team)I think it was 1.10.1.590#2020-08-1418:37Alex Miller (Clojure team)which was fixed in 1.10.1.636#2020-08-1418:37Alex Miller (Clojure team)it is hard to keep track of all these across the two repos#2020-08-1418:38Alex Miller (Clojure team)but I would say if you're going to use any prerelease, I would use the latest#2020-08-1418:40Alex Miller (Clojure team)I have 3 or 4 pending things to get into clj and then I'm hoping to get to another stable release#2020-08-1612:07borkdude@seancorfield Would it be possible to exclude clojure and clojure.spec.alpha sources/AOT-ed assets from an uberjar? The use case would be for babashka which already has clojure built-in (spec not yet, but maybe coming). So an uberjar with only (non-clojure) source files and no AOT-ed stuff would be leaner.
https://github.com/borkdude/babashka/issues/535#issuecomment-674518153
Since you're reading from a system property here https://github.com/seancorfield/depstar/blob/b7e9e4537e33bbf78fbdbc0ef56d9675cb494bc2/src/hf/depstar/uberjar.clj#L283
it might be a small tweak to override the classpath with a command line arg or file reference (java itself uses the @ convention for that).#2020-08-1612:36borkdudeE.g.:
clojure -A:depstar -m hf.depstar.uberjar MyProject.jar -v -m proj --classpath "$(clojure -Spath)"
#2020-08-1612:36borkdudeBtw, I think depstar will be able to compile with GraalVM so maybe I can even include it in babashka itself to create babashka uberjars#2020-08-1616:46Alexis VincentI’d like to distribute a deps.edn file of aliases for everyone to use in our company. But I want people to be able to have their own local ./clojure/deps.edn. Is there a way to specify a deps.edn file to merge in, either globally for all projects, or from a projects deps.edn?#2020-08-1616:58Alex Miller (Clojure team)no, not currently#2020-08-1617:29j0niIs it possible to add a list of jar files from a project sub-dir to the classpath without having to name them? I know about :local/root and I see this in the docs https://clojure.org/guides/deps_and_cli#local_jar but I have a lot of them and they aren't published anywhere. Do I just have to make up names for them and add them individually as :deps?#2020-08-1617:32Alex Miller (Clojure team)It’s kind of a hack, but java supports on the classpath for this (literal path/ so not shell expanded). I think if you added a :paths entry for “path/*” that should work#2020-08-1708:15rickmoynihanInteresting I was wanting to do something similar too, so I could deploy an app as a zip with a directory of jars in lib but start it with the clojure command line tooling; via a :local/root; with a different deps.edn for production, rather than the one we use in dev.#2020-08-1713:43Alex Miller (Clojure team)the other crazy thing you can do is to reference jars in your manifest (this was part of the original applet support) with the Class-Path attribute but I think that has to be explicit jars. afaik that does not support the glob stuff#2020-08-1714:55rickmoynihanyeah now you mention it I’ve seen manifests do this…. a long time since I’ve seen or written an applet tho! 😆#2020-08-1714:57Alex Miller (Clojure team)also relevant, the manifest has a line wrapping limit and so other arcane rules so it's best to use the apis to make it rather than doing it by hand#2020-08-1714:59rickmoynihanwow I had no idea… eitherway if I do this I think it’ll be via the :local/root method, largely motivated via the new -F -X stuff#2020-08-1617:32j0nithank you @alexmiller I will try that out#2020-08-1617:59j0niamazing, thank you Alex, it works#2020-08-1619:30borkdude@seancorfield Awesome: depstar works with GraalVM and can hence be included in babashka proper so it can make its own uberjars#2020-08-1619:34borkdude$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj
Building uber jar: foo.jar
$ bb -jar foo.jar
:foo - failed: int?
#2020-08-1620:24seancorfield@borkdude Cool. As for your earlier question, depstar deliberately just includes everything on the current classpath, aside from itself.#2020-08-1620:26ghadiThat was a fundamental design decision I regret#2020-08-1620:27ghadiIn retrospect I’d use t.d.a as a library to calculate what to tar up#2020-08-1620:27ghadirather than subtracting depstar itself from the classpath#2020-08-1620:29borkdudeyep. but that can be easily changed. I've already done that in my fork#2020-08-1620:31ghadiYou using t.d.a to calculate instead?#2020-08-1620:32seancorfieldTo be honest, using the classpath has made depstar a lot easier to use for people, and a lot more predictable.#2020-08-1620:33borkdude@ghadi I just added a :classpath option that can be used to override#2020-08-1620:33borkdudehttps://github.com/borkdude/depstar/blob/d3fb96c3731d4b7987e75927e6c13497da22fae2/src/hf/depstar/uberjar.clj#L360#2020-08-1620:34seancorfieldHmm, interesting approach. Since it's easy to get clojure to produce an arbitrary classpath from deps.edn aliases etc...#2020-08-1620:34borkdudeExactly.#2020-08-1620:36ghadithis is a similar thing to what I’m trying to avoid, vectors of strings#2020-08-1620:36borkdudeSo the classpath given to babashka, produced by clojure is just passed down to depstar:
$ bb -cp $(clojure -Spath) --uberjar foo.jar -m proj
which results in a foo.jar that can be invoked with bb -jar foo.jar.#2020-08-1620:38ghadiI understand. I’m just saying the shell is not leverage compared to a clojure library. One of the things I want to do is split a list of dependencies by their rate of change for smarter distribution to containers or Lambdas#2020-08-1620:41borkdudeYou mean, make uberjars for things not changing often?#2020-08-1620:42ghadiFor containers, you don’t want every commit to result in a 60MB new uberjar, you want some structural sharing across commits#2020-08-1620:42borkdudeI guess one could use the -A or -T option for that, to make groups of deps and then push that classpath into depstar#2020-08-1620:42ghadiYeah I know you can hack it in shell, but why not program it directly in bb or clojure?#2020-08-1620:43ghadimy hope with tools.build (not that I have anything to do with it currently) is that it encourages build tasks to compose better than the lowest common denominator of the CLI#2020-08-1620:43borkdudebb will have an --uberjar option, but it can also have a bb -e '(babashka.uberjar/run {:classpath ...})-ish interface#2020-08-1620:44borkdudecan have them both#2020-08-1620:44vlaaadWait are tools.build out?#2020-08-1620:44ghadino#2020-08-1620:45seancorfielddepstar currently has zero dependencies, except Clojure itself, so it can't interfere with anyone's project dependencies -- that's worked very well in practice. I like @borkdude’s idea of just giving it an explicit classpath to build form 🙂#2020-08-1620:47vlaaadUnless I want to have the whole build happen in a single vm, and it has to have other build-specific dependencies like data.xml for updating pom.xml. I use 2 vms for that now, but I'd rather not :)#2020-08-1621:02seancorfieldI'm not sure whether you're agreeing with depstar's approach or suggesting something different?#2020-08-1620:45borkdudebb won't do AOT btw (maybe it could but I don't think it's worth it). for JVM uberjars (with AOT-ed classes), just use clojure / depstar#2020-08-1620:45borkdudebb uberjars typically will contain sources and other assets, but no classes#2020-08-1620:47borkdudeI did have depstar compile with GraalVM as is and when I didn't remove the AOT-related code, it did compile, but the binary got somewhat bloated (+ 30MB, happens when it hits a specific part of Clojure). So maybe compile just works natively, since it's just going to transform .clj into classfiles, I haven't tried it.#2020-08-1620:51borkdudeI'll try it now... just for science#2020-08-1621:15borkdudeInteresting. Compilation of a class doesn't work in a GraalVM binary because the classpath inside the binary is not set correctly for clojure RT. I guess it won't work for that reason, although technically it could maybe work.#2020-08-1621:24borkdudeGoing to try this:
(binding [*compile-path* (str tmp-c-dir)]
(clojure.lang.RT/compile (str main-class ".clj")))#2020-08-1621:25borkdudecrap, it's private ;)#2020-08-1621:38borkdudeWith a reflective hack it works. I see all kinds of AOT-ed sourced, even aot__init.class, but no aot.class. Maybe :gen-class isn't part of RT/compile? Eh.... enough for today.#2020-08-1621:42borkdudeSo the AOT part could be useful when depstar had a binary CLI version, so to speak. I don't need it for bb, but it could be useful for fast build tooling in general.#2020-08-1621:45borkdudeTo make this work clojure.lang.RT should also have a pluggable classpath resolver. It might work for Clojure files, but I wonder how it will work if code relies on .class files from the Java world. Does the Clojure compiler kind of treat .class files as data, without actually having to have them on the classpath per se, as long as you point it to the right file? I.e. no runtime need, just able to read and analyse statically?#2020-08-1622:05borkdudeI see, Compiler relies on reflection when dealing with interop, so there it will likely break, since the reflection API only works with classes that are part of the same runtime. OK, this was a nice tangent.#2020-08-1708:56borkdudeCreating uberjars with depstar native in bb now also works on Windows:#2020-08-1713:40jimberlageAnyone ever seen a message like the following after updating the :sha of a git dependency:
Error building classpath. Destination path "my-lib" already exists and is not an empty directory
org.eclipse.jgit.api.errors.JGitInternalException: Destination path "my-lib" already exists and is not an empty directory
#2020-08-1713:43delaguardoyou can delete local artifacts in $HOME/.gitlibs/{libs,_repos} and restart the repl#2020-08-1713:44delaguardoI don’t know exactly what cause that problem. In my case it happens after changing :git/url#2020-08-1713:44Alex Miller (Clojure team)I believe this is the result of parallel downloading to gitlibs messing up one of the cached git dirs#2020-08-1713:50Alex Miller (Clojure team)if you clear your ~/.gitlibs (either fully or selectively) and then do clj -Sforce -Sthreads 1 that should force a re-download singlethreaded#2020-08-1713:51Alex Miller (Clojure team)we have a ticket for this, I haven't had a chance to really look at in depth yet#2020-08-1713:55jimberlageThanks! Trying that revealed another error - technically auth-related, but testing ssh pushes/pulls to github on this repo still works for me, so I'm wondering if this is somethimg on my end or a symptom of a different bug:
Error building classpath.
#2020-08-1713:59Alex Miller (Clojure team)you'll need to debug why it's not finding the key:
1. is ssh-agent running? if not, start it
2. does ssh-add -l list your public key? if not, add it
3. if you're on a Mac, check whether your ~/.ssh/config has Host * / IdentityFile in it, if so comment that out or remove it#2020-08-1713:59Alex Miller (Clojure team)after that, things get trickier to debug, could be key type#2020-08-1714:05jimberlageLooks like the bit about the IdentityFile line fixed it. Thank you#2020-08-1714:14Alex Miller (Clojure team):thumbsup:#2020-08-1911:10borkdudeWhy does clojure not warn about an unexisting :test alias?
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "9cfef3eef7f8e9f8f9dcd1deccaeacada5"}, :content ("[email protected]")}
#2020-08-1911:11borkdudeAlso, I figured it would be useful to have something like: clojure -CR:test to get the same as clojure -C:test -R:test: use case, I want to run a REPL with the same paths and deps as the :test alias, just not execute the main#2020-08-1913:11Alex Miller (Clojure team)you can use clojure -A:test to do that#2020-08-1913:12Alex Miller (Clojure team)oh, "and not execute the main" - well for that you need the -C / -R#2020-08-1911:28vlaaadI think :test exists in install deps?#2020-08-1911:58dharriganindeed, in my install on Arch, in /usr/share/clojure/deps.edn it has test as an alias#2020-08-1912:56souenzzoThis file
https://github.com/clojure/brew-install/blob/1.10.1/src/main/resources/deps.edn
The "global" deps#2020-08-1916:59seancorfield@borkdude https://github.com/seancorfield/depstar/issues/34#2020-08-1917:30Alex Miller (Clojure team)-Scp is an existing clj option btw#2020-08-1917:40borkdudeThat's useful, but it only affects the runtime classpath. This feature is about overriding the classpath from which the uberjar is created, which makes it more flexible. The classpath can be generated from other tools like leiningen, boot, clojure, etc, whatever source.#2020-08-1917:46Alex Miller (Clojure team)gotcha, might be something good to add to tools.build's uber stuff then :)#2020-08-1919:41Janne Sauvala@alexmiller I keep hearing references to tools.build (since your Inside Clojure -post). Is there more detailed description of it somewhere already or can we expect something coming out soonish? 🙂#2020-08-1920:01seancorfieldI suspect the answers are "no" and "maybe" -- depending on your definition of "soon" 🙂#2020-08-1920:09Alex Miller (Clojure team)^^#2020-08-1920:09Alex Miller (Clojure team)trying to pull together a lot of strands right now, not sure on timing yet#2020-08-2005:59Janne SauvalaGotcha. I’m exited to hear tools.build could be a thing. I really enjoy working with tools.deps and the latest features look great :thumbsup::skin-tone-2:#2020-08-1917:03seancorfield@alexmiller https://github.com/seancorfield/depstar/issues/35 -- good idea you posted in #beginners !#2020-08-1917:30Alex Miller (Clojure team)we would definitely like to encourage more use of -X - considering a bunch of additional features that may continue to make it more attractive for tools#2020-08-1917:52vlaaadFeatures like...? :)#2020-08-1917:59Alex Miller (Clojure team)well, still designing :)#2020-08-1918:01Alex Miller (Clojure team)but really trying to simplify invocation in several ways#2020-08-1918:39seancorfieldUpdating hf.depstar.uberjar:
(defn run
"Generic entry point for all invocations.
Can be used with `clojure -X`:
In `:aliases`:
:depstar {:extra-deps {seancorfield/depstar {:mvn/version ...}}}
:uberjar {:fn hf.depstar.uberjar/run
:args {:aot true :jar-type :uber}}
Then run:
clojure -R:depstar -X:uberjar :jar MyProject.jar :main-class project.core
If the destination JAR file and main class are fixed, then could be
added to `:args` in `deps.edn`:
:depstar {:extra-deps {seancorfield/depstar {:mvn/version ...}}}
:uberjar {:fn hf.depstar.uberjar/run
:args {:aot true :jar MyProject.jar
:jar-type :uber :main-class project.core}}
Both `:jar` and `:main-class` can be specified as symbols or strings."
It would be nice if -X acted like -R .. -X .. so you didn't have to also specify -R:depstar here and could have the :extra-deps in the :uberjar alias.#2020-08-1918:41Alex Miller (Clojure team)what I'm working on right now is in that area#2020-08-1918:42Alex Miller (Clojure team)all over it!#2020-08-1917:30Alex Miller (Clojure team)and easier to use#2020-08-1917:42borkdude@seancorfield One other thing that came up in #beginners: the non-standard way of providing the name of the uberjar. It would be easier to compose if that could be done using a named parameter like --uberjar foo.jar#2020-08-1917:43borkdudeSee https://clojurians.slack.com/archives/C053AK3F9/p1597841581095000#2020-08-1918:40seancorfield@borkdude Yeah, I'm making it :jar for the -X option and it will also be -J / --jar on the command-line (as well as just accepting the filename anywhere).#2020-08-1918:47dakraNice :)#2020-08-1918:41seancorfieldSo @daniel415’s example will work exactly as he would like.#2020-08-1918:41borkdude:thumbsup:#2020-08-1920:17seancorfieldOK, in light of earlier discussions, the develop branch of depstar has several new features requested this morning: support for clojure -X, support for overriding the classpath used for building the JAR, and reworked argument parsing so the JAR filename can be specified "anywhere" (before/after/in the middle of the options). Please take it for a spin and report back any issues (either on GitHub or here). seancorfield/depstar {:git/url "" :sha "c461ba830e069edf0f011169c18595a4a8f917fc"} Thanks in advance#2020-08-2012:41timo👋
How do I pass a version to clj -Spom?#2020-08-2012:43timoI need it for CI. or is there a more clever way to update the version in the pom.xml for clojars-deployment?#2020-08-2012:48delaguardoI’m using sed 🙂
https://github.com/xapix-io/axel-f/blob/master/.github/workflows/publish.yml#L45-L46#2020-08-2013:15timosure, this is another option#2020-08-2013:15timothanks#2020-08-2012:48dominicm@timok generally you cannot. clj -Spom only updates your deps (and generates a basic outline).#2020-08-2012:48dominicmThere's garamond which can do that though#2020-08-2012:50jose@timo you can use the maven versions plugin: mvn versions:set versions:set-scm-tag -DnewVersion="${version}"#2020-08-2012:51timoah thanks @dominicm @jlle. then I will keep the pom.xml in the repo.#2020-08-2012:52joseif you want to publish from github, I wrote a github action: https://github.com/jlesquembre/clojars-publish-action#2020-08-2013:09timothanks @jlle but it's actually circleci but I will have a look :thumbsup:#2020-08-2014:41joseDo you think that the new -T CLI option can be a better alternative to user.clj? I can imagine moving some repl helpers I have in my user.clj to a function and invoke that function explicitly with -T#2020-08-2014:47Alex Miller (Clojure team)no, that doesn't seem right#2020-08-2014:48Alex Miller (Clojure team)if you're starting a repl, you can use the clojure.main -i to do things like that#2020-08-2014:48Alex Miller (Clojure team)-T is for invoking external tools (although I think -T is actually going to go away anyways)#2020-08-2016:06seancorfieldWas -T intended to be a specialized -R/`-C` combination?#2020-08-2016:07seancorfieldI was looking on the CLI reference page for where you can use :deps in an alias and that seems to just be -T -- but I thought it was valid in some other use case...?#2020-08-2016:07borkdudeI think -T did not include the deps on the top level. Which I can see being useful. Kind of like a system wide alias, but for projects#2020-08-2016:10seancorfieldIt feels "simpler" to just allow :deps / :paths in any alias with the expectation that it overrides the top-level values... but I guess there's always the question of what happens if you specify two aliases and the first has :deps / :paths to override and then the second alias has :extra-deps / :extra-paths to add to those? Will people expect that to work on will :deps / :paths always "win"?#2020-08-2016:10Alex Miller (Clojure team):deps/paths replacement was previously only available via -A, was pulled out as -T in past#2020-08-2016:10Alex Miller (Clojure team)we are reworking all of this stuff right now#2020-08-2016:10seancorfieldToo many unanswered questions 🙂 Needs more Hammock Time!#2020-08-2016:10Alex Miller (Clojure team)already did it#2020-08-2016:10borkdudepaths replacement = classpath overrides?#2020-08-2016:11Alex Miller (Clojure team)no#2020-08-2016:11Alex Miller (Clojure team):paths replaces the :paths in your project deps.edn#2020-08-2016:11borkduderight#2020-08-2016:11Alex Miller (Clojure team)which is admittedly a very unusual use case#2020-08-2016:11borkdudecan be useful for a project within a project?#2020-08-2016:11Alex Miller (Clojure team)we're going to rename these to :replace-deps and :replace-paths#2020-08-2016:12borkdudeyeah, that's clearer#2020-08-2016:12seancorfieldAnd if there are multiple of those, it'll be like :main-opts and only one gets picked?#2020-08-2016:12Alex Miller (Clojure team)it's not worth me going through all this by piece - it's all getting updated#2020-08-2016:13Alex Miller (Clojure team)so let me go work on that :)#2020-08-2016:13borkdudean analogy with :classpath-overrides {foo/bar nil} could be :remove-deps and :remove-paths#2020-08-2016:15seancorfieldFWIW, I'm liking the -X stuff (modulo always wanting :extra-deps to be respected by it) and I'm thinking it will simplify several pieces of tooling at work, since we can remove multiple namespaces containing just a -main function that exists solely to parse/route arguments to other functions that take a hash map 🙂#2020-08-2016:16Alex Miller (Clojure team)that's the idea (and usage stuff will be improved)#2020-08-2016:16seancorfield(our current problem is a couple of our servers are too old to be able to run the install script so I have to update them manually -- but that's not Clojure's problem 🙂 )#2020-08-2016:16Alex Miller (Clojure team)also working on ideas to make ad hoc -F variant a bit more useful for the case of one namespace with many entry point fns#2020-08-2016:17Alex Miller (Clojure team)have I talked about -F yet? I can't even remember now#2020-08-2016:17borkdudeno, go on ;)#2020-08-2016:17seancorfield-F is like -X only without needing an alias 🙂#2020-08-2016:18Alex Miller (Clojure team)yeah, invoke ad hoc function#2020-08-2016:18borkdudewhy even have X then?#2020-08-2016:18Alex Miller (Clojure team)lets you package default args and not repeat the fn at cli#2020-08-2016:18Alex Miller (Clojure team)both useful in different contexts#2020-08-2016:18seancorfieldInvoking via an alias is more flexible. Yeah, default args, simpler name, will be even more useful when it respects :extra-deps 🙂#2020-08-2016:18borkdudeyou can stuff default args in the function itself though?#2020-08-2016:19seancorfieldSure, but you could have different aliases with different default args. That's a use case that we could leverage at work.#2020-08-2016:19borkdude-A:foo -F foo/bar, respects extra deps of foo alias?#2020-08-2016:20seancorfieldCurrently, you'd need -R:foo -X:foo for a function alias -- repetitive! -- so if -X respects :extra-deps you could just do -X:foo#2020-08-2016:22Alex Miller (Clojure team)this is coming#2020-08-2016:23seancorfieldYeah, at this point I'm just sort of assuming it will do that 🙂#2020-08-2016:21seancorfieldBut -F is definitely useful for ad hoc command-line work.#2020-08-2016:22borkdudeinstead of creating different aliases with different defaults, one could also just write multiple functions in one namespace. I don't directly see the benefit of one over the other, -A:foo -F foo/foo, -A:bar -F foo/bar.#2020-08-2016:22Alex Miller (Clojure team)sure if you bake it into the code#2020-08-2016:23Alex Miller (Clojure team)putting it in deps.edn is default but configurable#2020-08-2016:23borkdudeI guess having it closer together is one benefit#2020-08-2017:04dominicm-F seems like -e? But with less (#2020-08-2017:07Alex Miller (Clojure team)-e is arbitrary expression, -F is "invoke a functional entry point that takes a map" so it's much more specific#2020-08-2101:59steveb8nQ: Is this a silly idea? I want a combination of git deps and “-LATEST” behviour for my CI/test builds. For prod builds I want static git shas but for automated/test builds, it would be great to automatically use the most recent git sha. Is this even possible?#2020-08-2102:35souenzzoTemporal-basead versions are always bad IMHO
If your CI fail and you run it again, it may take a new commit and success again. So success and fail says nothing.
Pass the commit ID to your CI or fetch and use a sed / template.
Then in case of success/fail, you will know which code succeed/failed#2020-08-2102:35souenzzoAnd you will be able to run again the same CI in a older commit/retry#2020-08-2102:42steveb8nthx. fetch and sed is an interesting idea. I could even use bb for this script.#2020-08-2102:43steveb8nhow would the script apply the latest sha? by writing it into the deps.edn? or is there some other trick?#2020-08-2102:45steveb8nfyi: the reason I want this is to avoid the endless commits I currently do for all my git deps during dev/test. but I really like using git deps in CI#2020-08-2102:55seancorfieldRunning CI and production against different versions seems like a recipe for hard-to-debug disaster...#2020-08-2102:56seancorfieldI can understand wanting your local dev-only tooling to always pick up the RELEASE (or perhaps occasionally the LATEST) release but I really don't want any variance in versions between dev, CI, production for anything I depend on.#2020-08-2102:57seancorfield(so I guess my answer to your actual Q would be "yes, it's a silly idea" IMO)#2020-08-2103:19steveb8nthat is useful perspective. I’ll apply a hammock to this a little more. thanks#2020-08-2103:24steveb8nI should have mentioned that these deps are all projects that I control but have split apart for re-use and/or separation of concerns. so I can be sure I know when changes are being introduced. I would never do this for an external dependency#2020-08-2103:34seancorfieldConsider using :local/root maybe, if you have them all checked out locally?#2020-08-2103:34seancorfieldYou can always use a :dev alias to use :override-deps to replace the regular deps with your local ones.#2020-08-2103:43steveb8nagreed. I am using :local/root for these deps for localhost. the annoying commits are required when I push a lib which I depend on in N other projects and then have to update shas in all of them#2020-08-2103:44steveb8nideally CI/test would use latest for all these deps. then when I cut a prod release, I would update the shas (once) for all the deps#2020-08-2103:45steveb8nbut I can see that I could introduce human error later i.e. forget 1 sha update in the prod build. that would be bad#2020-08-2103:45steveb8nthat would qualify as “silly”#2020-08-2103:46steveb8nthis maintenance problem seems to be pushing me towards a mono-repo but I like keeping these things separate#2020-08-2104:07seancorfieldOur 100K+ line codebase at work is in a monorepo, with 30+ subprojects, and we build about a dozen deployment artifacts from that...#2020-08-2104:14steveb8nit’s something I will think about. thanks#2020-08-2106:39borkdude@U0510KXTU I’m using git submodules to prevent endless commits updating git deps. Easy to update them all to latest using one git command. Works with all build tools.#2020-08-2107:25steveb8nthanks. I’ll look into that as well. I’ve never used them before but, if they add no complexity, I’ll give them a try#2020-08-2107:26borkdude@U0510KXTU I'm using them in babashka, I have 4 of them now.#2020-08-2107:27steveb8ncool. I’ll look at your repos when I dig deeper#2020-08-2107:27borkdudeNot everyone likes them, I'm not entirely sure why, they work well for me.#2020-08-2111:29practicalli-johnI would certainly keep specific library dependency versions the same for any CI pipeline that had any chance of pushing things into production.
Are you able to set up separate CI pipeline that uses a different alias to pull 'latest' versions of libraries?
I prefer using :local/root rather than mono-repo or sub-modules as they feel simpler for the way I work. Nothing wrong with those approaches though...#2020-08-2111:31borkdudelocal/root works, but not if the repo you're including is not part of the codebase proper, unless you're writing a script to set up the filesystem in CI the exact same way as on your machine#2020-08-2111:34practicalli-johnIs that the case when using jars built from other projects, I thought that worked. Maybe I am mis-remembering.#2020-08-2111:35borkdudeI mean, with local/root to some other project, you'll also have to git clone that other project in CI right#2020-08-2111:36practicalli-johnAh, yes.#2020-08-2123:22steveb8nI feel the same @U05254DQM in that I like them all separate. it has the added benefit of allowing me to outsource work on projects without needing to grant access to all#2020-08-2123:23steveb8nso for me it’s.. local deps using local/root, CI using git/sha and prod using the same git/sha#2020-08-2123:24steveb8nand the maintenance annoyance is updating the git shas for the CI builds anytime upstream projects change.#2020-08-2123:25steveb8nfrom this thread, I’ve realised that this is also a good thing because it means that my CI builds always match prod builds. thus it’s probably the lesser of all evils, despite the chattiness of commits#2020-08-2408:15rickmoynihan@U04V15CAJ: I’ve done the git submodule thing many times in the past. I’ve found it works quite well if you’re careful and have infrequently changing dependencies, but it can become pretty painful if you change branches a lot and have fast moving dependencies.
The detached heads in the submodules can also lead to people unwittingly making many kinds of mistakes.#2020-08-2408:20borkdudegit pull --recurse-submodules#2020-08-2408:21borkdudeI almost never have such mistakes, but maybe for contributors it's different?#2020-08-2108:37borkdudeWondering, is tools.build also going to have a test runner like https://github.com/cognitect-labs/test-runner or is this a separate concern?#2020-08-2112:29Alex Miller (Clojure team)Not building any new test runner but you can make tasks that do whatever#2020-08-2110:34otfromHi, I'm trying to reference the head of the manual-connector-anchors branch on dali here: https://github.com/stathissideris/dali/tree/manual-connector-anchors#2020-08-2110:34otfromand I'm struggling a bit with the deps.edn. Does anyone have a good example of how to reference a branch on github at a :git/url ?#2020-08-2112:32Alex Miller (Clojure team)Git deps always use shas so you don’t need to specify the branch#2020-08-2112:32Alex Miller (Clojure team)Sha is sufficient#2020-08-2115:03otfromthx! I've got it working now. It was a problem with the deps.edn in the repo I was referencing.#2020-08-2115:03Alex Miller (Clojure team)ah#2020-08-2110:41otfromI have it working locally in the mean time#2020-08-2111:20otfromfixed: https://github.com/stathissideris/dali/pull/20#2020-08-2422:55tolitiusneed to build a thin jar with one namespace aot compiled. the namespace has no dependencies:
{:paths ["src" "classes"]
:deps {}
:aliases {:jar {:extra-deps {seancorfield/depstar {:mvn/version "1.0.97"}}
:main-opts ["-m" "hf.depstar.jar" "yang.jar"]}}}
$ clj -e "(compile 'yang.java)"; clj -A:jar
it does work, but it also brings these along:
0 Mon Aug 24 18:45:40 EDT 2020 clojure/core/
0 Mon Aug 24 18:45:40 EDT 2020 clojure/core/specs/
932 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__40.class
1178 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__65$fn__71.class
798 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__46.class
929 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__27.class
936 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__101.class
929 Mon Aug 24 18:45:39 EDT 2020 clojure/core/specs/alpha$fn__25.class
...
I might be missing some args in depstar?#2020-08-2422:58seancorfield@tolitius Clojure core uses specs for language features (as macros), so those get dragged in because of that.#2020-08-2422:59tolitiusthank you. is there a way to exclude these?#2020-08-2422:59tolitiuswhen assembling a jar maybe#2020-08-2422:59seancorfieldYou'd have to manually remove them after your compile step right now.#2020-08-2423:00seancorfieldFeel free to open an issue on GH with those repro steps and I'll see about adding an option to exclude stuff via regex.#2020-08-2423:01tolitiusgreat, would that be a depstar repo?#2020-08-2423:01seancorfieldseancorfield/depstar#2020-08-2423:01tolitiusyep, will do. thank you Sean#2020-08-2423:02seancorfieldThere's already quite a few changes on develop so if I incorporate that exclusion this week, I'll cut a 1.1 release.#2020-08-2423:03tolitiusoh.. nice. that’s fast#2020-08-2423:07seancorfieldI'm off work Thu/Fri so I expect I'll do a round of OSS updates and releases before next Monday 🙂#2020-08-2423:08tolitiushttps://github.com/seancorfield/depstar/issues/37
wut it’s going to take a week?? )
yea, of course this is not urgent. I can remove these manually for now. thanks again#2020-08-2423:25tolitiusin case someone sees this before the next monday:
clj -e "(compile 'yang.java)"; clj -A:jar; zip -d yang.jar clojure/\*;
does it#2020-08-2423:53seancorfield@tolitius If you don't mind depending on develop, you can test that right now.#2020-08-2423:59seancorfieldNote that it excludes files not directories, so you'd still get the (empty) clojure/core/ and clojure/core/specs/ folders if you --exclude "clojure/.*"#2020-08-2500:24tolitius:main-opts ["-m" "hf.depstar.jar" "yang.jar" "--exclude" "clojure/core/specs/alpha.*"]}
756 Mon Aug 24 20:21:36 EDT 2020 yang/java$map__GT_edn$iter__130__134.class
3064 Mon Aug 24 20:21:36 EDT 2020 yang/java__init.class
1316 Mon Aug 24 20:21:36 EDT 2020 yang/java$fn__128.class
2003 Mon Aug 24 20:21:36 EDT 2020 yang/java$map__GT_edn$iter__130__134$fn__135$fn__136.class
0 Mon Aug 24 20:23:37 EDT 2020 clojure/
0 Mon Aug 24 20:23:37 EDT 2020 clojure/core/
0 Mon Aug 24 20:23:37 EDT 2020 clojure/core/specs/
0 Mon Aug 24 20:23:37 EDT 2020 tolitius/
1867 Mon Aug 24 20:21:36 EDT 2020 tolitius/Yang.class
yep, it works )#2020-08-2500:32seancorfieldCool, thanks for testing!#2020-08-2511:47Janne SauvalaCan I specify version ranges in deps.edn for dependencies?#2020-08-2512:08vlaaadno#2020-08-2517:04seancorfield@janne.sauvala Yes, you can, but you shouldn't. The string used for the version in :mvn/version is just passed through to the underlying library, e.g.,
(! 1176)-> clj -Sdeps '{:deps {hiccup {:mvn/version "(,1)"}}}'
DEPRECATED: Libs must be qualified, change hiccup => hiccup/hiccup
Downloading: hiccup/hiccup/1.0.0-SNAPSHOT/maven-metadata.xml from clojars
Downloading: hiccup/hiccup/1.0.0-SNAPSHOT/hiccup-1.0.0-20120218.070543-1.pom from clojars
Downloading: hiccup/hiccup/1.0.0-SNAPSHOT/hiccup-1.0.0-20120218.070543-1.jar from clojars
Clojure 1.10.1
user=> #2020-08-2517:06seancorfieldCurrent version is 1.0.5 but you can get the highest version below that
(! 1177)-> clj -Sdeps '{:deps {hiccup {:mvn/version "(,1.0.5)"}}}'
DEPRECATED: Libs must be qualified, change hiccup => hiccup/hiccup
Downloading: hiccup/hiccup/maven-metadata.xml from clojars
Downloading: hiccup/hiccup/1.0.4/hiccup-1.0.4.pom from clojars
Downloading: hiccup/hiccup/1.0.4/hiccup-1.0.4.jar from clojars
Clojure 1.10.1
user=>
But version ranges are "Bad(tm)" and should be avoided.#2020-08-2517:06seancorfield^ @vlaaad FYI#2020-08-2517:15Alex Miller (Clojure team)it's not actually passed through, it's resolved in canonicalization to the max in range#2020-08-2517:16Alex Miller (Clojure team)but a) version ranges have many known weirdnesses and b) as a non-specific version, they also break clj's caching (you will retain the same cached classpath even if newer versions become available)#2020-08-2517:16Janne SauvalaInteresting to see it works, thanks @seancorfield! I realized I haven’t seen anyone using version ranges so I was wondering were those intentionally not supported#2020-08-2517:16Alex Miller (Clojure team)they are officially not supported#2020-08-2517:16Alex Miller (Clojure team)but unofficially handled because they exist in the wild, particularly with transitive deps#2020-08-2517:18seancorfield@alexmiller Good to know re: canonicalization -- I passed in a bad range and the exception came from Aether so I assumed it was just being passed through. My bad:
Unbounded version range (1,
org.eclipse.aether.resolution.VersionRangeResolutionException: Failed to resolve version range for hiccup:hiccup:jar:(1,: Unbounded version range (1,
at org.apache.maven.repository.internal.DefaultVersionRangeResolver.resolveVersionRange(DefaultVersionRangeResolver.java:137)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveVersionRange(DefaultRepositorySystem.java:247)
#2020-08-2517:19seancorfield> I haven’t seen anyone using version ranges
Because they're bad and folks avoid them 🙂#2020-08-2517:24Janne SauvalaYes. Too bad I have seen them used quite a lot in npm and python world. It is good that bad practice hasn’t polluted clojure community 🙂#2020-08-2517:19Alex Miller (Clojure team)well the resolution happens via aether#2020-08-2517:20Alex Miller (Clojure team)I think if you had more of stack, you'd probably see canonicalize in there somewhere#2020-08-2621:26jhacksTrying out antq (https://github.com/liquidz/antq) and depot (https://github.com/Olical/depot) with these one-liners I get many hundreds of lines of DEBUG statements:
- clojure -Sdeps '{:deps {antq/antq {:mvn/version "RELEASE"}}}' -m antq.core
- clojure -Sdeps '{:deps {olical/depot {:mvn/version "2.0.1"}}}' -m depot.outdated.main
Is this normal or is there something up with my system?#2020-08-2621:30seancorfield@jhacks What do you mean by "hundreds of lines of DEBUG statements"?#2020-08-2621:31seancorfieldI just tried antq.core on a project and got this:
> clojure -Sdeps '{:deps {antq/antq {:mvn/version "RELEASE"}}}' -m antq.core
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
| :file | :name | :version | :latest-version |
|----------+-----------------------------------------------------------------+------------+-----------------|
| deps.edn | com.cognitect/test-runner | | Failed to fetch |
| | com.h2database/h2 | 1.4.199 | 1.4.200 |
| | com.microsoft.sqlserver/mssql-jdbc | 8.2.1.jre8 | 8.4.0.jre14 |
| | com.zaxxer/HikariCP | 3.4.2 | 3.4.5 |
| | io.zonky.test.postgres/embedded-postgres-binaries-darwin-amd64 | 12.2.0 | 12.3.0 |
...
which seems perfectly reasonable.#2020-08-2621:32seancorfielddepot produced this
> clojure -Sdeps '{:deps {olical/depot {:mvn/version "2.0.1"}}}' -m depot.outdated.main
Downloading: olical/depot/2.0.1/depot-2.0.1.pom from clojars
Downloading: olical/depot/2.0.1/depot-2.0.1.jar from clojars
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
Checking for old versions in: deps.edn
org.clojure/java.data {:mvn/version "1.0.78"} -> {:mvn/version "1.0.86"}#2020-08-2621:36jhacks@seancorfield For this command clojure -Sdeps '{:deps {olical/depot {:mvn/version "2.0.1"}}}' -m depot.outdated.main
I get many lines like this before ending with the usual output:
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.headers - << Age: 7291592
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.headers - << X-Served-By: cache-bwi5137-BWI, cache-wdc5525-WDC
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.headers - << X-Cache: HIT, HIT
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.headers - << X-Cache-Hits: 1, 1
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.headers - << X-Timer: S1598477735.173106,VS0,VE1
17:35:35.193 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.client.DefaultHttpClient - Connection can be kept alive indefinitely
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<html>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<head><title>404 Not Found</title></head>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<body bgcolor="white">[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<center><h1>404 Not Found</h1></center>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<hr><center>nginx</center>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "</body>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "</html>[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "<!-- a padding to disable MSIE and Chrome friendly error page -->[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.wire - << "[\n]"
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection [id: 49][route: {s}->] can be kept alive indefinitely
17:35:35.194 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection released: [id: 49][route: {s}->][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 100]
17:35:35.195 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager is shutting down
17:35:35.196 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51708<->199.232.64.209:443 closed
17:35:35.196 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51708<->199.232.64.209:443 closed
17:35:35.197 [clojure-agent-send-off-pool-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down
17:35:35.198 [clojure-agent-send-off-pool-2] DEBUG org.eclipse.aether.internal.impl.TrackingFileManager - Writing tracking file /Users/joe/.m2/repository/selmer/selmer/resolver-status.properties
Checking for old versions in: deps.edn
cljs-ajax {:mvn/version "0.8.0"} -> {:mvn/version "0.8.1"}
com.google.javascript/closure-compiler-unshaded {:mvn/version "v20200504"} -> {:mvn/version "v20200719"}
day8.re-frame/http-fx {:mvn/version "0.1.6"} -> {:mvn/version "0.2.1"}
metosin/jsonista {:mvn/version "0.2.6"} -> {:mvn/version "0.2.7"}
org.clojure/core.async {:mvn/version "1.2.603"} -> {:mvn/version "1.3.610"}
org.postgresql/postgresql {:mvn/version "42.2.14"} -> {:mvn/version "42.2.16"}
re-frame {:mvn/version "1.0.0"} -> {:mvn/version "1.1.1"}
thheller/shadow-cljs {:mvn/version "2.10.17"} -> {:mvn/version "2.11.0"}
#2020-08-2621:36jhacksSorry about the formatting#2020-08-2621:40jhacksFor clojure -Sdeps '{:deps {antq/antq {:mvn/version "RELEASE"}}}' -m antq.core
I get lines like this:
17:38:47.822 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down
17:38:47.822 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager is shutting down
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51766<->151.101.193.128:443 closed
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51766<->151.101.193.128:443 closed
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager is shutting down
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51765<->199.232.64.209:443 closed
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51765<->199.232.64.209:443 closed
17:38:47.823 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down
17:38:47.825 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager is shutting down
17:38:47.825 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51769<->199.232.64.209:443 closed
17:38:47.825 [Finalizer] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:51769<->199.232.64.209:443 closed
17:38:47.825 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down
| :file | :name | :version | :latest-version |
|----------+-------------------------------------------------+-----------+-----------------|
| deps.edn | cider/piggieback | 0.5.0 | 0.5.1 |
| | cljs-ajax/cljs-ajax | 0.8.0 | 0.8.1 |
| | com.cognitect/test-runner | | Failed to fetch |
| | com.google.javascript/closure-compiler-unshaded | v20200504 | v20200719 |
| | day8.re-frame/http-fx | 0.1.6 | 0.2.1 |
| | metosin/jsonista | 0.2.6 | 0.2.7 |
| | org.clojure/core.async | 1.2.603 | 1.3.610 |
| | org.postgresql/postgresql | 42.2.14 | 42.2.16 |
| | re-frame/re-frame | 1.0.0 | 1.1.1 |
| | thheller/shadow-cljs | 2.10.17 | 2.11.0 |
#2020-08-2621:47jhacksI’m using clojure from homebrew and just updated it to see if that was causing it, but no change. There isn’t a debug setting somewhere with deps.edn?#2020-08-2621:51jhacksI tried it on another project that uses deps.edn (I only have two) and I don’t get the DEBUG output. It must be something specific to the project. Thanks for trying out the commands on your side @seancorfield!#2020-08-2621:54seancorfieldSomething in your project is enabling logging (at debug level) and those DEBUG messages are coming from the machinery that fetches metadata about artifact versions.#2020-08-2621:55seancorfieldYou may be able to resolve that by adding org.slf4j/slf4j-nop {:mvn/version "1.7.25"} to your command-line deps when running antq and/or depot @jhacks#2020-08-2621:59jhacks@seancorfield Wow! That got rid of the DEBUG statements on both of the commands. Thanks so much, that’s good to know about.#2020-08-2622:00seancorfieldI use it with clj-new to suppress logging in project templates as it is generating them 🙂#2020-08-2622:05jhacksThanks for this video, btw: https://www.youtube.com/watch?v=CWjUccpFvrg I found it very helpful. I’ve been using lein and just getting started with deps.edn. clj-new is on my list to check out. 🙂#2020-08-2622:14practicalli-johnhttps://practicalli.github.io/clojure/clojure-tools/using-clojure-tools.html covers using Clojure CLI tools and several community tools.
Sean and I both have examples of ~/.clojure/deps.edn configurations you may find of use
https://github.com/practicalli/clojure-deps-edn#2020-08-2622:20jhacksThanks @U05254DQM. I found this tip extremely helpful: https://practicalli.github.io/clojure/clojure-tools/deps-edn-aliases.html#only-one-main-namespace#2020-08-2622:21jhacksA lot of good info there 🙂#2020-08-2622:23seancorfieldThe example of mine that John refers to above is at https://github.com/seancorfield/dot-clojure#2020-08-2622:25practicalli-johnI learnt most things from Sean (and Alex too)#2020-08-2812:10rickmoynihanAre there any plans to support data-readers in deps.edn :args data?#2020-08-2812:14rickmoynihanThe use case is defining overrides for an integrant system, allowing #ig/ref.
Obviously it’s also easy to put that system in a separate file, and read that, so not a big deal, and I can imagine that it’d add complexity to tools.deps. Just curious.#2020-08-2812:43Alex Miller (Clojure team)I don’t think there’s any need for that#2020-08-2812:44Alex Miller (Clojure team)Or do you mean for data in aliases in deps.edn?#2020-08-2812:48dominicmfor context #ig/ref creates a record type. I suspect Rick is trying to pass in the integrant system to run via deps.edn aliases.#2020-09-0514:39seancorfield> I'm not sure I'm in agreement with you on the scope
I'm pretty sure we're just not in agreement on that 🙂#2020-09-0514:39Alex Miller (Clojure team)there is a middle ground - no warning for -A on first stable release#2020-09-0514:39Alex Miller (Clojure team)then warning#2020-09-0514:39Alex Miller (Clojure team)then (much later?) removal#2020-09-0514:41seancorfieldHaving been on the receiving end of complaints about incorrect instructions (or version-specific instructions that weren't absolutely clear on how to determine the version and how to "solve" the problem, on all platforms), I'm perhaps more sensitive to this whole issue than you... 🙂#2020-09-0514:41seancorfieldYes, bullet 3 is that middle ground.#2020-09-0514:41Alex Miller (Clojure team)I'm on the receiving end of those complaints too#2020-09-0507:45seancorfieldThis provides a migration path for the most common use cases that is minimally disruptive (most uses of -R today won't care about paths or JVM options I suspect). It improves naming for the three desired use cases: REPL, eXecution, and Main opts. It breaks the least amount of code out there.#2020-09-0507:50seancorfield(question for Alex when he gets back: why are there two identical copies of exec.jar in the CLI distribution -- one at the top level, one in libexec?)#2020-09-0514:31Alex Miller (Clojure team)I think there aren't two in the distribution, the install script copies the root one to libexec which is the structure brew expects, although I think maybe I didn't sync the clojure script up to that and it's using the root one. I'll check on that#2020-09-0516:03seancorfieldFYI, confirmed that script uses the root one:
-classpath "$cp:$install_dir/exec.jar"
#2020-09-0514:25Alex Miller (Clojure team)I think this is pretty good. I'll have to go back through my spreadsheets to evaluate more. I think the first bullet recommendation could even be for tool authors to do nothing, wait until most people are using a new version, then change their recommendation.#2020-09-0514:28Alex Miller (Clojure team)Keep in mind that most Clojure users are on Mac and install via brew and that brew aggressively updates you to latest version (our stable) even if you're upgrading something else. So within some period of time (couple months maybe), many people are using newest version.#2020-09-0515:27practicalli-johnThat only covers 55.36% of the survey responders, so this doesnt seem to be the strongest of arguments.
Of course I am happy to help encourage upgrading to newer versions in any way I can, once I've understood the changes myself.#2020-09-0515:31seancorfieldI wonder what percentage of the 35%-ish that are on Linux are using brew vs manually installing via the .sh file?#2020-09-0515:36practicalli-johnThere are legion of package managers for Linux that brew competes against, maybe a question for the next survey?
I find it easier to use the Linux script install.#2020-09-0515:51seancorfieldI use brew on Ubuntu (on WSL) because it's easy to keep clojure up-to-date that way but that's recent for me -- I used to use the manual install script, but I tended to only upgrade when I remembered to do it, or I tried to do something that my older, installed version didn't support 😐#2020-09-0521:18dominicmI've used Linux for many years, the brew stuff is extremely rare.#2020-09-0521:24seancorfieldYup, I'm just glad it exists for clojure. Getting stuff into the more standard installer repos on Linux seems very laborious? What relatively painless options exist for Linux package managers that the Clojure core team could leverage @U09LZR36F?#2020-09-0521:25dominicmHa! 😂#2020-09-0521:26dominicmIf only you knew the rabbit hole you'd just asked about#2020-09-0521:27dominicmEvery distribution has its own rules and timelines.
Consider Ubuntu LTS. They don't take changes. Canonical pay people to backport security patches. Every now and then you get a system upgrade (like mac) and that's when you get changes. New features, bug fixes, etc.#2020-09-0521:28dominicmThere are, of course cutting edge distributions like arch and void. They'll update whenever the package maintainer feels like it. Usually regularly for important changes.#2020-09-0521:29dominicmSome distributions care about reproducibility. They have other constraints I don't understand.#2020-09-0521:30dominicmI am a void clojure package contributor, and my life is pretty easy. The installer script was "ported" to the void build system in ~10 lines and it's just worked since then.
Maybe I'll need to make changes with the new system.#2020-09-0600:02practicalli-john@seancorfield To package Clojure for Ubuntu, then I'd suggest a personal package archive (PPA), new release of Clojue can be pushed to this as soon as they are ready. There is a specific process for creating package to learn. Packages can be built for each release of Ubuntu (e.g. current long term support and intermediate releases) and published on the PPA. The relevant packages can be submitted for each new Ubuntu release (6 month cycle).
There are also other options such as snaps and flatpack, but a proper *.deb package is much nicer in my opinion as it ensures there are no conflicts with other software installed via the package manager. There is a Leiningen package for Ubuntu, so some of the experience from that can be used.
For Arch, I seem to remember that @U11EL3P9U packages up clj-kondo , so I assume his work can be of help for packing to the Arch User Repository (AUR).#2020-09-0605:43dharriganThat's right, I do maintain a few Clojure packages on Arch, like clj-kondo, babashka, clojure-lsp. A pleasure to do so 🙂 However, the main Clojure package is part of the Community Repo that is under a different maintainership. It is reguarly updated (last update was 2020-08-26, which brought in Clojure 1.10.1.645).#2020-09-0514:28Alex Miller (Clojure team)Also keep in mind that we are in work on a 1.10.2 Clojure cycle and we tie brew versions to Clojure versions, so that might be a natural point to either coax an upgrade or drop some kind of compatibility#2020-09-0515:56seancorfieldI have updated the depstar README to show usage with the prerelease -X option: https://github.com/seancorfield/depstar/blob/develop/README.md#clojure--x-usage#2020-09-0515:56seancorfieldIt works quite nicely that way...#2020-09-0515:59seancorfieldI expect I'll update clj-new next, to support -X usage...#2020-09-0516:34borkdude> Keep in mind that most Clojure users are on Mac and install via brew
Have you considered that those Mac users will probably also use the linux scripts on CI? Do you get any stats from that usage? It may be less of an issue since the install script is versioned, but since people tend to develop CI scripts locally (I often do), it might become confusing.#2020-09-0516:36borkdudeThere might also be less Windows users using this because for some people it's harder to get going than lein. I got this report yesterday and it's not the first time I hear these kinds of issues.
> Clojure until 1.6 used to work on Windows from https://chocolatey.org/packages/clojure too, without problems. Clojure Tools broke everything and it seems that OSX ( Homebew ) is the only "first class citizen" now.
https://github.com/babashka/babashka.process/issues/15#issuecomment-687035007
Don't shoot the messenger, I don't really have a stake in Windows support because I know how to work around it :)#2020-09-0520:32Alex Miller (Clojure team)Oh I definitely think that’s a factor#2020-09-0516:49seancorfieldYeah, you have to be pretty persistent to use Clojure CLI/`deps.edn` on Windows, and willing to work with very "alpha" tooling. Which a lot of Windows users are not willing to do: they often have different expectations of the user experience than devs on macOS/Linux.#2020-09-0516:49borkdudeOne personal data point: I found it difficult to install clojure on Windows CI because the powershell stuff you had to enable and whatnot. That's why I also went with lein to run tests there:
https://github.com/borkdude/clj-kondo/blob/master/appveyor.yml#2020-09-0516:51borkdudeI do use clojure on the other two major OS CIs#2020-09-0516:53seancorfieldI'll be using Windows for development a lot more going forward (Microsoft had a great deal on Surface Laptop 3 so I just ordered an i7 with 16GB RAM and 256GB SSD). But I'll be leaning on WSL2 very heavily, and using (Linux) brew for installation.#2020-09-0517:03QuestI have an odd case -- at least from my pure Clojure background (didn't start in Java.)
I have a Java dependency that doesn't publish to maven. I can run ./gradlew build on it to get a library .jar.
What's the best way to include this into deps.edn dependencies?#2020-09-0517:06QuestCan I publish just a .jar to maven under my own repo?
I could try updating the library to properly publish to Maven & PR'ing it, but I'm just unfamiliar with the gradle ecosystem. The library author says he'll start publishing to maven soon...#2020-09-0517:08seancorfieldYou can use a local .jar file directly in deps.edn via a :local/root dependency. We use that for a couple of things that have to build locally because they aren't published anywhere.#2020-09-0517:09Questoh man, I immediately get what you mean, but how did I utterly fail to find this in a doc somewhere? I spent 20 minutes searching around#2020-09-0517:09Questanyway thanks Sean, you're a big help in many channels 🙂#2020-09-0517:09seancorfieldIt's mentioned here https://clojure.org/guides/deps_and_cli#_using_local_libraries#2020-09-0517:10seancorfieldAnd here in more detail https://clojure.org/reference/deps_and_cli#_dependencies#2020-09-0517:10seancorfieldIt doesn't "scale" so well if you're part of a team or want to use it in a CI pipeline, but it's fine for local development and testing.#2020-09-0517:11QuestAhh, I'll try just reading through the whole deps guide. I kept searching for "clojure deps include .jar library" and similar terms, never thought of "local library"#2020-09-0517:13QuestAs far as "scaling" this approach, I was just going to git submodule the dependency library.
I could see how many layers of builds would take a long time / cause odd failures. Thankfully still at "fun personal project" size for now#2020-09-0517:43Questdeps.edn didn't seem to pick up on the .jar (silent failure?) as I couldn't load any expected classes.
I realized there's no pom.xml, probably why.
At a minimum, I believe I need to tweak the library to use https://docs.gradle.org/current/userguide/maven_plugin.html#2020-09-0517:54seancorfield@U22M06EKZ You shouldn't need a pom.xml for this usage -- but if that Java library has any dependencies, you need some way to tell clojure about them (if the JAR contains a pom.xml the t.d.a machinery would take care of that, else you could just add those deps directly to your deps.edn file).#2020-09-0517:57Quest@seancorfield Hmm, I could uptake them into my deps.edn. Do you know of a way to determine if the jar was loaded or not?
I've just been trying to run an :import on a class that I expect to exist, but wondering it there's a better way to poke around#2020-09-0518:00seancorfieldUse the -Spath option on the clojure CLI to see what the classpath contains.#2020-09-0518:01seancorfieldOr even inside your REPL: (System/getProperty "java.class.path")#2020-09-0518:02seancorfielduser=> (run! println (clojure.string/split (System/getProperty "java.class.path") #":"))
/home/seanc/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/home/seanc/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
/home/seanc/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
src
nil
user=>
(example from a bare REPL)#2020-09-0518:05Quest^ I figured there was something from the REPL, but -Spath is a good tip. Appreciate the knowledge-share, you had a hand in me starting adoption of deps.edn a year ago too and I doubt I would've tried it otherwise :thumbsup:#2020-09-0518:07seancorfieldCool. We've never regretted switching to CLI/`deps.edn` in 2018 at work.#2020-09-0518:08seancorfield(and we're using the bleeding edge prerelease version right now -- as of yesterday)#2020-09-0517:26practicalli-johnFor the up-coming release of Clojure CLI tools:
I have been using clojure -R:cognitect-rebl -A:nrebl to run REBL with nREPL. The -R:congnitect-rebl load in dependencies from an alias that also has a :main-opts (which it not the main namespace to run) and -A:nrebl which has a :main-opts I do want to run.
If I create a separate alias for cognitect-repl without the :main-opts and use clojure -M:rebl-deps:nrebl it works.
Or I could just add all the rebl dependencies into the :nrebl alias.
Does that seem the right approach?#2020-09-0517:32seancorfield@jr0cket I suspect if you used -A:cognitect-rebl:nrebl it would run the :main-opts from :nrebl since they overwrite each other -- last one wins.#2020-09-0517:34seancorfieldYou could also use -A:cognitect-rebl:nrebl -M:nrebl which would work on both today's stable version and the current prerelease version without the -A warning about using -M to run the :main-opts (since the "winning" :main-opts here is using -M).#2020-09-0517:39seancorfieldI think the guidance from Alex has fairly consistently been to separate out :main-opts from other stuff if you need to mix'n'match main opts from several aliases. But the expansion of both -M and -X to respect :extra-deps and other stuff in the latest prerelease feels like the pressure is in the opposite direction -- toward bundling more things into a single alias for convenience @jr0cket Not sure how closely you've been following the discussions generally in this channel?#2020-09-0518:56practicalli-johnI've read the whole conversation in here and reading through the docs Alex wrote. Still digesting what it all means.
I'm currently testing all the aliases I have to understand the changes#2020-09-0519:07seancorfieldI'm working on updating clj-new to work with -X, much as I've updated depstar. I'll probably add :new-x and :depstar-x aliases to my .clojure/deps.edn file so folks can use those tools via -X as examples. It'll be a while before that can become the default 🙂#2020-09-0520:28Alex Miller (Clojure team)@jr0cket would love to hear your feedback too#2020-09-0601:12practicalli-john@alexmiller so far very positive changes. I like that -M feels similar in concept to the :main in Leiningen, in part easing the move from Leiningen to CLI tools.
My current thoughts are around edge cases where I have multiple aliases containing :main-opts and how to use them together with certainty rather than optomism. I have tried to elaborate some options in this gist
https://gist.github.com/jr0cket/6fae7576a87d5d32eda4be2253f41692#2020-09-0601:29seancorfield@jr0cket Multiple aliases containing :main-opts is an "issue" today (stable) as well as tomorrow (prerelease), and the answer is the same as it has always been -- per the documentation -- "If multiple maps with these keys are activated, only the last one will be used" -- see https://clojure.org/reference/deps_and_cli#_aliases which explains how each different type of option is "merged" when multiple appear. The same wording is present in the prerelease reference docs, but spread out over multiple sections, rather than gathered together in one section.#2020-09-0601:30seancorfieldIs that not "certainty" enough?#2020-09-0608:02practicalli-johnCertainly in my mind would be as part of the syntax rather than convention. If the convention changed in a release without requiring a change to the syntax of use.
Convention also means more to remember, especially as some parts of slides are merged and others only take the last. I assume (hope) nothing takes just the first.
Not saying it's an issue, but it adds to the learning curve.#2020-09-0608:11practicalli-johnAs it's a current issue and there isn't anything in the up coming changes to resolve it, then this aspect seems a moot discuss point. I'll make a decision and move on.#2020-09-0815:52Alex Miller (Clojure team)fwiw, I don't see this as an issue - the documented behavior is what it does and what I expect it to do in the future#2020-09-0517:41seancorfield(but of course, at this point, there's still some potential for changes in the prerelease that the -A/`-M` usage recommendations may change)#2020-09-0517:42hugodI happened to watch Spec’able before reading Alex’s post. Seems pretty ironic.#2020-09-0517:42seancorfieldSpec'able?#2020-09-0517:43hugodsorry, sepc’ulation#2020-09-0517:44seancorfieldAh yes, https://www.youtube.com/watch?v=oyLBGkS5ICk#2020-09-0517:44seancorfield(transcript: https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/Spec_ulation.md )#2020-09-0517:51seancorfieldTL;DR: don't break stuff, only add stuff and fix bugs 🙂#2020-09-0519:13andy.fingerhutat least once you decide you are out of alpha#2020-09-0522:45hugodThe deps/cli reference page only talks about the library being in alpha. The guide doesn’t mention it at all, and besides “But, that is not to say just leave your thing 0.0.967. At a certain point, you are going to have users, and whether you change it to 1.0 or not, they are going to be depending on your stuff.”#2020-09-0519:35seancorfieldFeedback on the "WARNING: When invoking clojure.main, use -M" message (for @alexmiller) -- this usage produces that warning: clojure -m my.entry.point -- is that intentional? Do you really want folks to write clojure -M -m my.entry.point instead?#2020-09-0520:19Alex Miller (Clojure team)Yes#2020-09-0520:34borkdudeIs this to discourage that usage in favor of -X? It seems to be another maybe unnecessary breaking thing that's been there for a long time#2020-09-0520:35Alex Miller (Clojure team)It’s not to discourage it as much as that we are trying to open up the repl arg space for other things#2020-09-0520:35Alex Miller (Clojure team)And that means making clojure.main usage explicit#2020-09-0520:41seancorfieldInteresting choice. A lot of tutorials out there talk about running simple Clojure projects with just clojure -m my.entry.point#2020-09-0520:41seancorfieldI think beginners will (quite rightly) ask "Why do we have to specify two m options?"...#2020-09-0520:43Alex Miller (Clojure team)“Lot” = ? 2 or 3?#2020-09-0520:44Alex Miller (Clojure team)And if these tutorials transition into talking about -X...#2020-09-0520:46Alex Miller (Clojure team)We want clj to do more. In some cases that means restructuring these options. It’s either that, or clj2 and clojure2 which do not seem better to me#2020-09-0520:46seancorfieldSure, if all the books and tutorials that exist today are updated to use -X instead of -m, it's not going to be a problem 🙂#2020-09-0520:52dpsuttonThanks for balancing all these things in a spreadsheet so lots of concerns are managed. On a Saturday no less. Looking forward to the previews#2020-09-0521:05seancorfield(I just want to be clear that I think all the new functionality is great -- all of my concerns are around how existing (stable) functionality is deprecated and/or removed over time)#2020-09-0521:06seancorfieldAnd, ultimately, I'm not going to be the one complaining about the changes since I'm already on the latest prerelease version on every machine and I'm aggressively adopting the new functionality!#2020-09-0521:08seancorfield(I am somewhat concerned about users of my CLI/`deps.edn`-based projects, of course 🙂 )#2020-09-0521:08seancorfieldComing in the next version of clj-new: https://github.com/seancorfield/clj-new#clojure--x-usage#2020-09-0521:09borkdudeThere was a time when the clojure CLI wasn't driven by tools.deps.alpha right? Wasn't it non-alpha back then? Isn't only tools.deps alpha but the CLI / shell script now also got alpha?#2020-09-0600:13Alex Miller (Clojure team)No, clj and tools.deps were developed together#2020-09-0521:09seancorfield(`depstar` added support for -X in its most recent 1.1.104 release, last week)#2020-09-0521:12borkdudeI could be misremembering things, but I think there was a clojure package already years ago, before tools.deps.alpha? Not sure what it did back then, but -m was maybe already a part of it back then#2020-09-0521:14seancorfieldI know there was an unofficial clojure script floating around which behaved differently than the official one, once it appeared.#2020-09-0521:14borkdudemaybe it just did java -jar clojure.jar #2020-09-0521:17seancorfieldYeah, I think that was pretty much it. My recollection of the official CLI was that it appeared in tandem with t.d.a during the 1.9 cycle once Spec was split out of the core clojure.jar since you could no longer just run a REPL with the JAR file alone so it was supporting machinery for that... but I'm having a hard time finding any of the original posts about it (maybe I should scan back through Inside Clojure since I bet Alex blogged about it).#2020-09-0521:18borkdudeah right, thanks#2020-09-0521:19seancorfieldHmm, the pre-1.10 Inside Clojure posts are fairly sparse...#2020-09-0521:21seancorfieldThis is the first mention of t.d.a on the Clojure mailing list: https://groups.google.com/g/clojure/c/FpMqtNu0TCE/m/wc2n1tPHBwAJ#2020-09-0521:22seancorfield(July 2017)#2020-09-0523:52seancorfieldBeen playing with the new features, and it's nice that you can have the following
:new {:extra-deps {seancorfield/clj-new {:mvn/version "RELEASE"}}
:exec-fn clj-new/create
:exec-args {:template lib} ; default
:main-opts ["-m" "clj-new.create"]}
and it works with -A:new today (or -A:new -M:new) and, with the prerelease it works with -M:new (or -A:new -M:new without giving a warning) and it also works with -X:new so you can have a single alias provide for both styles of execution.#2020-09-0715:52practicalli-johnForgetting migration for a moment, would you see :exec-fn and :exec-args replacing :main-opts in the long term?
Using these new keys seems a nicer approach for the design of the alias code itself.
Would this be an example of edn over strings that Alex mentions?#2020-09-0716:47seancorfieldI think we'll see a good uptake on the :exec-fn approach because you can avoid parsing strings to get your arguments and you can easily provide defaults -- configurable defaults, since each alias have have its own :exec-args map.#2020-09-0523:57seancorfieldThat's probably what I'll do in my dot-clojure repo by way of education for folks and as a way to encourage folks to transition (or at least see what the options available are).#2020-09-0601:18hugodI like the new features. :ns-default and :ns-aliases look very useful. The -P option is great, and could be even nicer if it (optionally) output a map of the resolved dependencies, which could be used to make reproducible builds.#2020-09-0601:40Alex Miller (Clojure team)Every build outputs a lib map in the cache dir with this info (-Sdescribe tells you where)#2020-09-0601:49hugoddidn’t realise that 🙂#2020-09-0601:18hugodTo be honest, I’ve always found the different merging rules confusing and never really understood their purpose. Why not just have -A to specify the aliases to merge, and -e, -x (equivalent of the :exec-fn key) or -m (equivalent of the :main-opts key) to specify or override what should be run. Maybe with a -r for a future repl specifier.
If the config is not specific, define a precedence for :exec-fn and :main-opts.
This would separate (decomplect, dare I say?) merging of aliases from specifying what to invoke, which I, for one, find simpler to understand.#2020-09-0601:41seancorfield@hugod In the stable version, -M didn't pick up any items from the alias(es) except :main-opts and in the first version of -X on the prerelease track didn't pick up any items from the alias(es) except the fn/args spec as I understand it. In the latest prerelease, both of those pick up everything in the alias(es) specified and then -M runs the :main-opts while -X runs the function invocation -- and -A is slated for being similar (picking up everything in the alias(es)), but ultimately just running the REPL, not executing main/function stuff. That's why I proposed keeping But, either way, it's a much more streamlined piece of logic: all three options pick up all options and then each one of -A (or maybe -R?), -M, and -X selects what execution model is used.#2020-09-0601:41seancorfieldIt's nice and streamlined. And I think it removes a lot of confusion about which parts of an alias get picked up by which options.#2020-09-0601:50hugodHaving -M, -X, and -A pick up everything certainly seems like an improvement. I would still find it simpler to just have the one option for specifying aliases, and then options for specifying what to run. Using lower case for the latter would allow for the cli to maintain backward compatibility iiuc.#2020-09-0601:52seancorfieldI suspect there would be confusion, since -e, -m, and -r are existing main options today (for clojure.main).#2020-09-0601:55seancorfieldYou'd also need to specify both an option to collect aliases and then an additional option to specify the action to take. With the prerelease direction, you'd only need one option: either -X for aliases and execute function, or -M for aliases and main function, or -A for aliases and the REPL (at some point, given -A currently still executes main opts).#2020-09-0602:03hugodI was suggesting -e and -m kept their current meaning, and implied invoking clojure main.#2020-09-0602:04seancorfieldBut :main-opts can contain a combination of those so I don't think that's going to be intuitive.#2020-09-0602:05seancorfield(`:main-opts` can also contain -i and -r options)#2020-09-0604:59mike_ananev@alexmiller Is it possible to add functionality of generation distributionManagement section to 'clj -X:deps mvn-pom' command?
I need to distribute my artefacts written in Clojure to corporate Nexus. To do this I need section in pom.xml:
<distributionManagement>
<repository>
<id>releases</id>
<url></url>
</repository>
<snapshotRepository>
<id>snapshots</id>
<url></url>
</snapshotRepository>
</distributionManagement>
Building and deploying artefacts is performed in corporate Jenkins, where I can't manually add this section.#2020-09-0605:02seancorfield@mike1452 Since clj -X:deps mvn-pom only updates dependencies and a few other bits, so what I tend to do is generate an initial POM, augment it manually, check it in to version control, then just run that command as needed to update the dependencies section (and it leaves all the rest intact).#2020-09-0605:02seancorfieldSo you could manually add that section, and clj will not overwrite it.#2020-09-0605:34vlaaadWeekly reminder that reading/updating pom.xml is a workaround for the problem, not a solution :)#2020-09-0607:24borkdude@alexmiller I know tools.deps isn't going to expand ~, but I'd like to know the reason for this. Reason I'm asking is that I'm considering doing this in .clj-kondo/config.edn and would like to know if I'm missing some problems#2020-09-0611:46Alex Miller (Clojure team)Because then everyone reading deps,edn has to interpret the file#2020-09-0611:47Alex Miller (Clojure team)It’s no longer just edn/read, it’s that plus run magic code#2020-09-0612:32borkdudeWhy does it have to be interpreted by anything else than tools.deps which already does interpretation of this entire model?#2020-09-0612:37borkdudeI mean, merge, assoc, etc still works for other tools in this case?#2020-09-0612:45Alex Miller (Clojure team)If you have alias data, anyone may be reading it via the basis (without tools.deps, which has a lot of deps)#2020-09-0612:47Alex Miller (Clojure team)There are a bunch of tools already reading and manipulating deps.edn. The less they have to know, the better#2020-09-0613:30borkdudeI don’t think this will be a problem for the linter config file. Thank you sir.#2020-09-0609:31mike_ananev@seancorfield Is it possible to exclude clj source files from uberjar using depstar? I need only aot compiled files in uberjar, when distributing it. Also, how do you recommend to compile java source files, when using depstar?#2020-09-0609:42mike_ananevIs it possible to add to depstar something like this https://gist.github.com/just-sultanov/e2f61734e22e2ed3981216ed9b880bc0 to compile java sources?#2020-09-0609:51borkdude@mike1452 depstar has this for removing itself:
https://github.com/seancorfield/depstar/blob/a542560766cfd2b51d415908885625aa64165cb4/src/hf/depstar/uberjar.clj#L292
I think that can be easily extended to remove other things#2020-09-0609:52borkdudeAh no, scratch that, that's only for removing something from the classpath.#2020-09-0609:52borkdudeI meant this: https://github.com/seancorfield/depstar/blob/develop/src/hf/depstar/uberjar.clj#L150#2020-09-0609:54borkdudeAh, and that's exposed via -X (not clojure's -X, but depstar's -X)#2020-09-0609:54borkdudeso probably something like -X "\.clj$" would work#2020-09-0612:53dominicmYou don't need to remove the clj files anyway. Or is there an additional piece of information?#2020-09-0618:33seancorfieldThe regex-based file exclusion was added in the most recent depstar release @mike1452 -- did the suggestions above work for you?#2020-09-0618:34seancorfield(you could also specify a classpath that only had the class path, without the src path I suppose, if every piece of source has been compiled?)#2020-09-0618:35seancorfieldAs for Java, that's outside the scope of a Clojure tool: compile it yourself and then add that to the classpath (depends on how/where you compile your Java source to .class files).#2020-09-0622:02dominicmHow can I remove org.clojure/clojure from the classpath? I seem to recall a trick to it.#2020-09-0622:03dominicm(I want to use org.clojure/clojure$sources instead)#2020-09-0622:39Alex Miller (Clojure team)Just specify a different coord for it? Or use :override-deps?#2020-09-0622:41Alex Miller (Clojure team)Keep in mind that everything calls clojure.main and you need a class for them jvm to invoke#2020-09-0622:45dominicmI tried using override-deps, maybe I used it wrong.#2020-09-0622:46dominicmError building classpath. Coordinate type not loaded for library org.clojure/clojure in coordinate {}#2020-09-0622:57dominicm(I also tried nil)#2020-09-0623:25seancorfieldYou need an alias to use :override-deps#2020-09-0623:26dominicmI used one. But I'm trying to override to remove it and {} didn't work at that removal coordinate.#2020-09-0700:46seancorfieldCan't you override it to point to local JAR? The sources JAR file?#2020-09-0701:25seancorfieldSince we're getting a lot of depstar questions here, I created a dedicated channel for support/questions/etc. Also for clj-new.#2020-09-0706:59borkdude@dominicm you can also use classpath overrides with nil. I’ve used this for creating babashka uberjars (see its README and look for uberjar)#2020-09-0709:53kirill.salykingoodmorning,
please advice how I can turn off the OmitStackTraceInFastThrow?
clojure -A:app -J "XX:-OmitStackTraceInFastThrow"
doesnt seem to work#2020-09-0709:54dharriganclojure -J-XX:-OmitStackTraceInFastThrow#2020-09-0709:54kirill.salykinthanks! it is all about the order#2020-09-0709:55dharrigannp#2020-09-0710:28practicalli-johnIn clojure pre-release (1.10.1.672) it seems that the -X flag does not download dependencies specified in the alias :extra-deps section. Is this the correct behavior?
From my interpretation of "-X now supports multiple aliases, aliases with other argmap options like :extra-deps, and ad hoc functions" I would have thought deps would be downloaded#2020-09-0710:29borkdudeis that in combination with -A?#2020-09-0710:29practicalli-johnNo, just by itself. I understood -A was being depreciated, so am trying to understand the scope of using -M (which seems fairly clear) and -X which I am still confused about (from a users ponit of view)#2020-09-0713:30Alex Miller (Clojure team)@jr0cket that is intended to work, so could be a bug#2020-09-0715:24practicalli-johnYes, it does work on a clean install of Clojure CLI with the ~/.clojure/.cpcache removed. This is making much more sence now. Thanks.#2020-09-0713:45rickmoynihanTo temporarily swap out a git dep for a local dev version, am I normally better adding an alias with :override-deps in it to a :local/root, or adding :classpath-overrides to my ~/.clojure/deps.edn ?#2020-09-0713:48dominicm@rickmoynihan I usually use an override-deps (actually extra-deps because it's the same otherwise) in ~/.clojure/deps.edn because:
• Nobody else can use it - has my homedir in
• Most people on my teams aren't contributing to those other libraries anyway. It's less reusable than one would think
#2020-09-0713:50rickmoynihanyeah for both options I’m talking about adding it to ~/.clojure/deps.edn… trying to understand which way is better.#2020-09-0713:51rickmoynihanOne (`:override-deps`) effects resolve-deps and the other make-classpath right?#2020-09-0713:52rickmoynihanPractically though; aren’t they both equivalent?#2020-09-0713:52dominicmI think for your case, they're essentially equivalent. I'd go with extra-deps just because it's more famiilar.#2020-09-0713:53rickmoynihanexcept :classpath-overrides assumes the lib only introduces a single classpath root, is that right?#2020-09-0713:54rickmoynihanso e.g. it would potentially miss including resources etc#2020-09-0713:55dominicmhttps://github.com/clojure/tools.deps.alpha/blob/f94815dd55bdf5eb30ac8fa075c39e757cbbcca5/src/main/clojure/clojure/tools/deps/alpha.clj#L321 yeah 🙂#2020-09-0713:55dominicmExactly, yeah#2020-09-0714:55borkdudeRe command line args, I have one tool that only has one CLI argument, --opts in which you pass a Clojure map. For that tool I think it made sense and it also simplified command line parsing, possibly also easier to keep it non-breaking. I'm not suggesting that other tools do this, but works for this one particular thing.
clojure -A:carve --opts '{:paths ["src" "test"]}'#2020-09-0723:05sogaiui have a tool that does something similar -- though it doesn't use anything like --opts before the string.
it's nice to not have to decide on command line arguments when you don't know what option and arg names are likely to be used more often.#2020-09-0715:03dominicmI suppose the clj way of doing that is would be clj -X:carve :paths '["src" "test"]'?#2020-09-0715:04borkdudeI guess so, but if we're going to use quotes, why not quote the whole thing anyway#2020-09-0715:05dominicmI mention it only because I think this is what clj is trying to make the standard for cli tools. While I personally think --path src --path test would be the best, standards are better than better.#2020-09-0715:08borkdudeor :paths src,test since comma is whitespace in Clojure, but not in bash?#2020-09-0715:08borkdude$ bb -e "*command-line-args*" -- :paths src,test
(":paths" "src,test")
#2020-09-0715:09dominicmYou're thinking in Clojure 🙂 I'm thinking of bash. Admittedly, my answer was without context. Generally I'd much prefer an api where paths was the & args at the end so I could just do:
clj -X:carve dev-*
#2020-09-0715:09dominicm(which expands to clj -X:carve dev-src dev-resources dev-etc)#2020-09-0715:12borkdudeIf bash would expand that comma separated, then it would work as passing one value.#2020-09-0715:12borkdudeAnyway, it doesn't.#2020-09-0715:14borkdudeYeah, if keywords would separate other values then :jars *.jar :paths src* would work#2020-09-0715:14borkdudeunless your dir starts with a colon :)#2020-09-0715:15borkdudebut I guess using a comma as explicit separator works:
$ bb -e "*command-line-args*" -- :foo t* , :nums 1 2 3
(":foo" "tags" "test" "test-resources" "," ":nums" "1" "2" "3")
#2020-09-0715:21borkdudeThis doesn't work in powershell (no expansion, comma is gone)
PS Microsoft.PowerShell.Core\FileSystem::\\wsl$\Ubuntu-20.04\tmp> cd C:/Temp
PS C:\Temp> bb -e "*command-line-args*" -- :foo t* , :nums 1 2 3
(":foo" "t*" ":nums" "1" "2" "3")#2020-09-0715:22borkdudeFinally, cmd.exe (no expansion, preserves comma)
C:\Temp>bb -e "*command-line-args*" -- :foo t* , :nums 1 2 3
(":foo" "t*" "," ":nums" "1" "2" "3")#2020-09-0715:36dominicmNo idea how cmd.exe handles wildcards. Ultimately there's some platform specific tweaks to match the ergonomics of a platform you're on.#2020-09-0715:37dominicmhttps://superuser.com/questions/1308329/move-folders-using-wildcard#1308364 welp#2020-09-0715:37dominicmhttps://docs.microsoft.com/en-us/cpp/c-language/expanding-wildcard-arguments?view=vs-2019 apparently you have to use a custom routine or something... gosh#2020-09-0715:38borkdudeI guess there's a reason WSL2 is getting popular#2020-09-0715:40dominicmBut also - this is what people on cmd.exe expect. That's fine and while I don't understand it, if that's what makes sense for their OS then fine.#2020-09-0715:42borkdudeI listened to a podcast by someone who is working on the new Terminal app. She said that they couldn't change cmd.exe's behavior because that would be breaking for lots of people. But everyone basically agrees that it sucks.#2020-09-0715:43borkdudeMaking the cmd.exe window smaller will make it faster, because there is less rendering, which is blocking#2020-09-0715:44borkdudeok sorry, got a little bit off topic#2020-09-0717:49lreadGetting back to clojure/clj command line arg compatibility, this is not helpful right now, but I am curious. If we could go back in time would these tools have been better initially named something like clj-alpha1 and clojure-alpha1? This would have drawn attention to their alpha-ness and also allowed a path for introducing breaking changes.#2020-09-0717:51lreadThen again maybe not… this would have implied clojure itself is alpha.#2020-09-0717:51lreadTricky little problem.#2020-09-0717:55borkdudeThe help could have mentioned the word alpha at least somewhere. From and end user's perspective, it's hard to see this is alpha, you'd have to know what .jar it's calling underneath which has alpha in the name. This entire page doesn't mention alpha neither: https://clojure.org/guides/deps_and_cli#2020-09-0717:59seancorfieldI just got a bug report against the new version of clj-new and it's obviously broken but it leads me to a question about t.d.a and migrations: clj-new uses t.d.a to resolve coordinates for templates and it was using t.d.a 0.8.677 and the clojure.tools.deps.alpha.reader namespace with default-deps and read-deps. I updated it to 0.9.782 and switched the code over to pull in the runtime basis... but of course that doesn't exist in the new CLI! Doh! I could revert to 0.8.677 and continue using the old logic, but I was wondering if there was a way in 0.9.782 to get at that same logic so that utilities could run with the basis, if it exists, else fall back to the equivalent of (read-deps (default-deps))? Mostly a question for @alexmiller I guess...#2020-09-0718:15Alex Miller (Clojure team)Those functions now exist in the tools.deps.alpha namespace#2020-09-0718:16Alex Miller (Clojure team)Or their equivalents - there were some name changes etc#2020-09-0718:18seancorfieldHmm, it looks like some of the logic for finding deps.edn has moved into the CLI script, and is no longer in t.d.a?#2020-09-0718:18Alex Miller (Clojure team)No, should all be there#2020-09-0718:19Alex Miller (Clojure team)Shape is a little different#2020-09-0718:19seancorfieldOK, I'll keep digging...#2020-09-0718:19Alex Miller (Clojure team)Sorry, not at a computer to be precise#2020-09-0718:21Alex Miller (Clojure team)It’s just root-deps and user-deps and ./deps.edn#2020-09-0718:21Alex Miller (Clojure team)Read and merge#2020-09-0718:32seancorfieldfind-edn-maps was what I needed so I think I'm good now...#2020-09-0718:52seancorfieldFor anyone else working on t.d.a-based tooling who wants to leverage 0.9.782 but continue to support older CLI versions, here's what I ended up with for the basis:
(delay (if-let [basis-file (System/getProperty "clojure.basis")]
(-> basis-file
(io/file)
(slurp)
(edn/read-string))
(let [{:keys [root-edn user-edn project-edn]} (deps/find-edn-maps)]
(deps/merge-edns (filterv some? [root-edn user-edn project-edn]))))))
That's to replace the equivalent of (read-deps (default-deps)) from t.d.a 0.8.677 -- it's not identical, but for the purposes of equivalence with 0.8.677 it seems close enough.#2020-09-0802:11cflemingIs there any way with deps to manage source artifacts for non-Clojure dependencies?#2020-09-0802:12cflemingOr rather, how could I specify that I would like them to be downloaded, and how would they be distinguishable from binary jars in the output?#2020-09-0802:25cflemingThis seems to be the answer:
{:deps {junit/junit {:mvn/version "4.13"}
junit/junit$sources {:mvn/version "4.13"}}}#2020-09-0802:30cflemingThey do end up on the classpath, which is unfortunate. I guess they could go in an alias.#2020-09-0803:15cflemingWhen using :local/root, the doc says:
> If the jar has been packaged with a pom.xml file, the pom will be read and used to find transitive deps
Is there a way to prevent this from happening? I would just like it to use the jar I give it and not look for transitive deps.#2020-09-0803:22cflemingLooking at the code, it seems like not. My use case here is trying to use deps to manage the dependencies for Cursive. I need to depend on the IntelliJ framework, so I have a script which downloads and unpacks that, and I’m trying to create a deps.edn which refers to the jars within it. However a bunch of those are third party jars which include poms.#2020-09-0803:22Alex Miller (Clojure team)you could use :local/root to a directory which has a deps.edn that includes the jar on the :paths#2020-09-0803:23cflemingOh, I can just put jar paths directly in :paths? That’s even better.#2020-09-0803:23Alex Miller (Clojure team)kind of a hack, but it works#2020-09-0803:23cflemingI guess that makes total sense, but never occurred to me.#2020-09-0803:23Alex Miller (Clojure team):paths are just things that get added to the classpath#2020-09-0803:23Alex Miller (Clojure team)on the source stuff, tools.deps is about making classpaths so that's what you're going to get#2020-09-0803:24cflemingRight. It’s actually much better for this use case since I don’t need to create a bunch of bogus dep symbols.#2020-09-0803:24cflemingOk. Using $sources works fine, so I’ll support that in Cursive, document how to do it and suggest putting them in an alias.#2020-09-0810:24practicalli-john@alexmiller I'm confused about using -M with a project rather than a specific alias.
In this example I use the hello-world example from https://clojure.org/guides/deps_and_cli#_writing_a_program
A project deps.edn file was created containing the dependency for clojure.java-time and the source code from that page copied into src/hello.clj
clojure -m hello runs the project and returns the time from running the -main function.
However this gives a warning:
WARNING: When invoking clojure.main, use -M
clojure -M runs a REPL
clojure -M -m hello runs the project and returns the time. But then I ask myself what is the purpose of -M
Creating an alias to run the project seems an interesting idea, as I could also set default arguments. I can only get this to work with :main-opts and not :exec-fn
This version of the alias works when calling clojure -M
:aliases
{:run-project {:main-opts ["-m" "hello"]}}
This version of the alias simply runs the REPL
:aliases
{:run-project {:exec-fn hello]}}
Am I missing something from :exec-fn ?
Edit: yes I am.
The -X option is for running :exec-fn and not the -M function, oops.
So clojure -X:run-project should work... except the hello-world project has an unqualified function and cannot be resolved.
Moving the source code to src/practicalli/hello.clj and calling clojure -X:run-project gives an execution error, (ArityException) as the -main function does not take any arguments (defn -main [] ,,,) . Changing the -main function to (defn -main [& args] ,,,) fixes that and calling clojure -X:run-project works.
From a user point of view, that seems quite a lot to extrapolate from the original warning message when running a project with clojure -m hello#2020-09-0814:49Alex Miller (Clojure team)The guide hasn’t been updated for the new stuff yet - I’ll do that with the stable release of it#2020-09-0814:57practicalli-johnGreat. That time frame wont impact me at all. Thanks If you want help with the docs, let me know (although you are the expert here)#2020-09-0811:57practicalli-johnThinking about this further, it does feel that the -X flag would be the most important flag in the long term, especially with :exec-arg allowing default arguments that can be over-written on the command line.
-M seems to be more of a (very useful) convenience function for the specific case of running -main-opts configurations. This does remove the need for command line args and bash quoting, so that is valuable. -M is also useful for those aliases that dont work with -X , eg rebel-readline I could only get working with -M .
As more maintainers adopt the new version of Clojure CLI, there is scope to configure all projects and tools to work with -X and eventually make -M redundant.
Using the -M flag seems to help a little when making the change from Leiningen to CLI tools, as setting the main namespace is a familiar concept.
I am sure I am over-simplifying this and probably missing out on a lot of understanding as of yet.#2020-09-0812:42hugodWould -M not continue to be useful for utilities that you expect to be invokable from the shell? How would you allow passing glob’ed files for example?#2020-09-0812:48practicalli-johnYes, it seems the -M option is valuable when the :main-opts data is more intricate. Interesting to see what can be done with :exec-args verses :main-opts .
I do like :exec-args as a means of using edn over strings.
I dont believe I've have any use cases for using glob'ed files, so cant comment on that without specific examples. I assume something like *.jpg would just be a command line argument after the -X:alias-name and passed to the function.#2020-09-0813:04hugodiuuc, -X expects arguments to be key-path edn-value pairs, so clojure -X:my-alias :x "1 2 3" ends up passing {:x 1} to the exec-fn.#2020-09-0813:18hugodIn general, if you want something that can be be composed with output from tools that don’t produce edn, I think -M is the easier (only?) option.#2020-09-0813:18practicalli-johnAh, so I assume clojure -X:alias :x '["1 2 3"]' is required for :exec-fn , which fits in the concept stucture over strings. Although more specific options seem preferable.
clojure -X:alias :x 1 :y 2 z: 3 .
This is less ambiguous and ties into the general approach I use for Clojure arguments.#2020-09-0813:18practicalli-johnStill lots to learn, so interesting to hear these thoughts. Thanks.#2020-09-0813:19hugodI’m still trying to grasp the implication of the changes too.#2020-09-0814:18Alex Miller (Clojure team)@U05254DQM I think mostly what you’ve gleaned is correct. -M is for pass through to clojure.main. (And originally that’s the only thing clj did.) Calling with no exec opt is for repl. Right now that’s done with clojure.main but you should consider that an implementation detail. -X is for the new function execution style invocation.#2020-09-0814:20Alex Miller (Clojure team)We expect most tools are better off with -X, but totally possible to support both of course#2020-09-0814:23Alex Miller (Clojure team)For your own app, not sure yet how this will be taken up. Using -X means you are at least somewhat bound to using clj as your launcher (as the exec stub code is not available anywhere else (yet)). Rich seems to think that’s fine, I’m a little less sure about it. :)#2020-09-0814:25Alex Miller (Clojure team)My biggest issue with the -X arg passing is that passing a string path as the v requires wrapping in multiple quotes which kind of ruins - expansion and other glob and auto completion stuff. I’d say that’s not totally finalized#2020-09-0814:43practicalli-johnThanks for confirming Alex. I am far more confident in my understanding of the changes. Apart from updating practicalli/clojure-deps-edn with either -M or -X variants instead of -A, there seems no impact on my development projects.
My notes from a project developer are shared on this gist: https://gist.github.com/jr0cket/ae7dd745eb45870109ace59fe835ce80#2020-09-0814:47practicalli-johnUpdating the Practicalli books with the new approach is pretty easy and can run with the 'new' and 'classic' options for a while.
I've planned screencasts of using Clojure CLI tools for Clojurists Together over the next 2 months. I would rather not re-do them as high quality screencasts are time intensive. I will need to decide when to do these videos, so any thoughts on a release date for the new features would help my own planning (I appreciate any comments on dates are no guarantee).
If a release is not planned until December, I'll opt for some lower quality screencasts and recreate them running up to a release.#2020-09-0814:48Alex Miller (Clojure team)I’m hoping to get to a new stable release in the next week or two #2020-09-0812:43jeremysHi, I have a couple questions about deps API. There is there a complement function to slurp-deps that also merges the 'user wide' deps.edn and the 'system wide' one ? Also the docs says that resolve-deps and make-classpath are supported. Then the question is: is make-classpath-map to be considered private api ? (and so subject to change?)#2020-09-0814:11Alex Miller (Clojure team)You can use find-edn-maps to read and merge-edns function to merge edn maps once they’ve been read#2020-09-0814:12Alex Miller (Clojure team)make-classpath-map is public but some details of the make-classpath details are going to change, probably this week#2020-09-0814:13Alex Miller (Clojure team)(Regarding path ordering)#2020-09-0814:16Alex Miller (Clojure team)The whole make-classpath area is in a bit of transition#2020-09-0814:25jeremysOk Alex, thanks a lot! I have been foolishly experimenting with the idea of making https://github.com/JeremS/mbt based on tools deps and I ran into those questions today. If I may I have one more concerning tools.gitlibs. Right now it uses jgit v4. Do you plan to use v5?#2020-09-0814:31Alex Miller (Clojure team)Yeah, I’ve spent some time on it but not enough to get it over the line. It also plays into the jsch ssh stuff and look at their newer impl with Apache Mina. #2020-09-0814:31Alex Miller (Clojure team)Just FYI, I have built a tools.build tool that will hopefully be making its way toward release soon#2020-09-0814:43jeremysOh man... my rationale was that tools.deps begs a tools.build... A proper tools.build is very good!
The jgit question was because the clj-git project uses the v5 making it, I believe, incompatible with tools deps
(breakage between jgit 4 and 5)
Thank you again for the info.#2020-09-0817:24seancorfield@U0FR867U1 If it helps you, take a look at https://github.com/seancorfield/clj-new/blob/develop/src/clj_new/helpers.clj#L60-L71 -- This uses the latest tools.deps.alpha (0.9.782) but still works with both CLI stable and CLI prerelease.#2020-09-0817:25seancorfieldRe: find/read/merge#2020-09-0819:38jeremysThank you @seancorfield the last 2 lines are almost exactly what I need!#2020-09-0815:04borkdudeMaybe aliases should also become fully qualified?#2020-09-0815:04borkdudeThen we can have a global (worldwide) alias registry =)#2020-09-0815:05borkdudeI wasn't trolling (at least, not consciously)#2020-09-0815:05mpenetI didn't mean it that way 🙂#2020-09-0815:07borkdudee.g. want to use seancorfield's aliases -A:org.seancorfield/rebl#2020-09-0815:14borkdudeI guess the problem with that is that aliases can change over time and effectively become a dep. Importing aliases from libs might be an idea, but maybe a bit too much.#2020-09-0816:02Alex Miller (Clojure team)aliases can be fully qualified now#2020-09-0816:02Alex Miller (Clojure team)people typically don't because more typing, but that's fully supported#2020-09-0908:39rickmoynihanI’ve been doing this for a while with local overrides, e.g. :local/libname will be defined in ~/.clojure/deps.edn for when I have a local checkout of it. I think it works quite well, also to distinguish user wide aliases from project ones.
cc @borkdude#2020-09-0908:41borkdudeyes, doing something something similar#2020-09-0816:04Alex Miller (Clojure team)one thing we've done a bit of thinking about is - how could I specify a tool to use and have it automatically pulled from github without needing to declare an alias at all#2020-09-0816:04borkdudeyes, that was a bit was I was thinking with this#2020-09-0816:05borkdudebit like lein new does maybe#2020-09-0816:05borkdudeAliases can be fully qualified now, but will it ever be deprecated that they aren't - as with deps#2020-09-0816:05dominicmNeed some aliases procurers#2020-09-0816:06Alex Miller (Clojure team)probably not going to fully tackle that for a bit, but I think it would be pretty useful. you need some kind of naming convention that takes you to a gh repo, that can be a deps.edn with deps etc#2020-09-0816:06Alex Miller (Clojure team)I think just github support would take you pretty far, but yes all that needs to be thought about#2020-09-0816:06Alex Miller (Clojure team)there is also some open space I've been working in around discovery with -X#2020-09-0816:06borkdudethis can probably be solved in a tool which you can put in your aliases#2020-09-0816:07Alex Miller (Clojure team)like could -X have support for listing the functions available to call#2020-09-0816:07borkdudeI wasn't trolling! geez, what is it with people today ;)#2020-09-0816:07Alex Miller (Clojure team)how can you support automatic documentation of a -X compatible function#2020-09-0816:08borkdudemaybe metadata?#2020-09-0816:08borkdude^:x-callable#2020-09-0816:08Alex Miller (Clojure team)all tbd, but with combination of mapping from alias to tool repo, and support for :ns-default (to find the default tool namespace), and ability to discover the -X functions, and help for each option....#2020-09-0816:09Alex Miller (Clojure team)that's an ecosystem#2020-09-0816:09borkdudeyes, the mapping from alias to tool repo was what I was getting at earlier#2020-09-0816:09Alex Miller (Clojure team)and then if only had a build system that could make use of those functions...#2020-09-0816:10borkdudebuild system?#2020-09-0816:10borkdudetroll#2020-09-0816:10Alex Miller (Clojure team):)#2020-09-0816:11dominicmMetadata wouldn't work by itself, you'd have to require every namespace.#2020-09-0816:11Alex Miller (Clojure team)anyhow, I'm on "vacation", so back to vacating...#2020-09-0816:11borkdudeenjoy!#2020-09-0816:11dominicmYes. Begone!#2020-09-0816:14borkdudeMaybe aliases aren't the right fit for name -> github repo and some other concept/option should be invented for that.#2020-09-0817:27seancorfieldSort of related to this line of thinking: clj-new supports templates on GitHub via naming conventions, so maybe have a look at the readme for the usage and see if it is close to what you are thinking @borkdude?#2020-09-0817:31borkdude@seancorfield You mean:
clj -A:new -M:new ...
?#2020-09-0817:33borkdudeor do you have some short form where you only mention the org + repo name, with automatically resolving the newest sha?#2020-09-0817:43seancorfieldCurrently, you need the full URL but that's really only to detect what type of template you're using. If you know it's GH, you could take just the org/repo-name and build the rest. And my dot-clojure file has an example of using tools.gitlib to figure out the HEAD of master SHA.#2020-09-0817:44seancorfield@borkdude See https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L174-L177#2020-09-0900:24cflemingJust double checking - there is no way to get the deps of a subproject specified via :local/root including deps from an alias, correct? i.e. I cannot specify an alias together with :local/root in some way?#2020-09-0900:26seancorfieldNope.#2020-09-0900:26seancorfieldDependencies are opaque in that respect.#2020-09-0900:28seancorfieldAlthough... I guess you could read the basis (in the prerelease CLI) and figure out the actual :local/root path and see if it has deps.edn and then use t.d.a to read that and then inspect its aliases... 👀#2020-09-0900:29seancorfieldBut a :local/root could point to a JAR, or a directory with pom.xml. Or project.clj 🙂#2020-09-0900:32cflemingRight.#2020-09-0900:33cflemingThis is just specifying deps in a deps.edn file, so I can’t do anything funky or programmatic.#2020-09-0901:13hugodWhat is the rationale for deprecating unqualified dependency symbols in :deps? (couldn’t see one in the release notes)#2020-09-0901:17seancorfield@hugod It's mentioned in this post https://insideclojure.org/2020/07/28/clj-exec/ near the end.#2020-09-0901:18seancorfield(IOW it was deprecated in an earlier prerelease)#2020-09-0901:20hugodThanks - I missed that release 🙂#2020-09-0901:25seancorfieldI spent the first week of August, scurrying around, fixing all our unqualified lib names at work, and all my open source projects! 🙂#2020-09-0902:31cflemingIs there any way to substitute values into a deps file that I’m not aware of, e.g. from env vars or properties? In my case I would like to be able to substitute a global version in several deps files. I’m aware of default-deps, but I’d need to be able to substitute it in a :local/root path.#2020-09-0902:33seancorfieldNope. No substitutions.#2020-09-0902:39seancorfield(I'm just a bundle of "nope" today, aren't I?)#2020-09-0902:40cflemingHaha#2020-09-0902:40cflemingNo worries, that was what I expected, just makes my life a little more tedious 🙂#2020-09-0913:19hugodFwiw, having played with the pre-release, I still think the following CLI switches would be simpler, as they separate the selection of aliases from what to execute, and would simplify the doc. A given set of aliases can be used for main, exec or repl, so it makes it easier to compose a command line from other tools if the passing of options is distinct from the selection of what to invoke.
clj [clj-opt*] [-A:aliases] [init-opt*]
clojure [clj-opt*] [-A:aliases] -x [a/fn] [kpath v]*
clojure [clj-opt*] [-A:aliases] [init-opt*] [main-opt] [arg*]
clojure [clj-opt*] -P [other exec opts]
or it could be
clojure [clj-opt*] [-A:aliases] -M [init-opt*] [main-opt] [arg*]
If that is really required, though I confess I don’t see the rationale for having to specify the -M given that you’re passing -m or have a :main-opts key.
The (at least my perceived) lack of usage of the current -T, `-R`, `-C`, and -O options supports the above.#2020-09-0913:21Alex Miller (Clojure team)Sorry, not doing that#2020-09-0913:29hugodThat’s clear 🙂#2020-09-0916:15seancorfieldAlso, I think there's more usage of -R out there than you suspect @hugod#2020-09-0916:22hugodI’m sure I don’t have a good perspective on that, but don’t think it really changes my view that just using -A would be conceptually simpler.#2020-09-0916:22hugodGiven the cli changes seem fixed, I’ll just go and update my nascent build tool invoker project to cope with multiple versions of the cli.#2020-09-0916:28seancorfieldI still think that a) given the expansion of -M to include more than just :main-opts b) the overlap between -A and -M (both run :main-opts now) c) the current uses of -R (doesn't run :main-opts but does "most" of what -A does -- based on resolve args being the most common; it would make more sense for -R to be kept and expanded as the "REPL" usage (and quietly undocumenting -A). So we'd have -R (REPL), -M (main), -X (exec) and they all three respect all arg types in deps.edn and there would be no overlap on :main-opts or :exec-fn.#2020-09-0917:20hugodI do find it unfortunate that the deps.edn isn’t authoritative. By this I mean that whatever you specify in deps.edn you still have to know whether to invoke it with -M or -X.#2020-09-0917:29seancorfieldSee https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L15-L18 -- you can support both 🙂#2020-09-0917:30seancorfieldThat works with -A today and either -M or -X tomorrow!#2020-09-0917:36hugodI don’t think that is quite what I meant. As a deps.edn writer I would like to provide an abstract, say test, alias for doing something, and have the user be able to invoke that independent of how I chose to implement that alias. Writing it both ways seems like duplication.#2020-09-0918:55Alex Miller (Clojure team)well the arg structures are different, not sure how that could work#2020-09-0919:45seancorfieldExactly, as a tool writer you need to specifically support -M style invocation and/or -X style invocation as two separate functions in your code: -main taking & args which are strings and some-exec-fn taking a single hash map as its argument.#2020-09-1013:04hugodmain obviously takes strings and an exec-fn wants an edn value, but that doesn’t need to dictate the format of arguments passed on the cli. The cli could print edn to strings before passing it as a main argument, or read literal strings to edn strings before passing to an exec-fn. As a strawman, with intentionally hideous names, consider this:
i) default to passing args as literal string values, or add a --string switch to explicitly specify this
ii) add an --edn switch, that causes the next argument to be read as EDN and passed as a value to an exec-fn or as a string to a main.
iii) add a --patch switch that takes an edn path vector and an edn value and patches the default value. if the value is prefixed with --string treat it as a string value rather than EDN. Allow main to have default arguments specified, in the same way as exec-fn.
iv) add a --rest switch that collects the remaining arguments into an EDN vector. --rest-strings could build an EDN vector from string arguments rather than edn values. This could be used to pass glob values, for example.
v) allow an exec-fn to take multiple arguments.
As described this is unwieldy, but if there were volition, I’m sure it could be improved upon. The main idea would be to divorce the specification of the format of the arguments passed on the command line from how they are passed to the implementing function.#2020-09-0919:08jeremysHi, I have been working on a https://github.com/JeremS/mbt based on tools.deps for some time. It has a mix of functionality inspired from several other projects and it's at a stage where I use it to version, build and deploy my projects. Maybe someone other than me might find it useful! Cheers#2020-09-0919:37seancorfield@alexmiller It looks like clojure -X:deps mvn-pom now adds the following dependencies to all pom.xml files:
<dependency>
<groupId>org.clojure</groupId>
<artifactId>tools.deps.alpha</artifactId>
<version>0.9.782</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.7.25</version>
</dependency>#2020-09-0919:39seancorfieldThat's going to cause a lot of projects to accidentally end up depending on tools.deps.alpha (and the slf4j-nop dependency is likely to be unwelcome in many cases I suspect).#2020-09-0919:40seancorfieldThese deps are in a freshly-generated pom.xml file -- and they are also added to any existing pom.xml, potentially overwriting the dependency for any projects that use a different version of t.d.a#2020-09-0919:43Alex Miller (Clojure team)ah, interesting#2020-09-0919:43Alex Miller (Clojure team)thx, will fix#2020-09-0919:59seancorfieldHow will you fix it? For example, my clj-new project does depend on t.d.a (and has a dependency on slf4j-nop because of that) -- but I know that's an outlier -- so you'd need to also not remove any existing dependency on those?#2020-09-0920:00Alex Miller (Clojure team)I'll fix it by calculating the basis, not using the injected basis#2020-09-0920:01Alex Miller (Clojure team)basically what it did before#2020-09-0920:02seancorfieldWon't that break anyone currently using clojure -A:some:aliases -Spom ?#2020-09-0920:11Alex Miller (Clojure team)it doesn't have to - the injected basis includes those argmaps#2020-09-0920:11Alex Miller (Clojure team)the calculated basis can use the project deps.edn + the injected basis argmaps#2020-09-0920:25seancorfieldRe: basis -- so -A and -X are distinguishable via the basis? Or am I misunderstanding?#2020-09-0920:26Alex Miller (Clojure team)oh, I see what you're asking. well let me think about it more#2020-09-0920:12Alex Miller (Clojure team)another option would be to take those additional aliases as args to the -X pom function#2020-09-0920:12Alex Miller (Clojure team)that would be an all new thing though#2020-09-0920:13Alex Miller (Clojure team)very similar thing exists in the datomic ion-dev tool which I was just helping with a bit :)#2020-09-0920:14Alex Miller (Clojure team)I am planning to reinstate the -Spom as well so looking at that right now too#2020-09-0920:26seancorfieldIf it's just going to be identical to -X:deps mvn-pom I wouldn't have thought it was worth the effort?#2020-09-0920:27Alex Miller (Clojure team)you were the one here arguing it was breaking people last week... :)#2020-09-0920:31seancorfieldHahaha... yeah, but you didn't seem very sympathetic to that argument, and I'm not sure having duplicate ways to do the exact same thing is better 🙂#2020-09-0920:14Alex Miller (Clojure team)I had some thoughts on the -Stree thing you ran into though - I think there are several better ways to detect changing deps tree rather than scraping the -Stree stdout. not sure what your requirements on that stuff are but might be useful to talk about#2020-09-0920:18Alex Miller (Clojure team)from a clj pov, we're writing edn files (.libs in particular) to the cache dir (can see location with -Sdescribe), so you could be slurping and diffing Clojure maps rather than comparing strings#2020-09-0920:27seancorfieldTrue, but a) how can you easily get the correct .libs file from a given invocation without scraping the output and b) that gives no sense of depth of transitivity, which is specifically what I rely on in -X:deps tree#2020-09-0920:28seancorfield"Because reasons"... we go one level down in the full tree (so we consider top-level deps and the first level of transitive deps, but we don't care about lower levels).#2020-09-0920:30seancorfieldWe've found that we can safely ignore the 2nd-order transitive deps and if we have a problematic 1st-order transitive dep, we just promote it to the top-level (and then its transitive deps are promoted to 1st-order transitive deps and we start to track them separately).#2020-09-0920:30Alex Miller (Clojure team)well the transitivity stuff is all broken in the -Stree output#2020-09-0920:31Alex Miller (Clojure team)but maybe you're just not encountering the cases where it's bad (mostly with exclusions)#2020-09-0920:31seancorfieldWe don't use exclusions, for the most part.#2020-09-0920:32seancorfieldThis is our only exclusion:
com.walmartlabs/lacinia {:mvn/version "0.35.0"
:exclusions [clojure-future-spec/clojure-future-spec]}#2020-09-0920:32Alex Miller (Clojure team)the main thing is, the code never actually builds the real expansion tree as a tree. -Stree is kind of faking from the libs map#2020-09-0920:33seancorfieldWe decided that managing conflicts via :exclusions was a bit of a nightmare so we control conflicts by promoting transitive deps to the top level.#2020-09-0920:33seancorfieldRight, but I'd have to write that "faking" code myself, whereas -X:deps tree already does a "good enough" job.#2020-09-0920:34Alex Miller (Clojure team)well, it exists as a function you could call (the one -Stree calls) :)#2020-09-0920:34Alex Miller (Clojure team)but that stuff is due for some significant changes soon as I fix up some of this stuff#2020-09-0920:34seancorfieldBesides, I've had this shell script running with -Stree for ages and don't really want to rewrite it as a Clojure program right now. Maybe when I'm bored and want to go shave some more yaks...#2020-09-0920:35Alex Miller (Clojure team)ok, just letting you know that I'm not considering stdout changes from -Stree (or its replacement) to be breaking :)#2020-09-0920:36Alex Miller (Clojure team)but I am planning to add some more api fns to get tree-like deps data#2020-09-0920:19Alex Miller (Clojure team)and from a tools.deps pov, you could probably make something that did this via the api instead, but not sure if that's worth the effort#2020-09-0920:21Alex Miller (Clojure team)I really consider -Stree / -X:deps tree to be human consumer tools and ones likely to hopefully improve over time, changing format. I don't consider the current particular output to be something a script should rely on.#2020-09-1000:55didibusI'm excited to try the new clj exec, at first glance, its giving me some powershell vibes... which I'm not sure about 😛#2020-09-1001:00didibusOne question, in the guide it says: clj -A:deps -X:install :jar '"/path/to.jar"', couldn't you include the deps in the :install alias itself? And thus you could then omit the -A:deps and just do: clj -X:install :jar ...#2020-09-1001:14seancorfieldIt says clj -X:deps mvn-install :jar '"/path/to.jar"'#2020-09-1001:14seancorfieldAre you looking at the old (stable) guide @didibus?#2020-09-1001:14seancorfieldThe prerelease is https://clojure.org/reference/deps_and_cli_prerelease#2020-09-1001:15seancorfieldBut, yes, in general you can combine the exec stuff and the dependencies.#2020-09-1001:16seancorfieldI have some examples of that here https://github.com/seancorfield/clj-new#clojure--x-usage#2020-09-1001:38didibusI probably was, it was from the first blog post about it on inside clojure#2020-09-1001:39didibusAh yes, the latest blog post says: "so you can supply the deps and the function to execute together" never mind then#2020-09-1001:42didibusSo that's awesome, now all we need is a convention to define alias in some file that you put in a git repo, and something like: clj -X:deps install-alias some-git-repo and a corresponding clj -X:deps remove-alias some-git-repo and we have ourselves a little NPM of some sort.#2020-09-1001:43didibusWell, it doesn't need to live inside -X:deps, but it would be an easier way to bootsrap it#2020-09-1001:45didibusYou could even do something like: clj -X:deps install-alias :git some-git-repo :symlink true and it could create a symlink in your user /bin to: clj -X:your-alias#2020-09-1001:45didibusSorry, none of this has to be included in deps, though I think it could be a good idea#2020-09-1001:46didibusThen clj could be used to install Clojure command line apps on your machine as well.#2020-09-1001:47didibusMight need some more hammock time#2020-09-1114:58souenzzoI'm using clojure 1.10.1.645 on linux
When I do clojure -e "(compile 'app.main)" it results in java .... -e '(compile '\''app.main)' wich works
When I create a alias :aot {:main-opts ["-e" "(compile 'app.main)"]}
It results in java commnad like -e '(compile' ''\''conduit.server)' which will do a EOF error, as it will try to eval (compile
Is it a know issue? I think that I already used it before (it looks like a regression for me)
PS: I'm using bash -x /usr/bin/clojure ... to know which "final java command" clojure is calling#2020-09-1115:00Alex Miller (Clojure team)nothing has changed in this area afaik#2020-09-1115:01Alex Miller (Clojure team)things in :main-opts go into a file, and then back out of a file which causes problems due to word-splitting in bash#2020-09-1115:01Alex Miller (Clojure team)so if you replace the space after compile with a , that will help#2020-09-1115:01Alex Miller (Clojure team)this is a known issue (but it has always been a known issue :)#2020-09-1115:30souenzzoShould I try to PR?
I worked with bash for some years 🙂#2020-09-1115:30Alex Miller (Clojure team)there is a ticket and about 4 different patches on there already :)#2020-09-1115:31Alex Miller (Clojure team)let me find it#2020-09-1115:33Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-56#2020-09-1115:35Alex Miller (Clojure team)similar but different issues on windows#2020-09-1117:01lreadHere’s related issue around Windows https://clojure.atlassian.net/jira/software/c/projects/TDEPS/issues/TDEPS-133#2020-09-1117:17seancorfieldI just routinely use , instead of whitespace when writing stuff in deps.edn and even on the command-line for clojure 🙂#2020-09-1117:20lreadaka the “corfield comma”!#2020-09-1117:23seancorfieldI can think of worse things to be known for 🙂#2020-09-1117:51souenzzoNow thinking about Windows.
I can't implement a bash-friendly solution, once it may be dificult to implement in "cmd", once it may need to "interpret" that file as bash.
How is going that idea of do like "deps.exe" and use a graal-binary to replace bash/bat?#2020-09-1118:22seancorfieldOr use WSL2 for Clojure development? 🙂#2020-09-1118:49lreadIf you want to work on plain old Windows and don’t need to stay with official Clojure tools, you might consider trying https://github.com/borkdude/deps.clj. Using its deps.exe in place of clojujre.exe made sense for me when testing my project under Windows on GitHub Actions.#2020-09-1117:17seancorfieldI just routinely use , instead of whitespace when writing stuff in deps.edn and even on the command-line for clojure 🙂#2020-09-1117:18Alex Miller (Clojure team)now of course, you can just use -X in many cases instead :)#2020-09-1117:22seancorfieldAlthough you can't use -X to do a simple compile because it requires a symbol and -X passes a hash map 🙂#2020-09-1117:29dominicmI reckon we'll see a pack of map versions of core functions.#2020-09-1117:48vlaaadAll you need is a single invoke :
(defn invoke [{:keys [fn args]}]
(apply fn args))
(invoke {:fn inc :args [1]}) => 2
#2020-09-1117:29dominicmOr maybe the work on kwargs, will include positional arguments too?#2020-09-1117:48vlaaadAll you need is a single invoke :
(defn invoke [{:keys [fn args]}]
(apply fn args))
(invoke {:fn inc :args [1]}) => 2
#2020-09-1118:49Alex Miller (Clojure team)A new prerelease version of clj is now available (1.10.1.681):
• Reinstate -R, -C, and -Spom from last week's prerelease (but still deprecated)
• TDEPS-155 - Some error message improvements for various bad coordinate cases#2020-09-1118:49Alex Miller (Clojure team)Based on feedback from last week, added some things back to ease the migration.#2020-09-1118:52Alex Miller (Clojure team)For the moment -Spom will not warn. The -X:deps mvn-pom version was really missing support for modifications to the classpath previously pulled from clj invocation itself. I've added that as an arg but needs more thinking.#2020-09-1118:54Alex Miller (Clojure team)@seancorfield I've also standardized all the deprecation warnings to be stderr, if you wanted to pipe redirect them to /dev/null and ignore that's at least possible now. also I fixed the exec.jar issue in the installation.#2020-09-1119:24seancorfieldThanks @alexmiller -- just to confirm, is -R exactly the same as before? It hasn't expanded to support other types of arg maps like -A?#2020-09-1119:38Alex Miller (Clojure team)yes, just re-added, not going to do the expanded thing#2020-09-1120:56markbastianI had an uberjar alias using depstar that was configured with main opts of:
["-m" "hf.depstar.uberjar"
"target/myproject-with-dependencies.jar"
"--compile" "*" ;Note - the * seems to be needed for older versions of the args, but not now.
"--verbose"
"-m" "myproject.main"]
It was using the RELEASE version of depstar. I noticed that now instead of putting the jar in target/myproject-with-dependencies.jar it creates a jar named (Note: Do not ".
Is this the right way to build the standalone jar with aot compilation? It looks like --compile will do transitive compilation from your entry point. Is there a way to specify aoting the whole project? I saw that you can do :aot true. If I did that would it go in the main args or next to the :main-args key in the deps file?#2020-09-1121:05markbastianActually, looks like :aot is part of the -X functionality. I missed that.#2020-09-1121:21seancorfield@markbastian There's a #depstar channel now for dedicated help...#2020-09-1121:23markbastianThanks, I should have thought of that.#2020-09-1202:32hugodWith cli 673, and 681
clojure -Stree => Option changed, use: clj -X:deps tree
clojure -X:deps tree => Unqualified function can't be resolved: tree#2020-09-1203:15seancorfield@hugod It works fine for me (aside from showing tools.deps.alpha as a dependency and everything it depends on -- which was the problem with -X:deps mvn-pom in the last release).#2020-09-1203:15seancorfieldDo you perhaps have your own :deps alias, shadowing the system installed one?#2020-09-1210:56hugodThat was it thanks. Should have thought of that. Might be nice to have a warning if you define your own :deps alias.#2020-09-1203:16seancorfield(! 993)-> clojure -Sdescribe
{:version "1.10.1.681"
:config-files ["/usr/local/Cellar/
^ @alexmiller FYI#2020-09-1203:29sogaiuhere i get:
$ clj -X:deps tree
Cloning:
Checking out: at e160f184f051f120014244679831a9bccb37c9de
Unqualified function can't be resolved: tree
$ clj -h | grep 681
Version: 1.10.1.681
#2020-09-1203:31sogaiulooks like i've got the following in my ~/.clojure/deps.edn:
:deps
{
:extra-deps {org.clojure/tools.deps.alpha
{:git/url ""
:sha "e160f184f051f120014244679831a9bccb37c9de"}}
}#2020-09-1203:33sogaiu...and with -Srepro i don't get that message#2020-09-1203:57seancorfieldYup, bad idea. You should only have that under an alias.#2020-09-1203:58seancorfieldYou're essentially breaking the :deps alias in the system installed deps.edn...#2020-09-1203:58seancorfield@sogaiu ^#2020-09-1205:38dominicmMaybe the system deps should have a namespace?#2020-09-1206:05seancorfieldOr maybe system aliases should be "reserved" so you can't overwrite them?#2020-09-1206:18sogaiu@seancorfield yes, thanks. i think it was a left-over from here: https://github.com/seancorfield/dot-clojure/blob/d4aba12ec24834200b70ee0c06a30fb0538bddc0/deps.edn#L167 - my bad for not keeping up-to-date#2020-09-1206:19seancorfieldYeah, I changed that when I realized that it conflicted with the system one 😞#2020-09-1206:19seancorfieldNow it's this: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L164#2020-09-1206:20seancorfield(and that's an outdated branch -- the latest versions of add-lib require builds from source because they include Java code)#2020-09-1206:20sogaiuah thanks!#2020-09-1206:21seancorfieldI update that file a lot -- so it's a good idea to keep git pulling it regularly 🙂#2020-09-1206:22seancorfieldI'm adding -X compatible stuff now as well, for folks on the latest prerelease of the CLI: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L15-L18#2020-09-1206:22seancorfieldBoth -M:new and -X:new will work.#2020-09-1206:23seancorfieldclojure -M:new lib myname/myproject or clojure -X:new :name myname/myproject (since :template is defaulted to lib).#2020-09-1313:35dominicmI suppose that the new -X stuff can't express regex directly, as it's EDN based. I only mention it because it's one of a handful of types that couldn't be passed to a function directly, and a wrapping string-based function would be needed for use from the CLI.#2020-09-1313:49Alex Miller (Clojure team)Yep#2020-09-1317:35seancorfieldYup, that's what I did for depstar's --exclude arg, but the command-line version is also a plain string so both logic paths have to turn it into a pattern.#2020-09-1318:26borkdudesame with EDN config, I also use strings as regex in clj-kondo's config in places#2020-09-1318:32kingcodeI’m wondering about this strange syntax in <install-dir>/deps.edn:
{…
{:aliases ...}
R
...
}
So it looks like an embedded deps map is used as a key pointing at the -R option? How does this work? I couldn’t find this documented anywhere on the clojure site’s Deps and CLI reference. (Using clojure 1.10.1.590)#2020-09-1320:32seancorfield@kingcode I don't see that in any of the versions I have installed -- is it possible you've accidentally edited it?#2020-09-1320:35seancorfieldHere's what's in my system install deps.edn for that version:
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "a9daccc8c7cae9edecfae2fde6f9849a99e0eae89e9f"}, :content ("[email protected]")}#2020-09-1320:40kingcode@seancorfield Sorry, I meant my config dir, not the install.. I think maybe installing rebl recently caused a bad edit - here is the effective content of .clojure/deps.edn:
{ ;...
{:aliases
{:rebl ;; for JDK 11+
{:extra-deps {com.cognitect/rebl {:mvn/version "0.9.241"}
org.openjfx/javafx-fxml {:mvn/version "15-ea+6"}
org.openjfx/javafx-controls {:mvn/version "15-ea+6"}
org.openjfx/javafx-swing {:mvn/version "15-ea+6"}
org.openjfx/javafx-base {:mvn/version "15-ea+6"}
org.openjfx/javafx-web {:mvn/version "15-ea+6"}}
:main-opts ["-m" "cognitect.rebl"]}
:rebl-jdk8 ;; for JDK 8
{:extra-deps {com.cognitect/rebl {:mvn/version "0.9.241"}}
:main-opts ["-m" "cognitect.rebl"]}}}
R
;;....
}#2020-09-1320:41seancorfieldThis is definitely not legal syntax#2020-09-1320:41Alex Miller (Clojure team)Agreed :)#2020-09-1320:42kingcodeOK…<sigh-of-relief> thx for your comments. I probably should remove the R and make the embedded map the entire effective content then?#2020-09-1320:44seancorfieldHard to say. My ~/.clojure/deps.edn looks like this: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#2020-09-1320:45seancorfieldBut :aliases should be a top-level key in the overall map.#2020-09-1320:46kingcodeThanks @seancornfield - that makes sense 🙂#2020-09-1321:19dpsuttonAliases always go in a map with the other aliases. That example is correct. And your deps.edn locally will largely look the same yes?#2020-09-1321:28kingcodewoops @dpsutton and @alexmiller Sorry indeed my mistake. The Rebl distro alias example works fine 🙂 Too much staring.#2020-09-1409:59kirill.salykinUPD: nvm, wrongly specified the maven repo, sorry
---
Hi
I am trying to install dependency from the s3 but facing the issue:
Error building classpath. Failed to read artifact descriptor for com.amazon.redshift:redshift-jdbc42:jar:1.2.47
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.amazon.redshift:redshift-jdbc42:jar:1.2.47
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:255)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
(Following this doc https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#configure-jdbc-connection-with-maven)
{:paths ["src" "resources"]
:mvn/repos {"redshift" ""}
:deps{
com.amazon.redshift/redshift-jdbc42 {:mvn/version "1.2.47"}
...
}}
Please help what I am doing wrong?#2020-09-1411:57practicalli-john@alexmiller Will the changes in the pre-release be released as version 1.10.2, or something similar such as 1.10.2-alpha or 1.10.2-RC ? This would make it very easy to document usage of the tooling, as an easy distinction can be made between 1.10.2 and pre-1.10.2 versions. Thank you#2020-09-1412:34Alex Miller (Clojure team)No, probably not#2020-09-1412:40practicalli-johnAh, so I'll just drop the older content then and add hints to require the latest version is running. If issues arise the the hint will be first to update the version of CLI tools. Thanks#2020-09-1417:44Lone Rangerstruggling with a proxy just trying to get clj -r going#2020-09-1417:45Lone RangerI've tried setting the settings in ~/.m2/settings.xml and running
clj -Dhttp.proxyHost=<host> -Dhttp.proxyPort=<port> -Dhttps.proxyHost=<host> -Dhttps.proxyPort=<port> -r
#2020-09-1417:45Lone Rangerany suggestions?#2020-09-1418:08borkdude@goomba You need to pass Java opts using -J-D#2020-09-1418:14Lone Rangerahhhhh#2020-09-1418:14Lone Ranger🙏#2020-09-1517:43Aaron CummingsDoes there exist a tool like Mr. Anderson (dependency shading) for tools-deps?#2020-09-1517:52andy.fingerhutI haven't actually used Mr. Anderson, but am aware of it and why someone might want to use it. Is Mr. Anderson in any way currently tied to Leiningen?#2020-09-1518:08Aaron CummingsIt is a leiningen plugin, but it looks like it could be used directly. But I'm having trouble describing what I'm trying to accomplish, which maybe means I'm reaching for the wrong thing. I'll need to think about this some more.#2020-09-1518:28andy.fingerhutI have never done this before, but the Mr. Anderson project README has a brief note and a pointer to this repository, which seems to use Mr. Anderson in a way that doesn't require Leiningen: https://github.com/Olical/conjure-deps#2020-09-1523:07sogaiuit does use it as a library there#2020-09-1600:41Aaron CummingsI'll take a look at that, thanks.#2020-09-1520:36rbergerI am wondering why I’m getting
shadow-cljs - starting via "clojure"
Cloning:
(Full stacktrace at https://gist.github.com/rberger/34d72e06736ed7cb449c88030c9815ba)
This happens if I have the following in my .ssh/config:
Host
User git
Port 22
HostName
IdentityFile ~/.ssh/github.rsa
AddKeysToAgent yes
TCPKeepAlive yes
IdentitiesOnly yes
Host *
User myname
Port 22
IdentityFile ~/.ssh/id_rsa
AddKeysToAgent yes
TCPKeepAlive yes
IdentitiesOnly yes
All my regular git activity works fine with this, and it seems I need this for macOS automatic key loading to work
If I remove the Host * entry the clojure tools git loading works fine.
I presume this is a jgit issue but I’m wondering if there is a way to make this work as I’m expecting or if I’m expecting wrong.
The Maven issues talk about setting to useNativeGit=true but I don’t know if there is a way to do it with the clojure tools
I had this problem once before and removed the offending config stuff, but I would like to figure out how to make this work with this or equivalent ssh config if possible.#2020-09-1520:50seancorfieldIdentityFile is the problem I believe, based on discussions I've seen here...#2020-09-1520:51seancorfieldBut @alexmiller can give you a more authoritative answer...#2020-09-1520:53Alex Miller (Clojure team)that's definitely where I'd start (by removing that)#2020-09-1520:56Alex Miller (Clojure team)since it looks like you are using a different github key, I'm not sure of the right combination of things to make it work. you might need to add that key with ssh-add#2020-09-1611:52practicalli-johnhttps://github.com/practicalli/clojure-deps-edn/tree/qualified-alias-keywords-and-new-flags is a refactor of the project wide aliases in light of the changes coming in Clojure CLI tools (version 1.10.1.681 onwards). This will go live when the Clojure CLI changes go live.
The aliases are using the -X flag for executing functions and providing default hash-map arguments (where the library supports this) and using -M flag for clojure-main (replacing -A and -R). Where chained aliases in a command have multiple main namespaces, the alias that should have its main run is added last in the chain.
Alias namespaces have been upgraded to be qualified names, further defining their purpose and making them easy to understand without looking at the details (at least that is the intent).
https://gist.github.com/jr0cket/ae7dd745eb45870109ace59fe835ce80 contains notes to be added to the Practicalli Clojure book on using the Clojure CLI tools with the recent changes. Any feedback on this is welcome, especially on how clear an accurate this is.
Thank you.#2020-09-1613:52Alex Miller (Clojure team)-P can be used with -X too#2020-09-1613:55Alex Miller (Clojure team)re "The order in which repositories are consulted is not guaranteed", recently I changed this to actually guarantee that the first two repos consulted are central and clojars (no guarantee after that). central and clojars are listed in the root deps.edn so do not need to be added to deps.edn or settings.xml unless you add mirrors etc.#2020-09-1613:56Alex Miller (Clojure team)"`business-area**" looks like bad markup#2020-09-1613:58Alex Miller (Clojure team)the automatically included "central" and "clojars" repos can be removed from consideration by nil'ing them in deps.edn {:mvn/repos {"central" nil}}#2020-09-1617:08seancorfieldcurl -O
chmod +x linux-install-1.10.1.681.sh
sudo ./linux-install-1.10.1.561.sh
Version is not correct in that last line.#2020-09-1617:09seancorfieldRun project with or without an alias:
`clojure -M:alias -m domain.app-name`
`clojure -M -m domain.app-name
`
Looks like you have extra backticks in that code block.#2020-09-1617:10seancorfield:project/outdated-mvn - check for newer dependencies (maven only)
The other two lines above this have the alias as code (with backticks) but this third one does not.#2020-09-1617:12seancorfieldhttps://cljdoc.org/d/seancorfield/next.jdbc/1.1.588/doc/getting-started#create--populate-a-database
I think it would be better to use CURRENT instead of a specific version there, unless you plan to ensure that you update the README any time you update the next.jdbc version used in any of your aliases?#2020-09-1617:12seancorfieldclojure -M:inspect/portal-cli ClojureScript node.js REPL with Portal dependency
That should be :inspect/portal-node#2020-09-1617:15seancorfieldAdd Reveal as tap> source
I thought Reveal did that automatically? I don't recall having to do this manually when I've used Reveal. Depends on how you start it. If you start it with vlaaad.reveal/repl the add-tap is automatic. Perhaps make it clearer that this section only applies if you have started the REPL some other way than via Reveal itself? Or maybe it was just my misreading of what's being discussed at this point in the README?#2020-09-1617:15seancorfield"DEPRICATED" => "DEPRECATED"#2020-09-1617:15seancorfieldI found several other aliases further down the README that are not highlighted as code with backticks.#2020-09-1617:22seancorfieldhttps://github.com/practicalli/clojure-deps-edn/blob/qualified-alias-keywords-and-new-flags/deps.edn#L444 -- midje-runner is not unqualified in the alias -- but should be midje-runner/midje-runner (since unqualified lib names just double up in the group and artifact slot), otherwise you might cause a clash with any midje-runner lib that actually ends up with the midje group ID.#2020-09-1617:24seancorfieldhttps://github.com/practicalli/clojure-deps-edn/blob/qualified-alias-keywords-and-new-flags/deps.edn#L575-L590 -- the documented -X entry points are hf.depstar/jar and hf.depstar/uberjar (the ones you have still exist for backward compatibility but are not documented).#2020-09-1618:52practicalli-johnThank you Alex, Sean, most helpful and much appreciated.#2020-09-1621:02lreadVery cool @U05254DQM! Not sure if anyone mentioned what looks like a typo:
* deps = `:deps`, `:extra-deps`, `replace-paths`
* path = `:path`, `:extra-paths`, `replace-deps
`
Last items seem reversed?#2020-09-1621:59practicalli-johnGood catch. @UE21H2HHD , I'll fix that copy past error, thanks.#2020-09-1623:30seancorfieldPosting here because it there's a larger audience who might have more insight into Maven and its shade plugin behavior: The next version of depstar will preserve license/copyright files coming from dependencies by concatenating them (but omitting exact duplicates). As far as I can tell from inspection of several "uber" JAR files I found in the wild, this is the behavior of Maven with the shade plugin, with the additional license resource transformer (which omits duplicates). I couldn't find anything in Maven's documentation about this behavior, and looking at the source of the shade plugin didn't help much. If anyone here has deep insight into that aspect of Maven and can confirm its actual behavior, that would be very helpful (otherwise I'll just run with the depstar behavior I described).#2020-09-1623:39Alex Miller (Clojure team)I've used the shade stuff several times and I'm not sure it is anything mandated by maven either way, you're in making stuff up territory :)#2020-09-1623:40Alex Miller (Clojure team)I actually have come to think that the "right" way to do this is store things like that in lib-specific sub dirs under META-INF#2020-09-1623:40Alex Miller (Clojure team)I don't think concatenating is wrong though#2020-09-1623:55seancorfieldYeah, my inspection of various "uber" JAR files seemed to confirm that if maven shade finds META-INF/LICENSE or META-INF/LICENSE.txt in multiple dependencies, it simply concatenates them together in the resulting JAR file. My reading of the Apache license resource transformer seemed to confirm that it attempts to avoid duplicating the text of any license.#2020-09-1623:55seancorfieldBut this is definitely all "undocumented territory" as far as I can tell.#2020-09-1701:03deactivateduserI wonder if there are legal implications when concatenating license texts?#2020-09-1701:07Alex Miller (Clojure team)seems dubious to me :)#2020-09-1714:13markbastianWhen I run clj -Spom with an existing pom that already has a <build> tag the generated <build> tag is then placed inside of the existing rather than merging the contents or just adding the <sourceDirectory>. Is this a bug or am I just doing it wrong?
To be clear, it does this:
<-- Contents above elided -->
</plugins>
<build>
<sourceDirectory>src/main/clojure</sourceDirectory>
</build>
</build>
</project>#2020-09-1714:17Alex Miller (Clojure team)what version are you on?#2020-09-1714:17Alex Miller (Clojure team)there have been bugs and fixes on this#2020-09-1714:17Alex Miller (Clojure team)clj -Sdescribe#2020-09-1714:23markbastian"1.10.1.561"#2020-09-1714:31Alex Miller (Clojure team)yeah, that's the newest stable but the fix is not included in a newer stable version yet (it's fixed in the prereleases but you'll get a bunch of other changes with that)#2020-09-1714:33Alex Miller (Clojure team)so you could move to a prerelease (1.10.1.681 is newest) or you could fall back to a prior stable version - 1.10.1.510 or earlier#2020-09-1714:33Alex Miller (Clojure team)that version was actually also broken but only for -Spom generating a new pom#2020-09-1714:34Alex Miller (Clojure team)or you can wait till the next stable version, hopefully coming within the next week#2020-09-1714:34borkdudeor generate the pom with the newest unstable and then move back to the newest stable :)#2020-09-1714:36borkdudeit's pretty easy to do this if you download the clojure shell script for a given release#2020-09-1714:38Alex Miller (Clojure team)I prefer "prerelease" to "unstable" :)#2020-09-1714:45borkdude@markbastian if you're on mac, this is how you can switch versions easily:
brew install clojure/tools/
#2020-09-1714:46borkdudeor if you're using linuxbrew, same#2020-09-1715:10markbastianCool, thanks for the help. Waiting for next week is great. I don't need to update the pom much.#2020-09-1716:42borkdudeWould it be useful if clj-kondo did some checking on deps.edn (statically) for common mistakes?
Feel free to post ideas here: https://github.com/borkdude/clj-kondo/issues/945#2020-09-1808:53Saikyunis there a way to get clj working on a machine without access to internet? specifically I'd want clj + project dependencies on a machine behind a very restrictive firewall. I'm able to transfer files to it, and when building I use uperdeps.#2020-09-1808:53SaikyunI know that with leiningen I could download a jar-file and a bat-file, then it didn't need access to internet. is it possible to do something like that with clj?#2020-09-1809:06dominicm@saikyun the installer just downloads a couple jar files and puts them on the filesystem, then puts a script in /bin/ to run that jar.#2020-09-1809:07dominicmThe void linux install script is the simplest version I know of, but others might have good examples too 🙂 https://github.com/SevereOverfl0w/void-packages/blob/master/srcpkgs/clojure/template#L15-L23#2020-09-1809:45Saikyunthanks for the example 🙂#2020-09-1809:07dominicmOne idea might be to run the installer into a directory with the -p flag.#2020-09-1809:07Saikyunthe target computer is on windows btw :x#2020-09-1809:18borkdude@saikyun Alternative if you end up running into issues: with deps.clj it's just a matter of downloading the executable and placing the tools jar in ~/.deps.clj (as documented here: https://github.com/borkdude/deps.clj/issues/20)#2020-09-1809:24Saikyun@borkdude thanks, something like that is what I wanted 🙂 what is the analogue to ~ on windows?#2020-09-1809:24borkdude%USERPROFILE%#2020-09-1809:24Saikyunthanks#2020-09-1809:33borkdude@saikyun Here is one user working around firewalls copying his .m2 deps with bb:
https://twitter.com/MrGung1/status/1305906562746253315#2020-09-1809:34Saikyuncool, thanks#2020-09-1809:35Saikyunwhere do you get deps.exe from?#2020-09-1809:35borkdudeGithub releases: https://github.com/borkdude/deps.clj/releases#2020-09-1809:36borkdudeor install via scoop etc, but since you're having firewall issues, Github releases is probably the easiest#2020-09-1809:37Saikyunyeah, thanks a lot.#2020-09-1809:37Saikyunand it's enough for deps.exe to be on PATH, right?#2020-09-1809:38borkdudeyes#2020-09-1809:38borkdudealso works from cmd.exe, no PS needed#2020-09-1809:38Saikyunokay, cool 🙂#2020-09-1809:38SaikyunI'm getting this error now:
Error building classpath. Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.1
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.clojure:clojure:ja
r:1.10.1
#2020-09-1809:39Saikyunthis is what prompted me to try to install tools.deps in an alternative way in the first place#2020-09-1809:39SaikyunI wonder if there's some old stuff lying around, trying to find tools.deps in the directory specified when I ran the .ps1 installation script...#2020-09-1809:39borkdudeprobably there's something wrong with your .m2 stuffs?#2020-09-1809:40Saikyunoh, right, I guess I need to put clojure in the .m2 folder? (e.g. by using -p...?)#2020-09-1809:41borkdudetools.deps is going to the internets when the dep is not yet in .m2#2020-09-1809:41Saikyunokay, that makes sense#2020-09-1810:14Saikyunwhen running clj -P , where do the dependencies end up? I'm looking in ~/.m2 but can't find clojure :S#2020-09-1810:14SaikyunI have clojure as an explicit dependency in deps.edn as well#2020-09-1810:15Saikyunnvm, it was in .m2/org/clojure.. , not .m2/clojure...#2020-09-1812:01Saikyungot it all working by removing my local, existing .m2, then ran clj -P, then I copied my new local .m2 to the firewalled machine and now it works 🙂#2020-09-1812:01Saikyunthanks a lot for the help#2020-09-1812:21borkdudeI think the .m2/repository is also configurable in deps.edn (can't remember it from the top of my head)#2020-09-1812:40Alex Miller (Clojure team)Yes, with :mvn/local-repo#2020-09-1812:47borkdudewhich probably also works with -Sdeps? then that could be used for populating an ad hoc repo @saikyun#2020-09-1812:49Alex Miller (Clojure team)Yep#2020-09-1814:10Alex Miller (Clojure team)I use this for testing "clean cache" cases all the time#2020-09-1814:10Alex Miller (Clojure team)(without having to muck with my actual ~/.m2/repository)#2020-09-2206:57Saikyunahh, thanks for the tip! great to know 🙂#2020-09-2206:57SaikyunI’m setting up a dev environment on a pretty locked down production machine, so these things are invaluable#2020-09-2207:00borkdudeOne issue I had while developing on a train is that I forgot to download deps for certain aliases #2020-09-2207:01Saikyunah, good point. will keep that in mind#2020-09-1819:09just.sultanovHi all! Does clojure take MAVEN_OPTS from environment variables? I can’t use keystore with clojure, but it’s work with mvn (directly)#2020-09-1819:14seancorfield@just.sultanov I just grepped the source of brew-install, homebrew-tools, and tools.deps.alpha and didn't see that var in any files so I would say "no".#2020-09-1819:15seancorfieldIt does handle ~/.m2/settings.xml for Maven stuff tho' so maybe you could put credentials in there?#2020-09-1819:24just.sultanovI need to use keystore.jks with self-signed corp certificates. I don’t have problems with credentials. Maven by default pick up settings from .mavenrc and MAVEN_OPTS, but I didn’t see any usages in sources of tools.deps#2020-09-1819:44Alex Miller (Clojure team)doesn't use it right now#2020-09-1819:45Alex Miller (Clojure team)you can log the request at https://ask.clojure.org#2020-09-2114:43timoWhat is the command to only resolve dependencies with tools-deps and not start a repl? I need it for CI.#2020-09-2114:44Alex Miller (Clojure team)clj -e nil is sufficient. current prereleases have a new -P (prepare) switch for this purpose.#2020-09-2114:46timothanks!#2020-09-2115:03deactivateduserFor better logging in multi-step builds, I sometimes do clojure -Srepro -e '(println "Dependencies downloaded")' .#2020-09-2114:47ghadi@timok you'll want clojure -e nil instead of clj if you don't have an interactive terminal#2020-09-2114:47ghadiin CI#2020-09-2114:47timook, cool#2020-09-2114:48timoI never understood the differences though#2020-09-2114:48dominicmClj calls rlwrap. So adds things like up/down.#2020-09-2114:49Alex Miller (Clojure team)right, sorry#2020-09-2114:50timoawesome, thanks for the answers :thumbsup:#2020-09-2202:26Alex Miller (Clojure team)clj 1.10.1.693 (prerelease) is now available:
• Classpath ordering changes. Classpath will always now be ordered extra-paths, paths, then libs. Libs are ordered by tree depth, then alpha. These changes both revert some ordering changes in the last stable release and go further in defining a reproducible classpath order.
• clj -Stree re-enabled for now while we decide on best way to convey classpath-changing options to -Stree and -Spom
• Windows scripts updated to latest
• This prerelease version should be considered a release candidate for the next stable version! If you try it and have feedback, please leave it here#2020-09-2203:20seancorfield@alexmiller I assume the -Stree restoration is because of this difference?
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "cfbcaaaea1ac8f8b8a9c849b809fe29e9afd9a85fe81"}, :content ("[email protected]")}#2020-09-2204:02Alex Miller (Clojure team)yes, but that's just a symptom of the greater problem, which is how to convey classpath changing arguments to tools like pom and tree without doing it through clj's args#2020-09-2204:03Alex Miller (Clojure team)Datomic's ion-dev has the same need (for lib bundling)#2020-09-2204:03Alex Miller (Clojure team)so we're looking at something common for all these but don't want to gate on figuring that out#2020-09-2204:10Alex Miller (Clojure team)The -X:deps tree stuff is going to have more features over time for other variants of viewing the tree etc#2020-09-2205:19seancorfieldInteresting. Thanks, Alex.#2020-09-2210:59dominicmI thought that the corfield comma was documented on https://clojure.org/guides/faq but it seems not. Is it documented on the site somewhere? It would be useful for the http://nrepl.org docs to link to as it is tripping users up.#2020-09-2211:05sogaiunot as corfield comma, but there are these: https://clojure.org/guides/weird_characters#_whitespace_character
https://clojure.org/reference/reader#_maps (2nd bullet point)#2020-09-2211:07sogaiuin #parinfer we were talking about use of the comma recently for a variety of things: https://clojurians-log.clojureverse.org/parinfer/2020-09-14/1600045197.010800#2020-09-2219:17ghadihow do people handle bumping the version inside pom.xml with minimal effort?#2020-09-2219:17ghadii'm not beneath sed but I was hoping for something intelligent#2020-09-2219:22Alex Miller (Clojure team)I've done manual, various Maven plugins, sed#2020-09-2219:22Alex Miller (Clojure team)the contrib projects actually have this done automatically via Jenkins Maven integration#2020-09-2219:29seancorfieldI edit it manually as the last step of preparing for a release -- and I edit both the <version> and the <tag> entries to be consistent (the latter just has v at the beginning). I tend to do a global find for a regex that captures the current release, the new (planned) release, and my generic placeholder (e.g., 1.1.next) which I may tag new functionality with in the readme, docs, code, and change log. Then I just work my way through those find results one at a time, and I can eyeball that all matches correctly refer to the new release and the old release as needed.#2020-09-2219:30seancorfield(so I might search for 1\.\1.(next|588|623) for example)#2020-09-2219:55vlaaadUgh, why not just create pom.xml from some tdeps-generated inputs?#2020-09-2219:57vlaaadBumping the version will be a different arg to a pom-generating fn#2020-09-2220:00borkdude@ghadi clojure.data.xml is included in bb, I've done XML rewriting with that#2020-09-2220:04borkdude(here's a script for getting the version out of a pom.xml: https://github.com/borkdude/babashka/blob/master/examples/pom_version.clj)#2020-09-2220:06hugodmakejack updates the pom based on a :version tag in a project.edn file. I haven’t got a task to bump the version yet.#2020-09-2220:07ghadised it is#2020-09-2220:08Alex Miller (Clojure team)https://github.com/cognitect-labs/transducers-java/blob/master/build/release#L26#2020-09-2220:09Alex Miller (Clojure team)in case it saves you any time... (made for Mac sed)#2020-09-2309:28timoone can use xmllint as well for getting the version out like here: https://github.com/replikativ/hasch/blob/87410246864f2e160cccac3a51f7a7e3139c3c45/.circleci/config.yml#L123 but setting it probably mvn plugin like @alexmiller does in the release script.#2020-09-2314:07lreadhttps://github.com/workframers/garamond offers a way to set the version in pom.xml. If you are using your own versioning scheme you can use its --force-version option.#2020-09-2314:09lreadIn the past I have also used maven to do this type of work, for example to update the tag and version:
mvn versions:set-scm-tag -DnewTag=$(git rev-parse HEAD) -DgenerateBackupPoms=false
mvn versions:set -DnewVersion=$(bin/generate_version) -DgenerateBackupPoms=false
Somehow it bothers me how long it takes maven to do this tho…#2020-09-2315:39Alex Miller (Clojure team)anyone tried the clj prerelease yet?#2020-09-2316:05vlaaadI haven’t, but I’d like to do it today#2020-09-2316:25Alex Miller (Clojure team):thumbsup:#2020-09-2316:33seancorfieldInstalled it Monday night and integrated it into our dev/test/build pipeline -- everything is working as expected. REPLs, aliases, running tests, building uberjars...#2020-09-2316:53Alex Miller (Clojure team)Thx, good to hear#2020-09-2318:44vlaaadI'm just trying to understand the basis stuff, is this intended that :parents of a thing contains a thing itself?
cljfx/cljfx$jdk11
{:mvn/version "1.7.9"
:deps/manifest :mvn
:dependents [cljfx/cljfx$jdk11 cljfx/cljfx]
:parents #{[cljfx/cljfx cljfx/cljfx$jdk11]
[cljfx/cljfx]}
:paths ["C:\\Users\\Vlaaad\\.m2\\repository\\cljfx\\cljfx\\1.7.9\\cljfx-1.7.9-jdk11.jar"]}#2020-09-2318:44vlaaad(that's an entry in :libs)#2020-09-2318:56Alex Miller (Clojure team)you should consider :parents to be undefined for now#2020-09-2318:57Alex Miller (Clojure team)in other words, that's internal implementation stuff leaking out and I make no guarantee it will be there or be anything in particular :)#2020-09-2318:58vlaaadah, okay. what's not internal there?#2020-09-2318:58Alex Miller (Clojure team)everything else#2020-09-2318:59Alex Miller (Clojure team)to your original question, I'm not sure why'd you see that though, if you could send me your deps.edn, I'd like to look at it#2020-09-2319:22vlaaadthis is deps.edn: https://github.com/vlaaad/reveal/blob/master/deps.edn#2020-09-2319:23vlaaadI can reproduce it by using clj without any other args and then loading the basis as described here https://clojure.org/reference/deps_and_cli_prerelease#_basis_injection#2020-09-2319:24vlaaad(clj on windows, if that matters)#2020-09-2319:46Alex Miller (Clojure team)doesn't matter. I look at it's a quirk of how classifier libs work in maven. when you have a lib foo and a classifier jar foo$a they technically both use the same pom. in this case, the root jar depends on the classifier jar, which has the same pom, so it looks like it depends on itself.#2020-09-2319:47Alex Miller (Clojure team)so, expected but weird :)#2020-09-2319:53vlaaadah, I see, thanks for clarification!#2020-09-2315:43delaguardohttps://github.com/DeLaGuardo/setup-clojure
btw, github action support prerelease as well. Might make the way to try it a little bit easier 😄#2020-09-2316:52ghadiworks beautifully @delaguardo. Thanks so much!#2020-09-2317:01seancorfield@delaguardo I'm using your setup for GH Actions on several of my projects so "Thank you!" -- one comment: using tools-deps for the CLI version is a bit confusing since tools.deps is a library with a different version numbering schema -- Clojure CLI 1.10.1.693 uses tools.deps 0.9.799 for example.#2020-09-2408:32delaguardothanks for the feedback!
I made a PR to address that confusion:
https://github.com/DeLaGuardo/setup-clojure/pull/13/files#diff-cdc05b01d72867060bc4d72cb108cf25
how do you think about it?#2020-09-2415:38seancorfieldThat looks good!#2020-09-2415:53delaguardomerged, thanks again for pointing on that#2020-09-2320:21grzmWhere might addressing https://clojure.atlassian.net/browse/TDEPS-46 “Repositories from dependencies’ pom.xml are not used” be on the horizon? Any pointers on where one might get started on fixing it?#2020-09-2321:53Alex Miller (Clojure team)Hey, sorry I didn't catch your question earlier in the flow...#2020-09-2321:54Alex Miller (Clojure team)I am honestly reluctant to fix this the more I think about it and talk to others about it#2020-09-2321:55Alex Miller (Clojure team)There are a lot of security concerns around getting someone's else repository, which may either be an internal repo or a repo that provides rogue versions of deps defined in central repos#2020-09-2321:55Alex Miller (Clojure team)So my recommendation right now would be to add those repositories explicitly to your deps/pom instead#2020-09-2322:02grzmThanks for the feedback! We ran into this today, so I wanted to see what the current thinking on this is. Cheers!#2020-09-2320:25seancorfield@alexmiller We haven't seen any impact of the changes to classpath order (yay!) but that made me wonder about pom.xml generation: is the order of the <dependencies> section there important, from a Maven p.o.v.? i.e., does it affect the order things show up on the classpath and does clojure -Spom generate the <dependencies> section in a reproducible order now based on the classpath?#2020-09-2321:01Alex Miller (Clojure team)good question. the order is important to maven (you should get deps in the order listed in the pom) and I have not explicitly looked at that ordering wrt deps#2020-09-2321:07vlaaadso that might be unstable if deployed library is used in a project that uses maven as a build tool?#2020-09-2321:11Alex Miller (Clojure team)well, deps.edn takes its deps in a map so from a clj pov, there is no order set by the user (the new changes order by alpha)#2020-09-2321:12Alex Miller (Clojure team)@seancorfield if you wanted to file a jira about this, I'd be happy to look at it#2020-09-2322:14seancorfield@alexmiller I don't care enough about the issue to file a Jira ticket. I was mostly just curious if you knew off the top of your head whether -Spom generated <dependencies> in the same order as they would now appear on the classpath.#2020-09-2322:14seancorfield(I didn't care much about the non-reproducible classpath ordering before either -- although some others did)#2020-09-2322:17Alex Miller (Clojure team)I looked at the code and the answer is no - they are not necessarily in the same order they'd appear on the classpath#2020-09-2405:38andy.fingerhutThis may be a distraction from real tools.deps work, but was wondering if others had come across a series of articles by Russ Cox on Golang and package versioning. This is one of those articles I have looked at recently: https://research.swtch.com/vgo-principles. One thing I found interesting is the number of different package managers that use SAT solvers to find what collection of modules and their versions to use. In case you are curious, Russ argues that maybe this is a bad idea.#2020-09-2413:05Alex Miller (Clojure team)Yeah I’ve read these. Some of the Go stuff is similar to the tools.deps resolution#2020-09-2415:00andy.fingerhutI've only read a couple of Russ Cox's articles, but he seems to do a thorough job of writing about pros and cons of various approaches, with examples.#2020-09-2504:46andy.fingerhutOut of curiosity, did you consider the go module version approach of using the minimum version among all requests for a dependency, rather than the latest, in Clojure CLI tools? It sounded unintuitive at first, but the examples in those articles do seem to make it pretty clear that it is at least one way to achieve better reproducibility in versions of libraries used by an application.#2020-09-2504:47Alex Miller (Clojure team)bleh#2020-09-2504:49Alex Miller (Clojure team)if you are developing for growth, not breakage, that doesn't make sense#2020-09-2514:49andy.fingerhutGo has apparently strongly recommended backwards compatibility in APIs and behavior since 2013, before Spec-ulation talk, for similar reasons. I think the "largest of the minimum required versions" of this latest Golang vgo tool is fairly reasonably argued for on the basis of reproducible builds, regardless of whether someone releases a new version of a lib tomorrow. Anyway, not trying to change Clojure CLI tools in this regard.#2020-09-2406:47vlaaadI like the new exec stuff.
User deps.edn :
{:aliases {:reveal {:extra-deps {vlaaad/reveal {:mvn/version "1.0.130"}}
:ns-default vlaaad.reveal}}}
Starting a repl is as simple as clj -X:reveal repl
❤️#2020-09-2406:55sogaiuyes, i liked that too.
was hoping i could stack -X to use multiple tools (like one could with -A), but i get the impression that's not part of the design intent.#2020-09-2406:56vlaaadI wonder if that can be done using injected basis#2020-09-2407:03sogaiuhaven't thought about that angle yet, may be there's something to that.
i did think one might be able to make one thing that's designed to do the stacking for anything else that might follow its conventions.#2020-09-2410:37practicalli-john@vlaaad I would encourage people creating tools for Clojure CLI to support theuse the -X flag. As well as using key value pairs for clarity you can also set default values, making the comand line even simpler
As example, using Sean's clj-new project and the alias below I can create a library project with a default name by just running the command clojure -X:project/new
:project/new
{:extra-deps {seancorfield/clj-new {:mvn/version "1.1.215"}}
:exec-fn clj-new/create
:exec-args {:template lib :name practicalli/playground}
:main-opts ["-m" "clj-new.create"]}
I can then choose to over-ride value in the :exec-args default values by including key value pairs on the command line
clojure -X:project/new :name practicalli/banking-on-clojure#2020-09-2410:42vlaaadI mean that's exactly what I do :)#2020-09-2410:46practicalli-johnSo I could have an alias similar to this, that would let me do clojure -X:inspect/reveal to run a repl and clojure -X:inspect/reveal :command prepl to run prepl instead
:inspect/reveal
{:extra-deps {vlaaad/reveal {:mvn/version "1.0.130"}}
:exec-fn vlaaad.reveal
:exec-args {:command repl}}
Does that make sense?
To support versions before the pre-release you could also include main-opts
:inspect/reveal
{:extra-deps {vlaaad/reveal {:mvn/version "1.0.130"}}
:exec-fn vlaaad.reveal
:exec-args {:command repl}
:main-opts ["-m" "vlaaad.reveal" "repl"]}
#2020-09-2411:10vlaaadthere is no vlaaad.reveal fn#2020-09-2411:10vlaaadyou can use :exec-fn vlaaad.reveal/repl#2020-09-2411:13vlaaadmy example earlier used ns-default because there are various types of repls that I might want to start, so I want that to be a command line option — it doesn’t prevent :exec-fn use#2020-09-2411:18vlaaadI don’t see why there should be a single top-level function that receives :command as argument, exec-fn already solves the problem of calling different functions in the ns. Various repls in vlaaad.reveal ns are independent functions with their own different arguments, smashing them together would be unnecessary complecting#2020-09-2411:42practicalli-johnI was only suggesting an example of a general form of alias, along the lines of the clj-new as that is all I am familar with at the moment. Looking forward to see what develops with reveal.#2020-09-2412:04sogaiunice pun 😉#2020-09-2412:43vlaaadyeah, I’ll think of a more versatile suggested alias to put into reveal docs once -X is no longer pre-release 👍#2020-09-2410:46dominicmA lot of tools support later command flags overriding earlier ones anyway 🙂#2020-09-2412:13sogaiui have something for starting a socket repl on a specific port.
invocation is like:
clj -X:alc.socket-repl :port 9123
there is also a mode where you don't specify a port and it tries to choose one for you:
clj -X:alc.socket-repl
Socket repl port: 38735
Clojure 1.10.1
user=>
here it will print the chosen port number out so it's less digging around you have to do to get this info to your editor or whatever other tooling.
it also makes the port info available programmatically -- jcmd <pid> VM.system_properties should have it show up in the way clojure arranges for it w/ clojure.server.repl=... (i have a separate tool that can obtain this info)
at the end it just hands things off to clojure.main/main with --repl.
i wish i could use this along with other folks' things -- so far -X doesn't look to be composable. what i should say is not necessarily that, but some way to compose things -- it doesn't have to be via -X.#2020-09-2415:44seancorfieldComposability of "main" functions is problematic though: there's no guarantee a "main" will exit (if it starts a server, for example) and aliases in general are merged to produce the final configuration that is used to start a (single) program.#2020-09-2415:46seancorfieldWhat has been suggested (several times, here in this channel) is to write your own "runner" function that takes a list of functions to execute and runs them one after the other. Now that programs can get at the basis easily (if they are run via the CLI tools -- not if they are run via a JAR!), you could pass a list of aliases and the "runner" could walk that list and exec each alias...#2020-09-2415:47seancorfield...you wouldn't get the same semantics of composition but you would get "multiple main" functions run (and you could control the semantics yourself).#2020-09-2415:48seancorfield(this is exactly why we have a build script that wraps the CLI: you can specific multiple "commands" to build and it runs the CLI for each command, by way of a shorthand)#2020-09-2422:33Alex Miller (Clojure team)has anyone tried the clj prerelease 1.10.1.693 on windows?#2020-09-2422:34Alex Miller (Clojure team)I've done a fair amount of testing with it, but would be great if someone else had used it#2020-09-2423:05sogaiuin case others want to try and don't want to look for instructions (https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows):
0) remove previous installation (for me this was deleteing the ClojureTools folder under Documents\WindowsPowerShell\Modules)
1) download: https://download.clojure.org/install/win-install-1.10.1.693.ps1
2) did the "unblock" thing via properties (right-click on file's icon in windows explorer and choose properties)
3) execute the ps script via powershell
4) choose 2 (for under Documents\WindowsPowerShell\Modules) and wait a bit
5) clj -h to verify version
6) freshly clone a clojure project that uses deps.edn
7) start it via clj
something close to those steps worked for me.#2020-09-2519:00seancorfield@alexmiller Now that 1.10.1.697 is a stable release, any more thoughts on this discrepancy?
(! 740)-> clojure -Sforce -Stree
org.clojure/clojure 1.10.1
org.clojure/spec.alpha 0.2.176
org.clojure/core.specs.alpha 0.2.44
(! 741)-> clojure -Sforce -X:deps tree
org.clojure/clojure 1.10.1
org.clojure/core.specs.alpha 0.2.44
org.clojure/spec.alpha 0.2.176
org.clojure/tools.deps.alpha 0.9.810
javax.inject/javax.inject 1
com.cognitect.aws/api 0.8.408
... lots more stuff ...#2020-09-2519:03seancorfield(I verified we do not see the same discrepancy with -Spom vs -X:deps mvn-pom)#2020-09-2519:05Alex Miller (Clojure team)I'll get to it#2020-09-2519:05Alex Miller (Clojure team)have not had a chance to finalize the plan for it yet#2020-09-2519:06Alex Miller (Clojure team)for now, -Stree exists#2020-09-2519:29Alex Miller (Clojure team)Clojure homebrew tap has the latest of course. Homebrew core and linux core should get this automatically now, but I have no idea how long that will take, looking forward to watching that :)#2020-09-2614:25sogaiu@borkdude it's a minor thing but doing deps -h gives the version string twice here:
$ deps -h
Version: 1.10.1.697
Version: 1.10.1.697
...
#2020-09-2614:27borkdudeok, going to fix now#2020-09-2614:44borkdude@sogaiu Fixed (and all remaining open issues as well). If there's nothing else, going to release#2020-09-2614:56borkdudeReleased: https://github.com/borkdude/deps.clj/releases/tag/v0.0.10#2020-09-2617:50practicalli-johnI've updated the practicalli/clojure-deps-edn GitHub project and replaced all uses of -A and -R flags for using all aliases, substituting them for -M without any issues... it actually makes things simpler 🙂
Details are in #announcements
Thank you Alex (and Sean)#2020-09-2619:24dominicmExecution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate clojure/run/exec__init.class, clojure/run/exec.clj or clojure/run/exec.cljc on classpath.
Full report at:
/tmp/clojure-16205359177864566580.edn
Presumably this means I need to correct something in the void installer, what is that?#2020-09-2619:25borkdude@dominicm Is this the normal clojure CLI?#2020-09-2619:26dominicmAh, a new exec.jar :)#2020-09-2619:27dominicmOK, made that part of the new void installer :)#2020-09-2619:27borkdudewhat's a void installer?#2020-09-2619:43dominicmVoid Linux installer :)#2020-09-2619:27borkdude@alexmiller Feedback on error message with the new exec:
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "c5a7aab7aea1b0a1a085888795f7f5f4fc"}, :content ("[email protected]")}
I think the latter could be improved#2020-09-2619:36Alex Miller (Clojure team)not sure I understand why that's failing#2020-09-2619:36borkdudebecause there's no val to the key#2020-09-2619:36borkdudeI think#2020-09-2619:37Alex Miller (Clojure team)oh, that's probably it#2020-09-2619:37Alex Miller (Clojure team)it looks at whether the non-alias arg count is even or odd#2020-09-2619:37Alex Miller (Clojure team)there is some fiddly code there trying its best to guess what's wrong#2020-09-2619:38borkdudecan aliases be whitespace delimited?#2020-09-2619:38Alex Miller (Clojure team)no#2020-09-2619:38borkdudethen where is the ambiguity?#2020-09-2619:38Alex Miller (Clojure team)I don't think there is here, just saying that's the code at faultl#2020-09-2619:39borkdudeah ok. want me to make a JIRA?#2020-09-2619:39Alex Miller (Clojure team)sure#2020-09-2619:43borkdudehttps://clojure.atlassian.net/browse/TDEPS-168#2020-09-2621:29vlaaadbtw, real world example where this might be super confusing: I tried passing var as an argument (`#'inc`), but shells treat # as a start of a comment, so...#2020-09-2623:59Alex Miller (Clojure team)Yeah, I’ll fix#2020-09-2619:40Alex Miller (Clojure team)there was a period of time where other variants were accepted, this might be a legacy of that, don't remember#2020-09-2619:40borkdudeUnrelated. If checking in binaries into git wasn't an anti-pattern, I think Clojure git deps could be used to implement an ecosystem of binaries more or less like brew, but using deps.edn. The classpath can then be used as the binary path, a bit similar to a nix env. I would have a purpose for this (babashka pods). The git deps stuff could be made to work with GraalVM as well I think, so everything could be very fast after deps have been downloaded (but even without GraalVM, it's already fast due to classpath caching).#2020-09-2622:19sogaiui experimented with this once using git lfs.#2020-09-2622:20sogaiuthe binaries in git thing and finding things under ~/.gitlibs that is#2020-09-2622:22borkdudeah interesting. what did come out of it?#2020-09-2622:26sogaiuwell for one thing i think i learned that not everyone has git lfs 🙂#2020-09-2622:26sogaiuthough it looks like typical git installs on windows get it#2020-09-2622:27sogaiuother than that it seemed to work pretty well#2020-09-2622:28sogaiuin addition to the size aspect of the repository, trying to but binaries for multiple platforms in one repository didn't feel great because the end user then gets extra cruft.#2020-09-2622:28borkdudeHere I'm sketching out an idea: https://github.com/babashka/babashka.pods/issues/19
One option is to have a .edn file inside the git dep which refers to the download#2020-09-2622:29sogaiuyeah, i saw that page first 🙂
the .edn file inside idea seems preferable to the binaries in the repository idea#2020-09-2622:29borkdudeyeah. also the binaries could be pulled the first time you are trying to find them#2020-09-2622:30borkdudethis idea could even work for non-gitlibs as well#2020-09-2622:30borkdudemaybe there could be a nice coupling with github releases somehow#2020-09-2622:30sogaiuwhat would the ultimate location of the binaries be?#2020-09-2622:31borkdudethe resolve function will download the link and the put them inside .foobar/your/dependency/<sha>/the-binary and then that binary file will be invoked#2020-09-2622:32borkdudewhere .foobar is some name for this package manager layered on top of tda#2020-09-2622:36sogaiuah ok, thanks for the explanation -- i see it at the issue 🙂#2020-09-2619:40borkdudeWonder what others think. Maybe instead of checking in the binaries themselves, there could be links to releases in the source, with checksums, etc. But most of the gnarly dependency stuff could be handled by tda.#2020-09-2619:49borkdudeImplementing something like brew this way would only require a couple of tens of lines of code maybe.#2020-09-2619:58borkdudeThis could be a package manager for macOS, linux and Windows at the same time#2020-09-2620:32borkdudeAha: https://github.com/aerys/gpm - unclear how it resolves deps#2020-09-2621:33borkdudeMade an issue here for further discussion: https://github.com/babashka/babashka.pods/issues/19#2020-09-2711:54borkdudeImplementation of the above idea: https://github.com/borkdude/cpm#2020-09-2712:10sogaiu@borkdude for linux, i changed the .cpm.edn file for babashka to be:
{:package/name org.babashka/babashka
:package/description ""
:package/version "0.2.1"
:package/license ""
:package/artifacts
[{:os/name "Mac.*"
:os/arch "x86_64"
:artifact/url ""
:artifact/executables ["bb"]}
{:os/name "Linux.*"
:os/arch "amd64"
:artifact/url ""
:artifact/executables ["bb"]}]}
then i was able to follow your readme.#2020-09-2712:11borkdudePR welcome :)#2020-09-2712:11sogaiulol, ok, i'll send one 🙂#2020-09-2712:14borkdudeI pushed an update to bb, now cpm is also executable using bb:
$ bb -cp src:test-resources -m cpm.main clj-kondo
/Users/borkdude/.cpm/packages/clj-kondo/clj-kondo/2020.09.09
which is fast :)#2020-09-2712:21sogaiuindeed:
$ time bb -cp src:test-resources -m cpm.main clj-kondo
/home/user/.cpm/packages/clj-kondo/clj-kondo/2020.09.09
real 0m0.015s
user 0m0.009s
sys 0m0.006s
#2020-09-2718:54dominicmVoid Linux has now updated to clj 697 😃#2020-09-2808:49viestihumdum, I sometimes get this error in CirleCI while building project:
clojure -A:... -m kaocha.runner analysis
Picked up JAVA_TOOL_OPTIONS: -Dclojure.main.report=stderr -XX:-OmitStackTraceInFastThrow
Cloning:
using cimg/clojure:1.10.1 as image, with Clojure cli version 1.10.1.536#2020-09-2808:50viestimight be totally relared to this project only, but just thinking out loud, that might there be a concurrency issue in downloading dependencies#2020-09-2813:09Alex Miller (Clojure team)You could try adding -Sthreads 1 and see if that helps#2020-10-0208:09viestithanks! 🙂#2020-09-2815:58jjttjjI just updated my clj tools to the latest version and am now getting
CompilerException java.lang.RuntimeException: Unable to resolve symbol: requiring-resolve in this context, compiling:(cljfx/context.clj:37:32)
which would seem to imply that an older version of clojure being used? Is there an obvious reason this would happen after the clj update? (Both cljfx and my project seem to explicitly depend on clojure 1.10.1)#2020-09-2817:34jjttjjOk I figured this out. I had a :local/root dependency which points to a standalone jar file which uses clojure 1.9, which seems like it causes the repl started from clj to be 1.9. Is this expected?#2020-09-2816:04borkdude@jjttjj clojure -Stree is your friend here#2020-09-2816:05borkdude(or whatever is the way of doing the tree now)#2020-09-2816:49Alex Miller (Clojure team)Still that for now#2020-09-2816:10jjttjjHmmm it looks like clj is giving me a clojure 1.9 repl even though my project only has 1.10 in the tree. might be a scoop issue going to try reinstalling clj
Edit: ok I think this is a scoop issue#2020-09-2816:37natea couple of things I've noticed that are different about the new cli tools:
- Only the first -M is taken. I used to be able to do -A:one -A:two before and both :one and :two were used (was useful for a script I had that wrapped a call to clojure and I could add extra args if I wanted)
- If I put -Spath at the end of my command, it's ignored if there's a -M.... Specifically clojure -Spath -M:repl prints a classpath, but clojure -M:repl -Spath runs the main specified in the :repl alias#2020-09-2816:50borkdude@nate -M:one:two#2020-09-2816:50Alex Miller (Clojure team)-A and -M have always supported multiple aliases so -M:one:two is what you need there. -M and -X should now be last as they grab all args afterwards#2020-09-2816:51Alex Miller (Clojure team)So clojure -Spath -M:repl#2020-09-2816:52nateright, I know that you can put them together, my point is that you used to be able to supply them separately#2020-09-2816:52Alex Miller (Clojure team)Now you can’t #2020-09-2816:52nateI definitely understand why the behavior changed#2020-09-2816:52natecool#2020-09-2816:53Alex Miller (Clojure team)Multiple -A should still work#2020-09-2816:53nateinteresting point about -A#2020-09-2816:54natedeprecation warning is pushing me away from that, for good reasons#2020-09-2816:54borkdudeclojure -A:foo -M:bar should also work I think? brings in deps/paths from foo and bar but executes main from bar?#2020-09-2816:54Alex Miller (Clojure team)yes#2020-09-2816:55seancorfieldThere are a lot more changes in this version than I think are perhaps obvious on the surface -- it's probably always been "best practice" to specify :main-opts separately via a single -M option after using one or more -A options to accumulate aliases but that's never really been codified/written down. This new release sort of "enforces" (or at least "strongly encourages") what I think the core team have always considered best practice...#2020-09-2816:56nateyeah, I agree with that#2020-09-2816:56seancorfield-M used to only bring in :main-opts but now brings in paths/deps as well which should not cause any surprises -- but it is quite an important difference.#2020-09-2816:57borkdudeare you referring to a previous pre-release or release?#2020-09-2816:58seancorfieldI'm talking about this in the context of "previous stable" and "current stable" releases.#2020-09-2816:58nateme too#2020-09-2816:58borkdudeI didn't even know there was an -M in a previous release#2020-09-2816:59borkdudeA minor thing but -M -m still feels weird to me :)#2020-09-2817:00Alex Miller (Clojure team)consider it encouragement to use -X :)#2020-09-2817:01seancorfield"old" CLI reference https://github.com/clojure/clojure-site/blob/58b9c790052c0d2dce8aeb1e3cd0872bd9dac296/content/reference/deps_and_cli.adoc#aliases shows -R, -C, -O, -M, and -A @borkdude#2020-09-2817:03borkdude-X is ok but I don't consider it a substitute for a command line interface for general usage#2020-09-2817:04borkdudei.e. there will still be a place for tools.cli based tools#2020-09-2817:04borkdudeunless I'm missing something#2020-09-2817:11borkdudeBut I guess the -m gets hidden in an alias mostly anyway :)#2020-09-2817:12tvaughanI was doing something like clojure -A:alias-with-main-opts -m some.other.main With the new release I get a deprecation warning to switch to -M, but with -M -m some.other.main are passed as command-line options to the main entry point associated with alias-with-main-opts. Does this mean there's now no way to override a main entry point?#2020-09-2817:13seancorfield@tvaughan I believe clojure -A:alias-with-main-opts -M -m some.other.main would be the equivalent now.#2020-09-2817:14seancorfield(I haven't tried that...)#2020-09-2817:18tvaughanThanks @seancorfield. Interestingly, this clojure -A:toolbox:dxf-tools -M -m server.core results in an error message from my application about -m not -M being an unknown option. The main-opts associated with dxf-tools are still being used#2020-09-2817:19seancorfieldHmm, sorry. Guess I'll have to do some digging (unless @alexmiller can give a quick answer off the top of his head).#2020-09-2817:20tvaughanNo worries. I appreciate the help. I can dig further too. I took a quick look at the changelog but didn't see anything about this specifically#2020-09-2817:23seancorfieldAh, I think this is one of those cases where the advice is to split up :alias-with-main-opts into two aliases -- one with :main-opts (and any deps needed solely for that) and one with all the (other) deps.#2020-09-2817:24seancorfield(the other option is to continue using the -R and/or -C options even though they are deprecated)#2020-09-2817:26tvaughanCool. Thanks. I thought that might be the case. Didn't want to change things without confirmation tho :)#2020-09-2817:34jjttjjOk I figured this out. I had a :local/root dependency which points to a standalone jar file which uses clojure 1.9, which seems like it causes the repl started from clj to be 1.9. Is this expected?#2020-09-2817:46Alex Miller (Clojure team)if you're including an uberjar in your classpath, then you've broken the ability of tools.deps to properly resolve libs#2020-09-2817:46Alex Miller (Clojure team)b/c it can't "see" inside the uberjar to tell what libs are included there or their versions#2020-09-2817:52jjttjjThat makes sense#2020-09-2818:15seancorfield@alexmiller Now that 1.10.1.697 is the stable release, there are four mentions of clj -m <some.ns> in https://clojure.org/guides/deps_and_cli that need to be updated to include -M if you want folks to avoid the clojure.main warnings (i.e., "WARNING: When invoking clojure.main, use -M").#2020-09-2820:02Alex Miller (Clojure team)Yeah it’s on my list to update this week#2020-09-2818:15seancorfieldThose all need to be clj -M -m <some.ns> now, right?#2020-09-2818:23seancorfield(also, I added a note on TDEPS-100 that indicates some confusion over -A since the reference page does not define clj-opt and clojure -h says that -A is an exec-opt which seems to be at odds with the CLI/deps reference text)#2020-09-2819:21borkdudeyes, now I feel the issue we discussed last weekend. I have this:
{;;:paths ["your-packages"]
;;:deps {your-org/your-packages {,,,}}
:aliases
{:glam {:extra-deps
{borkdude/glam {:git/url ""
:sha "4599fb019deae9418d76a9996ae19b4003f3cc96"}}
:main-opts ["-m" "glam.main"]}}}
and now I want to do:
bb -cp $(clojure -R:glam -Spath) -m glam.main --install clj-kondo/clj-kondo --global --verbose
but I have to make two aliases now, so from Clojure this becomes:
clojure -M:glam-deps:glam
:/#2020-09-2819:22borkdudewith -Spath it's pretty clear you don't want to run the main right#2020-09-2819:22borkdudeso why not ignore the main when doing this:
clojure -A:glam -Spath
#2020-09-2819:23borkdudethat doesn't seem to be breaking#2020-09-2819:33borkdudeah well, bash to the rescue:
{:aliases
{:glam/packages {:extra-deps
{borkdude/glam {:git/url ""
:sha "4599fb019deae9418d76a9996ae19b4003f3cc96"}
;; your-org/your-packages {,,,}
}
;; :extra-paths ["your-packages"]
}
:glam {:main-opts ["-m" "glam.main"]}}}
alias glam='clojure -M:glam/packages:glam'
$ glam --install clj-kondo/clj-kondo org.babashka/babashka
#2020-09-2820:03seancorfield@borkdude If you put -Spath first it will not run the main.#2020-09-2820:04borkdudeah really#2020-09-2820:04borkdudethanks!#2020-09-2820:05seancorfieldAnd that's supposed to be clear from the docs which show clj [clj-opt*] [exec-opt] as the syntax -- -Spath is a clj-opt, -M is an exec-opt#2020-09-2820:05borkdudeyep, makes sense, thanks for clearing that up#2020-09-2820:05seancorfield(as I just noted to @alexmiller it's not clear whether -A is meant to be a clj-opt or an exec-opt at this point, esp. given that at some future date it will stop running :main-opts)#2020-09-2820:06Alex Miller (Clojure team)It’s kind of in a gray area#2020-09-2820:06Alex Miller (Clojure team)But I think I would call it a clj-opt#2020-09-2820:10seancorfieldAt this point, I'm hoping that -A stops running :main-opts fairly soon and it can be clearly classified as a clj-opt and then -R and -C can be fully removed.#2020-09-2820:33borkdude@seancorfield I'm still getting the warning:
$ clojure -Sforce -Spath -A:glam
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
#2020-09-2820:33borkdudeI don't want main, I want path#2020-09-2820:33borkdudeI used to use -R -C for this#2020-09-2820:34borkdudecan I rely on this being a correct invocation and the warning begone next version?#2020-09-2820:34seancorfieldclojure -Sforce -Spath -M:glam should work.#2020-09-2820:34seancorfield-Spath implies -P I believe.#2020-09-2820:34borkdudeit works, but I get the warning#2020-09-2820:35seancorfield(! 696)-> clojure -Sforce -Spath -M:runner
src:resources:/Users/sean/.gitlibs/libs/com.cognitect/test-runner/f7ef16dc3b8332b0d77bc0274578ad5270fbfedd/src:/Users/sean/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/sean/.m2/repository/org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar:/Users/sean/.m2/repository/org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.jar:/Users/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/Users/sean/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
#2020-09-2820:35seancorfieldWithout -Spath that runs the tests (or attempts to -- it's missing the :test alias)#2020-09-2820:36borkdudeaha, with -M it doesn't warn, but with -A it does... that's weird#2020-09-2820:36borkdudeI think -A is the preferred way for creating paths right#2020-09-2820:36seancorfield(! 698)-> clojure -Sforce -M:test:runner
Running tests in #{"test"}
Testing tvaughan.example-test
FAIL in (a-test) (example_test.clj:7)
FIXME, I fail.
expected: (= 0 1)
actual: (not (= 0 1))
Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
(! 699)-> clojure -Sforce -Spath -M:test:runner
test:src:resources:/Users/sean/.gitlibs/libs/com.cognitect/test-runner/f7ef16dc3b8332b0d77bc0274578ad5270fbfedd/src:/Users/sean/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/Users/sean/.m2/repository/org/clojure/test.check/1.0.0/test.check-1.0.0.jar:/Users/sean/.m2/repository/org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar:/Users/sean/.m2/repository/org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.jar:/Users/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/Users/sean/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
(! 700)->
🙂#2020-09-2820:37borkdudeanyway, I'll be using -A (and advocating it in my README) and expect the warning to be gone some time in the future#2020-09-2820:40borkdudeclojure -Sforce -Spath -A:glam 2>/dev/null
#2020-09-2820:42borkdudeFeel free to comment on this README for improved clojure usage in case I missed something: https://github.com/borkdude/glam#2020-09-2915:53borkdudefyi, the above is a cross-platform package manager for binaries written using Clojure, Windows support is coming soon (with the intent of not using Powershell)#2020-09-3021:52markbastianIs there a way to ignore the global ~/.m2/settings.xml when resolving repos? I have a situation in which my global file has a forced mirror that doesn't seem to be resolving snapshots correctly but if I just add the :mvn/repos to my deps.edn file and mv settings.xml settings.bak it works.#2020-09-3021:53markbastianBut I don't want to have to do that every time I want to sync my snapshots to my internal repo.#2020-09-3022:12Alex Miller (Clojure team)That’s maybe the best option right now#2020-09-3022:12Alex Miller (Clojure team)You could also set that repo id to nil in your deps.edn and then add a new one?#2020-09-3022:23markbastianI’ll give that a shot. I’ve also got another workaround in which I run the clj command in a docker image with the source and ~/.m2/repository mounted in the container. Thanks!#2020-10-0105:53deactivateduserApologies if this has been asked & answered before (I did search, but couldn’t find anything relevant), but in recent versions (specifically v1.10.1.697) the following occurs:
$ clj -e '(println "Hello, world")'
WARNING: When invoking clojure.main, use -M
Hello, world
$
I understand that this can be “fixed” with clj -M -e '(println "Hello, world")' , but is that intentional?#2020-10-0108:27practicalli-johnIt seems so. The -M flag is required when invoking clojure.main.#2020-10-0116:26deactivateduserThat seems like an unfortunate UX decision…#2022-10-0708:05octahedrionyes! it's a bit annoying. version 1.11.1.1165 still requires -M to eval in order to avoid the WARNING: Implicit use of clojure.main with options is deprecated, use -M
clj -M -e '(+ 1 2)' rather than clj -e '(+ 1 2)'
in this case the warning comes across as scornful even though you've nothing wrong#2022-10-1008:51practicalli-johnWithout the -M execution option, the clojure tool has to infer (guess) that clojure.main is the execution option to use. As there is more than one execution option to choose from, the tool designer has deprecated this inference approach, so an explicit option is used to tell the clojure tool should do. (-M, -X, -T, -P)
Depreciation is a common first step in removing an approach to using a tool, API, library, providing a clear migration step. In future this inference may no longer work and should not be relied upon.
The depreciation warning has been presented for a relatively long time now, so surprised that the clojure tool is used without specifying an execution option.#2022-10-1009:20octahedrionyes the warning makes sense from the tool designer's point of view#2022-10-1009:25practicalli-johnI am only a user of the tool and it makes sense to me.
I appreciate the deprecation warning approach and updated my use of the tool after first seeing this useful warning message.#2020-10-0113:09Alex Miller (Clojure team)yes, all invocations of clojure.main should now be routed through -M#2020-10-0113:10borkdudeeven when $ clj -- -e '(+ 1 2 3)' which is quite unambigious?#2020-10-0113:11Alex Miller (Clojure team)it's unambiguous now but we're trying to open up arg space for repl execution#2020-10-0116:28deactivateduserIt seems to me that evaluation of a something provided on the command line should be considered at the same level of importance as running a REPL or invoking a main function, FWIW.#2020-10-0117:16Alex Miller (Clojure team)expression eval is done by invoking clojure.main. -M is how you (now) invoke clojure.main.#2020-10-0117:21vlaaadHow are we going to invoke clojure.main in the future? :)#2020-10-0117:21Alex Miller (Clojure team)-M#2020-10-0117:22Alex Miller (Clojure team)where future = now#2020-10-0117:34vlaaadAh, I misunderstood your previous message#2020-10-0117:35Alex Miller (Clojure team)yeah, I meant vs previous, not vs future#2020-10-0117:34Alex Miller (Clojure team)clj was originally built solely around clojure.main. we are in the process of making it bigger than that and this is part of segmenting off that part of the functionality.#2020-10-0118:21deactivateduserI understand the improvements that brings to the implementation, I’m more making a comment about the (negative, imho) UX that creates, especially for newcomers.#2020-10-0118:21deactivateduseri.e. why should a newcomer have to know that in order to evaluate an expression on the command line, they are actually invoking clojure.main? That seems like incidental complexity to me…#2020-10-0118:22deactivateduserTo non-Java/Clojure folks, clj is clojure.main.#2020-10-0118:22Alex Miller (Clojure team)I don't think evaluating an expression on the command line is actually a thing most people do compared to running a repl#2020-10-0118:25deactivateduserSure, but that doesn’t mean that CLI evaluation isn’t worth supporting in a clear and concise (and backwards compatible, though I know we’re in a period of time where that is being consciously and deliberately broken) manner.#2020-10-0118:30Alex Miller (Clojure team)is your complaint about the warning message mentioning clojure.main?#2020-10-0118:30deactivateduserNo.#2020-10-0118:31Alex Miller (Clojure team)ok, then what is the complaint?#2020-10-0118:31deactivateduserMy complaint is why does anyone need to know they’re invoking clojure.main in order to evaluate something on the command line. Shouldn’t this be a “first class” operation, regardless of how it’s implemented under the hood?#2020-10-0118:32deactivateduserAfter all that’s how most other languages handle this - Principle of Least Surprise etc. etc.
$ python -c 'print("Hello, world")'
Hello, world
$ perl -e 'print "Hello, world"'
Hello, world
etc.#2020-10-0118:37Alex Miller (Clojure team)ok, thanks for the feedback#2020-10-0118:39deactivateduserAnd to be clear, I don’t do this all that often, but it is a handy feature (e.g. in shell scripts).#2020-10-0118:40Alex Miller (Clojure team)the most common use of it that I'm aware of is to trigger deps loading without doing something else, which you can now do with -P#2020-10-0122:19deactivateduserSure, but -P won’t evaluate an arbitrary expression. I like to (println a specific message to make build log searching easier.#2020-10-0118:38borkdude> I don't think evaluating an expression on the command line is actually a thing most people do compared to running a repl
I do this all the time on the command line#2020-10-0118:38borkdude-M -m -M foo.clj -M -e, don't know if I can get used it ;)#2020-10-0118:38Alex Miller (Clojure team)why vs running a repl?#2020-10-0118:40borkdudeA lot of things I do in my open source projects is testing out command line applications which are usually one shot. Yes, I also use a REPL, but actually more often I execute one time invocations#2020-10-0118:41Alex Miller (Clojure team)so is that -e or -m ?#2020-10-0118:41borkdudelet me just do a zsh history thing#2020-10-0118:42Alex Miller (Clojure team)(note new -P to prepare deps w/o executing, which is one prior use case for -e)#2020-10-0118:52borkdudeok, I wrote this horrible script now:
#!/usr/bin/env bash
history | bb -iO '(->> *input*
(map #(as-> % x (str/trim x) (str/split x #"\s+") (rest x) (take 2 x)))
(keep #(when (and (str/starts-with? (first %) "cl") (second %) (> (count (second %)) 1)) (subs (second %) 0 2)))
frequencies (sort-by second >) (take 10))'
["-A" 326]
["-S" 208]
["-e" 172]
["--" 79]
["-M" 76]
["/t" 30]
["-J" 22]
["-m" 18]
["-X" 12]
["-R" 8]
which indicates I use -e way more often on the command line#2020-10-0118:52borkdudebut so far I haven't used -M a lot, usually -A#2020-10-0118:53borkdudebecause instead of -m there's usually an alias for it. But with -e there isn't. So when invoking main directly, -e is my primary use case.#2020-10-0118:58Alex Miller (Clojure team)thanks for the data#2020-10-0119:03borkdudeI think clj /tmp/foo.clj is also one of my primary use cases, this is where ["/t" 30] probably comes from#2020-10-0119:03borkdudebut because the filenames can change, it doesn't really show well here#2020-10-0119:05seancorfieldI just checked my history, my dot-clojure file, and my deps.edn at work and I almost never use -e without also using -m it seems. Just as a contrary data point to the above.#2020-10-0119:06Alex Miller (Clojure team)thx#2020-10-0119:07borkdude$ alias cljm='clojure -M'
$ cljm -e "(+ 1 2 3)"
6
it works#2020-10-0120:29practicalli-johnFor my own brief experiences, I only every used -e once as a saw an example in the Clojure CLI docs. I am not aware of any use cases where I would want to evaluate clojure on the command line, especially if I can run the repl just as quickly. If I was going to script something with Clojure, then I'd try using Babashka, it looks great. I am fascinated as to what the scenarios could be for using the -e . If anyone has a link to examples, please let me know.
The only thing I find a little strange is when running an application from the command line, eg. clojure -M -m namespace.main , from a usability point of view the -M looks a little redundant.. After thinking about this for a few days, I realize I don't actually run the application that way. I'll either run in the REPL or create a jar/uberjar and run it that way. I though about using an alias called :project/run that ran the unpackaged project, which I could include in a project template (eg. for clj-new), but not sure how valuable that would be.
I find the -X flag more appealing (and more Clojurey) approach, as I've been writing command line -main functions to take hash-map as an argument for a while now. It seems an idiomatic approach to have all entry point functions in a project take a hash-map where possible.#2020-10-0120:31dpsuttonin fish i have these handy shell functions
function clj-doc --description 'Get the docstring of a function'
clj -e "(require 'clojure.repl)(clojure.repl/doc $argv[1])"
end
one clj-doc and the other clj-source. easy to update to -M -e though#2020-10-0120:31borkdudeWell, sometimes I use -e to require some namespace or a compile invocation for example#2020-10-0120:32borkdudeNo big deal, can be easily hidden in a script, Makefile or whatever, I just don't find -M -m aesthetically pleasing, which may not be an important argument. Whatever :)#2020-10-0122:20lreadFor what it is worth, I am updating rewrite-cljc to use the new cli syntax, and updating from -A -m to -M -m is not causing me any terrible discomfort. simple_smile#2020-10-0122:28markbastianOut of curiosity, what is the purpose of the sourceDirectory tag addition in the pom target? I don't see where it is used by other mvn tasks and the clojure-maven-plugin puts the same data into its own config (which I realize is unrelated to tools.deps, but it's the only mvn plugin I can think of that is directly related to building Clojure).
For example:
<build>
<plugins>
<plugin>
<groupId>com.theoryinpractise</groupId>
<artifactId>clojure-maven-plugin</artifactId>
<version>1.8.4</version>
<extensions>true</extensions>
<configuration>
<sourceDirectories>
<sourceDirectory>src</sourceDirectory>
</sourceDirectories>
</configuration>
</plugin>
</plugins>
<!-- How does maven use this? -->
<sourceDirectory>src</sourceDirectory>
</build>#2020-10-0122:33deactivateduserPretty sure Maven uses that to know where to go to build Java source. IOW it’s probably not relevant for a deployed artifact.#2020-10-0122:35markbastianAh, that makes sense - mixed language project.#2020-10-0517:13zetafishHi all, I try to create an uberjar with depstar using the -X method. I.e. i updated my deps.edn to have an alias specifying a exec-fn and exec-arg. When running on my laptop clojure -X:uberjar a jar is generated just fine. However when I try to create the jar from a docker container it fails with
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
-X:uberjar (No such file or directory)
The docker images I tried:
clojure:openjdk-11-tools-deps-1.10.1.469
clojure:openjdk-11-tools-deps-1.10.1.561
#2020-10-0518:17kirill.salykinwhat do you think about https://github.com/juxt/pack.alpha?#2020-10-0518:18kirill.salykinit has interesting feature skinny jar#2020-10-0519:54zetafishI want a self contained jar that I can run with java -jar app.jar so skinny does not fit the bill.#2020-10-0517:20Alex Miller (Clojure team)you don't have the newest clj tools in your docker#2020-10-0517:21Alex Miller (Clojure team)you need clojure 1.10.1.697#2020-10-0519:52zetafishThank you Alex. I presume the 697 image will become available on docker hub soon. In the meantime I built the image locally and the -X:uberjar works fine then.#2020-10-0520:01Alex Miller (Clojure team)I don't have anything to do with the docker stuff, so no control over that for me#2020-10-0517:57seancorfield@zetafish I'm planning an overhaul of my project's readme files this week to make that clearer (that some of the usage examples require 1.10.1.697). I also need to update all of the clj-new project stuff to generate projects that can work both ways.#2020-10-0518:19dominicmIs clojure.libfile still present in 697?#2020-10-0518:27seancorfieldI don't see it in the system properties when running code via 1.10.1.697 @dominicm -- there's clojure.basis now, which is the path to the combined EDN file.#2020-10-0518:28seancorfieldI see it was still present in 590:
> /usr/local/Cellar/clojure\@1.10.1.590/1.10.1.590/bin/clojure
Clojure 1.10.1
user=> (System/getProperty "clojure.libfile")
"/Users/sean/.clojure/.cpcache/2731801087.libs"
user=>
#2020-10-0518:28dominicmAh, that could explain the bug I'm seeing. I thought it was being retained for backwards compatibility.#2020-10-0518:59seancorfield@dominicm Lots of breakage between the previous stable and the current stable 🙂 but if you change (System/getProperty "clojure.libfile") to (some-> (System/getProperty "clojure.basis") (str/replace #"\.basis$" ".libs")) you can still get at the old file (although, as I recall, the basis file now contains the libs list as an element?)#2020-10-0519:01seancorfieldYeah, (some-> (System/getProperty "clojure.basis") (slurp) (edn/read-string) :libs) should be the same data.#2020-10-0519:50Alex Miller (Clojure team)yeah, I removed it in 697. we've never publicly doc'ed it and I was not aware of anyone using it (other than add-libs which won't work with latest anyways) so I didn't think there was any reason to retain it#2020-10-0519:51dominicmI was using it to find myself on the classpath and remove my paths#2020-10-0520:00Alex Miller (Clojure team)I'd say separating the tool from the basis is the better long term path for that#2020-10-0520:01Alex Miller (Clojure team)we have an ongoing conversation about some details of that right now#2020-10-0520:21dominicmIf I could reliably identify my lib in the classpath, then that'd be fine 🙂. Happy to switch over to the newnew system when that comes out too.#2020-10-0520:21dominicmA github search indicates I'm probably the only user anyway#2020-10-0520:21dominicmWe also use it in our internal tool based on the lib, but I can update that the same way#2020-10-0521:20seancorfieldThe problem with that (getting a file from the command line) is that the desired order for repos such as ours is:
System -> User (happy to exclude with -Srepro) -> repo-master (this is missing) -> Project -> -Sdeps
and because "last one wins" etc for some combinations of aliases, you can't always specify "repo-master` via some sort of -Sdeps-file option.#2020-10-0521:24seancorfieldAlthough, if you could specify an arbitrary list of files to read in order via -Sdeps-file (and they didn't have to be called deps.edn) then we could use -Sdeps-file ../versions/master-deps.edn:project-deps.edn and not have an actual deps.edn file in each subproject in the repo.#2020-10-0521:25borkdude-Sdeps-path might be a more appropriate name and I'd be happy to support it in deps.clj (and change the name). It already does this now but for just one file#2020-10-0521:27borkdude-Sdeps-overrides could also work as a name to indicate that it doesn't read any other files than these#2020-10-0521:29borkdudebut I'm relatively sure all of this can be done using scripting as well:
merge the configs yourself, provide it via -Sdeps and move the local deps.edn aside (hacky)#2020-10-0521:31borkdudealso I'm relatively sure these things are in some row/column in an excel sheet somewhere on Alex's computer :)#2020-10-0521:37seancorfieldOur deps files are pretty big -- too big to go on the command line I think as merged data -- and, as you say, that's a hacky approach.#2020-10-0521:39seancorfield(also, we're not about to switch from the official Clojure CLI to a third party tool at this point, esp when we would need multiple different binaries for different parts of our dev/test/staging pipeline, whereas we can have just one CLI install)#2020-10-0521:41seancorfieldAnd, yeah, I've already brought this up with Alex (which was how I learned that it's similar to something the Datomic Ions users have asked for) so, yeah, I'll bet it's already in a spreadsheet. I don't know what I'd prefer in terms of a solution. As long as it addresses the deps ordering shown above and is officially supported, I'll be happy with whatever Cognitect/Nubank think is the best approach 🙂#2020-10-0521:43borkdude@seancorfield small details: deps.clj also runs using clojure / JVM as a dependency but yeah I don't expect you to switch - I was just thinking out loud about this option. I'm blathering too much.#2020-10-0522:01seancorfieldAh, true, I'd forgotten there was a library version as well...#2020-10-0522:01borkdudeI just pushed 0.0.10 - there was something with the clojars deploy token, missed that. But now it works:
/tmp $ clojure -Sdeps '{:deps {borkdude/deps.clj {:mvn/version "0.0.10"}}}' -M -m borkdude.deps -Sdeps-file "$HOME/git/clojurescript/deps.edn" -Spath
src/main/clojure:src/main/cljs:resources:/Users/borkdude/.m2/repository/com/cognitect/transit-clj/0.8.309/transit-clj-0.8.309.jar:/Users/borkdude/.m2/repository/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar:...#2020-10-0522:02borkdude(so I'm in /tmp, but I'm using deps.edn from clojurescript in $HOME/git/clojurescript)#2020-10-0523:47Alex Miller (Clojure team)Just putting this here for now, but there is a new prerelease clj available - 1.10.1.708. Has a small fix to -X error handling and more importantly an update on how dep expansion works when multiple transitive versions of the same dep are in the tree to fix a case where a dep can be incorrectly omitted.#2020-10-0715:42Alex Miller (Clojure team)anyone tried this?#2020-10-0716:45seancorfieldI'm about to update things here...#2020-10-0718:52seancorfieldBuilding our uber JARs for our 14 processes seems to produce near-identical results between .697 and .708 -- there are variations from run-to-run based (I assume) on differing timestamps etc but nothing more than 150 bytes on JAR files that range from 30MB to 70MB, so that all looks reasonably sane.#2020-10-0718:53seancorfield(if dependencies were missing we'd have known about it by now on .697 and 150 bytes here and there isn't likely to indicate an entire dependency coming or going)#2020-10-0718:59Alex Miller (Clojure team)yeah, shouldn't be anything different other than the set of jars selected#2020-10-0523:49Alex Miller (Clojure team)The circumstances for the latter case are quite complicated so it's a little challenging for me to categorically say whether your deps are affected (they probably aren't), but if you were were interested in trying it, would be happy to have any feedback before I make it stable.#2020-10-0715:42Alex Miller (Clojure team)anyone tried this?#2020-10-0711:58timofor transitioning to clojure tools I am missing the lein install . How does one install a lib to local repository with clojure tools?#2020-10-0711:59dominicm@timok you don't really need to, you can reference it using :local/root#2020-10-0712:00timo😆 easy as that. thanks @dominicm#2020-10-0712:32practicalli-john@timok as mentioned local/root is very useful. clojure -X:deps mvn-install is a built-in Clojure CLI alias to deploy a Jar of the project locally in the ~/.m2/repository directory#2020-10-0712:37dharriganI find local/root very useful too. I pull down a 3rd party library, do some changes, and without having to deploy it anywhere, I can reference it in my own project by using the local/root structure to point to its sources.#2020-10-0723:00kennyOn 1.10.1.697, if I try to "https://clojure.org/reference/repl_and_main#_launching_a_script," I get the warning "WARNING: When invoking clojure.main, use -M." Is this correct?#2020-10-0723:16Alex Miller (Clojure team)yes, there have been changes in the clj arguments in 697#2020-10-0723:16Alex Miller (Clojure team)those examples are (now) out of date, will fix#2020-10-0723:16Alex Miller (Clojure team)forgot they were there!#2020-10-0815:41rickmoynihanDoes anyone know if https://clojure.atlassian.net/browse/TDEPS-163 is likely to be implemented any time soon?#2020-10-0815:44Alex Miller (Clojure team)probably not, but I did think more about it - the time when it's read by clj is in tools.deps.alpha, which is the process that builds the classpath. at that point you don't have any deps on the classpath (b/c that's what you're making). so it is more complicated than it initially appeared.#2020-10-0815:45ghadicatch 22#2020-10-0815:45rickmoynihanahh so the data-readers aren’t even available yet?#2020-10-0815:45Alex Miller (Clojure team)no#2020-10-0815:46Alex Miller (Clojure team)tools.deps could do what's in the ticket, and that would open that up for library use of tools.deps. but it won't help you via clj.#2020-10-0815:48rickmoynihannot sure I understand. Would it not be possible to read it with a :default reader in the first pass, and then re-read it later when you have that info?#2020-10-0815:57Alex Miller (Clojure team)no? you would need to start a whole new jvm to do that#2020-10-0815:52borkdude@alexmiller With regards to env vars your reasoning was: the less tools have to know, the better it is. Doesn't that also go for reader tags?#2020-10-0815:55borkdudewhen opening up reader tags, one could implement one which reads env vars (or wait, it was resolving your home dir), which brings the same trade-offs maybe#2020-10-0815:57Alex Miller (Clojure team)yeah#2020-10-0816:00rickmoynihanYeah I guess it makes sense not to do it then, thanks for considering it though 🙇#2020-10-0816:08vlaaadwhat if reading tags is done with tagged-literal?#2020-10-0816:09vlaaadand then basis can be read with proper reader tags when there is a classpath..#2020-10-0816:10vlaaadWhat’s the problem reader tags are supposed to solve though?#2020-10-0816:14rickmoynihangiving a data representation to otherwise un(print|read)able things.#2020-10-0816:24rickmoynihanSo my use case is essentially:
I’m using integrant to define systems of components.
Those systems are typically wired together in a base.edn file on the classpath. That file contains a map of integrant components.
However it would be useful to override some parts (key/values) in the integrant system via the command line, and start the thing with clojure -X
Or in prodution allow a user to specify their own profiles as :exec-args data in the deps.edn file directly.
One of the things you may want to override though is an #ig/ref which basically states this component depends on this other component. My current example is a program that I want to communicate with two different deployments of the same system. One of which is authenticated via an auth0 provider, the other is identical but authenticated via a basic auth provider.
I was hoping tagged reader support in the cli tooling would help with this; but it seems I’ll need to implement something to do this. It’s not too complicated; but it’s something I could use in a lot of projects, so I was hoping I could just leverage the cli tools rather than reimplement it each time; or require a library function to do it.#2020-10-0816:50Alex Miller (Clojure team)You can read the deps.edn yourself and have any data you like in the aliases#2020-10-0909:00rickmoynihanI’d be happy to do that, but it would also be useful if the cli tools didn’t choke on it:
{:aliases {:reader {;;:exec-fn foo/bar
:exec-data {:tag #foo [some stuff]}}
}}
$ clj -X:reader
Error building classpath. Error reading edn. No reader function for tag foo (/Users/rick/tmp/deps.edn)
If they could set :default to clojure.core/tagged-literal this way would I think work for me.
Is that what you mean by this?
> A good problem would be to avoid any error like that while computing the classpath from the deps.
> That I would be happy to do#2020-10-0909:01rickmoynihanahh also seen @U47G49KHQ appears to be suggesting this :thumbsup:#2020-10-0909:02vlaaadyep, that’s what I suggested#2020-10-0909:03rickmoynihanYeah it’s also what I actually meant here:
https://clojurians.slack.com/archives/C6QH853H8/p1602172111259500#2020-10-0909:04rickmoynihanThough I see now it wasn’t clear#2020-10-0913:03Alex Miller (Clojure team)I actually made this change yesterday in tools.deps https://github.com/clojure/tools.deps.alpha/commit/17f036429b28537c5bb83878cea1c4322fe21d15#2020-10-0913:04Alex Miller (Clojure team)Oh, I didn’t change it for -X param reading though, I’ll get that too before the next release#2020-10-0913:15vlaaadoh, that’s great!!!#2020-10-0913:29vlaaad#clj/requiring-resolve foo.bar/baz here I come#2020-10-0913:30rickmoynihanAhh fab! Thanks @alexmiller 🙇#2020-10-0913:44vlaaadOr bringing #= to edn with #clj/eval (some-code)#2020-10-0816:51Alex Miller (Clojure team)A good problem would be to avoid any error like that while computing the classpath from the deps.#2020-10-0816:51Alex Miller (Clojure team)That I would be happy to do#2020-10-0816:51vlaaadHmm, agree, supplying non-data args via -X is complicated. My api functions in Reveal are supposed to only accept functions as arguments, but to make them available from clj-exec, I have to accept symbols as well and introduce polymorphism just to satisfy clj-exec: https://github.com/vlaaad/reveal/blob/master/src/vlaaad/reveal.clj#L11#2020-10-0816:52vlaaadIf deps supported tagged literals, I wouldn't have to pollute my api#2020-10-0816:53ghadithere is no straightforward way to do it. I wouldn't gate yourself on deps#2020-10-0817:48dharriganI'm curious - what is the caching strategy used for looking up dependencies - I know that a particular dependency has been updated, it shows on clojars, but referencing it as 'RELEASE' in the deps.edn (my main deps.edn, i.e., $HOME/.clojure/deps.edn) I can see that it it's not being pulled down.#2020-10-0817:48hiredmanuse real versions#2020-10-0817:49borkdude@dharrigan it's a simple string hash on the user deps.edn, args and local deps.edn. Use -Sforce to update#2020-10-0817:50dharriganSure, I could do that - but I see for example in the popular seancorfield deps.edn, a lot of references to RELEASE as well. I mean it's not wrong to use RELEASE, it is supported. So surely, something must eventually attempt to find out if it is the latest release or not?#2020-10-0818:36seancorfield@dharrigan The only reason that some things in there use "RELEASE" is so that folks can "guarantee" a recent version of development tools when they first lift aliases from that file (or use the file whole). @hiredman is correct that this is "wrong" insofar as it breaks CLI caching and as @alexmiller says, this is not officially supported by the CLI/t.d.a so it's certainly not a good idea for anything other than quick access to dev tooling. I stopped using it in all the templates baked into clj-new a while back, although it is still used when looking up templates (on Clojars/Maven) to try to get the most recent stable release of a template -- and boot-new and lein new before it all use that same approach, with the same flaws: you can always use the -V / --version option to fetch a specific version of a template if you want/need. I am planning to make a pass over depstar and clj-new to make -X the favored way to invoke those now, including in the generated app template from clj-new, and at that point I'll also make a pass over my dot-clojure repo and start pinning versions, instead of "RELEASE", to ensure folks get a "recent-enough" version of things that actually support -X. Unfortunately, this sort of change is a big ball of yarn that causes a lot of stuff to need unraveling...#2020-10-0817:50hiredmanit is wrong#2020-10-0817:51borkdudeRELEASE is something that maven supports, but it's not encouraged by deps.edn, it just works because it works in maven. They will deprecate it.#2020-10-0817:51hiredmanRELEASE is a mutable reference#2020-10-0817:51hiredmanbuilding caching on top of that is terrible, and clj relies on caching#2020-10-0817:52dharriganOkay, fairy enough. I will use the version, not release 🙂#2020-10-0817:52hiredmanRELEASE is an easy vs. simple thing where people consistently choose what is easy#2020-10-0817:53dharriganIndeed 🙂 We're all lazy 🙂#2020-10-0817:53dharrigan$deity knows I didn't become a programmer to be active 🙂#2020-10-0817:53Alex Miller (Clojure team)RELEASE is not officially supported by clj#2020-10-0817:53Alex Miller (Clojure team)use at your peril (it breaks the cache)#2020-10-0817:54dharrigantbf, it (this library) was the only one in my deps.edn that has release for the coordinate#2020-10-0817:54Alex Miller (Clojure team)imo you should only ever use it in -Sdeps on the command line, not in a deps.edn file. I reserve the right to enforce that opinion at an arbitrary future time :)#2020-10-0817:55borkdudeit's on the list for clj-kondo warnings here: https://github.com/borkdude/clj-kondo/issues/945#2020-10-0817:56dharriganSomething else, does deps use the .m2/settings.xml at all?#2020-10-0817:56Alex Miller (Clojure team)yes, for some things (like server creds, mirrors, proxies)#2020-10-0817:56dharriganright right, good. thanks 🙂#2020-10-0817:56Alex Miller (Clojure team)see https://clojure.org/reference/deps_and_cli#_maven#2020-10-0817:56hiredmanproxies for fetching maven deps, not for git deps?#2020-10-0817:57Alex Miller (Clojure team)yes, that is still a todo#2020-10-0817:57Alex Miller (Clojure team)there is a ticket for it, iirc#2020-10-0817:57Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-124#2020-10-0817:58borkdudedeps.clj has support for http / https proxy settings: https://github.com/borkdude/deps.clj - not sure if that's what you mean
https://github.com/borkdude/deps.clj#proxy-environment-variables#2020-10-0818:01borkdudeI think it works in deps.clj because the proxy settings are also passed to the make-classpath invocation#2020-10-0818:02Alex Miller (Clojure team)that would make sense#2020-10-0818:02borkdudehttps://github.com/borkdude/deps.clj/blob/02144768876f68445ed2c04cbf90c0b85b0e7d06/src/borkdude/deps.clj#L393#2020-10-0818:12dpsuttoni can get the version from clojure by running clojure --help but it prints with lots of other text obviously. is there a way to get just the version?#2020-10-0818:13dpsuttonwe have a script that invokes -m and wondering whether its better to omit the -M and get a warning or include it and break usage for people who are on older versions#2020-10-0818:14Alex Miller (Clojure team)clj -Sdescribe will return edn content that includes a :version key#2020-10-0818:14borkdude@dpsutton
$ clojure -Sdescribe | jet --query :version
"1.10.1.697"
#2020-10-0818:15Alex Miller (Clojure team)note that the warnings are on stderr so you can redirect stderr to null too if you want to not see those#2020-10-0818:16borkdude$ clojure -Sdescribe | clojure -M -e "(require '[clojure.edn :as edn]) (:version (edn/read *in*))"
"1.10.1.697"
#2020-10-0818:16borkdude$ clojure -Sdescribe | bb '(:version *input*)'
"1.10.1.697"
ok, I'm done now :)#2020-10-0818:16Alex Miller (Clojure team)heh#2020-10-0818:17Alex Miller (Clojure team)but I have all these cat skins lying about...#2020-10-0818:18borkdudek, one more:
$ clojure -Sdescribe | grep :version | cut -d' ' -f2
"1.10.1.697"
#2020-10-0818:25lread@dpsutton, I’ve moved to the new clojure command line syntax for rewrite-cljc. I have its scripts fail with an error message if clojure version < 1.10.1.697. My scripts are all in babashka so the check was easy.#2020-10-0818:30dpsuttonthanks everyone#2020-10-0818:36seancorfield@dharrigan The only reason that some things in there use "RELEASE" is so that folks can "guarantee" a recent version of development tools when they first lift aliases from that file (or use the file whole). @hiredman is correct that this is "wrong" insofar as it breaks CLI caching and as @alexmiller says, this is not officially supported by the CLI/t.d.a so it's certainly not a good idea for anything other than quick access to dev tooling. I stopped using it in all the templates baked into clj-new a while back, although it is still used when looking up templates (on Clojars/Maven) to try to get the most recent stable release of a template -- and boot-new and lein new before it all use that same approach, with the same flaws: you can always use the -V / --version option to fetch a specific version of a template if you want/need. I am planning to make a pass over depstar and clj-new to make -X the favored way to invoke those now, including in the generated app template from clj-new, and at that point I'll also make a pass over my dot-clojure repo and start pinning versions, instead of "RELEASE", to ensure folks get a "recent-enough" version of things that actually support -X. Unfortunately, this sort of change is a big ball of yarn that causes a lot of stuff to need unraveling...#2020-10-0818:38dharriganw00t. It will certainly help those coming fresh to Clojure and seeing your excellent deps.edn example.#2020-10-0818:42seancorfieldOr the Practicalli example (nicer web site, more descriptive names, much better documentation). I'm considering linking to it from my dot-clojure repo and encouraging new folks to use the Practicalli stuff instead -- so I'm under less pressure to keep dot-clojure maintained and documented 🙂#2020-10-0821:29practicalli-john@seancorfield I would be honoured to be linked from your repo. Thank you.
The library versions used in the Practicalli aliases are updated at least once a month using the :project/outdated alias and version changes added to the changlog.
Versions are updated sooner if I spot a new version or a library maintainer let's me know.#2020-10-0900:29seancorfieldhttps://github.com/seancorfield/dot-clojure has been updated with caveats about it being my personal deps.edn file and to direct people to your repo if they want something well-documented and well-maintained!#2020-10-0821:39seancorfieldseancorfield/depstar {:mvn/version "1.1.126"} -- https://github.com/seancorfield/depstar/releases/tag/v1.1.126 -- just three small bug fixes; follow-up in #depstar#2020-10-0822:00seancorfieldI was planning to make -X the preferred way of using depstar in that release but I realized that -X:uberjar ends up putting the Clojure CLI's version of tools.deps.alpha into the JAR file -- because depstar assumes the "current classpath" is what you want in the JAR (and it excludes itself, assuming it is the only dependency you don't want in the JAR). -X:jar works just fine (because it excludes JAR files). I'll need to modify depstar to use t.d.a directly to build a classpath from deps.edn files (per the discussion the other day) but that gets complicated if you need a "user" deps.edn file and you're using CLJ_CONFIG with a monorepo -- as we are at work -- so that will have to wait for "depstar 2". Given the discussions around that and t.d.a itself recently in this channel, I'll probably defer that work until t.d.a has better support for that (since CLJ_CONFIG is part of the clojure script, not t.d.a).#2020-10-0822:01seancorfieldBTW, CLJ_CONFIG is documented officially (I thought it wasn't):
User - cross-project configuration (typically tools)
Locations used in this order:
If $CLJ_CONFIG is set, then use $CLJ_CONFIG (explicit override)
https://clojure.org/reference/deps_and_cli#_deps_edn_sources#2020-10-0822:13hugodIs it possible to ignore a project’s deps.edn when invoking clojure cli? It would sometimes be useful when passing -Sdeps, e.g. in a tool that bumps a project’s version, or builds docs.#2020-10-0822:14seancorfield@hugod Yes, that's what :replace-deps is for.#2020-10-0822:15seancorfieldSee https://clojure.org/reference/deps_and_cli#_replace_project_environment_tool#2020-10-0822:15hugodthank you - I was looking for a command line switch#2020-10-0822:16seancorfieldYou can use it in -Sdeps but you'd need an alias I believe.#2020-10-0900:31seancorfieldseancorfield/clj-new {:mvn/version "1.1.226"} -- https://github.com/seancorfield/clj-new/releases/tag/v1.1.226 -- preferred usage is via -X; templates updated to reflect -A/`-M`/`-X` changes in recent Clojure CLI versions; follow-up in #clj-new#2020-10-0919:00seancorfieldSigh... seancorfield/depstar {:mvn/version "1.1.128"} and seancorfield/clj-new {:mvn/version "1.1.228"} are available now to fix the bug I introduced in depstar 1.1.126 in license file handling and for clj-new to use the fixed version of depstar; follow-up in #clj-new and/or #depstar (sorry for anyone who used yesterday's broken versions!)#2020-10-1020:36borkdudeMade a small (proof of concept) tool to rewrite EDN files while preserving formatting, whitespace, comments, and reader ignores.
A few examples that might be handy for deps.edn: https://github.com/borkdude/rewrite-edn#examples#2020-10-1113:59Eamonn SullivanHi all. I have a real newbie question on deps.edn. I have the following to add cloverage to my project.
:cloverage
{:extra-paths ["test"]
:extra-deps {cloverage/cloverage {:mvn/version "RELEASE"}}
:main-opts ["-m" "cloverage.coverage"
"-p" "src"
"-s" "test"
"--fail-threshold" "90"]}
The cloverage docs say I should be able to provide a map of options somewhere, like using :fail-threshold instead of the args. How would I do that?#2020-10-1114:16Alex Miller (Clojure team)I don’t think this is a deps.edn question, not sure how cloverage works#2020-10-1114:18Eamonn SullivanAh, ok, sorry. I thought there was a key I could set like :options {:fail-threshold 90}. Told you it was a newbie question!#2020-10-1315:50tvaughan$ cat tests.edn
#kaocha/v1
{:plugins [:kaocha.plugin/cloverage
:noyoda.plugin/swap-actual-and-expected]
:cloverage/opts {:fail-threshold 100}}#2020-10-1315:51tvaughan@eamonn.sullivan ☝️#2020-10-1315:52Eamonn SullivanAh, thank you. Perfect!#2020-10-1114:19Eamonn SullivanYou can set project default settings for Cloverage in your project. That's all they mention in the docs and I have no idea that means, either. I'll just use the command-line arguments.#2020-10-1114:20Alex Miller (Clojure team)Yeah, don’t know offhand#2020-10-1114:27lread@eamonn.sullivan, I don’t know have active it is, but there is a #cloverage channel.#2020-10-1117:34practicalli-john@eamonn.sullivan kaocha test runner can also use cloverage too. I have aliases for both, but I don't think I used options yet. Will give them a try and interested to know how you get on
https://practicalli.github.io/clojure-webapps/#2020-10-1212:18borkdudeJust automatically qualified deps in a large deps.edn file (while preserving comments, etc):
https://github.com/borkdude/rewrite-edn/blob/master/examples/qualify_deps.clj#2020-10-1311:09orestisI’m seeing this in Github actions — when trying to do a -X — presumably some weirdness in Clojure version?
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
955
Could not locate clojure/run/exec__init.class, clojure/run/exec.clj or clojure/run/exec.cljc on classpath.
#2020-10-1311:25delaguardohow you setup clojure-cli for your action?#2020-10-1311:26orestisVia… your action 🙂#2020-10-1311:26orestisclojure -Stree gives me this
org.clojure/clojure 1.10.1
1089
org.clojure/core.specs.alpha 0.2.44
1090
org.clojure/spec.alpha 0.2.176
1091
org.clojure/tools.deps.alpha 0.9.816
#2020-10-1311:27orestisAnd clojure --help gives me
Version: 1.10.1.708
#2020-10-1311:27delaguardocould you change the version of my action?)
to this DeLaGuardo/#2020-10-1311:28delaguardonew jar (exec.jar) was introduced recently as a part of clojurecli installation
now it is fixed in master#2020-10-1311:29delaguardosorry for inconvenience (#2020-10-1311:30orestisoh 🙂#2020-10-1311:30orestisNo worries, I was just about to dive into the source code 🙂#2020-10-1314:57borkdudeScript to update pom.xml version, if that's useful to anyone:
https://github.com/borkdude/babashka/blob/master/examples/set_pom_version.clj#2020-10-1315:14flowthingCool! Guess you could also do mvn versions:set -DnewVersion=1.0.1, too, though?#2020-10-1315:15borkdudeShhh#2020-10-1315:15borkdudeHehe. I didn't know that. ^ @UME7B1EBV#2020-10-1315:16borkdudeI guess the script becomes more useful when you want to actually parse the version and bump it.#2020-10-1315:19flowthingCertainly. You might be able to do that with the Maven Release plugin, too, but I’m not sure.#2020-10-1319:39Søren SjørupThe maven version downloads everything on https://apache.org to do the job for me though 🙂#2020-10-1407:39flowthingSure, Maven downloading the entire internet before doing anything is a time-honored tradition. 🙂 It'll only do that the first time you run the command, though.#2020-10-1315:22Alex Miller (Clojure team)I think maybe I forgot to post it here, but I did a clj 1.10.1.716 prerelease this past weekend that includes a bump in many dependencies (maven resolver, aws api), and changes to make the edn reading more tolerant of unknown tagged literals in alias data#2020-10-1315:25seancorfieldI'll update my dev/test setup today to try it out, and integrate it into our work repo.#2020-10-1315:25Alex Miller (Clojure team)thx#2020-10-1319:20seancorfieldSo far so good. Now it's on our QA infrastructure and handling deps/building JARs all seems to be working just fine.#2020-10-1509:11rickmoynihanahh amazing I’ll make a note to give it a try at some point. Thanks 🙇#2020-10-1403:10zaneCan anyone think of a reason why someone might be getting USERAUTH fail when using clj to attempt to fetch a public repository via https? 😕#2020-10-1403:22Alex Miller (Clojure team)If you could share a repro that would help#2020-10-1403:24zaneSure thing.
clj -Sdeps '{:deps {probcomp/metaprob {:git/url "" :sha "8dc9d09f747c1e29886bb9628a0110c6f6bc6f5a"}}}'#2020-10-1403:24zaneWorks for me. Fails for my colleague.#2020-10-1403:24zaneI’ll get you the exact message.#2020-10-1403:25zaneHere’s what he’s seeing: https://a.cl.ly/llu2n0yR#2020-10-1403:26zaneI’m confused as to why it appears to be attempting to fetch via git ssh.#2020-10-1403:29Alex Miller (Clojure team)well it's a git dep#2020-10-1403:29seancorfieldFWIW, that command-line works fine for me on WSL2 (Ubuntu) on Windows 10 and also on macOS 10.12.#2020-10-1403:31seancorfieldCan't test it on Powershell since I don't have the Clojure CLI installed on Windows natively any more.#2020-10-1403:32Alex Miller (Clojure team)I agree though that it does look like it's trying to use ssh though#2020-10-1403:32seancorfield@alexmiller Where could the jgit stuff be getting auth from that might cause a failure with a public git repo?#2020-10-1403:32Alex Miller (Clojure team)it's not that that it's not getting auth but that it's looking for it in the first place#2020-10-1403:50zane> well it’s a git dep
Sorry, what I meant was: I’m confused that it appears to be trying to fetch via SSH.#2020-10-1403:51zanePerhaps I’m reading this output wrong?#2020-10-1403:52Alex Miller (Clojure team)is there any chance that they have a ~/.gitconfig with a url.<base>.insteadOf set?#2020-10-1403:53Alex Miller (Clojure team)that allows you to direct git to use a different url instead of the original one and can be used to swap https for ssh urls#2020-10-1403:53seancorfieldOr perhaps they have a deps.edn file in that directory that already specifies that library via {:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "f6919f82b6919f829e8394"}, :content ("[email protected]")}?#2020-10-1403:54Alex Miller (Clojure team)the -Sdeps is merged last so that shouldn't affect anything#2020-10-1403:54seancorfield(they're running that command in their downloads folder so who knows what is in there)#2020-10-1403:55zaneYeah, it’s possible.#2020-10-1403:55zaneI’ll ask.#2020-10-1403:55Alex Miller (Clojure team)should look for [url " and then an insteadOf line in /.gitconfig - something like that#2020-10-1403:56zaneGot it.#2020-10-1404:00Alex Miller (Clojure team)[url "
#2020-10-1404:00Alex Miller (Clojure team)I was able to repro with that#2020-10-1404:00zaneSeems like he’s signed off. I’ll check back in when I can get ahold of him again.#2020-10-1404:00zaneI see! Hopefully that’s the culprit.#2020-10-1404:00zaneThanks for the help!#2020-10-1404:01Alex Miller (Clojure team)could even be more generic like:#2020-10-1404:01Alex Miller (Clojure team)[url "git://"]
insteadOf = https://#2020-10-1404:01Alex Miller (Clojure team)can't say I've ever messed with that config before!#2020-10-1404:02zaneYeah, me either.#2020-10-1404:03zaneThe colleague having the issue is wanting to do some volunteer work on some of our private repositories, and I suspect he probably has a lot going on in his ~/.ssh/config from his job.#2020-10-1404:05zaneFrom talking to Ghadi earlier I know how to set up a new machine for use with clj + private repositories, but I’m not clear yet on how to configure a machine that has a lot of extant ~/.ssh/config settings.#2020-10-1404:06Alex Miller (Clojure team)well, the answer is probably ... it depends#2020-10-1404:07zaneFor instance, I know that because of JSch clj doesn’t play well with IdentityFile declarations, and should instead be set up to read keys from the ssh-agent, but I don’t think I can ask this person to change their ~/.ssh/config in a way that might impact their work setup.#2020-10-1404:08zane> it depends
Yeah, I suppose there’s no way out but through. I’m going to have to understand what their existing ~/.ssh/config settings are doing in detail.#2020-10-1418:36zaneTo close this loop, he had this in his ~/.gitconfig:
[url "
#2020-10-1418:37zaneThanks, @alexmiller and @U04V70XH6!#2020-10-1418:37seancorfieldWas he OK with removing that to get clj to work?#2020-10-1418:38seancorfield(I'm kind of curious as to why anyone would want all lein install with deps.edn?#2020-10-1419:29seancorfieldclojure -X:deps mvn-install#2020-10-1419:29seancorfieldhttps://clojure.org/reference/deps_and_cli#_local_maven_install#2020-10-1419:29seancorfield(you'll probably need to generate a pom.xml file first -- I can't remember)#2020-10-1420:07Alex Miller (Clojure team)no, you don't - you can either rely on the pom inside the jar or supply the g/a/v externally#2020-10-1420:07Alex Miller (Clojure team)or supply the pom#2020-10-1419:29seancorfieldOh, and that's assuming you're on 1.10.1.697 or later.#2020-10-1419:33dominicm@alexmiller how safe is it to rely on tdeps putting src/main/clojure on the classpath indefinitely?#2020-10-1419:34seancorfield@dominicm I didn't think it did? :paths ["src"] is in the system deps#2020-10-1419:36dominicmOops, via pom.xml*#2020-10-1420:06Alex Miller (Clojure team)I'm confused by what you mean#2020-10-1420:21dominicm@alexmiller there's always src/main/clojure added when doing a local/root to a pom.xml right now.#2020-10-1421:46Alex Miller (Clojure team)So local dep with pom.xml#2020-10-1421:46Alex Miller (Clojure team)Not planning to change that#2020-10-1501:07Jacob O'BryantThe new git-resolve-tags command appears to have a bug. Someone reported this to me today and I've been able to reproduce as well:
$ cat deps.edn
{:deps
{github-jacobobryant/biff
{:git/url ""
:tag "HEAD"}}}
$ clj -X:deps git-resolve-tags
Error building classpath. Library github-jacobobryant/biff has :tag but no :sha.
Add :sha or run `clj -Sresolve-tags` to update deps.edn.
$ clj -Sresolve-tags
Option changed, use: clj -X:deps git-resolve-tags
$ clj --help | head -n 1
Version: 1.10.1.697#2020-10-2021:24Alex Miller (Clojure team)fix for this will be in the next prerelease version, coming shortly#2020-10-2021:37Alex Miller (Clojure team)available in 1.10.1.723#2020-10-1503:03Alex Miller (Clojure team)heh, thanks for the report#2020-10-1900:27athomasoriginalIs there an equivalent of :local/root in the alternative tools e.g. lein or boot?
The reason for the question is because this makes debugging 3rd party library (say reagent) wildly easy. Just clone the lib, add it as a :local/root and then start editing the library as if it was part of your project. Seems like reason enough for Clojure Tools to be used over alternatives.#2020-10-1900:35seancorfield@tkjone I think that is what checkouts is for in Leiningen? https://github.com/technomancy/leiningen/blob/stable/doc/TUTORIAL.md#checkout-dependencies (I've never used that but it sounds a bit like using :local/root in deps.edn)#2020-10-1900:36seancorfieldThere's also a plugin for Leiningen that lets you specify your dependencies via a deps.edn which, presumably, would also serve you for this purposes I think...#2020-10-1900:37seancorfieldhttps://github.com/RickMoynihan/lein-tools-deps -- yes, it supports :local/root and :git/url, according to its README.#2020-10-1900:39seancorfield(`checkouts` would require that the checked-out library be a Leiningen project as well -- Rick's plugin should work for non-lein projects, but for transitive dependencies you'd need it to have pom.xml or deps.edn in it I think)#2020-10-1921:48grzmWas there a change recently (past month or so) in clojure.tools.deps that changed the order of the classpath? In clojure 1.10.1.507 I’m seeing clojure -Spath puts deps.edn :paths first (e.g., with {:paths ["src/main/clojure"] ,,,}, src/main/clojure is the first item in the classpath), and in clojure 1.10.1.645 I’m seeing :paths listed last.#2020-10-1921:50dpsutton> 0.9.799 on Sep 16, 2020- Fix classpath ordering for paths and libs
#2020-10-1921:51grzmCheers! Is there a quick way to correlate tools.deps versions other than looking at the change log?#2020-10-1921:52dpsuttoni'm not sure. i just remembered that change being discussed vaguely. sorry#2020-10-1921:52dpsuttoncorrelate versions to what?#2020-10-1921:52grzmNo worries, you already did more than I could do on my own 🙂#2020-10-1921:53grzmThe clojure-tools release and the clojure.tools.deps releases. Looking at the clojure -h on an installation, I see the former. I’d like to get at the latter more easily.#2020-10-1922:26Alex Miller (Clojure team)https://clojure.org/releases/tools#2020-10-1922:27Alex Miller (Clojure team)That’s kind of a consolidated release page for clj and tools.deps (versions of that included for reference) only for stable versions of clj#2020-10-2018:49grzmYeah, found that. I was hoping to have something similar to the output of clojure -h on the command line, to aid in debugging rather than referring to a webpage. That works, though.#2020-10-2018:54Alex Miller (Clojure team)you can tell by doing clj -A:deps -Stree | grep tools.deps.alpha#2020-10-2019:54grzmCheers!#2020-10-2019:55grzmwrites that down#2020-10-2000:22cch1I’m getting up to speed with the deploy process using tools from the greater tools.deps ecosystem and I ran into a strange issue today. Essentially, I was finding that libs installed locally did not bring their transitive dependencies along with them. I used clojure.tools.cli.api/mvn-install to do the install.#2020-10-2000:24cch1A bit of investigation shows that mvn-install does not copy the pom file to the local repo -interestingly @seancorfield has observed that slipset/deps-deploy does install the pom file. So is that a bug or a limitation/design decision in c.t.d. mvn-install?#2020-10-2000:24cch1@alexmiller ^#2020-10-2000:32seancorfield@alexmiller I confirmed with @cch1 that running clojure -X:deps mvn-install omits the .pom file and when trying to use that dependency in a clean folder, t.d.a. reaches out to Clojars to look for the "missing" .pom file. When I re-ran the same scenario but using slipset/deps-deploy for the local install, it did copy pom.xml to <library>.pom in ~/.m2 and when trying to use that in a clean folder, t.d.a. did not reach out to Clojars to look for the .pom file. Definitely feels like a bug in t.d.a's mvn-install to me.#2020-10-2000:43Alex Miller (Clojure team)can you back up to the full command you're running?#2020-10-2000:44Alex Miller (Clojure team)are you supplying a pom? using the one in a jar file? supplying additional attributes?#2020-10-2000:45seancorfieldThere is a pom.xml in the same directory. I ran just clojure -X:deps mvn-install :jar '"the-lib.jar"' -- do we need extra args to also copy the pom.xml file? It reads the pom.xml file to get the group/artifact/version data...#2020-10-2000:46seancorfieldThe same pom.xml file is inside the JAR as well (so that's the one it reads, based on the reference docs).#2020-10-2000:48seancorfieldAh, if I say :pom '"pom.xml"' as well, then it does copy that to <library>.pom -- is that an intentional difference @alexmiller?#2020-10-2000:53seancorfieldHere's a full repro session:
(! 692)-> rm -rf ~/.m2/repository/seancorfield/
(! 693)-> jar tvf next-jdbc.jar |fgrep pom
110 Mon Oct 19 11:24:06 PDT 2020 META-INF/maven/seancorfield/next.jdbc/pom.properties
1996 Mon Oct 19 11:24:06 PDT 2020 META-INF/maven/seancorfield/next.jdbc/pom.xml
(! 694)-> clojure -Sforce -X:deps mvn-install :jar '"next-jdbc.jar"'
Installing next-jdbc.jar
Installed to /Users/sean/.m2/repository/seancorfield/next.jdbc/1.1.610
(! 695)-> ls -lR ~/.m2/repository/seancorfield/
total 0
drwxr-xr-x 4 sean staff 136 Oct 19 17:51 next.jdbc
/Users/sean/.m2/repository/seancorfield//next.jdbc:
total 8
drwxr-xr-x 4 sean staff 136 Oct 19 17:51 1.1.610
-rw-r--r-- 1 sean staff 305 Oct 19 17:51 maven-metadata-local.xml
/Users/sean/.m2/repository/seancorfield//next.jdbc/1.1.610:
total 96
-rw-r--r-- 1 sean staff 164 Oct 19 17:51 _remote.repositories
-rw-r--r-- 1 sean staff 44300 Oct 19 11:24 next.jdbc-1.1.610.jar
(! 696)-> rm -rf ~/.m2/repository/seancorfield/
(! 697)-> clojure -Sforce -X:deps mvn-install :jar '"next-jdbc.jar"' :pom '"pom.xml"'
Installing next-jdbc.jar and pom.xml
Installed to /Users/sean/.m2/repository/seancorfield/next.jdbc/1.1.610
(! 698)-> ls -lR ~/.m2/repository/seancorfield/
total 0
drwxr-xr-x 4 sean staff 136 Oct 19 17:52 next.jdbc
/Users/sean/.m2/repository/seancorfield//next.jdbc:
total 8
drwxr-xr-x 5 sean staff 170 Oct 19 17:52 1.1.610
-rw-r--r-- 1 sean staff 305 Oct 19 17:52 maven-metadata-local.xml
/Users/sean/.m2/repository/seancorfield//next.jdbc/1.1.610:
total 104
-rw-r--r-- 1 sean staff 188 Oct 19 17:52 _remote.repositories
-rw-r--r-- 1 sean staff 44300 Oct 19 11:24 next.jdbc-1.1.610.jar
-rw-r--r-- 1 sean staff 1996 Oct 19 11:16 next.jdbc-1.1.610.pom
(! 699)-> ls -l pom.xml
-rw-r--r-- 1 sean staff 1996 Oct 19 11:16 pom.xml
#2020-10-2000:54seancorfieldI see Installing next-jdbc.jar in the first case and Installing next-jdbc.jar and pom.xml in the second case so I suppose it is telling you what it is doing...#2020-10-2000:56seancorfieldWithout the .pom file in the repo, attempts to use the dependency will still reach out to Clojars/Maven for that .pom file and if it isn't found, it doesn't look like the pom.xml inside the JAR is sufficient for t.d.a to pick up the dependencies.#2020-10-2000:59Alex Miller (Clojure team)it will only use the pom if you pass it explicitly (doc string: https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/mvn-install)#2020-10-2001:01Alex Miller (Clojure team)reaching out to maven is surprising though, so I'll look at that tomorrow#2020-10-2001:03seancorfieldDownloading: seancorfield/next.jdbc/1.1.610/next.jdbc-1.1.610.pom from clojars
^ That's what I get trying to use the installed dependency without the .pom file in the repo. If I install the pom.xml file as well (or use slipset/deps-deploy install) so that I get the .pom file in the repo, that line does not appear when trying to use the dependency.#2020-10-2001:03seancorfieldFeel free to DM me tomorrow if you need me to run more test cases.#2020-10-2001:03seancorfield(and there was no attempt to download the .jar file, only the .pom file)#2020-10-2001:04Alex Miller (Clojure team)sorry, maven looking for the pom file is normal behavior, I thought you were saying it was reaching out for the pom file when installing#2020-10-2001:05seancorfieldRight, but that will fail if there is only a local install -- and the problem Chris was seeing was that if no .pom file existed, transitive dependencies were not being found.#2020-10-2001:06Alex Miller (Clojure team)well that's to be expected#2020-10-2001:06Alex Miller (Clojure team)the question is what you supply on install#2020-10-2001:06Alex Miller (Clojure team)you have to either supply a pom, or use the pom in the jar, or is this the case where you did neither?#2020-10-2001:07seancorfieldThere is a valid pom.xml inside the JAR -- I showed that above.#2020-10-2001:07Alex Miller (Clojure team)sorry, too much reading, and I'm a drink into the night :)#2020-10-2001:07seancorfieldThe problem is that the .pom file is not created in the local repo on the install. Unless you explicitly specify :pom '"pom.xml"'#2020-10-2001:08Alex Miller (Clojure team)ok, that's surprising to me, should be using the one in the jar in that case#2020-10-2001:08seancorfieldOK, we can follow-up tomorrow. Feel free to DM me about it.#2020-10-2001:08seancorfieldSee my console session above.#2020-10-2001:23Alex Miller (Clojure team)yeah, that code is not there. it is using the pom in the jar to pull G/A/V but it's not actually installing it as a distinct artifact. not sure why I thought that would happen automatically, maybe tricked myself with some caching or something#2020-10-2001:23seancorfieldSo, definitely a bug? Want me to create a JIRA?#2020-10-2001:24Alex Miller (Clojure team)definitely a bug, sure on jira#2020-10-2001:24Alex Miller (Clojure team)should be easy to fix#2020-10-2001:28seancorfieldhttps://clojure.atlassian.net/browse/TDEPS-169#2020-10-2001:42seancorfieldseancorfield/depstar {:mvn/version "1.1.132"} -- calls shutdown-agents at the end to account for "badly-behaved" code being AOT'd that has side-effecting top-level forms that start threads (e.g., Neanderthal) -- follow-up in #depstar#2020-10-2001:52cch1Sorry I’m late catching up here. I don’t have much more to offer than what Sean noted above. Let me know if I can help test.#2020-10-2021:36Alex Miller (Clojure team)Clojure tools 1.10.1.723 (a prerelease) is now available:
• Fix clj -X:deps tree adding tools.deps.alpha to tree
• Fix clj -X:deps mvn-pom adding tools.deps.alpha to pom deps
• Fix clj -X:deps git-resolve-tags not working
• Fix clj -X:deps mvn-install on jar to also install embedded pom
#2020-10-2021:47Alex Miller (Clojure team)the first 3 were actually all a bug in the default :deps alias using :extra-deps rather than :replace-deps (duh)#2020-10-2114:03Alex Miller (Clojure team)if anyone tries that prerelease and has good/bad feedback, please let me know, I will probably release it as stable by tomorrow#2020-10-2116:35seancorfieldI updated our repo at work to use it immediately but I haven't exercised it much yet (and in particular I haven't exercised the options you fixed yet!). I'm just about to update our "ancient" script which uses -Stree to use -X:deps tree instead and see if it still works correctly...#2020-10-2116:36seancorfieldOh, I misspoke. Apparently I made that change a while back so I have already tested that it didn't break anything...#2020-10-2116:37seancorfield...and I can confirm that all the maven-resolver-* deps disappeared, as expected. So, yup, that worked.#2020-10-2116:44seancorfieldOK, I just ran clojure -Spom across a dozen projects and then clojure -X:deps mvn-pom and confirmed there's no differences between those two now.#2020-10-2116:49Alex Miller (Clojure team)there is still a difference in that your command line classpath-modifying arguments won't change the generated pom with the latter#2020-10-2116:50seancorfieldRight, I don't rely on that in any of the projects where I create pom.xml files at the moment.#2020-10-2116:51seancorfieldBut, yeah, we couldn't use this in our monorepo.#2020-10-2116:52Alex Miller (Clojure team)still a TBD but I think we will probably have some semi-standard set of args that you can use to do that on mvn-pom if desired (or in tree). not sure yet whether that is args that mirror clj switches or pass through of some kind that gets parsed etc.#2020-10-2116:56seancorfieldThat did cause me to notice that
CLJ_CONFIG=../versions clojure -A:defaults -Spom
and
CLJ_CONFIG=../versions clojure -R:defaults -Spom
behave differently. The latter works, in that it merges the :defaults alias (with all our :override-deps) into the lib map which -Spom sees but the former fails to do that.#2020-10-2116:58seancorfieldWith -A:defaults I get this error
Error generating pom manifest: No coordinate type found for library compojure/compojure in coordinate {}
because that's how we specify common deps that have :override-deps in the :defaults alias in our "master" deps.edn file.#2020-10-2118:21Alex Miller (Clojure team)I can see why that is - the gen script piggybacks the normal cp script and one of the arg names changed in the latter so it's getting missed#2020-10-2118:21Alex Miller (Clojure team)at the time that got changed, the gen script had been removed so didn't show up in the refectoring but was later re-added#2020-10-2118:38Alex Miller (Clojure team)@seancorfield fixed in 1.10.1.727#2020-10-2118:39seancorfieldI will update to that shortly 🙂#2020-10-2119:21seancorfieldUpdated. Confirmed fixed. Thanks @alexmiller#2020-10-2210:05practicalli-johnShould the -M flag for Clojure CLI tools have an issue when also supplying -Sdeps options? (the -A flag works without issue for the same command)
/usr/local/bin/clojure -M:env/test -Sdeps '{:deps {}}'
This gives an error, whether or not the :deps contains any dependencies.
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
-Sdeps (No such file or directory)
I first experienced this issue when running Clojure CLI tools with Cider (Emacs) and using the -M flag, specifically with cider-jack-in which injects the relevant dependencies for cider-nrepl using `-Sdeps argument. I can disable the actual dependencies, but not yet
Testing the same command on the terminal, gives the same error.
Using the command without the -M:env/test alias, only the -deps arguments works
Using the -A flag works with S-deps argument
Starting a repl with that command, minus the -Sdeps argument works via the command line and I can connect
I have tried this with several versions of Clojure CLI tools installed, from 1.10.1.697 onwards, including the latest pre-release. All seem to have the same issue.#2020-10-2210:14borkdude@jr0cket The -S... options should go before -M#2020-10-2210:16borkdudeclj-opts before exec-opts (although they are mentioned in the other order in the help text, that order is documented as Run main clojure [clj-opt*] -M[:aliases] [init-opt*] [main-opt] [arg*] )#2020-10-2210:19borkdude(I've made the same mistake before)#2020-10-2210:20practicalli-johnYes, putting the -M after -Sdeps does seem to work on the terminal.
That ordering is not the case with -A flag and so sounds like a change is required for Cider to work correctly with jack-in. The ordering can be changed by editing the the command line each time, but using .dir-locals.el config to set the alias, the -M and -A flags are put before the -Sdeps option
I assume -A will also work after -Sdeps option, to support those who havent upgraded to Clojure CLI tools#2020-10-2210:21borkdudethe same rules apply for -A#2020-10-2210:22borkdudeStart a REPL clj [clj-opt*] [-A:aliases] [init-opt*]
#2020-10-2210:23practicalli-johnExcept -A doesnt uphold that rule in practice...#2020-10-2210:23borkdudeIt's a case of Hyrum's law, but I guess the tools doing it wrong should be changed.#2020-10-2210:27practicalli-johnConfirmed that editing the cider command line and putting -M flag in the correct place does work :)
/usr/local/bin/clojure -Sdeps '{:deps {nrepl {:mvn/version "0.8.2"} refactor-nrepl {:mvn/version "2.5.0"} cider/cider-nrepl {:mvn/version "0.25.3"}}}' -M:env/test -m nrepl.cmdline --middleware '["refactor-nrepl.middleware/wrap-refactor", "cider.nrepl/cider-middleware"]'
I'll raise an issue with Cider to see if they can fix this. Thanks.#2020-10-2211:47dominicmThat's annoying. It looks like we'll need to write a parser for the arguments or something and split on it? Not too terrible, but it is if you want to be backwards compatible.#2020-10-2211:50practicalli-john@dominicm I just submitted an issue for this on the cider repository
https://github.com/clojure-emacs/cider/issues/2916#2020-10-2311:10practicalli-johnAnd there is a pull request that fixes this issue https://github.com/clojure-emacs/cider/pull/2917#2020-10-2211:51dominicmHmm, I guess I'll have to check if my scripts use prefix or suffix form. That pattern is used in other tools too.#2020-10-2212:09borkdudeor just don't use cider-jack-in. I never use it.#2020-10-2213:22dominicmI do. It's a useful tool, especially for beginners. Having less pieces to understand is very useful.#2020-10-2213:24borkdudeHaving less magic to understand is also useful ;)#2020-10-2213:26dpsuttonI also really like that CIDER can start up almost any project for you automatically without thought. It displays its startup command in the repl so you know there actually isn’t any magic. It’s just an extra dep and a main so you can run from the clj easily#2020-10-2213:54dominicmThere's no magic, just trying to build on top of simple tools underneath. It's more like an abstraction.#2020-10-2213:54borkdudeIt feels like too much magic to me. Also I don't want the REPL to be a subprocess of my editor..#2020-10-2217:33dominicmimplementation detail, subproc part could change :)#2020-10-2213:55dpsuttonthe subprocess part is fair 🙂 If there's a problem in your lisp interpreter from 1970 your repl goes down ...#2020-10-2214:12Alex Miller (Clojure team)Clojure tools (clj) https://clojure.org/releases/tools#v1.10.1.727 is now available:
• Fix clj -X:deps tree adding tools.deps.alpha to tree
• Fix clj -X:deps mvn-pom adding tools.deps.alpha to pom deps
• Fix clj -X:deps git-resolve-tags not working
• https://clojure.atlassian.net/browse/TDEPS-152 - Fix clj -X:deps mvn-install on jar to also install embedded pom
• Fix clj -Spom not respecting dep modifications from -A (regression)
#2020-10-2217:32rschmuklerIs there any way to install dependencies from a given alias without running an associated :main-opts in the alias? Essentially I want a -e "" that ignores the main-opts#2020-10-2217:33rschmukler(specifically from the CLI, I realize its very doable via the clj package)#2020-10-2217:42rschmuklerRight now my best shot is using babashka to construct a new deps.edn and invoking clj -Sdeps
clj -Sdeps "`cat deps.edn | bb -I -e "{:deps (merge (:deps (first *input*)) (get-in (first *input*) [:aliases :native-image :extra-deps]))}"`" -e ""
But this feels like quite the hack#2020-10-2217:48borkdude@rschmukler (I don't think you need -I here, since you're reading only one EDN value)#2020-10-2217:49borkdudecat deps.edn | bb -e "{:deps (merge (:deps *input*) (get-in *input* [:aliases :native-image :extra-deps]))}"#2020-10-2217:50rschmuklerRight on! I misunderstood the CLI. I thought -I coerced to EDN from string literal#2020-10-2217:50borkdude-I is for reading multiple EDN values from stdin#2020-10-2217:50seancorfieldThe official way is to use the -P option: https://clojure.org/reference/deps_and_cli#_prepare_for_execution#2020-10-2217:52seancorfieldSo clojure -P -M:my:aliases:here assuming you're using a recent release of the CLI.#2020-10-2217:52rschmukler@seancorfield perfection! Sorry that I missed that. Might be worth adding that to the clj --help docs too#2020-10-2217:53seancorfieldIt's in that help
exec-opts:
-A:aliases Use aliases to modify classpath
-X[:aliases] Use aliases to modify classpath or supply exec fn/args
-M[:aliases] Use aliases to modify classpath or supply main opts
-P Prepare deps - download libs, cache classpath, but don't exec
#2020-10-2217:53rschmuklerThen clearly my clj must be outdated lol!#2020-10-2217:53rschmuklerRight'o - thank you for the help!#2020-10-2217:53seancorfield(! 652)-> clojure -Sdescribe
{:version "1.10.1.727"
:config-files ["/usr/local/Cellar/#2020-10-2217:54rschmukler:version "1.10.1.645
#2020-10-2217:54rschmuklerThere we are#2020-10-2217:56seancorfieldYeah, 645 was one of the prerelease builds while a lot was changing in the CLI.#2020-10-2217:59rschmuklerNow I've just gotta figure out why Manjaro hasn't kept up#2020-10-2218:08rschmuklerUpdated my clojure and sure enough, it's all there for me. Thanks again @seancorfield#2020-10-2218:18practicalli-johnClojure CLI 1.10.1.697 is the minimum version for the -P and new mode for the -M flags. 1.10.1.727 is the latest (officially release today)#2020-10-2218:19Alex Miller (Clojure team)https://clojure.org/releases/tools for your version questions#2020-10-2218:19seancorfieldI'll be glad when -A changes to no longer run :main-opts 🙂#2020-10-2218:26seancorfield@U064X3EF3 This is still the plan, at some point, right?#2020-10-2218:31Alex Miller (Clojure team)yes#2020-10-2218:32Alex Miller (Clojure team)I expect that to be ~months#2020-10-2218:21practicalli-johnI did a little cheat sheet of command line flags to help me remember https://github.com/practicalli/clojure-deps-edn#clojure-cli-main-flag-options#2020-10-2222:11seancorfield@jr0cket I see
Run the project clojure -X:project/run -m domain.main-namespace
but there's no :project/run alias in your deps.edn so I'm not sure what this is meant to do? (and it seems weird to have -X and then have -m domain.main-namespace as well?)#2020-10-2222:12seancorfieldAlso, you have
Package library clojure -X:project/jars
That should be -X:project/jar singular, yes?#2020-10-2309:35practicalli-johnThanks for the review, those issues are now fixed, along with further explanation.
Using a :project/run alias seems to make most sense as a project level deps.edn, so the specific namespace and function names are in the config. I'm adding this to the templates I'm creating with clj-new.#2020-10-2222:03Drew VerleeWhen I run clj i get an exception about building the class path. I tried deleting my cpcache. If you can think of something else i can try please chime in.
gist file is my deps.edn. the comment contains the command run and output (error): https://gist.github.com/drewverlee/893a150c780a86fbf73cd789b48d9b40
Thanks in advance. ill post in this thread if i figure it out.#2020-10-2222:05dpsutton#:com.datomic{ion-dev #:mvn{:version "0.9.276"}} the ion-dev does not begin with a colon but in #:mvn{:version it does. can you try fixing that?#2020-10-2222:10dpsuttonscratch that. namespaced symbols look strange to me in that fashion 🙂#2020-10-2222:12Drew VerleeIt is strange, I'll clean it up. But I'm guessing that's not it. That's from copy pasting#2020-10-2222:12Alex Miller (Clojure team)I don't think that's it - this look like maybe a bad version range in some transitive dep?#2020-10-2222:13Alex Miller (Clojure team)although I know all those deps so not sure what you'd find there :)#2020-10-2222:14seancorfield{:deps #:com.datomic{ion-dev #:mvn{:version "0.9.276"}}
under an alias, that should be :extra-deps, not :deps (or :override-deps)#2020-10-2222:15Alex Miller (Clojure team)typo: mvn/verison#2020-10-2222:15seancorfieldAh yes, on com.datomic/ion {:mvn/verison "0.9.48"}#2020-10-2222:15Alex Miller (Clojure team)newest version of clj will give you a better error message there#2020-10-2300:08Drew Verleei see the typo now. ugh. I should turn on a spell check before i post something to catch things like this.#2020-10-2300:42Alex Miller (Clojure team)On latest clj, you'll get: Error building classpath. No :mvn/version specified for com.datomic/ion#2020-10-2307:08dominicmNow that add-lib isn't functional, if I wanted to load dependencies for a particular function call with exec-opts (without the user having to know which dependencies to add to deps.edn for that call) how would I do that?
I want to provide some beginner tools as part of something, but I don't want to bloat the primary use case.
If I had bash-style exec, I'd probably just restart the jvm that way.
I want the set of functionality to be an open-set though, so it might expand in the future, and the user shouldn't need to fiddle around to get that working. I'm certainly aiming for "easy" here.#2020-10-2307:24dominicmJust looked at what ion does, that has ion-dev which you install in your user deps.edn, that strikes me as an odd choice.#2020-10-2310:14dominicmI suppose this could be considered the ability to import deps.edn aliases into your project somehow :)#2020-10-2311:17Alex Miller (Clojure team)Putting ion-dev in user deps.edn is recommended - it’s a general tool for deploying ions#2020-10-2311:20Alex Miller (Clojure team)We will be updating add-lib to work again btw with some additional apis and ultimately in master#2020-10-2311:21Alex Miller (Clojure team)But not sure I have a good answer for this use case right now#2020-10-2311:46dominicmnp. Maybe I'll start with bloat, then swap to add-lib later.#2020-10-2316:41seancorfieldI built a version locally, using Maven, based on a t.d.a add-lib branch a while back I think. Whatever was the equivalent of 0.8.667 it seems.#2020-10-2316:45seancorfieldI built it on April 6th, so it's probably the head of the add-lib branch, which was 0.8.667 plus the calc-basis stuff, looking at the commit logs. @dominicm I guess keeping your own version around for this isn't going to work for your use case?#2020-10-2316:46dominicm@seancorfield it would yeah, I had assumed that stuff was completely broken by the internal changes. I'll give it a go, thanks!#2020-10-2316:50seancorfield-rw-r--r-- 1 sean admin 58110 Apr 6 2020 tools.deps.alpha-0.8.667-add-lib.jar
In case you want a sanity check on the size of the JAR after you build it. Then I just use :local/root to depend on the JAR.#2020-10-2316:52dominicmI'll probably have to deploy it somewhere, but that's fine :)#2020-10-2319:03Alex Miller (Clojure team)I just pushed an add-lib3 branch that's up to date with master#2020-10-2319:04Alex Miller (Clojure team)note that add-lib is now add-libs and there are some other additional apis (not final) in there too - find-revs and find-lib#2020-10-2319:08Alex Miller (Clojure team)also, at some point in the past I made it degrade gracefully if it doesn't have the compiled s3 class stuff so it's a little easier to hack onto a classpath like it was originally#2020-10-2319:10dominicmGreat, I guess I know what I'm doing tomorrow then!#2020-10-2319:11dominicmMy goal is that beginners can do (require 'beginner) or something, and it'll go ahead and fetch tools like spyscope and dot-clojure2 for them, and load them up.
Perfect compromise between beginners and advanced users (who will set that stuff up for themselves eventually)#2020-10-2321:29seancorfield@alexmiller Thanks for the add-lib3 branch -- very nice that it works from source again, albeit without S3 support! Just updated my .clojure file for that.#2020-10-2308:30Eamonn SullivanGood morning, does anyone use CircleCI with tools-deps and have an example I can see? I'm just exploring a few alternatives to our Jenkins. I use Github actions on my own stuff (e.g., https://github.com/eamonnsullivan/github-pr-lib I create for scripts and deploy to clojars) and I'm also going to take a look at AWS CodeBuild/Pipeline, but examples to study are mostly lien-based.#2020-10-2308:33borkdude@eamonn.sullivan Almost all of my projects do#2020-10-2308:34Eamonn SullivanAh, perfect. Thank you!#2020-10-2308:34borkdudehttps://github.com/borkdude/clj-kondo/blob/master/.circleci/config.yml#2020-10-2309:16practicalli-john@eamonn.sullivan There are images you can choose from that already have Clojure CLI tools included.
https://practicalli.github.io/clojure/testing/integration-testing/circle-ci/
If you want the very latest version of Clojure CLI tools, then use Borkdude's approach (which I am going to add to that page I mentioned right now, as I want to use 1.10.1.727)#2020-10-2309:46practicalli-john@eamonn.sullivan I have a collection of different approaches I'm going through in this ticket https://github.com/practicalli/clojure-content/issues/183
Would be interested to hear what you end up using. Thank you.#2020-10-2319:59Eamonn SullivanSo, you asked how I got on: I decided to try one of the pre-built images. The test/pull-request checks are working, but not the deploy. I get a TransferFailedException. I'm literally only hours old as a CircleCI user and I'm probably doing something basically wrong.#2020-10-2713:39Eamonn SullivanSo, just to finish this thread off, in case anyone is still reading: I did eventually get it to work, providing the same functionality that I have (on other repos) using Github Actions, which was the goal. Thanks very much for the help and the pointers! https://github.com/eamonnsullivan/github-api-lib/blob/main/.circleci/config.yml#2020-10-2715:50practicalli-johnThanks for sharing#2020-10-2310:27dominicmHum, I think there's a mistake in exec.clj. https://github.com/clojure/brew-install/blob/f734dacaab43c0cd44c5ffffb8fa3a10ab64d4a0/src/main/clojure/clojure/run/exec.clj#L20 this throws, but almost all calls also throw, https://github.com/clojure/brew-install/blob/f734dacaab43c0cd44c5ffffb8fa3a10ab64d4a0/src/main/clojure/clojure/run/exec.clj#L26#2020-10-2311:23Alex Miller (Clojure team)Yeah that doesn’t look right#2020-10-2311:44dominicmI also noticed that read-args uses a loop but could be mapv quite easily#2020-10-2311:52Alex Miller (Clojure team)fixed the double throw#2020-10-2312:05Alex Miller (Clojure team)agreed on read-args - that stuff has been through a lot of versions where the positionality was more important, going to leave it for now as it's reasonably likely to go through more changes#2020-10-2314:21dominicm@alexmiller I'm currently working on a main which will have exec-like uses, particularly the notion of updating a map in response to those args. Might be useful to rely on the exec code once it's stable. Then I can just point at your docs :p#2020-10-2314:23dominicmObviously read-args is pretty simple, and so is the reduce itself. So maybe that's not really useful. But I'm not sure how it'll evolve. The tagged-literal reader might be subtle.#2020-10-2314:23Alex Miller (Clojure team)I would not rely on the exec code right now - I do not (yet) consider that to be "public api"#2020-10-2314:24Alex Miller (Clojure team)I think it would actually be useful to move some or all of it into clojure itself (similar to clojure.main) but that's probably longer term#2020-10-2314:27dominicmYeah, I saw it was all private and ducked out :)#2020-10-2311:09arohnerWhat is the recommended way to deploy a jar to a private S3 repo using tools.deps?#2020-10-2314:08dominicmThis seems like an argument against exec-opts to me: clj -X:user/new create :template "\"$PWD/template::new-app\"" :name my.project.name (this doesn't work for me, but that's unrelated, just the awkward way to pass in pwd)#2020-10-2314:17Alex Miller (Clojure team)what doesn't work?#2020-10-2314:18Alex Miller (Clojure team)and it's awkward because? quoting?#2020-10-2314:24dominicm@alexmiller yeah, the quoting is awkward, compare to main-style:
--template "$PWD/template::new-app"
Doesn't work because I'm passing wrong args to clj-new, but I didn't need that debugging.#2020-10-2316:50seancorfield@dominicm an FYI on usability there: both clj-new and depstar ended up allowing symbols or strings in most argument slots because of quoting so :template foo works because it is silently converting the symbol to a string behind the scenes. But of course you can't use symbols in all cases (and particularly yours).#2020-10-2316:50dominicm@seancorfield heh, that feels like a dangerous path with spaces and backslashes, etc.#2020-10-2316:51dominicmI wonder if a directory named "4" would work, for example#2020-10-2316:51seancorfieldYup, it only works for "strings" that could be legal Clojure symbols... No, because that would be read as a number, not a symbol.#2020-10-2316:51dominicmEven my example is dangerous because it doesn't escape backslashes.#2020-10-2316:52dominicmSeems like you wouldn't want to script anything on that basis then.#2020-10-2316:52seancorfield#2020-10-2316:56seancorfield(because I just call str on it -- I thought I was being fancier than that)#2020-10-2316:59dominicmStr on a number works I guess?#2020-10-2317:04dominicmAnyway. Not a problem/solution situation really. Just an observation that this decision had this property.
I'm sure eventually we'll know when to apply each decision (or find a new decision) to get desired properties.#2020-10-2317:13seancorfieldYeah, that's why I crossed that out. I tried :4 as one of the args and got the equivalent of ":4" 🙂 Much depends on whether you want a plain string or, ultimately, a qualified symbol or relative path or...#2020-10-2318:36souenzzoI can't find how to do clj -m app.main in "modern #tools-deps#2020-10-2318:38Derekclj -M -m app.main, I believe#2020-10-2318:44borkdudecorrect#2020-10-2319:02Alex Miller (Clojure team)just wait till we get to post-modern clj#2020-10-2319:03souenzzocan I stay using clj -m app.main on my deploy/scripts?#2020-10-2319:09Alex Miller (Clojure team)for now, yes (eventually that will stop working but not soon)#2020-10-2319:06borkdudein post-modern clj there is no one big story of arguments, but lots of different forks of clj where everyone does their own thing#2020-10-2319:08borkdudeno one believes that hammock time eventually leads to the ultimate truth anymore. instead writing clojure programs becomes an expression of doubt#2020-10-2319:08borkdude:P#2020-10-2319:08dominicmI haven't released stable software since spec-ulation#2020-10-2319:09borkdudeso hasn't core maybe? I should check#2020-10-2319:09Alex Miller (Clojure team)1.10.1 :)#2020-10-2319:10Drew VerleeWhen i start an nrepl server through cider and eval datomic.client.api/client it complains about missing hmc auth. Indeed, if i check my cider-classpath that dep is missing.
The issue is the same as discussed here: https://clojurians-log.clojureverse.org/datomic/2020-07-22, which the solution being to update clojure. I updated via linuxbrew and now my version of clojure tools is 1.10.1.727. The issue remains.
My only idea is that if i ask cider to desribe the connection it says clojure is on 1.10.1 not 1.10.1.727 i'm not sure if its truncating or if it somehow connecting to an old version#2020-10-2319:11Alex Miller (Clojure team)727 is the version of the clojure tools. the actual clojure version really is 1.10.1#2020-10-2319:12Alex Miller (Clojure team)(unobviously, any version of the clojure tools can use any version of clojure - the clojure tools part really just tells you what it was built with and what version of clojure you get by default if you don't specify one)#2020-10-2319:13dominicm❯ clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.0.0"}}}'
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.pom from central
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.jar from central
Clojure 1.0.0-
user=> *clojure-version*
{:major 1, :minor 0, :incremental 0, :qualifier ""}
user=>
#2020-10-2319:16dpsuttonpossible the erroneous classpath computed by an older tools deps is cached? just delete the cache?#2020-10-2319:16borkdudeor use -Sforce#2020-10-2319:18Drew VerleeI deleted .cpcache (which i assume is the only cache) and it didn't work. I tried -Sforce, which starts a repl but then I can't connect to it with cider. I just get a "connection failed".
I'm going to try restarting my computer and assume this is a cider thing. Thanks for the suggestions.#2020-10-2319:19dpsuttonits not CIDER. and restarting the computer won't clear the cache that is suspected to be at issue (if we are correct)#2020-10-2319:21Alex Miller (Clojure team).cpcache is the classpath cache so sounds like you've done that already#2020-10-2319:21dpsuttonoh wow. i totally glossed right over that message. sorry#2020-10-2319:22Alex Miller (Clojure team)oh that hmac auth thing was actually a tools deps bug and should be fixed in latest#2020-10-2319:23dominicmI wonder if there's 2 copies of clojure in $PATH or something...#2020-10-2319:23Alex Miller (Clojure team)yeah, can you repro outside cider?#2020-10-2319:23Alex Miller (Clojure team)if not, I'd guess you're somehow seeing a different version of clojure, or something is explicitly using an older version of tools.deps.alpha#2020-10-2319:24dominicmcider literally just runs clojure -Sdeps '{nrepl/nrepl … cider/cider-nrepl …}' -m nrepl.cmdline …#2020-10-2319:25Drew Verleecider does just run clojure -Sdeps, but ill try using it directly just in case.#2020-10-2319:26dominicmcider's jack-in isn't magic, it's just saving you knowing which dep & which main to start.#2020-10-2319:27dpsutton(and now prints that and optionally the cljs startup to remove any semblance of magic)#2020-10-2319:40Drew VerleeThe issue was corrected by having cider look at a version of clojure tools that had the bug fix. I had two versions of clojure installed (no useful reason for that). So i uninstalled both and reinstalled using linux brew.
As dominicm points out, you can look at what version of clojure cider is using in the message. It was different from what i got at the command line. So after verifying one version worked but not another, it was fairly obvious.
When i run clj /clojure from the command line it tells me the version of clojure, but in this case it seems it was the version of tools deps that was different between the two.
e.g /usr/local/bin/clojure starts a repl with "clojure 1.10.1" and the other version in /home/drewverlee/.linuxbrew/bin/clj says "clojure 1.10.1" so part of me assumed they were a symlink/alias.#2020-10-2320:02Alex Miller (Clojure team)cool#2020-10-2413:21dominicm{:aliases {:foo ["bar"]}} and clj -A:foo produces interesting results :P
e.g.
:resolve-args {\b \a},
:classpath-args {\b \a},
#2020-10-2413:33dominicmI was curious to see what would happen :)#2020-10-2414:43Jakub Holý (HolyJak)Hi! What is the relationship of versions between Clojure and tools.deps? Latest Clojure download is https://download.clojure.org/install/linux-install-1.10.1.727.sh, latest https://github.com/clojure/tools.deps.alpha#release-information. If I install that Clojure, which deps will I get? I.e. what command-line options will clojure support? Thank you!#2020-10-2414:44Alex Miller (Clojure team)The change logs for the two are cross linked and you can find official release info at https://clojure.org/releases/tools#2020-10-2414:45Alex Miller (Clojure team)The docs on the Clojure site always reflect the latest stable release (or use -h or the man page)#2020-10-2414:46Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#2020-10-2414:51Jakub Holý (HolyJak)Thank you! I should have noticed the " See the changelog for version information." at https://clojure.org/guides/getting_started#2020-10-2415:10Jakub Holý (HolyJak)Q1: Does :local/root need to be an absolute path? If so - I guess it isn't possible to use something like "~/path/to/it" (to make it portable across users)?#2020-10-2415:13Jakub Holý (HolyJak)Q2: Why am I getting
> Error building classpath. No coordinate type found for library cryogen-asciidoc/cryogen-asciidoc in coordinate {:local-root "/Users/me/external/cryogen-all/cryogen-asciidoc"}
when head /Users/me/external/cryogen-all/cryogen-asciidoc/pom.xml returns successfully (modified)
...
<groupId>cryogen-asciidoc</groupId>
<artifactId>cryogen-asciidoc</artifactId>
...
?#2020-10-2415:14dpsuttonhttps://github.com/lambdaisland/chui/blob/master/deps.edn#L4 is an example of relative paths. I think if you use an absolute path like ~/some/path it will not be portable at all#2020-10-2415:18Alex Miller (Clojure team)You can’t use ~ - that’s a shell thing#2020-10-2415:19Alex Miller (Clojure team)But either absolute or relative are fine#2020-10-2415:21Alex Miller (Clojure team)Not sure if that solves your q2 or not but if not share your deps.edn#2020-10-2416:20Jakub Holý (HolyJak)Here is it
{:deps {org.clojure/clojure {:mvn/version "1.10.1"}
ring-server/ring-server {:mvn/version "0.5.0"}
ring/ring-devel {:mvn/version "1.7.1"}
compojure/compojure {:mvn/version "1.6.1"}
cryogen-core/cryogen-core {:mvn/version "0.3.2"}
#_#_cryogen-core/cryogen-core {:local-root "../../external/cryogen-all/cryogen-core"}
#_#_cryogen-asciidoc/cryogen-asciidoc {:mvn/version "0.3.2"}
cryogen-asciidoc/cryogen-asciidoc {:local-root "../../external/cryogen-all/cryogen-asciidoc"}}
:aliases {;; Run with `clojure -M:build`
:build {:main-opts ["-m" "cryogen.core"]}
;; Start a server serving the blog: `clojure -X:serve`
;; (requires tools-deps 0.9.745+)
:serve {:exec-fn cryogen.server/serve
:exec-args {:port 8888}}}}
This is from https://github.com/holyjak/blog.jakubholy.net, the local dependency is https://github.com/cryogen-project/cryogen-asciidoc/#2020-10-2416:56Alex Miller (Clojure team)It’s :local/root not :local-root#2020-10-2417:00Jakub Holý (HolyJak)OMG, thanks a lot! I must be blind.#2020-10-2415:36dpsuttonI’m surprised you can’t use ~. I figured it would end up in a path object and the jvm would work it out for you#2020-10-2415:40borkdudejvm doesn't do this#2020-10-2415:40borkdudeit's a bash thing. On Windows this doesn't work either#2020-10-2415:40dpsuttonMakes sense. Not sure where I got that expectation#2020-10-2415:41borkdudeWhat could work OS-independent is interpolation of environment variables. But even then you would have to use different names for linux and windows#2020-10-2415:42borkdudeso probably a script that generates deps.edn for you as a pre-processing step is the least worst way of dealing with such concerns#2020-10-2415:42borkdudeI haven't had an app where I did this, except for one where I keep deps in sync between project.clj and deps.edn#2020-10-2415:50Alex Miller (Clojure team)Java has a system property for this - user.home#2020-10-2415:50Alex Miller (Clojure team)(System/getProperty "user.home") is platform independent#2020-10-2415:51Alex Miller (Clojure team)but no equivalent to that in deps.edn (yet, or maybe ever)#2020-10-2415:55dominicm@holyjak what's the home-relative use case?#2020-10-2416:13Jakub Holý (HolyJak)None, if I can (I can) use relative psths#2020-10-2416:21dominicmah, OK :)#2020-10-2416:10dharriganI would love to see deps.edn support some type of user home substitution. I have the a main $HOME/.clojure directory, in there I have my deps.edn file, that I pull in other libraries with that are all local root (i.e., setting up a REPL with some middleware etc..). Right now, since I’m on Linux I have to put an absolute path in, i.e., :base {:extra-deps {local-base/local-base {:local/root "/home/david/.clojure/libs/local.base" :deps/manifest :deps}}#2020-10-2416:10dharriganI cannot bring that across to my mac environment#2020-10-2417:06Alex Miller (Clojure team)Making the .clojure/deps.edn portable across machines wrt paths is not a goal - imo if you’re on a different machine, you will need to adjust to that machine#2020-10-2417:07Alex Miller (Clojure team)Which is not to say we won’t ever add something like this, but I’m not eager to do so#2020-10-2416:10dharrigansince that is /Users/home/david#2020-10-2416:11dharriganI thus have to maintain two separate deps.edn files that are near identical, apart from the path stuff.#2020-10-2416:12dharriganbut and I’m just trying this out…having it as a relative path may work!#2020-10-2416:13dharrigani.e., :base {:extra-deps {local-base/local-base {:local/root "libs/local.base" :deps/manifest :deps}}#2020-10-2416:14dharrigannope#2020-10-2416:14dharriganSwitching to a relative path, ends up with#2020-10-2416:14dharriganExecution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate local/rebel/main__init.class, local/rebel/main.clj or local/rebel/main.cljc on classpath.
#2020-10-2416:15dharriganwhich leaves me with having to put the absolute path in my deps.edn, and maintaining two separate deps.edn (which I suspect, if I was also on windows would incur more mangement)#2020-10-2416:18dharriganIt would be nice IMHO if deps supported at least having :local/root resolve the user’s home.#2020-10-2416:27dominicmHmm, I'm slightly surprised that resolution isn't relative when doing that. The personal tooling case is the only one I have so far too :)#2020-10-2416:57Alex Miller (Clojure team)I’m not, that’s what I’d expect - top level deps are relative to where you’re running clj#2020-10-2416:59dominicm@alexmiller I suppose they could be resolved with a relative *dir* like when doing transitive :local/root though?#2020-10-2416:59dominicm(obviously there's problems there due to the merging)#2020-10-2416:59Alex Miller (Clojure team)Yeah, that’s problematic#2020-10-2417:10dominicmI suppose paths could be canonicalized pre-merging.#2020-10-2418:13borkdudethat's what I do in clj-kondo config. Each config can refer to local dirs relative to itself and these get resolved during config reading.#2020-10-2418:14borkdudethis way you can have hooks in your home dir config, not relative to your project#2020-10-2418:44Alex Miller (Clojure team)That’s possible, I’ll think about it #2020-10-2418:59dharriganthat would be nice#2020-10-2418:59dharrigan🙂#2020-10-2608:12onetomi have an alias defined in my deps.edn as follows:
:ion-dev
{:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.276"}}
:main-opts ["-m" "datomic.ion.dev"]}}
if i run /nix/store/6n60jkgpxb1a95wjx3ikhn9nvp5w33yh-clojure-1.10.1.727/bin/clojure -A:ion-dev -Sdeps '{:deps {nrepl {:mvn/version "0.8.0"}...' -m nrepl.cmdline ... it complains
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
WARNING: When invoking clojure.main, use -M
does this mean
1. i have to move the :main-opts into a different alias?
2. the command line should have a -M (without any aliases as a parameter) if i just want to run clojure.main?#2020-10-2608:16onetomim wondering what should be the naming convention for aliases in cases when one want to use some extra dependency both as just a library and as a command-line tool...
-A:ion-dev -M:ion-dev/run maybe?#2020-10-2608:40onetomam i supposed to combine aliases which doesn't contain :main-opts with aliases which does?
like -M:ion-dev:ion-dev/run?
when my deps.edn has:
:ion-dev
{:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.276"}}}
:ion-dev/run
{:main-opts ["-m" "datomic.ion.dev"]}}#2020-10-2608:46dominicm@onetom needs a -M before the -m, so clojure -A:ion-dev -Sdeps … -M -m nrepl.cmdline …#2020-10-2610:29practicalli-john@onetom it is not required to have separate aliases, you can use the original :ion-dev alias and just use -M instead of -A . However, the -M (and -A flags) should come after the -Sdeps option and before the -m option (the -A flag does still work out of position though)
clojure -Sdeps '{:deps {nrepl {:mvn/version "0.8.0"}...' -M:ion-dev -m nrepl.cmdline
It is also possible to chain the aliases under -M , especially if there is an alias for nrepl too (this example taken from https://github.com/practicalli/clojure-deps-edn/)
:middleware/nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "0.8.2"}}
:main-opts ["-m" "nrepl.cmdline"]}
And then simplify the command
clojure -M:ion-dev:middleware/nrepl
#2020-10-2612:02onetomsure, but i need the :extra-path in :ion-dev on its own too, without the :main-opts and so far i haven't found any option combination which wouldn't print some kind of warnings.#2020-10-2612:38onetomi went thru the new deps and cli docs and found this section:
https://clojure.org/reference/deps_and_cli#_replace_project_environment_tool
but it's unclear to me what does `tool` process refer to.#2020-10-2612:55practicalli-john:replace-deps will just use the deps specified in the alias, dropping any libraries from the main :deps section. This is useful if you just want to run an external tool, e.g. such as clj-new that creates a new Clojure project.
Using the -M option only calls the main namespace from the last alias in a chain, so with -M:ion-dev:middleware/nrepl only the main namespace from :middleware/nrep is called and the :ion-dev is simply adding the library as an additional dependency.
I didnt see any other main namespaces on the example shared.
I am not clear on why an :icon-dev with only extra paths is required in the scenario shared so far.#2020-10-2613:07onetomto start a development repl (with cursive or cider)#2020-10-2613:08onetombut i want to retain the option of running datomic.ion.dev/-main, since that's the official way to deploy datomic cloud projects.#2020-10-2613:09onetomwhen i have a repl running already (started with repl related :main-opts), im just using datomic.ion.dev/{push,deploy,deploy-status} directly#2020-10-2613:33Alex Miller (Clojure team)There is no option to use the extra-deps but not the main-opts from an alias#2020-10-2613:33Alex Miller (Clojure team)You should separate those into two separate aliases if you need that#2020-10-2614:09arohnerSomething in my deps is using a version range or SNAPSHOT because every time I run clj, it runs
Downloading: com/amazonaws/aws-java-sdk-cloudwatch/maven-metadata.xml
. Is there a clj command that will help me find which dep that is?#2020-10-2614:11arohnerit’s not -Strace or -Stree, because those only tell me the resolved version#2020-10-2614:16Alex Miller (Clojure team)if you do tree, it's probably whatever is above that dep in the tree ?#2020-10-2614:17Alex Miller (Clojure team)I don't think I have any other magic command to tell you more unfortunately#2020-10-2614:20arohnerthat helps some, thanks#2020-10-2614:21arohnerI was able to find the dep depending on cloudwatch (amazonica), but something else is causing
Downloading: org/clojure/clojure/maven-metadata.xml
. I guess I can wipe out ~/.m2/repository and do some grepping#2020-10-2614:22Alex Miller (Clojure team)alternately, use -Sdeps '{:mvn/local-repo "tmp"}'#2020-10-2614:22Alex Miller (Clojure team)build up a new tmp repository instead#2020-10-2615:58arohnerfound it. de.kotka/lazymap specifies:
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure</artifactId>
<version>[1.2,1.6)</version>
<scope>compile</scope>
</dependency>#2020-10-2615:59arohnershould that always trigger a snapshot check?#2020-10-2616:04Alex Miller (Clojure team)all version ranges will trigger a resolution to the "newest" stable version#2020-10-2616:05arohnereven when bounded like that?#2020-10-2616:05Alex Miller (Clojure team)yes#2020-10-2616:06Alex Miller (Clojure team)you could :exclude org.clojure/clojure down that path#2020-10-2616:06Alex Miller (Clojure team)although based on where the canonicalization happens, that may not help#2020-10-2616:09arohnerOn https://clojure.org/reference/deps_and_cli, I don’t see any definition for the contents of the :exclusion key#2020-10-2616:11Alex Miller (Clojure team)that may be underdocumented :)#2020-10-2616:11Alex Miller (Clojure team):exclusions [org.clojure/clojure]#2020-10-2616:11Alex Miller (Clojure team)in the coordinate#2020-10-2616:11Alex Miller (Clojure team)it looks like lazymap 3.1.1 dropped the version range#2020-10-2616:14Alex Miller (Clojure team)I don't think exclusions will help you here#2020-10-2616:15Alex Miller (Clojure team)at the point where expansion goes to resolve the deps, it's resolving all of them, the exclusions and stuff are handled at a higher level#2020-10-2616:15arohnerPutting lazymap 3.1.1 as an explicit dependency fixed it for me#2020-10-2616:15Alex Miller (Clojure team)yeah, that should work#2020-10-2616:15arohnerlazymap isn’t a dep of my project, it’s a dep of a dep. Putting lazymap in :override-deps doesn’t seem to do anything. Is it supposed to?#2020-10-2616:39seancorfield@arohner :override-deps only works inside an alias#2020-10-2617:02arohnerthanks#2020-10-2711:07mkvlrHey 👋 We’d like to bring in modules from another private repo using gitlibs. Ideally we’d want to use ssh auth. There’s no way to add support for another coordinate or get it to use tools.gitlibs from https://github.com/ghadishayban/tools.gitlibs/pull/1 without forking tools.deps (and having to distribute it to everybody on the team), is there? Or any other solutions to this problem that folks can recommend? Thanks!#2020-10-2711:11borkdudeIf all else fails, you could manually pull the code and add it via local/root?#2020-10-2711:14mkvlryep, guess that’s one option. Was also looking at your https://github.com/borkdude/deps.clj since we do require babashka on the project but the compute classpath part still happens on the jvm using the normal t.d.a jars right?#2020-10-2711:14borkdudecorrect#2020-10-2711:16borkdudeit pulls tools jars that are exactly the same as the jars that clj uses. however, we could make that configurable ;)#2020-10-2711:17borkdudebut maybe best to not create forks of official tools when it can be fixed in those official tools#2020-10-2711:18mkvlrright#2020-10-2714:04kirill.salykinhi
please advice - is there a way to include some java source files and use it from the clj?
for instance I want ot have this file used https://dzone.com/articles/java-tips-creating-a-monitoring-friendly-executors
is it possible to achieve with a c.t.d.?
thanks!#2020-10-2714:06Alex Miller (Clojure team)you can compile the java source into java class files, then add the path containing those to your paths#2020-10-2714:07Alex Miller (Clojure team)you can either use javac (the Java compiler) directly or there are various tools available to help with that https://github.com/clojure/tools.deps.alpha/wiki/Tools#2020-10-2714:10kirill.salykini see, thanks for the options#2020-10-2714:29borkdudewould it be an idea to extend -X with being able to pass a single map?
clj -X:graph graph :deps '"mydeps.edn"' :trace true :output '"trace"'
clj -X:graph graph '{:deps "mydeps.edn" :trace true :output "trace"}'
The quoting of strings seems a little clumsy on the command line, so quoting the entire map can deal with that as a whole.#2020-10-2714:30borkdudeI've been adopting the latter style for more tools where I just pass the entire map as --opts, e.g.:
$ echo '[1 2 3]' | puget --opts '{:print-color false}'#2020-10-2714:31borkdudeAnother question: is the -X CLI parsing logic available as a library? I think this could be used in other places than clj as well (e.g. GraalVM compiled binaries).#2020-10-2714:38Alex Miller (Clojure team)answering the last question first, no but that's something we're actively thinking about - I think it's something that could end up in Clojure itself (like clojure.main) but I am still actively changing the interface to it and will need further work before I think it's there#2020-10-2714:40Alex Miller (Clojure team)on the former question, that's a good question, will think about it#2020-10-2714:41Alex Miller (Clojure team)I mean really the idea was to put the base map in alias data and then just overrides on the cli, but some tools would benefit more than others from this#2020-10-2721:13dominicmI keep thinking I've missed a way to have some kind of :deps-aliases in an alias to use the extra deps from another alias, similarly to how I can with :extra-paths.#2020-10-2721:16Alex Miller (Clojure team)I'm confused by what you mean about :extra-paths#2020-10-2721:17Alex Miller (Clojure team)unless you mean paths as alias data#2020-10-2721:17dominicmI mean that I'm surprised there's no equivalent to :extra-paths [:build-paths] such as :extra-deps [:build-deps]. Obviously that syntax doesn't work, but that general idea.#2020-10-2721:17dominicmI do yeah :)#2020-10-2721:17Alex Miller (Clojure team)ah, ok#2020-10-2721:18Alex Miller (Clojure team)would certainly complicate the (already too complicated) merging logic#2020-10-2721:19dominicm:extra-deps [:build-deps {…}] maybe…#2020-10-2721:21Alex Miller (Clojure team)I'd probably be tempted to define something mappy instead so the merging would continue to work (like destructuring syntax or spec selects)#2020-10-2721:22dominicm:extra-deps {:clojure.tools.deps/aliases [:build-deps] my.cool/lib {}} ?#2020-10-2721:22Alex Miller (Clojure team)well, I'm not going to design it right now :)#2020-10-2721:22Alex Miller (Clojure team)if you wanted to file a jira or an ask request, would be glad to keep that idea around#2020-10-2721:25dominicmI'll open a JIRA. :)#2020-10-2721:26Alex Miller (Clojure team)thx#2020-10-2915:41dpsuttonworking to give an error message to verify that the tools deps version is too old to support -M -m. Does anyone know offhand if a version supported -X but not -M -m? Hoping can check something real quick with -X and that means it necessarily supports -M#2020-10-2915:44dpsuttonas it stands, added a simple (ns check) (defn -main [& _] (System/exit 0)) but would prefer to not need a new main and new ns if X and M go hand in hand#2020-10-2916:27dpsuttonie, could just do clojure -X clojure.core/identity :a :a and if that returns 0 then good to go for -M -m our.stuff#2020-10-2916:38lread@dpsutton I expect this is not what you are looking for, but I write my scripts in Babashka where I find a Clojure version check to be explicit and straightforward: https://github.com/lread/rewrite-cljc-playground/blob/60ad8c118a86221e35692f2ce10281e7cef0f8a4/script/helper/env.clj#L39#2020-10-2916:39practicalli-john@dpsutton https://clojure.org/releases/tools might have the info about -X but not -M -m support... although I believe -X support wasnt added until https://clojure.org/releases/tools#v1.10.1.697 and -M -m was in by then.
I use https://clojure.org/releases/tools#v1.10.1.697 as the minimum requirement for Clojure CLI tools with my aliases and content from Practicalli#2020-10-2916:42seancorfield-M has been in the CLI for a long time but it's meaning changed around 1.10.1.697#2020-10-2916:43seancorfieldThe difference between 1.10.1.561 (stable) and the later version is just that -M now respects resolve-args and classpath-args in aliases as well as main-opts. Previously, it ignored resolve-args and classpath-args. Does that help @dpsutton?#2020-10-2916:45dpsuttonYeah that does. I think that means we should use the check for X support to make sure we have a recent enough version#2020-10-2916:45dpsuttonAlthough I think our use case is simple enough that as long as M support is there it’s fine#2020-10-2916:59seancorfield@dpsutton I just ran some tests on a bunch of prerelease versions between the last two stable releases, and there are several where that command returns a zero status but doesn't actually work.#2020-10-2917:00seancorfield(but I believe they would support -M with aliases by that point... I just haven't tested that 🙂 )#2020-10-2917:02dpsuttonIn what way do they not work?#2020-10-2917:02dpsuttonAnd thanks so much!#2020-10-2917:02seancorfield(! 1626)-> /usr/local/Cellar/clojure\@1.10.1.636/1.10.1.636/bin/clojure -X clojure.core/identity :a :a
No alias specified with -X
Thu Oct 29 09:58:15
(sean)-(jobs:0)-(/Developer/workspace/wsmain/build)
(! 1627)-> echo $?
0
#2020-10-2917:03seancorfieldThe ability to pass the fn to execute on the command-line came fairly late in that set of prerelease builds.#2020-10-2917:04seancorfield1.10.1.672 seems to be the first prerelease build that allowed it.#2020-10-2917:05dpsuttonOh ok. It’s only prereleases? If so they can diagnose it on their own :)#2020-10-2917:05dpsuttonBleeding edge is sometimes bleeding#2020-10-2917:06seancorfieldYeah, 1.10.1.561 (previous stable) throws an exception and gives a non-zero exit status.#2020-10-2917:06seancorfield(so does 1.10.1.590 which was the first of the prerelease builds)#2020-10-2917:06dpsuttonBeautiful. How do you change versions so quickly? Would have done that if I knew a good way to swap them around easily#2020-10-2917:07seancorfieldI installed them by version with brew and then I can use the full path (as shown above) to run any version I want 🙂#2020-10-2917:08kirill.salykinhow are you doing this?#2020-10-2917:09kirill.salykinjust brew install @version ?#2020-10-2917:09seancorfieldbrew install clojure/tools/ per the CLI install docs#2020-10-2917:09kirill.salykinclear, thanks a lot!#2020-10-2917:10seancorfieldSee https://github.com/clojure/homebrew-tools (which is linked from the CLI install docs on the CLI guides page)#2020-10-2917:07seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "63000f0c0916110623524d52534d524d565552"}, :content ("[email protected]")}#2020-10-2917:08seancorfieldI didn't have 561 until just now, but installed it by version to test.#2020-11-1103:18Alex Miller (Clojure team)those are it right now#2020-11-1103:21natecool#2020-11-1103:21natethank you#2020-11-1109:30henryw374I'm looking for a tool that will help pull out unnecessary :exclusions - ie keeping the set of resolved dependencies unchanged for some combination of aliases. anyone know of such a thing? thanks#2020-11-1110:08borkdude@henryw374 I once diagnosed a dependency problem like this using clojure.spec:
https://github.com/borkdude/dependency-problem#2020-11-1112:09henryw374another thing I'm looking for is a documented description of how deps chooses which version of a dependency to use. I expect it's there but I just can't see it on http://clojure.org and so far google has failed#2020-11-1112:15delaguardohttps://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions.clj#L95-L104
the best place I can think of)
and there is how this defmulti is implemented for different coordinate type
https://github.com/clojure/tools.deps.alpha/search?q=compare-versions#2020-11-1112:27henryw374thanks#2020-11-1113:08Alex Miller (Clojure team)I did a talk at the conj about it, that’s probably the best description right now. The summary though is that it tries to use the newest version it can.#2020-11-1113:10Alex Miller (Clojure team)https://youtu.be/7CM7Ef-dPWQ#2020-11-1114:46andy.fingerhut"the newest version it can" could be easily misinterpreted as "the newest version available on Maven central and/or Clojars", e.g. similar to behavior of using a -SNAPSHOT or -RELEASE version.#2020-11-1114:47andy.fingerhutWould it be accurate (but slightly longer) to say "the newest version among all of those versions explicitly requested by at least one of your dependencies (but top level can override that)" ?#2020-11-1114:53Alex Miller (Clojure team)yes#2020-11-1113:09Alex Miller (Clojure team)But top level dep versions always win#2020-11-1113:17henryw374gottit! thanks#2020-11-1114:57imreis there any way to tell the clj command to ignore my custom .clojure/deps.edn?#2020-11-1114:58imreah, I may have found it, -Srepro#2020-11-1416:07rickmoynihan@alexmiller: The other week I built a dumb tool to override pom version numbers, but realised that clojure -X:deps mvn-pom almost has this functionality baked in it’s just not exposed.
Is there any reason why tools.deps can’t let you pass [:params :version] here: https://github.com/clojure/tools.deps.alpha/blob/65b1f039929acdd852aae60c87b2e57606e099a3/src/main/clojure/clojure/tools/cli/api.clj#L73
?#2020-11-1416:16Alex Miller (Clojure team)no real reason - the internals have been refactored to make this feasible but really I expect this to probably go away in favor of doing this via tools.build (which uses most of this same code)#2020-11-1416:18rickmoynihanok in that case my tool wasn’t a complete waste of time 😂#2020-11-1416:33vlaaadAh, I hoped tools-build would have a solution for xml madness, that's great to hear!#2020-11-1416:39seancorfield@rickmoynihan...and this is why I've been pushing back on the depstar discussion until I see what tools.build actually brings to the table: I don't want to do work that would be made redundant by tools.build because I'd much rather use an "official" tool chain, maintained by Nubank, than have to keep maintaining and working with ad hoc third party stuff 🙂#2020-11-1615:33rickmoynihanyup I’ve never questioned that… I’m just trying to improve this area for myself and maybe others in the interim#2020-11-1513:38borkdudeborkdude/babashka {:git/url ""
:sha "41ede17be04965c975e1c0f00e221b02bc5cf879"}
I'm getting:
$ clojure -Sforce -P -A:native
Error building classpath. Manifest type not detected when finding deps for borkdude/babashka in coordinate {:git/url "", :sha "41ede17be04965c975e1c0f00e221b02bc5cf879"}
Why is this? With :deps/manifest :deps added it "works" but it can't find resources in that project#2020-11-1513:40borkdudeMaybe it has issues with projects with git modules?#2020-11-1513:41borkdudeI'll just use the mvn/version one#2020-11-1514:28Alex Miller (Clojure team)There is a known issue with git submodules#2020-11-1514:29Alex Miller (Clojure team)There’s a ticket for that, not sure if that’s what you’re seeing#2020-11-1515:16borkdudeNo worries, I already solved it a different way#2020-11-1618:43jplazaHi! I’m having this issue https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#long-classpath is there any other workaround besides the {:mvn/local-repo "/path/to/repo"} one? That is not working either#2020-11-1618:52borkdude@jplaza Hmmm. You could spit out the classpath to a file and then start Clojure using java -cp @classpath-file clojure.main but this is a bit spartan maybe.#2020-11-1618:53Alex Miller (Clojure team)there is a ticket with some work on it but I don't know that they are workarounds currently#2020-11-1618:54Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-120#2020-11-1713:06teroheI tried the @$CpFile suggested and seemed to work. Added a sample diff here: https://github.com/clojure/brew-install/pull/4#issuecomment-728905508
Doesn't check the Java version though so would need something extra for Java 8 (if needed).#2020-11-1618:56borkdude@jplaza
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "d4b6bba6bfb0a1b0b194999684e6e4e5ed"}, :content ("[email protected]")}#2020-11-1618:57borkdudeWhat this does: java reads the classpath from a file here.#2020-11-1618:57borkdudeTools.deps / clojure CLI might leverage the same workaround#2020-11-1618:57Alex Miller (Clojure team)seems like it could easily do so since it always has a classpath file already :)#2020-11-1618:58borkdudeyeah#2020-11-1618:59Alex Miller (Clojure team)this would actually be trivial#2020-11-1618:59Alex Miller (Clojure team)if you wanted to hack your ClojureTools.psm1 and try it#2020-11-1619:01Alex Miller (Clojure team)basically where you have -classpath $CP if you changed that to "@$CP" (probably not the right syntax but that's the idea) in like line 396#2020-11-1619:02Alex Miller (Clojure team)$CP here is the name of a classpath file#2020-11-1619:03Alex Miller (Clojure team)the exec case is maybe the hardest as it adds the special exec jar: -classpath "$CP;$InstallDir/exec.jar" so might need to emit a well-known variant for that case or something#2020-11-1619:04Alex Miller (Clojure team)if someone wanted to work on a patch, would be happy to integrate that#2020-11-1619:04borkdudeyou would have to append $InstallDir/exec.jar to the classpath file and then read that. Probably better to append it to some other file classpath-exec and then use that one#2020-11-1619:05borkdudeafaik you can only use one @ and not in combination with non-file classpath#2020-11-1619:06Alex Miller (Clojure team)yes that would be my expectation. I'm suggesting that right now we write 12341234.cp and we could also write 12341234-exec.cp (which included the exec jar) and use that instead for the -X case#2020-11-1619:06borkdudeyes#2020-11-1619:10borkdudeYou might want to check compatibility of this over several java versions#2020-11-1619:14borkdudeThe java docstring doesn't mention this explicitly#2020-11-1619:16Alex Miller (Clojure team)been around for a long time#2020-11-1619:16Alex Miller (Clojure team)if it works on Java 8, then it's ok by me :)#2020-11-1619:17borkdude> been around for a long time
where do you find docs about this?#2020-11-1619:18Alex Miller (Clojure team)looks like maybe java 9?#2020-11-1619:18borkdudeI'll try java 8#2020-11-1619:19borkdudenope, didn't work#2020-11-1619:19Alex Miller (Clojure team)def doc'ed here https://docs.oracle.com/javase/9/tools/java.htm#2020-11-1619:20Alex Miller (Clojure team)I think maybe it was added in java 9 based on my googling. I thought it was older than that#2020-11-1619:21Alex Miller (Clojure team)well, could make it version dependent - if java version is 9+, use @#2020-11-1619:21borkdudeRight, it seems the @argument file thing applies to every option#2020-11-1619:21Alex Miller (Clojure team)or maybe invert, if Java version is 8, don't use @#2020-11-1619:23borkdudeYeah, that makes sense. That will assume you will not re-use your cache for different java versions, so maybe include the java version in the cache hash if you haven't already#2020-11-1619:24Alex Miller (Clojure team)I don't think you need that#2020-11-1619:25Alex Miller (Clojure team)the files are the same either way, it's just whether or not you can use it via @#2020-11-1619:25borkduderight. how will the CLI know which java version is used?#2020-11-1619:25Alex Miller (Clojure team)by calling java -version ?#2020-11-1619:26borkdude$ time java -version
java -version 0.07s user 0.06s system 94% cpu 0.136 total
#2020-11-1619:27Alex Miller (Clojure team)what's the error if you use the @file on java 8?#2020-11-1619:27borkdude$ java -cp @classpath clojure.main
Error: Could not find or load main class clojure.main
#2020-11-1619:27Alex Miller (Clojure team)not too distinguishable then#2020-11-1619:28borkdudeMaybe this can be controlled via an environment variable#2020-11-1619:28Alex Miller (Clojure team)I was wondering if you could just assume it would work, detect the failure and retry#2020-11-1619:28borkdudeCLOJURE_CP_FILE=true#2020-11-1619:28Alex Miller (Clojure team)yeah, it could be an env var too. I was trying to avoid that :)#2020-11-1619:28Alex Miller (Clojure team)might be the cleanest though#2020-11-1619:29borkdudefor those experiencing this problem, they would set it, others won't#2020-11-1619:29Alex Miller (Clojure team)that's fine#2020-11-1619:29borkdudeand only on Java 9+.#2020-11-1619:30borkdude@jplaza what Java version are you using?#2020-11-1619:32borkdude@alexmiller alternative dirty workaround: use add-lib to load the libs later on ;)#2020-11-1619:34Alex Miller (Clojure team)bleh#2020-11-1619:34borkdudeyeah, I said dirty, but it's always good to list alternatives, even when they're bad ;)#2020-11-1619:36borkdudeanother alternative: won't fix until Java 8 isn't supported anymore :)#2020-11-1619:36Alex Miller (Clojure team)who knows if that day will ever come#2020-11-1619:37borkdudeanother alternative: reverse engineer the java command line tool and build a nicer front-end which supports this for older java versions ;)#2020-11-1619:41jplazaThanks for the fast reply @alexmiller @borkdude! I’m helping a teammate with that issue. She has java 9.#2020-11-1619:42jplazaI told her the problem is that she’s using Windows haha! She’s installing Windows Subsytem for Linux#2020-11-1619:42borkdudeas a quick workaround: https://clojurians.slack.com/archives/C6QH853H8/p1605552978403300#2020-11-1619:43borkdude#2020-11-1619:46jplaza😆#2020-11-1619:47borkdude> Windows command-line length limit
Does this apply to cmd.exe, powershell.exe or is it a conhost limit? Do you experience the same problem with the new Windows Terminal?#2020-11-1619:48jplazaNot sure but happens with both cmd and powershell#2020-11-1619:48borkdudewsl2 is pretty good ;)#2020-11-1619:51seancorfieldYeah, I gave up on the whole Powershell thing and just do everything on WSL2 now. Esp. with VS Code's Remote-WSL2 extension.#2020-11-1619:53borkdudeHave you also tried portal btw?#2020-11-1620:01seancorfieldYes, but I prefer to dock a GUI next to my editor rather than switch back and forth between an editor and a browser.#2020-11-1619:52seancorfield(I run VcXsrv aka Xlaunch so I can start X11-based GUIs from WSL2, such as REBL or Reveal)#2020-11-1620:51jplazaCool! Any tips on using Calva + WSL2?#2020-11-1620:53seancorfield@jplaza I use Clover with VS Code. I like working with a plain Socket REPL in processes -- I avoid nREPL and all that stuff. Clover is "Chlorine for VS Code" and I was previously using Chlorine on Atom.#2020-11-1620:59jplazaOK I will try that. I already cloned your deps.edn https://github.com/seancorfield/dot-clojure to play with the sock repl#2020-11-1621:00borkdude@jplaza Many people use Calva. Check out #calva. There's a lot going on there.#2020-11-1621:01borkdude@jplaza Also check out out the clj-kondo plugin for VScode </shameless-plug>#2020-11-1622:34jplazaWe use clj-kondo in our team! Awesome tool! I actually mentioned our startup in the issue so you can list us there 😉#2020-11-1621:32gmorphemeI assumed clojure -X:deps mvn-pom :lib 'foo/bar' :version '"0.0.1-SNAPSHOT"' would generate generate me a POM with the specified details but I’m not having any luck - I just get defaults. (Maybe mvn-pom doesn’t pass the args on through to sync-pom?). Was this not an intended usage? Or am I missing something? Or just an oversight?#2020-11-1621:57Alex Miller (Clojure team)it does not pass those through right now#2020-11-1621:57Alex Miller (Clojure team)mvn-pom may go away in favor of tools.build once that is out so I'm resisting the urge to make it better#2020-11-1709:11gmorphemeunderstood, thanks#2020-11-1709:27rickmoynihanIn the mean time you might try this:
https://github.com/rickmoynihan/pom-update#2020-11-1709:29rickmoynihanhot off the press so YMMV, I’ve not even properly announced it yet.#2020-11-1709:41gmorphemeThanks - works for me if I generate the POM with tools.deps then update the version with pom-update. I needed to add :pom/write-strategy :overwrite :pom/input-file '"pom.xml"' explicitly though.#2020-11-1711:02rickmoynihanoh yes thanks that’s intentional, I just forgot to add that to the README.md when I wrote it up last night.
I’ll do that now.
Thanks for letting me know.#2020-11-1711:03rickmoynihanFixed.#2020-11-1711:03rickmoynihanMy recommendation is that you include those in your deps.edn rather than via command line, as they won’t vary across calls.#2020-11-1711:03rickmoynihanSee the updated README.md#2020-11-1622:12pez@jplaza if you find out stuff about using WSL2 with Calva, please consider updating this page: https://calva.io/wsl/#2020-11-1622:35jplazaWill do #2020-11-1714:53imreHey folks. I just ran into a problem at deps resolution: Error building classpath. Could not find artifact gcardone:junidecode:jar:0.2 in central ()
My project is not directly referencing this lib and I don't know where in the dep tree it is referenced from. Is there any way to trace this with the clojure command line?#2020-11-1714:55borkdudeclojure -Stree#2020-11-1714:55imreUnfortunately that fails with the same error without printing anything useful. So do -Spath and -Strace#2020-11-1714:56borkdudeoh right :)#2020-11-1714:56borkdudethen bisect?#2020-11-1714:57borkdudemaybe also try -Sforce which will force recreation of classpath#2020-11-1714:57imrebisect it is then#2020-11-1714:57imre😞#2020-11-1714:57imreforce does not help#2020-11-1714:59dominicmI'm surprised trace doesn't output anything. That would be pretty useful for this case.#2020-11-1714:59dominicmIs the secret verbose flag still a thing the?#2020-11-1715:00Alex Miller (Clojure team)no#2020-11-1715:00Alex Miller (Clojure team)there isn't a good answer for this right now other than tracing down the deps tree semi-manually#2020-11-1715:00Alex Miller (Clojure team)I've been looking at some new tools for questions like this#2020-11-1715:01Alex Miller (Clojure team)or you could -Spom and mvn dependency:tree#2020-11-1715:01Alex Miller (Clojure team)might get more info that way#2020-11-1715:01imrehmm could try that yeah#2020-11-1715:01imrewell, that also fails with the same I'm afraid#2020-11-1715:02Alex Miller (Clojure team)maybe mvn -X dependency:tree ?#2020-11-1715:03imreSorry, I meant -Spom fails#2020-11-1715:04Alex Miller (Clojure team)oh, right :)#2020-11-1715:21imrefound it in the end by reverting and updating deps one by one#2020-11-1915:24Michaël SalihiHi!#2020-11-1915:26Michaël SalihiIs it possible to use :local/root with a local Java library ?#2020-11-1915:31Alex Miller (Clojure team)only if you independently compile the .java files into .class and include the class dir as a source path#2020-11-1915:37Michaël SalihiPerfect, that's what I thought.
Thanks#2020-11-2014:05millettjonin deps.edn, is it possible to have one alias include another?#2020-11-2014:10dominicm@millettjon You can have an alias include directories from another like so:
:aliases {:foo-paths ["src"] :some-alias {:extra-paths [:foo-paths "some-alias-path"]}}
#2020-11-2014:14delaguardoare you sure?
clj -Sdeps '{:paths ["foo"] :aliases {:foo {:extra-paths ["bar"]} :bar {:extra-paths [:foo "baz"]}}}' -A:bar -Spath
=> baz:foo:...
without paths from alias :foo#2020-11-2014:16millettjonLooks like this is the JIRA ticket: https://clojure.atlassian.net/browse/TDEPS-170#2020-11-2014:16dominicm@U04V4KLKC You changed my example :)#2020-11-2014:19millettjonThanks. I would use both in this case.#2020-11-2014:24delaguardo@U09LZR36F right, I didn’t know that you can have a vector of strings as a value for alias keyword. Not sure if this is a part of official api thought#2020-11-2014:30borkdudeHuh, is this new?#2020-11-2014:31borkdudeAnd does it work or not? I see conflicting evidence with @U04V4KLKC's example?#2020-11-2014:34delaguardo:aliases {:foo ["X"]
:bar {:extra-paths [:foo "Y"]}
is working
clj -Sdeps '{:aliases {:foo ["X"] :bar {:extra-paths [:foo "Y"]}}}' -Srepro -A:bar -Spath
=> X:Y:...
but it also is not valid acording the spec in tools.deps.alpha — https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/specs.clj#2020-11-2014:34millettjonI can see the jira ticket but don't see how to vote. Maybe I don't have the right access?#2020-11-2014:35millettjonYeah I didn't see it mentioned anywhere but this is the second time I have wanted to use include functionality.#2020-11-2014:35borkdudeThat is a common issue with JIRA. It sucks.#2020-11-2015:14dominicm@U04V4KLKC this is part of the official API, yes.#2020-11-2015:14dominicmThis is new in tools deps#2020-11-2015:19delaguardodo you know where this declared? Can’t find it in cli guide or reference#2020-11-2015:23delaguardoalso (if this is new) when it was introduced?#2020-11-2015:24delaguardonwm, found it in changelog#2020-11-2015:25borkdudewhen?#2020-11-2015:26delaguardoin 0.9.745#2020-11-2015:26delaguardo• Added support for using data stored in aliases as :paths#2020-11-2015:26borkdudedo you have a link to the changelogs? I never know where to look: in brew install clojure or tools-deps?#2020-11-2015:26delaguardohttps://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md#2020-11-2015:29delaguardobut this is soooo confusing
clj -Sdeps '{:aliases {:foo ["x"]}}' -A:foo -Spath
=> "src:..."
seems like this isn’t working in that case#2020-11-2015:29borkdudeI would have never guessed that
> Added support for using data stored in aliases as :paths
this would mean the above#2020-11-2015:29delaguardosame#2020-11-2016:49seancorfield@U04V4KLKC There's an example in the docs of this https://clojure.org/reference/deps_and_cli#_paths#2020-11-2016:50seancorfield{:paths [:clj-paths :resource-paths]
:aliases
{:clj-paths ["src/clj" "src/cljc"]
:resource-paths ["resources"]}}#2020-11-2016:50seancorfieldHere are the CLI release notes: https://clojure.org/releases/tools#2020-11-2016:52seancorfieldAh, I see Alex linked you to the same example in the docs (but at least the CLI releases link is new to this thread).#2020-11-2016:53delaguardoyes, this one is new, thanks!#2020-11-2016:54seancorfieldYou can also use aliases for :exec-args to point to a data structure that is then used as the arguments to the :exec-fn#2020-11-2016:55delaguardoas a complete replacement of exec-args? like :exec-args :one-of-aliases ?#2020-11-2016:55seancorfieldSo far those are the only two places where the CLI/t.d.a itself allows/uses aliased data. Yes, like that.#2020-11-2016:56seancorfield(the :exec-args usage is not documented on that page, unfortunately)#2020-11-2016:57seancorfieldWith the Clojure basis available to any program run from the CLI, you can also read the merged deps.edn data and access aliased data inside your program at runtime.#2020-11-2017:00delaguardoI wish to have ability to provide custom data literals for deps.edn reader. then those tiny references to data in aliases can be colored as #aliases/ref :some-alias and allows to be used as a partial replacement for exec-args#2020-11-2017:01delaguardowithout this twick it is hard to say what is going on here :paths [:foo "bar"] without reading documentation (with blanks 😞 )#2020-11-2017:04seancorfieldFeel free to post requests to improve the docs on http://ask.clojure.org -- or if you've signed the Contributor's Agreement, you could submit PRs directly to https://github.com/clojure/clojure-site/pulls (but probably best to discuss with @alexmiller first to see how he'd like this clarified). Suggestions for changing the deps.edn file format itself can go on http://ask.clojure.org#2020-11-2014:10dominicmFor including dependencies, there's a JIRA for that and you might be able to vote on it there if you have an account.#2020-11-2014:22jmayaalvis there a way to tell cli where to search for maven settings.xml ? maven allows to override the location with -s (ex: mvn -s myptah/settings.xml install)#2020-11-2015:43Alex Miller (Clojure team)no#2020-11-2015:43Alex Miller (Clojure team)if you want to file an https://ask.clojure.org request for that, I don't think we have any such question/ticket right now#2020-11-2015:48jmayaalvWill do ! thanks @alexmiller#2020-11-2015:54delaguardoI have a question -
should this clj -Sdeps '{:aliases {:foo ["x"]}}' -A:foo -Spath
work the same as clj -Sdeps '{:aliases {:foo {:extra-paths ["x"]}}}' -A:foo -Spath ?
right now I’m getting different results. In first case there is no “x” in my path#2020-11-2016:03Alex Miller (Clojure team)the first is not a thing#2020-11-2016:03Alex Miller (Clojure team)not sure what you're expecting there?#2020-11-2016:04Alex Miller (Clojure team)unless you do
clj -Sdeps '{:paths [:foo] :aliases {:foo ["x"]}}' -A:foo -Spath#2020-11-2016:04delaguardojust curiosity
as an example this clj -Sdeps '{:aliases {:foo ["x"] :bar {:extra-paths [:foo]}}}' -A:bar -Spath is working wierdly#2020-11-2016:04Alex Miller (Clojure team)as it should#2020-11-2016:06delaguardothat is what I found today in the slack thread
was surprised because can’t remember mention of that in cli guide or reference
also specs in tools.deps.alpha is not allowing such aliases neither as in API.md#2020-11-2016:08Alex Miller (Clojure team)there is an example in https://clojure.org/reference/deps_and_cli#_paths but may be under-doc'ed :)#2020-11-2016:09Alex Miller (Clojure team)specs may be lagging#2020-11-2016:10delaguardolooks like :aliases become a little bit overloaded key#2020-11-2016:20Alex Miller (Clojure team)not overloaded at all, just very generic. an alias is a name for a data structure.#2020-11-2016:23delaguardoand this data now is context specific + depend on it’s shape, isn’t it?#2020-11-2016:26Alex Miller (Clojure team)dependent on how you want to use it, it's just data#2020-11-2016:30delaguardosure, but this clj -Sdeps '{:aliases {:foo "x"}}' -A:foo -Spath
will throw
clj -Sdeps '{:aliases {:foo "x"}}' -A:foo -Spath
Error building classpath. nth not supported on this type: Character
java.lang.UnsupportedOperationException: nth not supported on this type: Character
this exception is telling me that data is used in a context hidden from me. So for me this is not just data#2020-11-2016:31Alex Miller (Clojure team)well, I'd say that's a bug#2020-11-2016:32Alex Miller (Clojure team)should either be ignored or have a better error in that case, feel free to log at http://ask.clojure.org#2020-11-2016:35delaguardosure, I will report it
btw, this clj -Sdeps '{:paths [:foo] :aliases {:foo "x"}}' -Spath
doesn’t give me an exception#2020-11-2016:36Alex Miller (Clojure team)iirc, it flattens so it's little loosey goosey about the colls there atm, still deciding what to allow#2020-11-2016:43schmuddeI have an issue bringing in a .jar that relies on a pom.xml that relies on Dropwizard. Having a little trouble figuring out where the issue really lies. clj -Spath gives me this error for all the .jar's dependencies: [ERROR] Failed to determine Java version for profile....
I can run the jar just fine with plain old Java. It only becomes a problem when I try to bring it into my project. Here it is in my deps.edn: com.plaid/quickstart {:local/root "src/usermanager/java/target/quickstart-1.0-SNAPSHOT.jar"}#2020-11-2016:44schmuddeThe docs tell me this is the right behavior ("If the jar includes a pom.xml file, it will be used to find transitive deps"). FWIW, each dependency is enumerated thusly: <groupId>io.dropwizard</groupId>, <artifactId>dropwizard-bom</artifactId>, <version>${dropwizard.version}</version>. With the version of Dropwizard resolving as 1.3.2. That what it seems to be choking on, based on the error.#2020-11-2016:45schmuddeShould I just build my own uberjar?#2020-11-2016:47Alex Miller (Clojure team)resolving pom files is kind of a pain in the ass (super poms, variables, env vars, etc) - it is certainly possible for the pom inside a jar to be insufficient to actually resolve the dep set#2020-11-2016:49schmuddeWell, I have the source. So I can roll it again. Maybe that's the easiest way? this .jar came in the repository.#2020-11-2016:49Alex Miller (Clojure team)if you have a concrete repro, would be happy to see a question on https://ask.clojure.org with that so I can look at it. at the very least, would like to fail gracefully and let you supply those deps explicitly.#2020-11-2016:50schmuddeDebugging it from the clojure side seems pretty silly if it's not immediately obvious what's going wrong.#2020-11-2016:50schmuddeI do. It's Plaid's API Java code... so it's also something that isn't too obscure.#2020-11-2016:51Alex Miller (Clojure team)"Failed to determine Java version for profile" means that there is a maven profile that uses maven's Java version stuff (lets you select per-java version deps)#2020-11-2016:51schmuddeI can put the question there. I'll have to give a bit of a think of how to phrase it succinctly.#2020-11-2016:51Alex Miller (Clojure team)I think maybe that requires JAVA_HOME env var#2020-11-2016:52Alex Miller (Clojure team)you could try setting that and seeing if it works#2020-11-2016:52schmuddeGot it. I'll play a little more and post something on http://ask.clojure.org. Thanks @alexmiller!#2020-11-2111:55mishahowdy!
https://akovantsev.github.io/corpus/clojure-slack/tools-deps
this channel's log as a single page (as preserved by https://clojurians-log.clojureverse.org/tools-deps)
double-click on words to filter (cmd+f to highlight as per your browser), click on dates to expand filtered-out.#2020-11-2112:57borkdudenice work. fwiw, it's also here: https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/tools-deps#2020-11-2115:18misha> Log in to Zulip
opieop#2020-11-2113:58borkdudeWe are using CLJ_CONFIG to set dependency versions properly from a "one source of truth" deps.edn file that is located in ../base.
It works great. Our only problem is generating the pom file:
$ export CLJ_CONFIG=../base
clojure -A:base -X:deps mvn-pom :lib dre.annotator/annotator
Error generating pom manifest: Bad coordinate for library selmer/selmer, expected map: nil
#2020-11-2113:59borkdudeshould we be using -X:base:deps ? this also doesn't work#2020-11-2114:33borkdudeWe're using the :default-deps feature#2020-11-2114:34borkdudeRepro:
{:aliases {:versions {:default-deps {clj-kondo/clj-kondo {:mvn/version "2020.10.10"}}}}
:deps {clj-kondo/clj-kondo nil}}
clojure -A:versions -Spath
works, but:
$ clojure -A:versions -X:deps mvn-pom
Error generating pom manifest: Bad coordinate for library clj-kondo/clj-kondo, expected map: nil
#2020-11-2217:46Alex Miller (Clojure team)Note that -X:deps mvn-pom is not affected by changes in the classpath used to run it (so using the :versions alias with it won't affect the pom that is generated). This differs from -Spom which does work from the classpath you're choosing via clj. We are still debating the best way to supply a set of classpath-modifying options to mvn-pom (and other programs like tree).#2020-11-2217:49Alex Miller (Clojure team)I wasn't able to repro the error you show above with that or with variants. The code you point to should only be happening if a coord can't be resolved so that's too probably too late to be fixing this error, but I'd like to have a good repro first#2020-11-2217:51Alex Miller (Clojure team)also btw, mvn-pom does not take a :lib argument, I'm curious how you got the idea it did#2020-11-2218:33borkdude@U064X3EF3 Tried this again.
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "e88a879a838c9d8c8da8a5aab8dad8d9d1"}, :content ("[email protected]")}#2020-11-2218:34borkdudeI don't know where the :lib argument came from. This was written by a colleague who went with vacation and this blocked our deploy.#2020-11-2218:35borkdudefwiw @seancorfield also has said he has the exact same problem at his job#2020-11-2219:08borkdudeSame with -Srepro:
$ clojure -Srepro -A:versions -X:deps mvn-pom
Error generating pom manifest: Bad coordinate for library clj-kondo/clj-kondo, expected map: nil
#2020-11-2222:24Alex Miller (Clojure team)thx, that repros for me#2020-11-2222:53Alex Miller (Clojure team)so this is exactly what I said above - the :versions alias affects the classpath of the mvn-pom program, but is not used when computing the pom deps (only the project deps.edn is used for that), and yes that's the same issue Sean has. as he said, clj -Spom works differently and that's the recommended path right now.#2020-11-2222:59borkdudealright, thanks#2020-11-2114:55borkdudemy team lead is now threatening with migrating everything to mvn ;)#2020-11-2114:58borkdudeProbably this is too strict:
https://github.com/clojure/tools.deps.alpha/blob/9b3744fc9ab3b64d1e39765afc6e478c7c77cea4/src/main/clojure/clojure/tools/deps/alpha/extensions.clj#L66-L72
The map? check should be extended to allowing nil as well#2020-11-2115:18misha> Log in to Zulip
opieop#2020-11-2118:16seancorfield@borkdude use -Spom instead#2020-11-2118:16seancorfieldWe have the exact same problem at work#2020-11-2118:18seancorfield(also we use {} instead of nil for overridden deps - and then :override-deps#2020-11-2118:40borkdudelol, I made this hack:
clojure -M:base:deps -e "(require '[clojure.tools.cli.api :as api]) (require '[clojure.tools.deps.alpha.extensions :as ext]) (alter-var-root #'ext/throw-bad-coord (constantly (fn [x y] [x y]))) (alter-var-root #'ext/throw-bad-manifest (constantly (fn [x y z] z))) (api/mvn-pom {:lib 'dre.annotator/annotator})"
but it didn't work. I'll try -Spom .#2020-11-2118:41borkdude@seancorfield I'm seeing:
Skipping paths: resources
Skipping coordinate: {:local/root ../genseng, :deps/manifest :deps, :deps/root ../genseng, :parents #{[]}, :paths [/Users/borkdude/dre/DocSearch/genseng/src]}#2020-11-2118:41borkdudewe have a couple of local/root deps which it seems it doesn't pick up?#2020-11-2118:42borkdudeNot sure if this is bad. We use the pom.xml for the uberjar with depstar:
clojure -A:base:jvm -M:depstar -m hf.depstar.uberjar target/annotator-standalone.jar -C -m dre.standalone
#2020-11-2119:40seancorfieldThe pom.xml isn't terribly important to depstar -- it gets artifact, group, and version from it. That's all.#2020-11-2119:43seancorfieldWe generated a pom.xml for each subproject that we build uberjars for but the dependencies aren't important. Then we do the CLJ_CONFIG=../versions clojure -M:defaults:depstar ...etc... much like you @borkdude#2020-11-2119:44borkdudeyep, same. thanks!#2020-11-2119:44borkdudeIn the olden boot days I wrote my own solution for this: https://github.com/borkdude/boot-bundle#2020-11-2314:27imreAlex, would it at all be in line with the direction of the clojure tools to allow -Spath '{:replace-deps {my/lib {,,,}}'. Or perhaps a switch like -Srepro that would turn off any local deps.edn sources?#2020-11-2314:29imreSo that running tools from the command-line that don't need the project's classpath could become a little less verbose?#2020-11-2314:30imreSee https://github.com/tonsky/uberdeps/issues/33 for example#2020-11-2314:30dominicm@imre what were you doing when you wanted that feature?#2020-11-2314:31imreuberdeps, for example, as linked above#2020-11-2314:31dominicmAh, I'd put an alias into -Sdeps for that, and activate it that way.#2020-11-2314:43imreYeah, I understand that's the current solution 🙂 Just asking whether it could be expected that an alias won't be needed sometime in the future#2020-11-2314:51Alex Miller (Clojure team)you can clj -Sdeps '{:aliases {:foo {:replace-deps {my/lib {,,,}} :main-opts ["-m" "my.main"]}}' -M:foo#2020-11-2314:53Alex Miller (Clojure team)I think there are definitely more things to do on tools and ad hoc tool usage like this is something that could be included#2020-11-2315:03imreThanks both of you. -M:foo is what we've been using since we realized it doesn't work as we expected without an alias 🙂 It is grand for the moment, although it won't work if there is an invalid deps.edn in the current folder. I look forward to future developments wrt tools and ad hoc usage improvements.#2020-11-2319:40seancorfield@borkdude (and others): I'm going to add a question on ask.clojure about having additional deps.edn files read in via clj/t.d.a. so I'd like some input from folks on what problem(s) this would solve for them -- in a thread please.#2020-11-2319:41seancorfieldThe first scenario that comes to my mind is a monorepo but Alex has asked for the problem statement to be couched in more general terms.#2020-11-2319:44borkdudeis this related to the CLJ_CONFIG + mvn-pom problem?#2020-11-2319:44seancorfieldSo I'd state the problem in these terms: 1. a development team works together on multiple projects that each have their deps.edn file, and 2. those developers have their own preferred tooling configured in their own ~/.clojure/deps.edn file, and 3. the team as a whole wants a level of consistency in terms of certain library or tooling versions across multiple projects, by providing some standardized aliases and some "pinned" versions (e.g., :override-deps). Currently, 1. and 2. are supported by the CLI/t.d.a. but there's nowhere to hang the multi-project standardization.#2020-11-2319:45Alex Miller (Clojure team)@borkdude no, not really. that problem is about how to convey cp modifications to -X programs#2020-11-2319:45seancorfieldCurrently, folks who want 3. are using CLJ_CONFIG to provide a team-standard deps.edn file but losing 1. in the process.#2020-11-2319:46seancorfieldThe cp/deps modification to -X programs is a separate (but also a real) issue that would benefit from some sort of standardized approach that could be used by tooling in general (not just mvn-pom). So let's keep that discussion separate.#2020-11-2319:47seancorfield@borkdude Does my three-point problem statement match up with your CLJ_CONFIG use case, or is yours a separate scenario?#2020-11-2319:58borkdude1) yes, 2) yes. 3) we use :default-deps for this with nil versions (as documented).#2020-11-2320:04borkdudeRight now it works well enough for me. Any tools I need for development I can also just put in our work repo's deps.edn. But it would be nice if I should not have to do that.#2020-11-2320:06borkdudeIn clj-kondo I have a concept called config-paths which is a list of dirs that possibly have a configuration for clj-kondo to be merged in. It's similar to CLJ_CONFIG, but CLJ_CONFIG only supports one and not a list of dirs.#2020-11-2320:07borkdudeBasically a classpath for config.#2020-11-2320:07borkdudeAnd your home dir's clj-kondo config is implicitly part of your config paths, unless you opt out of that#2020-11-2320:35seancorfieldhttps://ask.clojure.org/index.php/9849/teams-common-dependencies-tooling-across-multiple-projects#2020-11-2321:20AleedI’ve been experimenting with a library, Interdep, for managing multiple dependencies and aliases in monorepos. It provides utilities for processing multiple deps.edn configs, so that a Clojure program could be started using that final config using Bash or Babashka (which is what the example in repo uses)
https://github.com/rejoice-cljc/interdep#2020-11-2321:22AleedI put a few constraints to get multiple unified deps configs to work well, i.e. subrepo aliases must be namespaced, since that makes them simple to merge and call from another directory#2020-11-2322:11seancorfield@UPH6EL9DH Interesting, but it looks/feels clunky as a workflow and is still just papering over the underlying problem -- which should really be solved inside t.d.a itself.#2020-11-2322:11seancorfield(monorepo is only a special case of the ask that I posted)#2020-11-2322:20AleedI’d be interested in hearing what you feel is clunky about it. Unifying all subrepo deps and starting a clojure program with that unified config seems less clunky (imo) than overriding the cross-user project config and having to put all user dev related aliases into git history.
i would have waited for this to be addressed in tools.deps but from slack searches, this issue has been present for over 2 years and only suggested solution in convos has been overriding the CLJ_CONFIG value, or using Lein. So I figured some experimentation in user space would be useful.#2020-11-2322:23seancorfield"having to put all user dev related aliases into git history" -- huh? No, no one is suggesting that.#2020-11-2322:23borkdudeI think you're right: it solves a problem users are experiencing now. Eventually this library might become obsolete, but in the meanwhile, I think some people may benefit from it.#2020-11-2322:23seancorfieldUnless you mean "common tooling for the monorepo" -- in which case, yes, that sort of stuff should be under git.#2020-11-2322:25seancorfield@UPH6EL9DH Maybe I'm reading your project wrong, but it looks like you would need to run that preprocessing step to create the actual deps.edn file after any changes?#2020-11-2322:29borkdude@U04V70XH6 I think what he's doing is spitting out a deps.edn at CLI invocation time and then your script invokes clojure using that deps.edn:
https://github.com/rejoice-cljc/interdep/blob/c248b47d9e4a6f45982cac41f2ab3990d6f5e310/example/scripts/run.clj#L60#2020-11-2322:29borkdudeso it writes on the fly, while you're invoking your commands#2020-11-2322:30borkdudeso the used deps.edn is ephemeral#2020-11-2322:31borkdudewhich is a robust way of working around the current issues, since reading a single deps.edn is well supported#2020-11-2322:31borkdudeI understand that this is not the way things should end, but it's a nice temporary hacky workaround :)#2020-11-2322:34Aleed> “having to put all user dev related aliases into git history” -- huh? No, no one is suggesting that.
@U04V70XH6 I imagined that by overriding CLJ_CONFIG and not having a user cross project config, that devs would need to put their own custom configurations in git history. but perhaps you work around that, too, not sure.#2020-11-2322:38borkdude@UPH6EL9DH I see your example script runs clojure in a .main subdir. Doesn't this cause issues with relative file paths in an app?#2020-11-2322:38Aleedright now this is a simple library that provides utilities for processing nested deps.edn configurations, along with constraints that standardize the experience, document it, and make it more robust. it’s up to the user how they want to start their program, since there’s many different ways someone may want to do that.
the way the example in the repo does it, is it calls the two helper utilities (`multi-repo/process-deps` and multi-alias/with-profiles) and those return a map with the values of the main deps config and the activated aliases. then it writes that deps config to a subdir and uses babaskha to start the clojure process from there
https://github.com/rejoice-cljc/interdep/blob/c248b47d9e4a6f45982cac41f2ab3990d6f5e310/example/scripts/run.clj#L57-L72#2020-11-2322:41borkdudeI think it's better to start clojure from the directory you are in. So maybe it should support reading files that are not named deps.edn so it can spit out to deps.edn locally (because clojure doesn't currently have the option to point at another local deps2.edn file)#2020-11-2322:41borkdudeNote that https://github.com/borkdude/deps.clj (also works with babashka btw) supports -Sdeps-file. I'll end the hackathon discussion here. Going to bed now :).#2020-11-2322:48Aleed> I see your example script runs `clojure` in a `.main` subdir. Doesn’t this cause issues with relative file paths in an app?
@borkdude multi-repo/process-deps accepts :out-dir option that makes any paths relative to the desired directory
but reading deps file from root directory is a good idea. I’ll see if that can be done with tools.deps; thanks for feedback, rest well!#2020-11-2323:29seancorfield@UPH6EL9DH
> I imagined that by overriding CLJ_CONFIG and not having a user cross project config, that devs would need to put their own custom configurations in git history. but perhaps you work around that, too, not sure.
Yeah, it's definitely a compromise. We have an agreed subset of tooling that's in the "master" deps.edn file, that satisfies everyone's needs for working on the monorepo. Luckily, we're all fans of fairly minimal tooling so we work with either plain REPLs or Socket REPLs. Mostly, the shared dev aliases are around testing and packaging (which of course we want standardized across the team, across all subprojects) and we have a :dev alias that points at a subproject in the monorepo that knows how to start various REPLs/tooling, based on whatever happens to be on your classpath (i.e., based on the set of aliases you used).#2020-11-2323:37seancorfieldI think if we decided to switch to some sort of rewriting tool at startup, we'd want to keep it pretty simple: just slurping the "master" deps.edn file, reading it as EDN, sucking out a couple of the aliases, and just assoc'ing those into the local project deps.edn and writing it back out. FWIW, we already have a tool that merges all the subproject deps.edn files and writes to an everything pseudo-subproject that we can use as a convenient place to start a REPL if we want the entire codebase available (which is normally how I work). We manually build it, as needed, and we keep it under git. It still assumes the "master" deps.edn file is in play with its :defaults alias to specify :override-deps for everything.#2020-11-2319:50seancorfieldA separate issue that needs to go on ask.clojure to get votes and feedback is that it is fairly common for tooling to be want to read the deps.edn environment as if it were under a set of aliases, i.e., for a tool to want to see the world as if a specific set of aliases were provided, that are different to the aliases needed to run the tool. Discussion in a thread...#2020-11-2319:51seancorfieldThis touches on how -X.. mvn-pom works, as well as other tools on https://github.com/clojure/tools.deps.alpha/wiki/Tools such as depot: the environment needed to run the tool is not the same as the environment the tool wants for analysis.#2020-11-2319:54seancorfielddepstar "works around" this problem by relying on its own runtime environment being the same as what goes into the JAR file but just excluding itself. This is only tenable because it is a very small tool that has zero external dependencies. You can override this and tell depstar to use a different classpath, but that approach doesn't work for tools that actually want to look at the versions of dependencies, such as mvn-pom, depot, etc.#2020-11-2319:58Alex Miller (Clojure team)I do have this problem staked out in a table btw, just trying to decide on the right answer#2020-11-2320:00Alex Miller (Clojure team)not sure yet whether it should be a feature, a set of conventions, a set of conventions + code to use those conventions, etc.#2020-11-2320:01seancorfieldAh, OK. So no need for an "ask" on this one?#2020-11-2320:01Alex Miller (Clojure team)the datomic ion dev tools is another example#2020-11-2320:02Alex Miller (Clojure team)would be fine to have one, just by virtue of working on it for the datomic tools it's well on our radar already :)#2020-11-2320:14seancorfieldOK, I'll add a question for it then...#2020-11-2320:16borkdude> depstar "works around" this problem by relying on its own runtime environment being the same as what goes into the JAR file but just excluding itself.
I also have a couple of tools that just receive a --classpath arg. But it depends on the tool. For mvn pom that might not be so convenient, and getting a list of aliases to resolve versions with might be better.#2020-11-2320:20seancorfieldhttps://ask.clojure.org/index.php/9848/tooling-based-tools-alpha-construct-basis-specified-aliases -- feel free to add more commentary/use cases/votes there.#2020-11-2321:43vlaaadreplied there 😄#2020-11-2321:44borkdude@U47G49KHQ Can you elaborate on how that would help with invoking mvn-pom?#2020-11-2321:47vlaaadnot sure that's what was asked.. did I read it wrong?#2020-11-2321:48borkdudeThe question is how can tools like mvn-pom be made aware of aliases that should be considered in adjunction to the local deps.edn#2020-11-2321:54vlaaadah, well... tools like mvn-pom should accept basis as argument instead of a list of aliases. Converting aliases to basis should be the outer-most place in such tools.#2020-11-2321:56borkdude@U47G49KHQ how do you pass a "basis" from the command line?#2020-11-2321:59vlaaadyou don't, that's why you need to use t.d.a as a library#2020-11-2322:00seancorfieldThat's not a useful answer to this question.#2020-11-2322:01seancorfieldI mean, yes, of course the tool has to use t.d.a but the point of this is to standardize an API so that all tooling can work from a set of aliases without having to reinvent the wheel and rely on knowing so much about how t.d.a works under the hood.#2020-11-2322:02seancorfieldAnd this also builds on the other question I posted on ask.clojure which allows for additional deps.edn files to be included in the process.#2020-11-2322:02seancorfieldWe don't want every single tool out there to have to write that boilerplate code.#2020-11-2322:03vlaaadhere is hoping for tools.build 😄#2020-11-2322:03seancorfieldThat still wouldn't address either of these two issues.#2020-11-2322:04seancorfield(and we also wouldn't want to tie arbitrary tooling to tools.build usage either)#2020-11-2322:06vlaaadI have a feeling it is possible to build a variation of tools-deps that accepts multiple deps.edn files and configuration with aliases as args using tools.deps... as a library, that is then used with -X#2020-11-2322:06seancorfieldThese two issues both need to be solved inside t.d.a really.#2020-11-2322:08seancorfieldSince -X mvn-pom needs a standardized way to do this, the solution will essentially be right there inside t.d.a -- once Alex et al have figured out the best way to solve it. I created the ask as a public placeholder for something that is already on Alex's plate, in a table of pros/cons for various solutions.#2020-11-2322:15vlaaadI think my suggestion answers your question 😕 There is a problem with mvn-pom , but any tool built on deps.edn can already build a basis given a set of aliases — which is what you asked#2020-11-2322:27seancorfieldYour suggestion is actually the problem that I want solved. It isn't a solution 🙂#2020-11-2322:27Alex Miller (Clojure team)It’s not just aliases, there are other relevant params like -Srepro#2020-11-2322:28seancorfieldWe don't want every tool to have to write the same code over and over.#2020-11-2322:29seancorfieldYes, incorporating how -Srepro affects the merging of deps.edn would be important too. I don't use that flag right now because of course we have to rely on CLJ_CONFIG to "insert" that extra deps.edn file, which is why I didn't think to include it in the question 😐#2020-11-2400:32seancorfieldI've updated https://ask.clojure.org/index.php/9848/tooling-construct-specified-aliases-without-boilerplate to try to make it clearer what is being asked here -- and also to include the point that Alex reminded me of, regarding -Srepro.#2020-11-2320:20seancorfieldhttps://ask.clojure.org/index.php/9848/tooling-based-tools-alpha-construct-basis-specified-aliases -- feel free to add more commentary/use cases/votes there.#2020-11-2320:35seancorfieldhttps://ask.clojure.org/index.php/9849/teams-common-dependencies-tooling-across-multiple-projects#2020-11-2321:20AleedI’ve been experimenting with a library, Interdep, for managing multiple dependencies and aliases in monorepos. It provides utilities for processing multiple deps.edn configs, so that a Clojure program could be started using that final config using Bash or Babashka (which is what the example in repo uses)
https://github.com/rejoice-cljc/interdep#2020-11-2321:34Alex Miller (Clojure team)A prerelease version of clojure 1.10.1.739 is now available:
• When resolving pom models for local deps, use System properties so jdk profile activation works
• In tools.deps, updated specs to cover alias changes#2020-11-2321:46seancorfieldIs there an example of what that first bullet means that you can link me to (or easily describe)?#2020-11-2321:56dominicmI'm guessing that it means that when specifying a jvm-specific dependency, that now works. @U47G49KHQ :)#2020-11-2322:00vlaaadhuh, me?#2020-11-2322:01dominicm@U47G49KHQ didn't you ask for this in relation to jfx deps?#2020-11-2322:02vlaaadwell, cljfx uses activation profiles, but it always worked ¯\(ツ)/¯#2020-11-2322:03dominicmOh. So now I'm extra confused about what's new!#2020-11-2322:05seancorfieldUpdated. Will let you know if I run into any issues (but I doubt either of those changes will affect our workflow).#2020-11-2322:16Alex Miller (Clojure team)See https://ask.clojure.org/index.php/9847/how-do-i-resolve-maven-dependencies-for-a-local-jar for an example failure case#2020-11-2322:18Alex Miller (Clojure team)Wouldn’t be an issue with maven deps, only with local#2020-11-2418:06schmudde@U064X3EF3 I grabbed the pre-release on my Linux box and gave it a run. Fixes the problem with the local .jar + maven deps. Everything else is running smoothly. Much gratitude!#2020-11-2418:08Alex Miller (Clojure team)I just released it as an official release, same bits so no need to re-install#2020-11-2321:35Alex Miller (Clojure team)let me know if you see anything weird#2020-11-2522:33Alex Miller (Clojure team)I spent some time today hacking on a new deps tree that has a lot more info in it. in particular:
• lists deps not in one place but in ALL places they're used (minus some stuff in a suppression list which defaults to clojure/spec/core.specs)
• lists deps that are NOT included with reason why
• special support for the case of discovering a newer version of a dep that was previously included during the expansion - these are marked with * and previously included have reason :superseded. this happens to be the locus of many tricky cases and bugs so often I'm looking specifically for this#2020-11-2522:33Alex Miller (Clojure team)#2020-11-2522:34Alex Miller (Clojure team)example ^^, interested if you have any feedback. not yet packaged for public use as I'm still tinkering#2020-11-2613:04rickmoynihanlooks really useful! Presumably this info is also available as data via the tdeps api?#2020-11-2615:42Alex Miller (Clojure team)Not really, it’s built from the trace log so technically yes, but haven’t decided yet how to break the steps apart yet#2020-11-2615:43Alex Miller (Clojure team)What would you want to do with it as data?#2020-11-2615:45rickmoynihanI have no idea, beyond maybe visualising it. I just assumed that the rendering would’ve been built on top of such a representation.#2020-11-2619:13borkdudeEDN (or XML? :)) output would be a nice to have, although currently I'm not using it programmatically#2020-11-2619:15borkdudeLooks nice and useful!#2020-11-2620:37Alex Miller (Clojure team)Certainly feasible - would help to know what people would want to do with it#2020-11-2709:15rickmoynihanI don’t know, I guess the most important data is the concluded set of deps you’re running with — which is obviously already available… You’re probably right that the reasons tools.deps made a versioning decision are something of secondary interest only.
I guess if it were in data something like cursive or cider could use that info to annotate all the deps they know about in their UI trees etc.#2020-11-2709:16rickmoynihanI could certainly imagine people using such a tool in my editor or IDE, but the CLI interface you’ve made is totally sufficient for me.#2020-11-2712:11rickmoynihan:thinking_face: would it also be useful to see which alias was ultimately responsible for each dep being pulled in?#2020-11-2523:04seancorfieldNice! I'd only need to make a minor tweak to my "ancient" shell script to pick up those changes (adding an fgrep -v ' X ' filter) and then just cleaning up the leading . or * and maybe running it through uniq which I don't currently have to do.#2020-11-2523:04seancorfieldThat's really nice and informative. Thank you!#2020-11-2618:14ghadiReally good stuff. That output effectively answers the top questions I have for project deps#2020-11-2817:10winkywoosterI swore I saw an example of a clojure shell script that included the deps.edn in the file itself, but no amount of googling is turning anything up. Was I dreaming, or is it possible?#2020-11-2817:11nbardiukhttps://gist.github.com/ericnormand/6bb4562c4bc578ef223182e3bb1e72c5#2020-11-2817:11nbardiukor if you need oneliner https://gist.github.com/nbardiuk/166dc6695bf55a351fa3070fd4065b8a#2020-11-2817:11winkywoosterAwesome! Thanks @U076FM90B#2020-11-2818:52dominicmThe shebang won't work on Linux, so don't do that#2020-11-2818:55borkdudeYep. The ericnormand example should work correctly on linux and macOS#2020-11-2818:55borkdudeBut not on Windows, because Windows doesn't have shebangs#2020-11-2818:56dominicmThere's a Monroe shebang around somewhere that also works on both.#2020-11-2818:56borkdudeThat's why you should write a babashka script to launch Clojure cross platform ;P#2020-11-2818:57dominicmhttps://clojure.org/guides/faq#clj_standalone here you are :)#2020-11-2818:57borkdudeI think you could pull that off with environment variables#2020-11-2818:58borkdudeoh right, because strings are just constants :)#2020-11-2819:09borkdudeCross platform launcher script. Should work on linux, macos and Windows:
https://github.com/babashka/book/issues/8#2020-11-2819:19borkdudeI just found a cross platform way (which really means: also works on Windows) to the single script with deps "exec" bash trick to invoke a Clojure JVM script, using babashka:
https://github.com/babashka/book/issues/8#2020-11-2819:19borkdudeYou could also do this with JVM Clojure only, but then it's slightly slower because of the startup of the first JVM#2020-11-2910:38borkdudeI'm having a problem with a pretty normal deps.edn:
{:aliases {:json {:extra-deps {cheshire/cheshire {:mvn/version "5.10.0"}}}
:test {:extra-paths ["test"]
:extra-deps {cheshire./cheshire {:mvn/version "5.10.0"}
cognitect-labs/test-runner
{:git/url ""
:sha "cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b"}}
:main-opts ["-m" "cognitect.test-runner"]}}}
on CircleCI:
https://app.circleci.com/pipelines/github/babashka/babashka.curl/5/workflows/d90fe806-57a4-4c12-bfb3-0277ff1c5be2/jobs/16
Cloning:
Error building classpath. : Auth fail
org.eclipse.jgit.api.errors.TransportException: : Auth fail
huh, wat?#2020-11-2910:38borkdudeThis was after upgrading to 739#2020-11-2910:47borkdudeI'm suspecting it's a CircleCI problem and not a tools.deps problem... not sure.#2020-11-2910:51borkdudeIt seems to clone the repo with ssh//, I'm not sure why#2020-11-2910:52Alex Miller (Clojure team)There are git config settings that can force the transport#2020-11-2910:54Alex Miller (Clojure team)git config insteadOf #2020-11-2910:56borkdudein .git/config? or on CircleCI?#2020-11-2910:57Alex Miller (Clojure team)The former#2020-11-2910:57borkdudeWhat are you doing up at 5AM?#2020-11-2910:57Alex Miller (Clojure team)¯\(ツ)/¯#2020-11-2910:58borkdudeBut .git/config isn't part of the git repo, that's a local thing right?#2020-11-2911:01Alex Miller (Clojure team)The only other time I’ve seen this was with a global git config on a local machine#2020-11-2911:01Alex Miller (Clojure team)Not sure how it would manifest on circle#2020-11-2911:02Alex Miller (Clojure team)But that’s the only reason I’m aware of that the transport would change like that#2020-11-2911:18borkdudeOn the CircleCI machine:
$ cat /home/circleci/.gitconfig
[url ""]
insteadOf =
[gc]
auto = 0
#2020-11-2911:19borkdudeThat seems to be the culprit, but why the heck is this there.#2020-11-2911:45lispycloudsAs someone writing a CI myself, I would speculate that this is there for private repos. Its easier to pass the creds via SSH key pairs than an HTTP clone where git would go through an interactive prompt for username and passwords. Just my assumptions :thinking_face:#2020-11-2911:45lispycloudshttps://discuss.circleci.com/t/cloning-another-private-repo-in-the-build/25505/5#2020-11-2911:47borkdudeShould people not write their own instead of config for this, instead of creating errors like this maybe?#2020-11-2911:48lispycloudsthats what CircleCI recommends, to add your ssh keys. but yes, this is a problematic assumption i would agree#2020-11-2911:51lispycloudsAlso I would imagine this is an issue on CI's outside the source platform, GH actions simply sidesteps this issue as being right there with your code 😛#2020-11-2911:52borkdudeGH has other issues though. Their UI becomes completely unresponsive sometimes and has other weird glitches.#2020-11-2911:54lispycloudsyeah pros and cons of it all i guess#2020-11-2911:55borkdudeAnd, I could login via SSH in CircleCI to at least debug this problem :)#2020-11-2911:57lispycloudsthats definitely a plus point. its getting more and more harder to SSH into things nowadays with all ephemeral magic machines on Kubernetes, rainbows and unicorns 😕#2020-11-2912:00lispycloudsthe pro being extremely quickly provisioned runners but all a big powerful totally opaque box#2020-11-2912:01borkdudeyeah, that's what I dislike about it, it's opaque and when it doesn't work, you have nothing to go by#2020-11-2911:23borkduderm -rf /home/circleci/.gitconfig
seems to "fix" the problem. 🤷#2020-11-2918:03ghadihttps://media4.giphy.com/media/ID4NXWnwuLnLq/giphy.gif#2020-12-0118:13cap10morganIs it possible / advisable to get the active deps.edn alias(es) from the Clojure code invoked by them? For example, to access app-specific settings defined in that alias?#2020-12-0118:14dominicm@cap10morgan look at clojure.basis system property#2020-12-0118:14dominicmI think it's OK some of the time to do that, yeah.#2020-12-0118:14dominicmI do it to identify build directories, for example, so I can stop them being refreshed#2020-12-0118:14seancorfield@cap10morgan You can't tell what aliases were used to invoke your code but, as Dominic says, you can read the basis file in -- if you were started via the Clojure CLI.#2020-12-0118:15seancorfieldThe basis is not available for code running via java -jar etc.#2020-12-0118:15dominicmI haven't actually looked, but I thought the active aliases were in the basis.#2020-12-0118:16seancorfieldYes, all merged aliases are in the basis. So you can slurp it, read the EDN, and then navigate to :aliases -> any alias.#2020-12-0118:17seancorfieldI'm thinking it would be nice, when building an uberjar, to be able to read the basis and write it into the JAR as a resource that could be read at runtime.#2020-12-0118:17dominicmI mean the activated ones though. Currently rebuilding my computer so I can't check :)#2020-12-0118:17seancorfieldNot sure what you mean by "the activated ones"?#2020-12-0118:17dominicmKnowing that -A:dev was called, for example#2020-12-0118:18seancorfieldThe basis is the merged deps.edn (and command-line) stuff, plus :libs and :classpath etc. No, you cannot tell what aliases were used for invocation.#2020-12-0118:29cap10morganit looks like what I want is in :resolve-args of the basis. thanks!#2020-12-0118:51seancorfieldDoes that actually include the aliases used? Or just the result of processing those aliases?#2020-12-0118:52seancorfieldA quick test locally shows that :resolve-args of the basis has keys :override-deps :extra-paths :extra-deps :main-opts :jvm-opts -- so that's not the aliases, just the result of the aliases @cap10morgan#2020-12-0118:53cap10morganYeah, the result is what I want. I have app-specific keys in one or more aliases and I want to access the ones that got activated. I don't actually care which alias key(s) did it.#2020-12-0118:54seancorfield@cap10morgan I'm confused. What "app-specific keys" are you talking about?#2020-12-0118:55cap10morganwell, in general, anything you want that doesn't step on something already defined. So, for my case, I'm using :pom2project/injections.#2020-12-0118:55cap10morganAlongside the :extra-deps, :main-opts, etc.#2020-12-0119:01seancorfieldAh, so you're outside the bounds of what deps.edn and t.d.a support there.#2020-12-0119:02seancorfieldIt just happens to work right now but I doubt that is guaranteed...#2020-12-0119:06seancorfield@alexmiller Can it be relied on that :resolve-args will contain any and all keys from the merged aliases selected, not just the known, supported keys?#2020-12-0119:13Alex Miller (Clojure team)I'll say undefined :)#2020-12-0119:15Alex Miller (Clojure team)aliases (that are used to modify resolve-deps and make-classpath) are a feature of the clojure script, not of tools.deps or the basis#2020-12-0119:16Alex Miller (Clojure team)that is, they are just a mechanism with which to convey :extra-deps etc. if you look at something like calc-basis, it does not take or return anything related to aliases#2020-12-0119:19Alex Miller (Clojure team)if you want to read data from an alias, I would encourage you to convey the aliases to use via an argument, read the deps.edn (or calculate a basis), and extract the aliases that way. relying on stuffing extra stuff into the aliases used to make classpaths is probably not a good idea#2020-12-0119:39seancorfieldThanks, Alex. That's a good clarification (feature of the Clojure CLI script -- rather than t.d.a), and another reason why the "selected aliases" information is not available in the running program.#2020-12-0119:40seancorfield^ @cap10morgan Does that change how you feel about adding unknown keys in aliases?#2020-12-0119:52cap10morganThanks for that background. For now this approach makes the most sense for what I need so I'll probably stick with it. If I have to change it later, no big deal. Strictly for internal tooling for now.#2020-12-0203:01Alex Miller (Clojure team)can anyone give me a sanity check on installing clojure with the new version of homebrew from clojure tap? it is not working for me, trying to ensure it's not just me#2020-12-0203:02Alex Miller (Clojure team)brew update
brew cleanup clojure
brew uninstall -f clojure
brew install clojure/tools/clojure
#2020-12-0203:02Alex Miller (Clojure team)something like that#2020-12-0203:05Alex Miller (Clojure team)ah, I had to brew untap clojure/tools before it worked#2020-12-0203:05Alex Miller (Clojure team)they've got new shallow clone logic and I think it's just not finding new stuff because I had previously installed latest there#2020-12-0203:38seancorfieldIt worked fine for me, using brew on Ubuntu, but (as usual) I had to manually run brew link --overwrite clojure afterwards.#2020-12-0203:38seancorfieldNow I have 1.10.1.739 installed as the stable version @alexmiller#2020-12-0204:16Alex Miller (Clojure team)thx#2020-12-0204:17Alex Miller (Clojure team)I assume if you brew -v you have 2.6.x ?#2020-12-0204:30seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "93f6f2fdf0d3d7d6c0d8c7dcc3bea0a3dad0d2a4a5"}, :content ("[email protected]")}#2020-12-0204:30seancorfield(Ubuntu/WSL2 on Windows 10 -- I can try this on my Mac tomorrow)#2020-12-0218:36seancorfield@alexmiller It worked on macOS 10.12 with a couple of caveats: the uninstall step refused:
Error: Refusing to uninstall /usr/local/Cellar/clojure/1.10.1.561
because it is required by unravel, which is currently installed.
and, as on Linux, I needed to run this afterwards:
brew link --overwrite clojure
#2020-12-0218:37seancorfieldThe install step worked fine (aside from the link).#2020-12-0218:53Alex Miller (Clojure team)my tap was wholly borked, I had to untap before I got things working again, but I suspect the history of that on my machine was a unique journey :)#2020-12-0204:33Alex Miller (Clojure team)cool, thx#2020-12-0204:33Alex Miller (Clojure team)just trying to get ahead of any possible breakage#2020-12-0205:47flowthingWorked OK for me, FWIW.#2020-12-0220:26socksyToday I discovered that clojure is not thread safe. I'm not sure if it's attempting to be or not, but just in case it is, it's because of the following lines:
if [[ ! -e "$config_dir/deps.edn" ]]; then
cp "$install_dir/example-deps.edn" "$config_dir/deps.edn"
fi
If you run a clojure command in two subprocesses, e.g. clojure -M:slow-process-1 & clojure -M:slow-process-2 & on a fresh install (e.g. within a CI build image), then some of the time you'd get an error like cp: cannot create regular file '/home/circleci/.clojure/deps.edn': File exists.
My attempt at a workaround right now is to run clojure at least once before running the subprocesses, to ensure that -e "$config_dir/deps.edn" will always return true#2020-12-0220:30seancorfieldFor anyone using my dot-clojure repo, I just pushed an update that adds a :dev alias that mimics what we have at work for starting REBL or Reveal or Rebel Readline or a plain REPL (depending on what's on your classpath) as well as a Socket REPL (saved to .socket-repl-port so you get the same port number next time you start it in that directory); if it starts Reveal, it also adds in the auto-table-view stuff that I use at work. If adds a dev.clj file in ~/.clojure which is loaded via -e using (System/getProperty "user.home")#2020-12-0316:18nateThis is amazing. Thanks for sharing. There's a lot too look at and learn from.#2020-12-0317:03seancorfieldThe code in dev.clj could be cleaned up. It certainly needs a bit of refactoring after I added in the complexity needed to run Rebel and Reveal together!#2020-12-0220:46borkdude@socksy that's just bash running in two different processes trying to create the file at the roughly the same time I think.#2020-12-0220:47borkdudeoh that was your point, sorry#2020-12-0220:49socksyyou're right, but it hadn't even occurred to me that that would be an issue when we made this regression. We had two slow code format checking tools which we were running in a bash script in parallel to speed up CI builds, and had just switched from lein to clojure to run them and suddenly half our CI builds were failing. Was quite a debug :)#2020-12-0303:28Alex Miller (Clojure team)I just released a prerelease clj 1.10.1.745 with the new dep tree printing under clj -X:deps tree (want to get some feedback but I will extend to -Stree before I release it). Options available (won't be available under -Stree):
• :file <path> - make tree from a trace.edn file created by clj -Strace instead of the local deps.edn
• :indent <int> - tree spacing (default = 2)
• :hide-libs <set-of-libs> - libs to hide in the tree under the root, defaults to #{org.clojure/clojure}#2020-12-0317:22seancorfieldI can't test this in our setup at work because there's no way to tell -X:deps tree about the aliases it needs to merge in (from our "master" deps.edn file specifically, but it's general problem if you want a tree based on a set of aliases).#2020-12-0303:29Alex Miller (Clojure team)I'm also thinking about having a flag to choose whether to display excluded libs (-Stree does not, the new format does)#2020-12-0303:29Alex Miller (Clojure team)anyways, feedback welcome#2020-12-0303:31Alex Miller (Clojure team)there are also apis to get as data, might be useful to have a version to print or spit the tree as edn instead of printing? dunno#2020-12-0308:04dominicmI could use edn output, then pipe into jet for manipulation.#2020-12-0316:06jcfHey all. I'm clutching at straws here so apologies for the noise. 🙂
Anyone tried installing deps today and noticed classpath issues like this:
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()
I thought it was some silly Docker stuff I was doing but I'm seeing the same problem inside a GitHub action now. Given the line that says "Downloading" I'm confused as to why a classpath can't be built.#2020-12-0316:06jcfI started out creating a question here because I thought it was a problem with the Datomic Maven repo but I can download the JAR with aws s3 cp just fine.
http://ask.datomic.com/index.php/546/could-not-find-artifact-com-datomic-ion-jar-0-9-48-in-central#2020-12-0316:07jcfThis works:
aws s3 cp .
#2020-12-0316:08jcf$ clojure -h | head
Version: 1.10.1.739
#2020-12-0316:09jcfIt works on my machine. 😄#2020-12-0316:09Alex Miller (Clojure team)do you have AWS creds set?#2020-12-0316:10jcfOn my machine, yep. Defaults and profiles.
In the Docker container I had AWS_ACCESS_KEY_ID and secret exported at build time, but have since moved that so the creds are only there at runtime.
Inside the GitHub Action I've not exported anything or configured AWS in any way.#2020-12-0316:10jcfAre you thinking my AWS creds are making things work on this host machine?#2020-12-0316:10jcfI read that for Maven repos on S3 there's some AWS API interaction required to get the region. I guess that could be failing.#2020-12-0316:11Alex Miller (Clojure team)yes, it needs to determine the bucket location#2020-12-0316:11jcfYou can stick a ?region=... on the end of the :mvn/repo URL I think. I could try that.#2020-12-0316:11jcfI don't know which region the Datomic Cloud bucket is in but I should be able to find out.#2020-12-0316:11Alex Miller (Clojure team)it's us-east-1#2020-12-0316:12jcfI'll try that now. Thanks Mr. Miller. Uno momento.#2020-12-0316:13jcfBuilding a Docker container, and running a job (?) via GitHub Actions.#2020-12-0316:14jcfNo joy with GitHub:
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: joda-time/joda-time/2.10/joda-time-2.10.jar from central
Downloading: commons-codec/commons-codec/1.15/commons-codec-1.15.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()
#2020-12-0316:16jcfDocker sits here for a few seconds and then continues on its way:
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: tigris/tigris/0.1.2/tigris-0.1.2.jar from clojars
Downloading: javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar from central
Downloading: com/cognitect/aws/appsync/809.2.784.0/appsync-809.2.784.0.jar from central
Downloading: com/amazonaws/aws-java-sdk-s3/1.11.210/aws-java-sdk-s3-1.11.210.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()
ERROR: Service 'clojure' failed to build : The command '/bin/sh -c clojure -Srepro -P -A:dev:test:cider-clj' returned a non-zero code: 1
#2020-12-0316:16jcfI'm not sure why tools.deps is downloading things twice.#2020-12-0316:16jcfShould there be two lines "Downloading" com.datomic/ion ?#2020-12-0316:17Alex Miller (Clojure team)yes, pom and jar#2020-12-0316:17jcfAhh, that makes sense.#2020-12-0316:18jcfHmm. I'm still stumped in that case. :thinking_face:#2020-12-0316:25jcfOkay, I can reproduce locally.
mv ~/.aws{,-ignore}
mv ~/.m2/repository/com/datomic/ion{,-ignore}
Then when I jack in I see the Ion dependency being downloaded and then the same error:
error in process sentinel: Could not start nREPL server: Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true
DEPRECATED: Libs must be qualified, change compliment => compliment/compliment
DEPRECATED: Libs must be qualified, change nrepl => nrepl/nrepl
DEPRECATED: Libs must be qualified, change refactor-nrepl => refactor-nrepl/refactor-nrepl
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud
Downloading: org/clojure/clojure/maven-metadata.xml from datomic-cloud
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central ()
#2020-12-0316:29jcfInterestingly there's nothing in my ~/.m2/repository/com/datomic/ion/0.9.48 directory…#2020-12-0316:30jcfI can see two .part files show up momentarily, and then they disappear after I'm told classpath construction failed.#2020-12-0316:32Alex Miller (Clojure team)it has to do with the aws creds of whatever user you're using#2020-12-0316:34Alex Miller (Clojure team)it does not have iam access to s3 ops or something like that#2020-12-0316:35jcfSo to pull down the Ion dependency I need to configure AWS and ensure I have at least some S3 permissions?#2020-12-0316:40Alex Miller (Clojure team)you need to be using aws user credentials that don't prevent s3 use for HeadObject, GetObject, and maybe GetBucketLocation#2020-12-0316:41jcfOkay, I'll create an IAM policy that grants that access. 👍
I can test that pretty quickly with Docker. GitHub Actions will take a little longer. 🙂#2020-12-0316:42jcfDos momentos.#2020-12-0316:42Alex Miller (Clojure team)the objects you're going after definitely still exist and are public in the bucket, the issue here is around IAM on the accessing user#2020-12-0317:26jcf@alexmiller I see you've answered my question over on http://ask.datomic.com.
I've added the following policy and I'm seeing the same error:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::*"
},
{
"Sid": "",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::datomic-releases-1fc2183a/*"
}
]
}
There is no HeadObject action as far as I can tell; you get that with GetObject. Is there something else I'm missing?
I can't find anything in the Datomic Cloud or tools.deps docs otherwise I wouldn't be pestering you like this. Sorry!#2020-12-0317:27jcf#2020-12-0317:29jcf#2020-12-0317:35Alex Miller (Clojure team)sorry I'm not an IAM guru#2020-12-0317:36Alex Miller (Clojure team)is there some kind of simulator or something that can answer whether this user has access to some resource?#2020-12-0317:36jcfI've added the credentials to my local machine and I'm trying out the operations using the AWS CLI.#2020-12-0317:37jcfCopying the JAR worked. Just checking I can fetch the bucket location.#2020-12-0317:38jcfOkay, so I'm not allowed to get the bucket location. That's not surprising though as it's not my bucket. I shouldn't be able to grant myself permission to interact with buckets owned by other people.#2020-12-0317:38jcf$ aws --profile REDACTED s3api get-bucket-location --bucket datomic-releases-1fc2183a
An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
#2020-12-0317:39jcf$ aws --profile REDACTED s3 cp .
download: to ./ion-0.9.48.jar
#2020-12-0319:05Alex Miller (Clojure team)this bucket has a policy that GetBucketLocation is open to *#2020-12-0319:06Alex Miller (Clojure team)you may be hitting something region related though - what's the default region for this user?#2020-12-0319:06Alex Miller (Clojure team)buckets are inherently region-specific#2020-12-0319:07Alex Miller (Clojure team)this is the case where adding the ?region=us-east-1 suffix to the url should help I think#2020-12-0408:36jcfThis user defaults to eu-west-2. I tried the suffix on the repo URL but no dice.#2020-12-0408:36jcfAnd massive thanks for your help on this, @alexmiller. :man-bowing:#2020-12-0414:44Alex Miller (Clojure team)can you try setting your default region creds to us-east-1?#2020-12-0414:46Alex Miller (Clojure team)either via the region in your credentials file or setting AWS_REGION#2020-12-0415:38jcfComing right up! Thanks! 🙂#2020-12-0416:21jcf@alexmiller I think I've found the right combination of permissions and Docker configuration to get things working! 💥
The IAM policy I shared previously should do the trick. There were two things I had to do to get Docker working:
1. Use COPY to add an AWS directory to the container at the beginning of the build; use of environment variables is frowned up for security reasons, and is hard to get right with enviroment: and env_file: not working at build time. If you want env vars at build time you need to add ARGs to your Dockerfile and then you can set args: within your build context. 🙂
2. Replace use of the new -P option and invoke clojure -Srepro -A:dev:test:etc -e 'nil'.
I mount the same ~/.aws volume at runtime but could swap out new credentials if necessary (which could make sense if you want different access at build vs. runtime).
I've got a running container with all the Datomic goodness available including connecting the SOCKS proxy up to a running system. I can connect Cider from Emacs and get all the code reloading goodness you'd expect. I've created a client and I can list databases in the Solo topology running on AWS.
I have to say another massive thank you for all your help with this! Thank you! And merry Christmas! 🎅 🎄#2020-12-0416:22Alex Miller (Clojure team)well I'm glad it's working. I'm curious why you needed the alternate clojure command, seems like if those aliases are relevant you could use clojure -Srepro -A:dev:test:etc -P#2020-12-0416:28jcfI chalked the -P up to using a newer version of tools.deps on my host machine to what's in the most recent Docker image for Clojure. I might have just gotten the order of the arguments wrong in that case. 🤷#2020-12-0416:29Alex Miller (Clojure team)yeah, that version could be lagging#2020-12-0416:29Alex Miller (Clojure team)but if you're missing -P that's a good workaround#2020-12-0317:22seancorfieldI can't test this in our setup at work because there's no way to tell -X:deps tree about the aliases it needs to merge in (from our "master" deps.edn file specifically, but it's general problem if you want a tree based on a set of aliases).#2020-12-0318:01seancorfield@alexmiller Any further thoughts on TDEPS-174 and/or TDEPS-175? I'm finding more and more that those are obstacles to things we want to do at work 😞#2020-12-0319:08Alex Miller (Clojure team)haven't worked on it yet#2020-12-0320:00borkdudeWhy does clj download this long list of deps with an empty deps.edn? Among these is an AWS api lib. Why? I'll post this as a snippet.#2020-12-0320:01borkdude#2020-12-0320:10borkdudeWhen setting :mvn/local-repo "/tmp/mvn/foo" the list is even longer. I thought the tools jar was an uberjar that contained all clj needs#2020-12-0320:10dominicm@borkdude Is it possibly the deps in the system deps.edn?#2020-12-0320:11borkdudeI guess so yes:
:aliases {
:deps {:replace-deps {org.clojure/tools.deps.alpha {:mvn/version "0.9.833"}
org.slf4j/slf4j-nop {:mvn/version "1.7.25"}}
:ns-default clojure.tools.cli.api}
:test {:extra-paths ["test"]}
}#2020-12-0320:12borkdudeThe question remains: why is there an AWS api lib among the deps of tools.deps.alpha. That feels not right.#2020-12-0320:13Derekfetching from an s3 repo, maybe?#2020-12-0320:14borkdudeAh right, that must be it!#2020-12-0320:14borkdude#2020-12-0320:18borkdudeThe tree looks great. But to get the deps tree of tools.deps I had to add it to the main deps, because you can't activate an alias. So running into the same problem that Sean ran into.#2020-12-0320:32dominicm@borkdude I don't fully understand why those aren't part of uberjar either though :)#2020-12-0321:00Alex Miller (Clojure team)the :deps alias doesn't use the uberjar#2020-12-0321:02Alex Miller (Clojure team)running -X:deps tree is in no way magical and doesn't use any parts of clj - it's just running a program that is in tools.deps.alpha library#2020-12-0321:03Alex Miller (Clojure team)the uberjar is used by clj to get the classpath of the program to run#2020-12-0321:04Alex Miller (Clojure team)@borkdude I'm a little confused by "But to get the deps tree of tools.deps I had to add it to the main deps, because you can't activate an alias. So running into the same problem that Sean ran into."#2020-12-0321:05Alex Miller (Clojure team)or nvm, I think I am reading it properly now#2020-12-0321:06borkdude@alexmiller it's not magical, but the deps it downloads are probably already in the uberjar. but I get the point of not using that as that would be magical. unless :deps was by convention a built-in thing that always uses the tools jar.#2020-12-0321:06Alex Miller (Clojure team)correct#2020-12-0321:07Alex Miller (Clojure team)it's intentionally not that because there is no way to add additional deps and then de-dupe across the uberjar#2020-12-0321:07Alex Miller (Clojure team)uberjars are inherently unfriendly to combination#2020-12-0321:07borkdudegood point#2020-12-0321:08Alex Miller (Clojure team)uberjars are evil, but a useful evil if you keep them to yourself#2020-12-0321:09Alex Miller (Clojure team)anyways, I'm going to change -Stree to do the format above too, so you'll be able to do all the things you do now with deps#2020-12-0321:10Alex Miller (Clojure team)and whenever we get to it, you should be able to use options to modify the -X:deps tree program too#2020-12-0321:11Alex Miller (Clojure team)one question I have is whether -Stree should show something like above or if it should be filtered to just the included libs like it does now#2020-12-0321:34borkdudefiltering can always be done with grep#2020-12-0321:36borkdude$ clj -X:deps tree | grep -v "X "
#2020-12-0321:59Alex Miller (Clojure team)I guess what I'm asking is would you prefer to usually see excluded or not?#2020-12-0322:09seancorfield(FWIW, I've spent a chunk of today moving us off the CLJ_CONFIG "hack" and we've decided to bite the bullet and generate our deps.edn files from a master template and a subproject template -- handling all the :override-deps directly ourselves... if the CLI / t.d.a ever supports an additional "shared project" deps.edn file, we'll switch back to that format... just got tired of dealing with tooling that doesn't honor CLJ_CONFIG and/or aliases 😞 )#2020-12-0322:17borkdudeSo you basically went the interdeps-like route?#2020-12-0322:18seancorfieldThere were things I didn't like about how interdeps worked so I did not go back and look at it when I rebuilt our stuff today.#2020-12-0322:19borkdudeI figured you didn't want to take on some dep, that's why I added the -like suffix#2020-12-0322:19seancorfieldWe ended up with <subproject>/<subproject>-deps.edn for the templates and a <subproject>/<subproject>-deps.md5 file for the checksums so we can regen on-demand automatically in our build script.#2020-12-0322:21seancorfieldThe checksum for each subproject includes the checksum for the master file as well, so we regenerate any <subproject>/deps.edn file if either the master template or the subproject template has changed. We have not yet tackled the issue of transitive :local/root dependencies (we probably won't, unless it actually bites us).#2020-12-0322:22borkdudeI'll probably roll my own solution for this too, like I did in the boot era, if this remains unsupported.#2020-12-0322:24borkdudeDo you actually parse the EDN and merge yourself, or do you use simple string templating#2020-12-0322:29seancorfieldMerge the EDN. I lift the :defaults aliases out of the master template and pull it's :override-deps out and then update the project template deps with those overrides.#2020-12-0322:30borkdudeyeah, easy enough.#2020-12-0322:12seancorfieldFour hours of restructuring our "build" script etc and I can now run the new -X:deps tree on our subprojects 🙂#2020-12-0322:15seancorfieldSo, yeah, the new tree structure is very nice -- thanks @alexmiller -- I haven't yet seen a * case in any of our subprojects...#2020-12-0322:16Alex Miller (Clojure team)I changed that before I put it out, but you can check for the trailing :superseded tag#2020-12-0322:17seancorfieldOh, I thought * indicated the newer selected version?#2020-12-0322:17Alex Miller (Clojure team)not in the release, no#2020-12-0322:17seancorfieldAh, OK. I see some :older-version tags. Haven't seen a :superseded tag yet#2020-12-0322:18Alex Miller (Clojure team)are you still doing a flat list of deps?#2020-12-0322:19Alex Miller (Clojure team)if so, you'll never see it#2020-12-0322:20Alex Miller (Clojure team)or if you're always doing some stuff to use a consistent set of transitive dep versions, same thing#2020-12-0322:22seancorfieldJust run a full pass looking for :superseded and, yup, plenty are showing up. Nice!#2020-12-0322:22seancorfield(mostly Jackson "of course")#2020-12-0322:23Alex Miller (Clojure team)I mean, that is actually something to maybe look at given that Jackson version changes are (often) breaking#2020-12-0322:35Alex Miller (Clojure team)there's like a next level of potential analysis here to gather the set of versions seen#2020-12-0322:37Alex Miller (Clojure team)etc etc#2020-12-0322:44seancorfieldYeah, it looks like we unexpectedly have some lower level pieces of Jackson that aren't being explicitly overridden...#2020-12-0322:47seancorfieldHere's partial output:
. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.10.2 :newer-version
. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.10.2 :newer-version
X com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11 :superseded
. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11
. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11
X com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.6 :parent-omitted
X com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.6 :parent-omitted
X com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11 :superseded
I'm not sure how to interpret that @alexmiller?#2020-12-0322:52seancorfieldThese are a top-level dep in a :local/root dep of this project:
. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11
. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11
I guess we'll need to add it in some of our projects that only depend on other :local/root projects#2020-12-0322:53Alex Miller (Clojure team)the tags you'll see at the end are really an artifact of the order they're found so that's not super relevant, but can be relevant if the set of considered versions are substantially different#2020-12-0322:54Alex Miller (Clojure team)you probably aren't even use those sub parts of jackson anyways#2020-12-0322:54seancorfieldAh, OK. So nothing to worry about here because it picks the same version anyway?#2020-12-0322:57Alex Miller (Clojure team)prob not#2020-12-0520:48seancorfieldFor anyone using my dot-clojure repo along with Reveal, I've documented the :dev alias and all the things it adds for Reveal: https://github.com/seancorfield/dot-clojure#the-dev-alias -- making it much more like Cognitect's REBL (automatically display each tap>'d value in a table with metadata; displaying URLs inline as web pages, etc).#2020-12-0621:56practicalli-johnFYI: I noticed that the excellent antq project wasnt check for new library versions when added as a dependency via :replace-deps so I've created a PR (its a very simple change). I've been using :replace-deps specifically for tooling that doesn't need to use the project deps, such as clj-new.
https://github.com/liquidz/antq/pull/45#2020-12-0622:31seancorfield@jr0cket That served as a nice reminder that now we've abandoned the CLJ_CONFIG "hack" at work, we can get rid of our weird, custom "outdated deps" script and switch over to antq which works really well on our new setup!#2020-12-0622:33practicalli-johnantq has some nice coverage, it even reported a newer version in my github action config for linting my user level configuration. Very impressive.#2020-12-0700:32deactivateduserI’ve been super impressed with antq too. Big 👍 from me!#2020-12-0622:34seancorfieldWith the custom formatter, I even managed to exactly replicate the output of our custom shell script that it replaces. And it found a few more outdated deps than our script.#2020-12-0622:37practicalli-johnNice.
I've found the local/root config option very handy for hacking on an open source project that I've forked locally. I just added an alias so I could test my code before committing
:hack/antq
{:replace-deps
{antq/antq {:local/root "/home/practicalli/projects/clojure/community-tools/antq"}}
:main-opts ["-m" "antq.core"]}#2020-12-0622:38borkdudeI use that as well (usually :clj-kondo/dev, :babashka/dev, etc) but I have not used or found the need to use :replace-deps with it.#2020-12-0622:40borkdude:extra-deps works fine for this use case, it will override deps with the same qualified symbol I think, but maybe there's something subtle going on that I'm missing#2020-12-0622:42ghadi:local/root is super useful in dev mode with multiple repos#2020-12-0622:42ghadiI've been enjoying all the new knobs (I know that one isn't so new)#2020-12-0622:42borkdudeyeah, it's super useful#2020-12-0622:49practicalli-john@borkdude :replace-deps was suggested to me for aliases that run tools which do not need the :deps from the project. The project deps wont be loaded, so don't add to execution time (although it probably has to be a large set of deps to notice). Its also in the official docs now
https://clojure.org/reference/deps_and_cli#_replace_project_environment_tool#2020-12-0622:50practicalli-johnI still used extra deps for project related aliases#2020-12-0622:50borkdudeoh so because you normally use antq as a tool, you just copied that alias and made the local/root hack version?#2020-12-0622:51practicalli-johnYes,#2020-12-0622:51borkdudethat makes sense#2020-12-0622:52practicalli-johncopy paste is a wonderful excuse not to think 🙂 So i didn't really think about how I was pulling in the dependency. Using extra-deps probably is more approapriate for hack/antq#2020-12-0622:53ghadi:replace-deps also useful for when you do want to read the project deps or construct the runtime basis#2020-12-0622:53ghadibut not be in the project classpath#2020-12-0622:53borkdudeso what if you are combining multiple aliases for dev, like :clj-kondo/dev:cider-nrepl then I guess :replace-deps would still work in both aliases, combining the deps of both aliases?#2020-12-0622:54practicalli-johnSound like something interesting to test...#2020-12-0622:55borkdude@ghadi I need an example to be able to understand what you said#2020-12-0622:56ghaditools like depot or antq want to analyze the project dependencies#2020-12-0622:56seancorfieldclojure.basis will still be computed from the project deps.edn file (and user deps.edn etc).#2020-12-0622:56ghadithose are good candidates for :replace-deps#2020-12-0622:57ghadiwithout revealing much, I'm working on a tool right now that is installed as a :replace-deps tool. It need to read the project's deps.edn and produce a bunch of data derived from it#2020-12-0622:58borkdudeWhere is clojure.basis documented, I haven't looked into it (I probably encountered it while porting the new clojure bash to clojure). I can't see it here: https://clojure.org/guides/deps_and_cli#2020-12-0700:16Alex Miller (Clojure team)It is intentionally not yet documented#2020-12-0700:17Alex Miller (Clojure team)As we are probably going to add some api around it#2020-12-0622:58ghadiwithin my tool I remember :project-dir as (io/file ".") and call tools.deps as an API#2020-12-0622:58ghadiloading in the deps.edn from the project and shredding it#2020-12-0622:59ghadihttps://clojure.github.io/tools.deps.alpha/clojure.tools.deps.alpha-api.html#clojure.tools.deps.alpha/calc-basis
@borkdude ^ doc there#2020-12-0622:59borkdudeshredding = rewriting?#2020-12-0623:00ghadideriving info from it for the purposes of my tool#2020-12-0623:00ghadi(i need to produce several zip files with different partitions of the classpath)#2020-12-0623:01seancorfield@borkdude https://insideclojure.org/2020/07/28/clj-exec/ -- explains the "runtime basis"#2020-12-0623:01borkdudeit sounds like you're working on some caching tool?#2020-12-0623:02ghadisort of#2020-12-0623:03seancorfieldhttps://clojure.org/reference/deps_and_cli#_basis_and_classpath talks about the "runtime basis" but doesn't seem to clarify that you can read it via the JVM property 😐#2020-12-0623:07borkdudeThanks, I missed that because I only searched for clojure.basis#2020-12-0623:03ghadiyou shouldn't actually rely on the JVM property#2020-12-0623:04ghadiuse tools.deps programmatically.#2020-12-0623:04ghadiI assume there will eventually be some stable API for an application to discover its own runtime basis#2020-12-0623:10seancorfieldA stable API would be nice. I wrote a few questions on http://ask.clojure.org recently about providing easier ways to use t.d.a with higher-level APIs to avoid the boilerplate. A nicer API than (-> (System/getProperty "clojure.basis") (slurp) (edn/read-string)) would be appreciated 🙂#2020-12-0623:12seancorfieldAlthough right now it doesn't require you to depend on t.d.a so there's that small bonus (but I get the impression that Clojure itself will start to depend on t.d.a soon, or parts of it perhaps, maybe for the add-libs style functionality?).#2020-12-0623:13ghadimost tools people write do not need to be within the project classpath#2020-12-0623:13ghadithe depstar approach was wrong#2020-12-0623:13ghadimost tools want access to the application's basis to analyze it#2020-12-0623:13ghadie.g. see what libs are outdated#2020-12-0623:14ghadiI don't think clojure will ever depend on t.d.a., but I'm curious to know why you said that#2020-12-0623:14seancorfieldWell, now we've abandoned the CLJ_CONFIG hack at work, I'd be happy with a version of depstar that constructed the project basis instead of relying on the runtime classpath environment.#2020-12-0623:15seancorfieldBut I don't want to break anyone's projects that are already depending on how depstar works today, so I'll have to give that some careful thought.#2020-12-0623:16ghadione advantage of running as a -X tool with :replace-deps is that you don't have to be frugal about which dependencies you allow yourself to use#2020-12-0623:17ghadiyou can access the application basis to produce an uberjar with whatever nice deps you want to use#2020-12-0623:17seancorfield@ghadi Alex has talked several times about add-libs style functionality being part of Clojure itself and that's currently in t.d.a -- so my assumption is that at least part of t.d.a's functionality will be rolled into Clojure directly, or Clojure will come to depend on (a future variant of) t.d.a like it currently depends on Spec etc.#2020-12-0623:17ghadi😕#2020-12-0623:17borkdude> A nicer API than (-> (System/getProperty "clojure.basis") (slurp) (edn/read-string)) would be appreciated
It think having a generic way of storing properties in clojure would be nice. Instead of the Java properties String -> String. we could have something like clojure.core/*properties* with keyword -> parsed EDN, without re-inventing a new variable for every new thing we want to store.#2020-12-0623:18ghadiI use the runtime basis for that borkdude. here is an ex:#2020-12-0623:18borkdude(I'm also thinking of other use cases than tools.deps)#2020-12-0623:19ghadi{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.10.2-alpha4"}
org.clojure/data.json {:mvn/version "1.0.0"}
cognitect-labs/secret :local/root "/Users/ghadi/dev/secret"}}
:aliases {:secret-deploy {:replace-deps {cognitect-labs/deploy {:local/root "/Users/ghadi/dev/deploy"}}
:exec-fn cognitect-labs.secret.deploy/magic
:exec-args {:compile-nses [hello-clojure
cognitect-labs.secret.runtime]}}}}#2020-12-0623:20ghadithen I run clojure -X:secret-deploy#2020-12-0623:20ghadithe code within cognitect-labs.secret.deploy/magic reads the local project's deps.edn and "does things" with it#2020-12-0623:21ghadibut you can put arbitrary data in aliases now, like the :compile-nses stuff#2020-12-0623:23borkdudeyeah, you can store data in deps.edn, but it can be handy to have a general *properties* thing with the actual runtime data as well. e.g. instead of *command-line-args* we could have (:command-line-args *properties*) without ever introducing a new dynamic var for environment specific things again#2020-12-0623:23ghadiyeah, that's the injected basis#2020-12-0623:24ghadijust need a stable API 🙂 . I am 90% sure the clojure.basis property is going away#2020-12-0623:28ghadiI see that clojure.basis is mentioned in Alex's blog#2020-12-0623:28borkdude(afk now, sleeping)#2020-12-0623:28seancorfieldFWIW, clj-new uses the runtime basis right via the clojure.basis property but switching to a nice API that reads the deps and computes it would be nicer.#2020-12-0623:31seancorfieldRight now it still depends on being introduced via :extra-deps but it would make more sense to use :replace-deps and calc-basis -- but that goes to the "standard way to provide aliases to tools" Q I posted on http://ask.clojure.org @ghadi#2020-12-0623:34ghadiyeah that's an interesting API design problem#2020-12-0623:34ghadihow do you talk about the aliases to run the tool and then also about the tool's inputs?#2020-12-0623:35seancorfieldYeah, and right now every tool does something different and ad hoc 😐#2020-12-0623:35ghadiin my example above, I'd add :aliases as a parameter within the :exec-args map#2020-12-0623:36ghadi(but I haven't the need for this tool)#2020-12-0623:37seancorfieldYeah, maybe as everything catches up with t.d.a and switches over to the -X approach and :replace-deps we can standardize things.#2020-12-0700:26seancorfieldI think one thing that would be really helpful would be if Cognitect could at least document a minimal, standard way to use t.d.a to get the project basis -- that's not entirely clear just from the API docs.#2020-12-0700:26Alex Miller (Clojure team)Yeah, I have been thinking about just that#2020-12-0700:27Alex Miller (Clojure team)Having just done it yet again for tree stuff :)#2020-12-0700:27seancorfieldI think it's this code, but I'm not certain:
user=> (require '[clojure.tools.deps.alpha :as t])
nil
user=> (keys (t/find-edn-maps))
(:root-edn :user-edn :project-edn)
;; -Srepro would omit user-edn here:
user=> (def deps (t/merge-edns (let [{:keys [root-edn user-edn project-edn]} (t/find-edn-maps)] [root-edn user-edn project-edn])))
#'user/deps
;; if you want to apply aliases at the tool level:
user=> (def combined (t/combine-aliases deps [:test :runner]))
#'user/combined
user=> (keys combined)
(:extra-paths :extra-deps :main-opts)
user=> combined
{:extra-paths ["src/test/clojure"], :extra-deps {org.clojure/test.check #:mvn{:version "1.0.0"}, com.cognitect/test-runner {:git/url "", :sha "f7ef16dc3b8332b0d77bc0274578ad5270fbfedd"}}, :main-opts ["-m" "cognitect.test-runner" "-d" "src/test/clojure"]}
user=> (def basis (t/calc-basis deps {:resolve-args combined :classpath-args combined}))
#'user/basis#2020-12-0700:28Alex Miller (Clojure team)The thing is you don’t necessarily want to do all of those steps#2020-12-0700:28Alex Miller (Clojure team)It’s in parts so you can choose which ones to do#2020-12-0700:28seancorfieldYeah, if you don't need to apply aliases, you can omit the combined piece and pass an empty map to calc-basis?#2020-12-0700:28Alex Miller (Clojure team)And modify in between if needed#2020-12-0700:29seancorfield(I'm cheating a bit with passing combined as both :resolve-args and :classpath-args there)#2020-12-0702:58Alex Miller (Clojure team)Prerelease of Clojure tools 1.10.1.749
• New option for `-X:deps tree` - `:format :edn` to print the tree as edn data
• Updated `-Stree` to also use the new format#2020-12-0708:05dominicmBenefit of system property over depending on tda is that I can make deps.edn optional AND I don't need to bloat the classpath.
I read the basis to locate output directories which tools namespace doesn't handle well.#2020-12-0708:39borkdudeI would prefer not to depend on tools.deps in clojure itself, but a decoupled system that can represent properties in EDN from whatever tool likes to put that in at startup. Comparable to System properties, but real data.
E.g.: https://clojurians.slack.com/archives/C6QH853H8/p1607297023257300#2020-12-0708:43borkdudeSimilarly spec could have been made optional with some more general hook system in the compiler for checking macros, possibly.#2020-12-0713:45Alex Miller (Clojure team)certainly we'll not make clojure depend on tools.deps#2020-12-0713:46Alex Miller (Clojure team)there is a basis api in the add-lib3 branch - some of that may move into either a very skinny lib or clojure itself tbd (that part would have no additional deps)#2020-12-0713:47Alex Miller (Clojure team)is the *properties* idea above about :aliases or more generic?#2020-12-0713:59Alex Miller (Clojure team)if not coming from :aliases, where do you imagine it's coming from?#2020-12-0714:51borkdude> certainly we'll not make clojure depend on tools.deps
<3 <3#2020-12-0714:54borkdudeAbout *properties*, I was thinking of a general way to set key/vals in a Clojure runtime, without having to deal with strings only (like in System/properties). This way you can avoid introducing more dynamic vars.#2020-12-0714:55borkdudee.g. *command-line-args* could just be a key in that thing, and so could *clojure-version*#2020-12-0714:55borkdudeand perhaps :clojure-basis or :clojure/basis#2020-12-0714:59borkdudeAnd one could perhaps set them from the command line with -J-Dclojure.property.clojure/basis={:a 1} so (= {:a 1} (:clojure/basis *properties*))#2020-12-0715:00Alex Miller (Clojure team)I think you are glossing over potentially important details, but I'll keep it in mind#2020-12-0717:43dominicmOne benefit of a dynamic var would be binding it for different class loaders.#2020-12-0804:03Alex Miller (Clojure team)I released clj 1.10.1.754 tonight, which has a bug fix for -Stree from yesterday and updates the windows scripts. If anyone wanted to give it a try I can promote it to stable. You should see new format for clj -Stree and same for clj -X:deps tree (see https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/tree for more options)#2020-12-0804:05Alex Miller (Clojure team)edn format tree nodes described at https://clojure.github.io/tools.deps.alpha/clojure.tools.deps.alpha-api.html#clojure.tools.deps.alpha.tree/trace-%3Etree#2020-12-0804:32seancorfieldI updated my two dev systems and tested on a few projects, looks good. Updated the work system and ran -Stree against a few of our subprojects, looks good. @alexmiller#2020-12-0804:37onetomis it possible to run multiple functions via clojure, using multiple -X options?#2020-12-0804:38Alex Miller (Clojure team)no#2020-12-0804:38Alex Miller (Clojure team)once you do -X, all args afterwards go to the function you're running#2020-12-0804:44seancorfield@onetom You could always do -e "(do (invoke-fn1) (invoke-fn2))" 🙂#2020-12-0804:45seancorfield(well, using requiring-resolve for each function)#2020-12-0804:55Alex Miller (Clojure team)A new version of the clojure tools is now available (https://clojure.org/releases/tools#v1.10.1.754)
• New, more informative tree format for `clj -Stree` / `clj -X:deps tree`
• Added https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/tree for use with `clj -X:deps tree`
• Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.9.857#2020-12-0804:56Alex Miller (Clojure team)winding it's way through the homebrew tubes#2020-12-0805:06onetom1st attempt via a Nix derivation:
: clj -X:deps tree
Downloading: org/clojure/tools.deps.alpha/0.8.677/tools.deps.alpha-0.8.677.pom from central
Downloading: org/clojure/tools.deps.alpha/0.8.677/tools.deps.alpha-0.8.677.jar from central
Unqualified function can't be resolved: tree
: which clj
/nix/store/g7cpwnmzfl9ikxnm7gbnq3qqgj0mynwk-clojure-1.10.1.754/bin/clj
#2020-12-0805:09onetomclj -Stree seems to work though. it has nice X-es and dots in front of the dependency names#2020-12-0805:10seancorfield@onetom Sounds you have a :deps aliases that doesn't come from the system deps.edn file?#2020-12-0805:10seancorfieldThat version of t.d.a is old.#2020-12-0805:11seancorfieldThe new CLI uses 0.9.857#2020-12-0805:12seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "bac9dfdbd4d9fafeffe9f1eef5ea97898af3f9fb8d8c"}, :content ("[email protected]")}#2020-12-0805:13onetomi know it's the old one, that's why im surprised and showed what happened.#2020-12-0805:13seancorfieldLike I said, you have a :deps alias in your user deps.edn I suspect.#2020-12-0805:15onetomah, indeed. i probably copied it from your ~/.clojure/deps.edn#2020-12-0805:25seancorfieldhttps://github.com/seancorfield/dot-clojure/commit/431a9f0ed6d763260bfa2c9b568f405b5bd59453 July 28th I changed that 🙂#2020-12-0805:08seancorfield@alexmiller I notice that mvn-pom takes an argument argmaps as a vector of aliases -- I understand why it's called that but it's... less than ideal 🙂 aliases seems like a more intuitive name if we're going to standardize on a way to provide a list of aliases to tooling?#2020-12-0805:09seancorfield(is the tree function possibly going to take an aliases (or argmaps) argument so we can see trees based on, say, test dependencies?)#2020-12-0806:05Alex Miller (Clojure team)both this and previous are again https://clojure.atlassian.net/browse/TDEPS-175#2020-12-0806:05Alex Miller (Clojure team)you can do this for now with -Stree#2020-12-0806:06Alex Miller (Clojure team)or you can use -Strace and whatever to create a trace.edn and feed it to -X:deps tree with :file#2020-12-0806:07seancorfieldYeah, I know what the workarounds are 🙂#2020-12-0806:08seancorfieldI'm asking about a "better future" 🙂#2020-12-0806:08Alex Miller (Clojure team)well, this is the same conversation we've already had, not sure what more you're looking for#2020-12-0806:09seancorfieldI was at least hoping for a comment on the name of the argument, given our past conversations.#2020-12-0806:09Alex Miller (Clojure team)I'm not sure how to convey "haven't worked on it yet" more than previously :)#2020-12-0806:09seancorfieldFair enough. I was fishing for an opinion or a direction, that's all.#2020-12-0806:10seancorfieldI mean, I'm about to do this for depstar so calling it :argmaps because that's what mvn-install uses is one approach, but calling it :aliases seems a bit more intuitive.#2020-12-0806:12seancorfield(one big "downside" to the exec stuff is that the names of arguments become part of the API so this is more important than it used to be)#2020-12-0805:52onetomI saw in https://clojure.org/reference/deps_and_cli there is a :default-deps option for factoring out version info for common dependencies.
What's the recommended way to provide that option from a central location?
(Something like https://github.com/walmartlabs/shared-deps is for lein)#2020-12-0805:54seancorfield@onetom There are only three deps.edn files: the system (that you can't touch), your user file (in ~/.clojure), and your project file.#2020-12-0805:55onetomi thought about using CLJ_CONFIG to override the per-user default within project environments, but that would also prohibit developers to access their personal tools...#2020-12-0805:55seancorfieldWe used to do that. It "works" but it introduces all sorts of problems regarding tools. We stopped doing that.#2020-12-0805:57onetomso do you require your devs so define certain aliases in their ~/.clojure/deps.edn in order for the walmart projects to work or that's the thing you mean you stopped doing?#2020-12-0805:57onetomi for example moved the ion-dev deps into the project's own deps.edn because we were accidentally using an older version on one of the machines and it failed in mysterious ways...#2020-12-0805:58onetomi just don't want to keep copy-pasting a lot of "boilerplate" setup, while we are in a monorepo, where it could be just centralized.#2020-12-0805:58seancorfieldUnfortunately, there's no "good" solution for monorepos at this point. Or, more generally, for providing "shared project deps". Feel free to upvote this issue https://ask.clojure.org/index.php/9849/teams-common-dependencies-tooling-across-multiple-projects#2020-12-0806:00seancorfieldWe used to use the CLJ_CONFIG "hack". Now we're using a different "hack" that involves generating deps.edn from template files. Neither are good.#2020-12-0806:05onetomgood to hear that "it's not just me" 🙂
the problem is that i don't even know what would a great solution look like... otherwise i would hack together something#2020-12-0806:05onetomthanks for sharing the problem statement!#2020-12-0806:06seancorfieldI think a reasonable solution is to be able to specify one-or-more additional deps.edn files that should be processed in this order: system -> user -> <specified deps.edn files in order> -> project deps.edn -> command-line -Sdeps#2020-12-0806:06seancorfieldbut that touches a lot of parts of tools.deps.alpha#2020-12-0805:54onetomi see and can probably do clj -Sdeps $(cat ../default-deps.edn) ..., but can i do that at some more fundamental level?#2020-12-0805:54seancorfield:default-deps only works in an alias BTW, as does :override-deps#2020-12-0817:15cddrI'm trying to build a clojure project in circleci and it depends on the cognitect-labs test-runner. This all works fine locally but when circleci tries to build, I get an authentication error when it tries to download dependencies. I've seen other folks with this problem and the solution is usually to fix the entry in deps.edn to use the "https:" pointer to the test-runner rather than the "/cdn-cgi/l/email-protection" pointer but my project already does that.
https://app.circleci.com/pipelines/github/cddr/test-machine/6/workflows/05e75093-3a28-438c-97b0-7bd0f9bc5132/jobs/6
I'm using the "clojure -P -M:runner" trick to download dependencies and save the cache before actually trying to run the tests and it seems to be at this point where it fails. Would you expect that to work ok? The code for the project itself is here: https://github.com/cddr/test-machine#2020-12-0817:19cddrAnother question this has prompted is, where does clojure download repos it clones when a :git/url style dependency is used? Would be cool to delete previously downloaded repo in my local setup in order to test this locally without pushing to circleci each time I change something.#2020-12-0817:20Alex Miller (Clojure team)You may have an insteadOf rule in your ci git config that is forcing ssh urls#2020-12-0817:21cddrAh interesting. I'll try to verify that. Thanks for the hint#2020-12-0817:21Alex Miller (Clojure team)I have helped multiple people with the same problem#2020-12-0817:22Alex Miller (Clojure team)This swaps before clj even sees it#2020-12-0817:23cddr{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "54373d26373831373d1461656c316532636037306435"}, :content ("[email protected]")}#2020-12-0817:23cddrright enough#2020-12-0817:26Alex Miller (Clojure team)Not sure if this is some recent change in default config #2020-12-0817:30cddrYeah I'm using one of their pre-built base images containing tools.deps. I'll report back with where it sets this ~/.gitconfig if I can find it 🙂#2020-12-0818:12practicalli-john@cddr there is a discussion in #circleci on this very issue, with more details#2020-12-0818:18borkdude@cddr yep, had exactly the same issue. "fixed" it by adding rm -f ~/.gitconfig to my circleci builds#2020-12-0818:34kirill.salykinHi
is there a way to use paths/deps from an alias but skip the main-opts
for instance, for this test alias
:test {:extra-paths ["test"]
:extra-deps {lambdaisland/kaocha {:mvn/version "1.0.690"}
lambdaisland/kaocha-cloverage {:mvn/version "1.0.63"}
lambdaisland/kaocha-junit-xml {:mvn/version "0.0.76"}
ring/ring-mock {:mvn/version "0.3.2"}
mockery {:mvn/version "0.1.4"}
http-kit.fake/http-kit.fake {:mvn/version "0.2.1"}}
:main-opts ["-m" "kaocha.runner"]}
is there a way to skip the main-opts ?
It seems that it is being used and this fails cider-jack-in
I am looking for analog of R/C in old syntax#2020-12-0818:36seancorfield@kirill.salykin The general recommendation is to separate that sort of thing into multiple aliases.#2020-12-0818:36kirill.salykini see, thanks#2020-12-0818:36seancorfieldSo you have a :test alias with the deps/paths you need to access/run your test code in the REPL and a :runner alias that is the Kaocha deps :main-opts (and any additional deps needed)#2020-12-0818:37kirill.salykinis there are reason for such change? -R -C seemed quite useful to me#2020-12-0818:37seancorfield@alexmiller ^#2020-12-0818:38borkdude^#2020-12-0818:38seancorfieldI suspect cider-jack-in should be adding its own :main-opts after the :test alias (so that it "wins" in the merge process)#2020-12-0818:40kirill.salykinIndeed, this is exactly what happens
jack-in command: /usr/local/bin/clojure -Sdeps '{:deps {nrepl {:mvn/version "0.8.3"} cider/cider-nrepl {:mvn/version "0.25.5"}}}' -A:dev:test -m nrepl.cmdline --middleware '["cider.nrepl/cider-middleware"]'
#2020-12-0818:40seancorfieldNo, that's not what I meant.#2020-12-0818:40kirill.salykinoh#2020-12-0818:41seancorfieldI literally meant :main-opts so that it is merged last and "wins"#2020-12-0818:41kirill.salykin-A:test sets -m which seems conflicting with -m from the cider#2020-12-0818:41seancorfieldUltimately, -A will stop running :main-opts at some point according to @alexmiller#2020-12-0818:42kirill.salykinah, so we need to migrate anyway#2020-12-0818:43seancorfieldclojure -Sdeps '{:deps ... :aliases {:cider/jack-in {:main-opts ["-m" ...]}}}' -M:dev:test:cider/jack-in --middleware ...#2020-12-0818:43seancorfieldSince there is no way to specify :main-opts without an alias (at least, not right now).#2020-12-0818:43kirill.salykinprobably I should reach to cider people to discss this#2020-12-0818:43kirill.salykinthanks for help!#2020-12-0818:44dpsuttonthat's a clever idea. offhand do you know if that works for the past several versions of t.d.a?#2020-12-0818:44dpsutton(don't research, just if you know off-the-cuff)#2020-12-0818:44seancorfieldWhich "that" are you asking about?#2020-12-0818:45dpsuttonthe main opts in the inline -Sdeps and last merged wins#2020-12-0818:45seancorfieldThat's always been the way :main-opts works in aliases.#2020-12-0818:45dpsuttonawesome. thanks @U04V70XH6#2020-12-0818:45seancorfield(I assume you mean "Clojure CLI" not "t.d.a"?)#2020-12-0818:46dpsuttonyes. i always blank on how to specifically refer to the command line stuff, not the language, not the library it is built on, etc#2020-12-0818:46dpsuttoni'll go with Clojure CLI from now on#2020-12-0818:48seancorfieldThis is another case where I think -R should have been retained for "REPL" operations and expanded to include both resolve-args (old behavior) and classpath-args -- in the same way -M was expanded to include resolve-args and classpath-args -- in addition to just :main-opts (old behavior). And -A deprecated. That would have been an easier migration path for people. But that ship has sailed 😞#2020-12-0818:49seancorfield@U11BV7MTK Mostly I nitpick on that because the versions of Clojure, the Clojure CLI, and the underlying t.d.a are all different.#2020-12-0818:51kirill.salykin> This is another case where I think `-R` should have been retained for “REPL” operations and expanded to include both resolve-args (old behavior) and classpath-args -- in the same way `-M` was expanded to include resolve-args and classpath-args -- in addition to just `:main-opts` (old behavior). And `-A` deprecated. That would have been an easier migration path for people. But that ship has sailed
that would be ideal indeed#2020-12-0818:52dpsuttoni appreciate it. i hope it didn't sound snarky. i always grasp for the right way to phrase it so thanks#2020-12-0818:40seancorfield(I always start my REPL manually and then connect to it from my editor)#2020-12-0818:41borkdudeme too, never use cider-jack-in, don't want my editor to bring down the dev process, ever#2020-12-0911:10practicalli-johnThis is a good point I had not considered.
This point and given the different opinions on how to resolve cider-jack-in with clojure cli tools, I will switch to cider-connect and update the book to use that as the main approach for anything but a beginning.
I have all the aliases I need to connect CIDER, Calva and Conjure to a repl run in a terminal, and they are updated monthly.
Thank you.#2020-12-0818:41seancorfieldUltimately, -A will stop running :main-opts at some point according to @alexmiller#2020-12-0818:48seancorfieldThis is another case where I think -R should have been retained for "REPL" operations and expanded to include both resolve-args (old behavior) and classpath-args -- in the same way -M was expanded to include resolve-args and classpath-args -- in addition to just :main-opts (old behavior). And -A deprecated. That would have been an easier migration path for people. But that ship has sailed 😞#2020-12-0822:41practicalli-john@kirill.salykin you can use an alias like :middleware/cider-clj in practicalli/clojure-deps-edn and a .dir-locals.el to switch off the auto-injection of cider dependencies, however it seems very strange to run two separate tools from one Clojure command, espcially as Kaocha does not interact with the REPL as I understand it, its only looking as the file system. This just seems weird I am afraid...#2020-12-0907:07kirill.salykinThe intention is not to run two separate tools - but to use the deps and paths from alias (ignoring the -m). Similar to what was possible to achieve using -c and -r opts#2020-12-0909:20practicalli-johnThe simplest approach is to have two aliases, one for test and one for a test runner. That is what I do any what I have seen from others.
Defining specific is also more efficient, as you are only loading in libraries that you are actually using.
Loading in kaocha and it's great many dependencies adds to startup time and also more potential for library version conflicts, especially on larger Clojure projects.#2020-12-0909:22kirill.salykinI understand that two separate aliases will work
But I was looking for “-C” / “-R” analogs which allows you just pick paths/deps from the alias
(now it is deprecating, so was asking how one can to do something similar with a new syntax)#2020-12-0909:23practicalli-johnThe issue you raise is easily solved, just add an alias that runs the nREPL.middleware to the end of the chain of aliases. I used your :test alias with the :middleware/cider-clj from my practicalli/clojure-deps-edn and Cider jack-in works without any issues#2020-12-0909:24kirill.salykinoke, thanks#2020-12-0909:28practicalli-johnOr you can just set cider variables to switch the cider auto-inject off Nd just use whatever aliases you wish, so long as a cider one is last.#2020-12-0909:34practicalli-johnOn the command line, clojure -M:foo:bar:baz:lizard:spock would only run the main-opts from :spock even if all of the aliases have :main-opts configuration.
Again this is simpler than having lots of different flags to remember.#2020-12-0910:40borkdudeWIP experiment to make a flexible scripting way of invoking tools.deps from the command line with babashka: https://twitter.com/borkdude/status/1336621562682880000#2020-12-0915:18borkdudeUtil function:
(defn- merge-defaults [deps defaults]
(let [overriden (select-keys deps (keys defaults))
overriden-deps (keys overriden)
defaults (select-keys defaults overriden-deps)]
(merge deps defaults)))
(defn merge-default-deps [deps-map defaults]
(let [paths (into [[:deps]]
(map (fn [alias]
[:aliases alias])
(keys (:aliases deps-map))))]
(reduce
(fn [acc path]
(update-in acc path merge-defaults defaults))
deps-map
paths)))
(merge-default-deps '{:deps {medley/medley nil}
:aliases {:foo {medley/medley nil}}}
'{medley/medley {:mvn/version "1.3.0"}})
;;=> {:aliases {:foo {medley/medley {:mvn/version "1.3.0"}}}, :deps {medley/medley {:mvn/version "1.3.0"}}}
I assume you're doing kind of the same in your setup @seancorfield?#2020-12-0915:23Alex Miller (Clojure team)why are you not using tools.deps to do this kind of thing?#2020-12-0915:28Alex Miller (Clojure team)oh, you're actually updating aliases#2020-12-0918:26borkdude@alexmiller This would be a way to generate a deps.edn value which can be used from scripting, assembled from as many deps.edn files as one wishes#2020-12-0918:27borkdudeas an alternative for the CLJ_CONFIG hack#2020-12-0918:28borkdudeBut one could also use the above util function to emit a deps.edn file on disk and then have clojure use that.#2020-12-0919:26seancorfield@borkdude Yeah, something like that -- I'm also processing :extra-deps in aliases so that :override-deps from our template affects those too -- although I've realized that my processing doesn't correctly override transitive deps (which is why we went down the :override-deps path in the first place) so I need to make a few adjustments... and then I'm still going to be stuck as far as needing an alias for :override-deps for transitive deps which I've no way of passing in to some tooling that needs it. So it's all a bit of a mess at this point 😞#2020-12-0919:27seancorfieldFair to say that I'm very unhappy about it all at the moment.#2020-12-0919:28seancorfieldThe CLJ_CONFIG hack "works" for the :override-deps and alias usage we had -- but that backs us into a corner as far as tooling is concerned (since several tools don't include the user-level deps.edn file and several tools don't accept aliases to apply -- and some tools "fail" in both those areas).#2020-12-0919:30seancorfieldAnd our new merge/generate deps.edn files from templates "works" for tooling but now has edge cases where transitive deps are not being overridden as we'd ideally like (and I'm actually surprised it hasn't bitten us yet since we've gotten this new "Frankenstein" build approach as far as our pre-production environment).#2020-12-0919:31borkdudeWe're still using the CLJ_CONFIG approach, but maybe in time I'll just write a babashka runner script (https://clojurians.slack.com/archives/C6QH853H8/p1607510436354200) that gives us all the flexibility we need. We can just merge maps the way we like it and then pass it to tools.deps.#2020-12-0919:31seancorfieldYeah, I saw that. But we're not about to start adding yet more tooling across our stack -- we're trying to standardize and reduce the custom tooling, not add to it 😐#2020-12-0919:34borkdudeUnderstandable. Sure! Just as deps.clj, I don't expect anyone to use it because it's a custom non-official port, except for some it solves a real problem. Just helped another Windows user who had issues.#2020-12-0919:34borkdudeLet's hope these are temporary workarounds.#2020-12-0923:15Felipehey, when I update tools.deps/clj to 1.10.1.739, REBL's (rebl/ui) fails to load with java.lang.NoClassDefFoundError: Could not initialize class cognitect.rebl.impl.monaco__init. Reverting to 1.10.1.507 seems to fix it.#2020-12-0923:17seancorfield@felipecortezfi I think I had to use -Sforce to recompute the dependency cache when I ran into that error ages ago.#2020-12-0923:18Alex Miller (Clojure team)just updating to newer clj should trigger a recompute but agree that first step is -Sforce#2020-12-0923:19Alex Miller (Clojure team)I'd also be curious which REBL version you're using, there have been a number of different variants of the REBL packaging that use different mechanisms to include transitive deps#2020-12-0923:19Felipelet me try -Sforce. I'm using rebl-0.9.242#2020-12-0923:21seancorfieldI think that's the current version (although I've stopped using it now so I may not have picked up the latest dev-local bundle).#2020-12-0923:24Felipeout of curiosity, what are you using in its place, if anything?#2020-12-0923:39seancorfieldI switched to Reveal. With the customizations in my dot-clojure repo, it provides a fairly similar experience. It provides more extensibility (partly because it is open source).#2020-12-0923:40seancorfieldSee https://github.com/seancorfield/dot-clojure#the-dev-alias for some information about the customizations I've built on top of Reveal.#2020-12-0923:41Felipethanks! I'll take a look#2020-12-0923:23Felipeno luck with -Sforce 😕#2020-12-0923:23Felipethere's an error before that monaco thing, though#2020-12-0923:24FelipeExecution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
netscape.javascript.JSObject
#2020-12-0923:31Alex Miller (Clojure team)the problem here is not finding some deps, so question is why#2020-12-0923:31Alex Miller (Clojure team)can you be more concrete about your deps.edn, and the command you're running?#2020-12-0923:41Felipeas simple as it gets, I think:
➜ test-rebl clj -Sdeps '{:deps {com.cognitect/rebl {:mvn/version "0.9.242"}}}'
Clojure 1.10.1
user=> (require '[cognitect.rebl :as rebl])
nil
user=> (rebl/ui)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
netscape.javascript.JSObject
user=>#2020-12-0923:41seancorfieldJava version? JavaFX not being available?#2020-12-0923:44FelipeI don't think that's the issue, since reverting tools.deps to an older version fixes the issue, but here goes:
➜ test-rebl java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-ga)
OpenJDK 64-Bit Server VM (build 25.222-bga, mixed mode)
#2020-12-0923:55Alex Miller (Clojure team)the problem here is that adoptopenjdk 8 doesn't have the javafx libs included#2020-12-0923:55Alex Miller (Clojure team)so you'll need to either use the Oracle Java 8 or use a newer adoptopenjdk with the open javafx libs#2020-12-0923:58Alex Miller (Clojure team)I don't actually think there is an easy to obtain set of javafx deps you can use with adoptopenjdk 8#2020-12-0923:58Alex Miller (Clojure team)in other words, you've picked the only jdk available that's difficult to run javafx apps on :)#2020-12-1000:01seancorfieldZulu OpenJDK 8 has a version with OpenJFX bundled. That's what I use locally when I still need to test on JDK8.#2020-12-1000:01seancorfieldhttps://www.azul.com/downloads/zulu-community/?package=jdk-fx#2020-12-1000:04seancorfield(right now I'm running OpenJDK 15+36 and specifying OpenJFX 15-ea+6 in my ~/.clojure/deps.edn file, just FYI @felipecortezfi)#2020-12-1000:55Felipehuh, why does rolling back make javafx available then?#2020-12-1000:55Felipewith an older version of tools.deps I do get to open a REBL window#2020-12-1001:03seancorfieldAn older version of the Clojure CLI, you mean (yes, it happens to use an older version of t.d.a but 1.10.1.507 is the CLI version, not the t.d.a version).#2020-12-1001:04Felipeyep#2020-12-1001:05Felipesorry, I always get mixed up between CLI / tools.deps#2020-12-1001:06seancorfieldI have to say that what you're seeing doesn't make much sense to me. I don't see how you can run REBL on OpenJDK 8 at all. And I can't see why just changing the CLI version would affect how REBL and its dependencies are handled.#2020-12-1001:06Alex Miller (Clojure team)Ditto, not sure why it would work on the old one#2020-12-1001:07Alex Miller (Clojure team)Unless how you’re changing the version also changes the jdk#2020-12-1001:29FelipeI don't think so, but I'll check tomorrow#2020-12-1001:09seancorfield(! 806)-> JAVA_HOME=$OPENJDK8_HOME /usr/local/Cellar/clojure\@1.10.1.561/1.10.1.561/bin/clj -Sdeps '{:deps {com.cognitect/rebl {:mvn/version "0.9.242"}}}'
Clojure 1.10.1
user=> (require '[cognitect.rebl :as rebl])
nil
user=> (rebl/ui)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
netscape.javascript.JSObject
user=> ^D
(! 807)-> JAVA_HOME=$OPENJDK8_HOME java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
I can repro your JSObject error with OpenJDK8 and CLI 1.10.1.561 and REBL 0.9.242 -- which is what I would expect.#2020-12-1001:12seancorfieldAnd I see the exact same error with the very latest CLI (again, exactly as I would expect):
(! 809)-> JAVA_HOME=$OPENJDK8_HOME /usr/local/Cellar/clojure\@1.10.1.754/1.10.1.754/bin/clj -Sdeps '{:deps {com.cognitect/rebl {:mvn/version "0.9.242"}}}'
Clojure 1.10.1
user=> (require '[cognitect.rebl :as rebl])
nil
user=> (rebl/ui)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
netscape.javascript.JSObject
user=> #2020-12-1001:14seancorfieldWith JDK 15 and CLI 1.10.1.561, I get this error -- which, again, is expected because there's no OpenJFX deps in play:
(! 810)-> /usr/local/Cellar/clojure\@1.10.1.561/1.10.1.561/bin/clj -Sdeps '{:deps {com.cognitect/rebl {:mvn/version "0.9.242"}}}'
Clojure 1.10.1
user=> (require '[cognitect.rebl :as rebl])
nil
user=> (rebl/ui)
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:606).
javafx.scene.web.WebEngine
user=> ^D
(! 811)-> java -version
openjdk version "15" 2020-09-15
OpenJDK Runtime Environment AdoptOpenJDK (build 15+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15+36, mixed mode, sharing)#2020-12-1001:15seancorfield(and the exact same error happens on CLI 1.10.1.754 with that setup)#2020-12-1001:16seancorfield@felipecortezfi What's in your ~/.clojure/deps.edn file?#2020-12-1001:22Felipe{:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
:aliases {:1.8 {:override-deps {org.clojure/clojure {:mvn/version "1.8.0"}}}
:1.9 {:override-deps {org.clojure/clojure {:mvn/version "1.9.0"}}}
:1.10 {:override-deps {org.clojure/clojure {:mvn/version "1.10.0"}}}
:1.10.1 {:override-deps {org.clojure/clojure {:mvn/version "1.10.1"}}}
:master {:override-deps {org.clojure/clojure {:mvn/version "1.11.0-master-SNAPSHOT"}}}
:new {:extra-deps {seancorfield/clj-new {:mvn/version "0.8.6"}}
:main-opts ["-m" "clj-new.create"]}
:deps {:extra-deps {org.clojure/tools.deps.alpha {:git/url ""
:sha "148cab8f154711f8495bd632ebdc53a14dff7f09"}}}
:depstar {:extra-deps {seancorfield/depstar {:mvn/version "0.5.2"}}}}}#2020-12-1003:20seancorfield@felipecortezfi OK, doesn't look like anything that could disrupt what you're trying to do here... very weird...#2020-12-1003:22seancorfieldI will say: rename :deps to :add-libs and update the SHA to match what's in my latest dot-clojure file. :deps is provided in the system deps.edn (baked into the CLI and t.d.a) and having a conflicting :deps alias will cause you all sorts of breakage/confusion now that the Clojure CLI has documented -X:deps based tooling!#2020-12-1015:54arohnerI think this is a bug, which would be very helpful in CI:
$ clojure -X bogus
Unqualified function can't be resolved: bogus
bash-3.2$ echo $?
0
#2020-12-1015:55arohnersimilarly:
clojure -X clojure.core/println :foo :::invalid
Unreadable arg: ":::invalid"
bash-3.2$ echo $?
0
#2020-12-1016:03Alex Miller (Clojure team)the bug being... the exit code?#2020-12-1016:04arohneryes#2020-12-1016:05arohnerruntime errors correctly set the exit code:
$ clojure -X clojure.core/assoc 1 2
Execution error (ArityException) at clojure.run.exec/exec (exec.clj:39).
Wrong number of args (1) passed to: clojure.core/assoc--5416
Full report at:
/var/folders/4l/tfxcfcs16t17zs2_pgkyxy540000gn/T/clojure-4043596194159065702.edn
bash-3.2$ echo $?
1
#2020-12-1016:09Alex Miller (Clojure team)yeah, I got it. that's done automatically if you throw out of main. I'll fix, thanks!#2020-12-1019:47Alex Miller (Clojure team)@arohner I released 1.10.1.763 prerelease with that change if you care to try it (or just wait for the next stable)#2020-12-1019:48Alex Miller (Clojure team)I think I might actually have had that in there originally but pulled it when I was writing unit tests, with the intention to put it back#2020-12-1019:49Alex Miller (Clojure team)all of the other programs run by clj were doing the proper thing already#2020-12-1020:59arohnerAwesome, thank you#2020-12-1021:16seancorfield(this version just missed our candidate build for our next release but I have it on dev for testing; we have .739 on production and .754 in the QA pipeline)#2020-12-1208:51practicalli-johnI am trying to remember what the -f option does in this alias, is it specifying the file to look in for the preceeding -i init path option?
Or does it have a different meaning?
:repl/rebel-nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "0.8.3"}
cider/cider-nrepl {:mvn/version "0.25.5"}
com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
:main-opts ["-m" "nrepl.cmdline"
"--middleware" "[cider.nrepl/cider-middleware]"
"-i"
"-f" "rebel-readline.main/-main"]}
Thanks.#2020-12-1219:10seancorfieldThat's an argument to nrepl.cmdline @jr0cket#2020-12-1219:11practicalli-johnAh, that makes sense, thanks.#2020-12-1219:16seancorfield-f is equivalent to --repl-fn so I guess it's how you specify a particular REPL to run instead of the default nREPL one.#2020-12-1219:16seancorfield(I had to go look in the source for that -- I don't use nREPL)#2020-12-1219:22practicalli-johnOh that's very useful. I'll take a look when I get chance. Thanks.#2020-12-1222:32borkdudesmall update from babashka.deps, a way to programmatically invoke a clojure process without additional installers, cross platform (it's using deps.clj under the hood (https://github.com/borkdude/deps.clj))
$ bb "(->> (deps/clojure '[-X:deps tree :format :edn] {:out :string}) deref :out edn/read-string :children keys (take 2))"
(org.clojure/clojure org.clojure/tools.cli)#2020-12-1222:36borkdude$ bb -e "(:exit @(deps/clojure '[-X clojure.core/prn :foo :bar]))"
{:foo :bar}
0#2020-12-1311:09borkdudeYou might be wondering, what the heck is borkdude doing. It's partially inspired by communications like this :)
> Babashka and deps.clj were critical for passing the Clojure pill in my team.
> It made installation and trying stuff so much "easier" because now it was reachable without any additional admin rights.#2020-12-1320:50borkdudeBabashka 0.2.5 released with tools.deps integration. More info here https://github.com/borkdude/babashka/blob/master/CHANGELOG.md#v025 and in #babashka. I'll stop blathering about bb for now ;)#2020-12-1514:44Alexis Vincent@alexmiller I’m running into a number of cases where I want to build a library (and publish it to a maven repository) that relies on a library that only provides a gitdep. Libraries or projects using this published library then dont have access to the gitdep since its not representable in the pom. Is there a recommended way to deal with this? Or is the advice to publish copies of these git libraries to maven ourselves?#2020-12-1514:48Alex Miller (Clojure team)that's the only path afaik#2020-12-1514:49Alex Miller (Clojure team)I mean, you can ask the owner to do so of course#2020-12-1514:50Alex Miller (Clojure team)if you have questions about how to best do so, you can ask at #clojars#2020-12-1514:52Alexis VincentThanks. Do you think its worth publishing a note about this case in the docs? I didn’t make the logical leap about the effects of designing against gitdeps. Im sure others will likely not either. https://github.com/juxt/pack.alpha/issues/91#2020-12-1514:53Alexis VincentRan into this again today with https://github.com/roman01la/uix#2020-12-1514:54Alexis VincentEven though I’d posted an issue about this exact case in. November, I forgot about it and ran into the same thing#2020-12-1514:55Alexis VincentI’m concerned we end up with a bunch of library authors not realising the results of not publishing to Clojars or something similar#2020-12-1515:04dominicmThe idea of pulling deps.edn from jars was floated as a solution for this at some point.#2020-12-1515:12Alexis VincentCurrently I’m forking libs, and publishing to an internal maven repo. But from a maintenance POV this is quite an ask, especially for folks who don’t necessarily have access to the same infrastructure setup we do. Also considering dep resolution will no longer work, I’m thinking a better way of dealing with this might make sense to consider.#2020-12-1518:08hlolliIs there a way to use tools.deps to know which SNAPSHOT version is currently resolved. For example 20191108.004735-9 from the functions I've tested in clojure.tools.deps.alpha.extensions, but I didn't test everything.
That first url is made up, but I pass kitchen-async/kitchen-async #:mvn{:version "0.1.0-SNAPSHOT"} {"central" {:url ""}, "clojars" {:url ""}} as parameters (depending on the function I'm testing).#2020-12-1518:15hiredmanI don't believe tools.deps does anything with SNAPSHOT, it just gets passed on to the maven code, which picks a version when it actually looks at what is in the repos#2020-12-1518:24hlolliok, so I should rather point my attention to aether for this?#2020-12-1518:26Alex Miller (Clojure team)From tools.deps - no. From aether, I’m not really sure how to do it there either#2020-12-1518:27Alex Miller (Clojure team)You request the snapshot version and it resolves that to a url but I don’t think you can see the time stamped version #2020-12-1518:28Alex Miller (Clojure team)I guess my question would be why do you care?#2020-12-1518:34hlolliIt's for a simple app I made in nix, where I create a file with all the endpoints and the artifact's hashsum. Appearently some other people are using it, and a user made a ticket that snapshot versions are returning 404.#2020-12-1518:35hlolliSnapshot is very against integrity, but if the artifact is timestamped, it can be "frozen" to the time which the endpoints are generated.#2020-12-1518:37Alex Miller (Clojure team)well, the Maven solution to that is a non-SNAPSHOT release#2020-12-1518:37hiredmanthat sort of compounds the error of using snapshots, different people running the tool will resolve the snapshot differently#2020-12-1518:37hiredmanso its frozen, but no one agrees to what it is frozen as#2020-12-1518:39hiredmanyou can directly depend on the timestamped version too#2020-12-1518:39hlolli@hiredman true that, the generated nix file will be frozen but not the file that generated it. It would be convenience that it would work. I think any serious release should be adviced against using snapshot versions in general.#2020-12-1518:40hlolliI will try other options, thanks for both of your inputs.#2020-12-1522:37LuHello! I am using a mac mini with an M1 chip and it seems I can’t add a private github repo in my deps.edn file. The JDK I am using is this:
openjdk 11.0.9.1 2020-11-04 LTS
OpenJDK Runtime Environment Zulu11.43+1015-CA (build 11.0.9.1+1-LTS)
OpenJDK 64-Bit Server VM Zulu11.43+1015-CA (build 11.0.9.1+1-LTS, mixed mode)
and you can find the error in the thread (not to pollute the main chat) 🙂.#2020-12-1522:37LuError building classpath. /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: dlopen(/private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp, 1): no suitable image found. Did find:
[watch:cljs] /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: no matching architecture in universal wrapper
[watch:cljs] /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: no matching architecture in universal wrapper
[watch:cljs] java.lang.UnsatisfiedLinkError: /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: dlopen(/private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp, 1): no suitable image found. Did find:
[watch:cljs] /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: no matching architecture in universal wrapper
[watch:cljs] /private/var/folders/kv/64_g981979325dj8jwwd90fc0000gn/T/jna-106985/jna3259090784966322613.tmp: no matching architecture in universal wrapper
[watch:cljs] at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
[watch:cljs] at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
[watch:cljs] at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
[watch:cljs] at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
[watch:cljs] at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627)
[watch:cljs] at java.base/java.lang.Runtime.load0(Runtime.java:768)
[watch:cljs] at java.base/java.lang.System.load(System.java:1837)
[watch:cljs] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761)
[watch:cljs] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
[watch:cljs] at com.sun.jna.Native.<clinit>(Native.java:131)
[watch:cljs] at com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory$CLibrary.<clinit>(JNAUSocketFactory.java:47)
[watch:cljs] at com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory.open(JNAUSocketFactory.java:114)
[watch:cljs] at com.jcraft.jsch.agentproxy.connector.SSHAgentConnector.open(SSHAgentConnector.java:93)
[watch:cljs] at com.jcraft.jsch.agentproxy.connector.SSHAgentConnector.<init>(SSHAgentConnector.java:54)
[watch:cljs] at com.jcraft.jsch.agentproxy.ConnectorFactory.createConnector(ConnectorFactory.java:104)
[watch:cljs] at clojure.tools.gitlibs.impl$fn__1250.invokeStatic(impl.clj:31)
[watch:cljs] at clojure.tools.gitlibs.impl$fn__1250.invoke(impl.clj:29)
[watch:cljs] at clojure.lang.Delay.deref(Delay.java:42)
[watch:cljs] at clojure.core$deref.invokeStatic(core.clj:2320)
[watch:cljs] at clojure.core$deref.invoke(core.clj:2306)
[watch:cljs] at clojure.tools.gitlibs.impl$call_with_auth.invokeStatic(impl.clj:51)
[watch:cljs] at clojure.tools.gitlibs.impl$call_with_auth.invoke(impl.clj:43)
[watch:cljs] at clojure.tools.gitlibs.impl$git_clone_bare.invokeStatic(impl.clj:73)
[watch:cljs] at clojure.tools.gitlibs.impl$git_clone_bare.invoke(impl.clj:70)
[watch:cljs] at clojure.tools.gitlibs.impl$ensure_git_dir.invokeStatic(impl.clj:112)
[watch:cljs] at clojure.tools.gitlibs.impl$ensure_git_dir.invoke(impl.clj:102)
[watch:cljs] at clojure.tools.gitlibs$resolve.invokeStatic(gitlibs.clj:33)
[watch:cljs] at clojure.tools.gitlibs$resolve.invoke(gitlibs.clj:29)
[watch:cljs] at clojure.tools.gitlibs$procure.invokeStatic(gitlibs.clj:47)
[watch:cljs] at clojure.tools.gitlibs$procure.invoke(gitlibs.clj:41)
[watch:cljs] at clojure.tools.deps.alpha.extensions.git$eval1316$fn__1318.invoke(git.clj:42)
[watch:cljs] at clojure.lang.MultiFn.invoke(MultiFn.java:239)
[watch:cljs] at clojure.tools.deps.alpha$expand_deps.invokeStatic(alpha.clj:422)
[watch:cljs] at clojure.tools.deps.alpha$expand_deps.invoke(alpha.clj:390)
[watch:cljs] at clojure.tools.deps.alpha$resolve_deps.invokeStatic(alpha.clj:495)
[watch:cljs] at clojure.tools.deps.alpha$resolve_deps.invoke(alpha.clj:475)
[watch:cljs] at clojure.tools.deps.alpha$calc_basis.invokeStatic(alpha.clj:648)
[watch:cljs] at clojure.tools.deps.alpha$calc_basis.invoke(alpha.clj:622)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:91)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$run_core.invoke(make_classpath2.clj:57)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:119)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$run.invoke(make_classpath2.clj:113)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:169)
[watch:cljs] at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:140)
[watch:cljs] at clojure.lang.RestFn.applyTo(RestFn.java:137)
[watch:cljs] at clojure.lang.Var.applyTo(Var.java:705)
[watch:cljs] at clojure.core$apply.invokeStatic(core.clj:665)
[watch:cljs] at clojure.main$main_opt.invokeStatic(main.clj:514)
[watch:cljs] at clojure.main$main_opt.invoke(main.clj:510)
[watch:cljs] at clojure.main$main.invokeStatic(main.clj:664)
[watch:cljs] at clojure.main$main.doInvoke(main.clj:616)
[watch:cljs] at clojure.lang.RestFn.applyTo(RestFn.java:137)
[watch:cljs] at clojure.lang.Var.applyTo(Var.java:705)
[watch:cljs] at clojure.main.main(main.java:40)
#2020-12-1522:39Alex Miller (Clojure team)you are doing something that selects a classifier dep based on architecture#2020-12-1522:39Alex Miller (Clojure team)for a dep that has native stuff in it#2020-12-1522:40dominicm@alexmiller make-classpath I think is triggering this? Looks like jsch to me?#2020-12-1522:40dominicmso probably jgit?#2020-12-1522:40Alex Miller (Clojure team)oh yeah, it's tools.deps itself#2020-12-1522:40Alex Miller (Clojure team)but yes, jsch for ssh git support#2020-12-1522:41seancorfieldSo, no ARM support for (that version) of jsch?#2020-12-1522:41Alex Miller (Clojure team)probably not anything easy to fix at the drop of a hat#2020-12-1522:42Alex Miller (Clojure team)can you avoid using a git dep?#2020-12-1522:43seancorfield(or more likely JCE directly in the JDK doesn't support ARM?)#2020-12-1522:43Alex Miller (Clojure team)or use an https git dep? that should work as the ssh code is in a delay that won't be forced#2020-12-1522:44LuThanks for the prompt responses 🙂 Yes, I can just have the repo live in the project and use :local/root#2020-12-1522:44seancorfield@UE35Y835W Is that JDK specifically for the ARM chip or are you relying on Rosetta emulation?#2020-12-1522:44LuThat’s specific for the ARM chip.. I haven’t tried yet an emulated one tbh#2020-12-1522:44seancorfield'k... hmm, I would have expected JCE to work then. Good job you have a workaround with a local dep.#2020-12-1522:45LuOut of curiosity I’ll give it a spin with an emulated jdk#2020-12-1522:48Alex Miller (Clojure team)seems like jna not quite there yet https://github.com/java-native-access/jna/pull/1238#2020-12-1522:52Lu@alexmiller When you say to try https you mean:
bar {:git/url ""
:sha "..."}
This is failing due to authentication - no CredentialsProvider has been registered#2020-12-1522:52Lubut that’s fine anyways.. I’ll simply get it in the repo for now 🙂 Thanks all again#2020-12-1522:52Alex Miller (Clojure team)yes, that's what I meant#2020-12-1715:43pithylessIf I fork a project and publish a jar under a different clojars groupid, can I somehow alias it back to the original groupid/name? Otherwise, my understanding is even if I use :override-deps the original project will be pulled in from any transitive deps, correct? Does that mean I need to track down all potential transitive deps that use this groupid/project-name and add an :exclusion?#2020-12-1715:44pithylessNormally, I'd just use the original groupid and point to a :git/url, but this particular project has some compiled java bits.#2020-12-1715:45borkdude@pithyless :classpath-overrides + nil might work too#2020-12-1715:45borkdudealthough that might still pull in the transitive deps#2020-12-1715:47borkdudebut if your fork pulls in the same transitive deps you will probably get the newest of those transitive deps#2020-12-1715:48pithylessOK, great; thanks @borkdude#2020-12-1716:26pithylessConfirming that :classpath-overrides + nil works like a charm; cheers!#2020-12-1718:09dominicmThere's a jira for this fwiw#2020-12-1720:35Eugenhi, can I run a java class as a Clojure alias? I'm trying to generate bindings for an antlr grammar using the java tool:
The gradle code I want to replicate is:
apply plugin: 'java'
repositories {
jcenter()
}
dependencies {
runtime 'org.antlr:antlr4:4.5.2'
}
task generateParser(type:JavaExec) {
main = 'org.antlr.v4.Tool'
classpath = sourceSets.main.runtimeClasspath
args = ['-Dlanguage=JavaScript', 'todo.g4', '-o', 'static/generated-parser']
}#2020-12-1720:35Eugenhow can I specify the main class: org.antlr.v4.Tool ?#2020-12-1720:42Alex Miller (Clojure team)you can specify clj -M -m org.antlr.v4.Tool if you just want to invoke that main class with the deps classpath#2020-12-1720:45Alex Miller (Clojure team)you can pack this into an alias too using :main-opts and :replace-deps or :extra-deps to specify the deps#2020-12-1720:46Alex Miller (Clojure team)and not sure if that -D is a jvm property or a class arg. If the former, use :jvm-opts for that one#2020-12-1720:46Alex Miller (Clojure team)(note that you can't remove Clojure itself from the classpath - it assumes you are running Clojure programs)#2020-12-1720:53Eugenthanks @alexmiller. How I can specify org.antlr.v4.Tool in deps.edn ?
Now I have:
:antlr-js-target {:main-fn ["org.antlr.v4.Tool"]
:main-opts ["grammars/timeclock.g4" "-o" "static/generated/js-parser"]
:jvm-opts ["-Dlanguage=JavaScript"]}#2020-12-1720:55Alex Miller (Clojure team):antlr-js-target {:main-opts ["-m" "org.antlr.v4.Tool" "grammars/timeclock.g4" "-o" "static/generated/js-parser"]
:jvm-opts ["-Dlanguage=JavaScript"]}#2020-12-1720:55Eugenthank you 🙂#2020-12-1721:12Eugenunfortunatelly it does not work 😞
hledger-grammars$ clj -M:antlr-js-target
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate org/antlr/v4/Tool__init.class, org/antlr/v4/Tool.clj or org/antlr/v4/Tool.cljc on classpath.
Full report at:
/tmp/clojure-8595960239732774942.edn
I have checked the dependencies and they should be on the classpath
{:clojure.main/message
"Execution error (FileNotFoundException) at clojure.main/main (main.java:40).\nCould not locate org/antlr/v4/Tool__init.class, org/antlr/v4/Tool.clj or org/antlr/v4/Tool.cljc on classpath.\n",
:clojure.main/triage
{:clojure.error/class java.io.FileNotFoundException,
:clojure.error/line 40,
:clojure.error/cause
"Could not locate org/antlr/v4/Tool__init.class, org/antlr/v4/Tool.clj or org/antlr/v4/Tool.cljc on classpath.",
:clojure.error/symbol clojure.main/main,
:clojure.error/source "main.java",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.io.FileNotFoundException,
:message
"Could not locate org/antlr/v4/Tool__init.class, org/antlr/v4/Tool.clj or org/antlr/v4/Tool.cljc on classpath.",
:at [clojure.lang.RT load "RT.java" 462]}],
:trace
[[clojure.lang.RT load "RT.java" 462]
[clojure.lang.RT load "RT.java" 424]
[clojure.core$load$fn__6839 invoke "core.clj" 6126]
[clojure.core$load invokeStatic "core.clj" 6125]
[clojure.core$load doInvoke "core.clj" 6109]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[clojure.core$load_one invokeStatic "core.clj" 5908]
[clojure.core$load_one invoke "core.clj" 5903]
[clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
[clojure.core$load_lib invokeStatic "core.clj" 5947]
[clojure.core$load_lib doInvoke "core.clj" 5928]
[clojure.lang.RestFn applyTo "RestFn.java" 142]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$load_libs invokeStatic "core.clj" 5985]
[clojure.core$load_libs doInvoke "core.clj" 5969]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$require invokeStatic "core.clj" 6007]
[clojure.main$main_opt invokeStatic "main.clj" 514]
[clojure.main$main_opt invoke "main.clj" 510]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause
"Could not locate org/antlr/v4/Tool__init.class, org/antlr/v4/Tool.clj or org/antlr/v4/Tool.cljc on classpath."}}#2020-12-1721:15Alex Miller (Clojure team)can you double-check with clj -Spath -M:antlr-js-target ?#2020-12-1721:16Alex Miller (Clojure team)oh, it's looking to load a clojure class here#2020-12-1721:16Alex Miller (Clojure team)maybe you can't actually do this :)#2020-12-1721:17Eugenclj -Spath -M:antlr-js-target
src:resources:/home/ieugen/.m2/repository/clj-antlr/clj-antlr/0.2.7/clj-antlr-0.2.7.jar:/home/ieugen/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/ieugen/.m2/repository/org/antlr/antlr4/4.8-1/antlr4-4.8-1.jar:/home/ieugen/.m2/repository/org/antlr/antlr4-runtime/4.8-1/antlr4-runtime-4.8-1.jar:/home/ieugen/.m2/repository/org/clojure/tools.logging/1.1.0/tools.logging-1.1.0.jar:/home/ieugen/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar:/home/ieugen/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/ieugen/.m2/repository/com/ibm/icu/icu4j/61.1/icu4j-61.1.jar:/home/ieugen/.m2/repository/org/abego/treelayout/org.abego.treelayout.core/1.0.3/org.abego.treelayout.core-1.0.3.jar:/home/ieugen/.m2/repository/org/antlr/ST4/4.3/ST4-4.3.jar:/home/ieugen/.m2/repository/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.jar:/home/ieugen/.m2/repository/org/glassfish/javax.json/1.0.4/javax.json-1.0.4.jar
#2020-12-1721:21Alex Miller (Clojure team)could just invoke with a Clojure expression, like?
:antlr-js-target {:main-opts ["-e" "(org.antlr.v4.Tool/main,\"grammars/timeclock.g4\",\"-o\",\"static/generated/js-parser\")"]
:jvm-opts ["-Dlanguage=JavaScript"]}
#2020-12-1721:22Alex Miller (Clojure team)as you can tell, you're kind of out of the main usage of this tool (running clojure programs)#2020-12-1721:26Eugenyes, seems like it is.
I am tryng to use this tool to do some project building.
Seems to be related to what it is used for in clojure#2020-12-1721:27Eugenalmost there, now it needs to pass a single array instead of 3 args#2020-12-1721:27EugenSyntax error (IllegalArgumentException) compiling . at (REPL:1:1).
No matching method main found taking 3 args for class org.antlr.v4.Tool
#2020-12-1721:28EugenI'll try to use object-array#2020-12-1721:32Eugenfailed with ["-e" "(org.antlr.v4.Tool/main,(object-array,[,\"grammars/timeclock.g4\",\"-o\",\"static/generated/js-parser\",],),)"]
class [Ljava.lang.Object; cannot be cast to class [Ljava.lang.String; ([Ljava.lang.Object; and [Ljava.lang.String; are in module java.base of loader 'bootstrap')
#2020-12-1721:32Alex Miller (Clojure team)oh right. it's probably actually a String[]#2020-12-1721:34Alex Miller (Clojure team)["-e" "(org.antlr.v4.Tool/main,(into-array,[\"grammars/timeclock.g4\",\"-o\",\"static/generated/js-parser\",],),)"]
#2020-12-1721:34Alex Miller (Clojure team)^^ will derive the array type from the first arg#2020-12-1722:09Eugenyes, this works but it does not take into account the jvm option: :jvm-opts ["-Dlanguage=JavaScript"] . It generates java classes instead of js.
I could set the property in the expression#2020-12-1722:23EugenI managed to get it working by checking the java code.
It seems they accept -Dlanguage as an argument also so this worked:
:antlr-js-target {:main-opts ["-e" "(org.antlr.v4.Tool/main,(into-array,[\"-Dlanguage=JavaScript\",\"grammars/timeclock.g4\",\"-o\",\"static/generated/js-parser\"]))"]}#2020-12-1722:24Eugenthank you @alexmiller for your help.
Could calling Java from deps.edn be made easier? Would JavaExec or an Exec be a feature?#2020-12-1722:41Alex Miller (Clojure team)it could be made easier, but I don't really see it as the purpose of this tool so I'm not really interested in adding stuff for it#2020-12-1816:54jjttjjI'm trying to run this project with this deps.edn https://github.com/JetBrains/skija/blob/master/examples/clojure/deps.edn but when I run clj in the directory i get
Error building classpath. Could not find artifact org.jetbrains.skija:skija-native:jar:0.0.0-SNAPSHOT in central ()
Anyone know why it's not respecting the :mvn/repos ?#2020-12-1817:02delaguardothere is no such package available in that repo
try to use this instructions https://github.com/JetBrains/skija#using-skija#2020-12-1817:04delaguardoreplace this
org.jetbrains.skija/skija-native {:mvn/version "0.0.0-SNAPSHOT"}
with that
org.jetbrains.skija/skija-%platform% {:mvn/version "0.6.41"}
do not forget to substitute %platform% with yours#2020-12-1817:06jjttjjThat was it, thank you !#2020-12-2121:12mike_ananevorg.jetbrains.skija/skija-macos {:mvn/version "0.6.43"}
#2020-12-2120:59seancorfield@alexmiller I'm working on depstar today and I was wondering how stable the t.d.a API is that underlies the -Spom option? For ease of use, I've added the ability to override the group/artifact and/or version in the pom.xml file when building a JAR for deployment and it seemed the most sane approach was to actually update the pom.xml since it will likely be used in a subsequent deployment step. I know this is sort of complecting two distinct steps but a) it is opt-in and b) allowing the GAV to be overridden without updating pom.xml seems fraught with potential pitfalls. Having crossed that bridge, I figured it would also be a nice ease of use feature to allow depstar to run the equivalent of clojure -Spom for users, since it already has the project basis (now).#2020-12-2121:16Alex Miller (Clojure team)Which thing are you asking about specifically?#2020-12-2121:33seancorfieldclojure.tools.deps.alpha.gen.pom/sync-pom#2020-12-2121:33seancorfield(it's very nice to use, BTW)#2020-12-2121:48Alex Miller (Clojure team)Should be relatively stable#2020-12-2121:54seancorfieldThanks! :thumbsup::skin-tone-2:#2020-12-2200:54seancorfieldRan into an interesting issue while working on depstar 2.0 today: I'd been assuming that you could use :replace-deps to add it as a dependency now that it calculates the project basis from the deps.edn files (and the :aliases you provide it with), and that works great for almost everything... except where it does AOT compilation, which has to be done in the context of the project's code so that the necessary dependencies are available. I looked at what Leiningen does and it seems to set up a specific classloader context for running compilation (based on the project's dependencies etc).#2020-12-2200:55seancorfieldAlthough calculating the project basis causes all the project's dependencies to become available (locally) for inclusion in the uberjar, they are not on the classpath used to invoke depstar if :replace-deps is used.#2020-12-2200:59seancorfieldThoughts/suggestions? What I've done for the time being is just documented that if you ask depstar to do AOT-compilation, you'll need to use :extra-deps instead of :replace-deps when specifying it as a dependency. What options do I have if I wanted to run clojure.core/compile in a different classloader context?#2020-12-2201:02seancorfield(that's not ideal because then you run the risk of depstar's transitive dependencies affecting your project's compiled code in some way)#2020-12-2201:51ghadistart another JVM just for the compilation from within depstar#2020-12-2201:51ghadithat’s the approach I’m taking on a similar tool#2020-12-2202:29seancorfield@ghadi Seems like a bit of a sledgehammer to crack a nut but it is also straightforward I guess. Are you shelling out? If so, how do you decide what command to use? Or is there a JVM-specific spawning process?#2020-12-2203:46seancorfieldHmm, with ProcessBuildler, that's not bad. And it can generally assume java -cp ... will work I guess?#2020-12-2204:22Alex Miller (Clojure team)Separate JVM is what I did in tools build#2020-12-2204:22Alex Miller (Clojure team)You can’t run clj if java doesn’t work so ...#2020-12-2204:23seancorfieldAnd the clojure script sets JAVA_CMD so I can rely on that too.#2020-12-2204:29seancorfieldHere's what I ended up with https://github.com/seancorfield/depstar/blob/develop/src/hf/depstar/uberjar.clj#L541 (it's all due for a big refactor -- most of today has been focused on just getting all the 2.0 functionality working and the next job is creating automated tests and then refactoring)#2020-12-2217:57kennyI see the output of -Stree has changed with the latest cli version. Are there some docs on what the notation used means?#2020-12-2218:10seancorfield@kenny X means that version of the dependency is not being used. The keywords at the end of some lines indicate why a particular version was used or not.#2020-12-2218:13kennyX means it's not being used because it's overridden by something else? It seems that last keyword is only every present on X's.#2020-12-2218:13seancorfieldExample from one of our services:
X compojure/compojure 1.6.2 :use-top
X ring/ring-core 1.6.3 :older-version
X ring/ring-core 1.6.0 :older-version
X ring/ring-core 1.6.0 :older-version
X ring/ring-core 1.7.1 :older-version
X worldsingles/lowlevel /Developer/workspace/wsmain/clojure/lowlevel :use-top
. ring/ring-core 1.8.2 :newer-version
X org.clojure/java.classpath 0.3.0 :older-version
Ring 1.8.2 is the :newer-version that is selected in preference to 1.6.3 which is :older-version, and Compojure 1.6.2 was specified as a top-level dependency (overriding any transient dependency on it).#2020-12-2218:14kennyOh I see. What does it mean when a dep doesn't have a keyword?#2020-12-2218:14seancorfieldThat's just a normal selection of the version.#2020-12-2218:15kennyAs in all deps have the same version of the dep?#2020-12-2218:16seancorfieldThe end result is just one set of versions. If there's no conflict of versions, there will be no annotation I believe.#2020-12-2218:17kennyRight. That's what I'm seeing. Just making sure it was the correct interpretation. This new output is great. How come it isn't documented anywhere? Alpha?#2020-12-2218:20seancorfieldAlthough:
X org.clojure/java.classpath 0.3.0 :older-version
. org.clojure/java.classpath 1.0.0
In this case 1.0.0 was selected implicitly (it's only a transitive dep in our code), and the older version was excluded because a newer version had already been found.#2020-12-2218:21seancorfieldI expect @alexmiller can provide more detail when he's around...#2020-12-2218:27Alex Miller (Clojure team)It’s a bit more complicated than that, and there are some cases where inclusions list a reason too#2020-12-2218:27Alex Miller (Clojure team)Maybe I will doc it today :)#2020-12-2218:27Alex Miller (Clojure team)I omit it in cases where it’s uninteresting#2020-12-2218:28Alex Miller (Clojure team)Like finding a new dep#2020-12-2218:28Alex Miller (Clojure team)Or finding same dep version already selected#2020-12-2218:33kennyIn that case, why wouldn't it be annotated with :newer-version like in your first example @seancorfield?#2020-12-2218:37seancorfield@kenny If it finds version N and later finds version N+1, I would expect it to annotate both, and then if it continues to find version N+1 it will not annotate those (since they are "finding dep version already selected")#2020-12-2218:39seancorfieldIf it only finds version N across all deps, it's not going to annotate either.#2020-12-2218:39seancorfieldAt least, that's my understanding of what @alexmiller just said.#2020-12-2219:07Alex Miller (Clojure team)it doesn't annotate both because it's a single pass breadth-first expansion#2020-12-2219:07Alex Miller (Clojure team)(or rather to be clear, all choices have a reason, I'm just suppressing some of them in the printing b/c they're not interesting)#2020-12-2219:08Alex Miller (Clojure team)when we first see a lib (version N) during expansion, it will have reason :new-dep#2020-12-2219:09Alex Miller (Clojure team)if you later find same lib version N+1, it will have reason :newer-version#2020-12-2219:11Alex Miller (Clojure team)but there are a lot of tricky cases#2020-12-2219:12Alex Miller (Clojure team)right now the reason codes :new-top-dep, :new-dep, and :same-version are suppressed#2020-12-2219:18borkdudeI'm getting a different resolution for a project that uses tools.reader 1.3.2 and pulls in a mvn lib that depends on tr 1.3.4 (I'm getting tools reader 1.3.4 in this case which I'm happy about), when using that lib via local/root (then it pulls in 1.3.2)#2020-12-2219:18borkdudeIs that expected?#2020-12-2219:18borkdude(I did try -Sforce)#2020-12-2219:20borkdudeI will just remove the tools.reader dep from the project since I always want to use what the local/root project is using anyway#2020-12-2219:27Alex Miller (Clojure team)not enough info there for me to tell#2020-12-2219:28Alex Miller (Clojure team)local/root dep changes are not automatically picked up, so you do need to -Sforce if those change#2020-12-2219:28Alex Miller (Clojure team)if you have an output from -Strace or -Stree, happy to take a look#2020-12-2219:42borkdudeoh yikes, I see what is happening. the mvn lib has AOT-ed class files... which should not happen.#2020-12-2219:55borkdudeFound the culprit: apparently when you have a :main in the top-level of defproject lein will compile on deploy ... which might make sense. I just used that main for testing some stuff. Problem solved.#2020-12-2220:53Alex Miller (Clojure team)@kenny @seancorfield new page on dep expansion, tracing, tree printing https://clojure.org/reference/dep_expansion#2020-12-2221:48kennyHow do you get to this page without the link?#2020-12-2221:56seancorfieldIt linked in two places on the deps/CLI reference page.#2020-12-2221:57kennyAh. Isn't it odd how it's not in the left nav though?#2020-12-2221:58seancorfieldIt's not a reference in and of itself: it's just extra material for an existing reference. Not every page is directly linked in the left nav.#2020-12-2222:00seancorfieldThe deps/CLI reference page is the entry point. The new page is linked from dep expansion under the Resolve dependencies section of Operation and also from tree printing under Other Programs (for -X:deps tree).#2020-12-2222:00kennyOh, okay. I could potentially see that being confusing for someone who is directly sent this link. That person has no way to "back out" to contextualize 🙂#2020-12-2221:05Alex Miller (Clojure team)undoubtedly more to say, but that is a dump of several things have only been in my head for a long time so felt good to get them out :)#2020-12-2222:55Eugenhi, what is the equivalent in deps.edn of :java-source-paths ["src/java/"] from leiningen ?
I would like to add deps.end for clj-antlr so I can use it from git.
The java directory contains java sources: https://github.com/aphyr/clj-antlr#2020-12-2223:04dominicm@eugen.stan no equivalent. You'll have to use javac yourself first.#2020-12-2223:06Eugenthanks, I keep finding these edge cases 🙂 .
Do you have any examples?#2020-12-2223:46seancorfield@eugen.stan He means you literally run the javac command yourself to compile the .java files to .class files.#2020-12-2223:48seancorfieldThis isn't an "edge case" -- this is just something that the Clojure CLI was not intended to do, by design. Leiningen is sort of a "kitchen sink": it has a whole bunch of tools included that have nothing to do with just "running Clojure code" (which is what the Clojure CLI is designed for).#2020-12-2223:49seancorfieldPeople are building "tools" that mimic some of the stuff that Leiningen does (or that is provided by one of the many Leiningen plugins) and those "tools" are just small Clojure programs that you run via the Clojure CLI.#2020-12-2223:50seancorfieldWhen you look at the general design principles behind Clojure -- and a lot of the things that Cognitect folks have produced -- there's a deliberate choice to keep things simple: each tool should do one job.#2020-12-2223:52seancorfieldSomeone could build a tool to compile Java source code -- a tool you could run from the Clojure CLI -- but you'd still need to run that tool and then run other tools and/or the CLI itself. There's still no "task pipeline" for the Clojure CLI. Maybe tools.build will provide that "task pipeline"? We'll have to wait and see what the Cognitect folks produce.#2020-12-2307:17Eugenthanks, I'll see what I can do then.
My goal, as I said is to be able to use my version of clj-antl (with patches) from git.
deps.edn seemed like a good fit until I got to the java classes for which I don't have a solutin.
I do have a direction now.
Let's see where it leads.#2020-12-2319:17tomgeorgeHi all, I'm learning about clj and deps.edn, and I had come up with an alias to connect to an nrepl server:
:dev {;; some dev deps }
:repl/connect {:extra-paths ["env/dev" "resources/dev]
:extra-deps {nrepl/nrepl {:mvn/version "0.8.3"}}
:main-opts ["-m" "nrepl.cmdline" "--connect"]}}}
This allowed me to run
clj -M:dev:repl/connect --host $HOST --port $PORT
Then I was reading about .clojure/deps.edn and figured I could move that repl/connect alias out, and I ended up with these snippets:
;; .clojure/deps.edn
:repl/connect {:extra-deps {nrepl/nrepl {:mvn/version "0.8.3"}}
:main-opts ["-m" "nrepl.cmdline" "--connect"]}}}
;; repository
:repl/connect {:extra-paths ["env/dev" "resources/dev"]}
I thought this would basically merge these into the first form, but not when I run
clj -M:dev:repl/connect --host $HOST --port $PORT
I see
Exception in thread "main" java.io.FileNotFoundException: --host (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
at clojure.lang.Compiler.loadFile(Compiler.java:7570)
at clojure.main$load_script.invokeStatic(main.clj:452)
at clojure.main$script_opt.invokeStatic(main.clj:512)
at clojure.main$script_opt.invoke(main.clj:507)
at clojure.main$main.invokeStatic(main.clj:598)
at clojure.main$main.doInvoke(main.clj:561)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:37)
It seems like I'm missing something but I don't know what. I'd like a generic repl/connect that I can use across projects and pass --host and --port to#2020-12-2319:45dorabIf I understand correctly, merge takes the last value of the key. So, since you are using the same key in your ~/.clojure/deps.edn and in your ${REPODIR}/deps.edn the one in the repo is "winning" and overriding the other one. It is probably best to use two different aliases and add them to the -M: command line option.#2020-12-2319:49dorab@seancorfield’s response in the main thread is more accurate than mine.#2020-12-2320:01tomgeorgethank you, I thought it merged the aliases themselves#2020-12-2319:45seancorfieldAliases overwrite each other in the system -> user -> project -> command-line order (so last one wins).#2020-12-2319:45seancorfieldThe merge happens on the :aliases key, not on the nested aliases themselves.#2020-12-2319:46seancorfieldSo, in your case, :repl/connect in your project overwrites your user-level version and thus the --host argument is passed to Clojure's main, because the nREPL stuff isn't present in the merged EDN.#2020-12-2319:47seancorfieldIf you have multiple non-conflicting aliases, the bodies of the aliases are merged, but with :main-opts the last one wins (those are not merged). /cc @dorab#2020-12-2319:55DerekIt also looks like your repository’s :repl/connect alias’ :extra-paths may be more suited to be in :dev?#2020-12-2320:00tomgeorge@seancorfield oh I see, I thought the merging happened on the aliases themselves. Thanks!#2020-12-2408:35Eugenhi is there a way that I can access (read) the merged deps.edn from the code I am calling ?
:my-alias-target {:extra-deps {....}
:main-opts ["-e" "function/class call that can use deps.edn contents"]}
#2020-12-2408:53borkdude@eugen.stan Maybe looking at the clojure.basis will help#2020-12-2408:53borkdude(slurp (System/getProperty "clojure.basis"))
#2020-12-2417:16seancorfieldPer @alexmiller it's possible (probable?) that the clojure.basis system property will go away (which would make it impossible for a program to know how it was invoked). It's pretty straightforward to calculate the project basis (from the various deps.edn files) using tools.deps.alpha but you need to tell it what aliases you need merging (and whether you want the user deps.edn file or not -- equivalent to -Sforce).#2020-12-2417:58Alex Miller (Clojure team)I don’t think I said that#2020-12-2417:58Alex Miller (Clojure team)We are going to add an api for access #2020-12-2417:59Alex Miller (Clojure team)The system property would then be part of impl#2020-12-2418:01seancorfieldAh, sorry, I misunderstood. It sounded like you wanted to get rid of the system property altogether.#2020-12-2418:03seancorfieldI misremembered: @U050ECB92 said "I am 90% sure the clojure.basis property is going away"#2020-12-2418:36ghadiThat’s me hedging that it’s going to be turned into an impl detail#2020-12-2507:55EugenThanks for clarifying#2020-12-2410:32Eugenthanks @borkdude, it's exactly what I need#2020-12-2416:35andy.fingerhutI am using Clojure CLI tools version 1.10.1.763, and with both that and a slightly earlier version when attempting a command with a single alias that uses a single :git/url dependency (which has its own :git/url dependencies), I get an error message "Error building classpath. No known relationship between git versions", but no other clue about which dependency is causing that message. Should it give a dependency name in that message? Or perhaps there are some extra command line options I could use to make it show more details?#2020-12-2419:12Alex Miller (Clojure team)What that error means is that the same git lib was found but there is no ancestor relationship between the two shas and thus the “newest” version can’t be determined#2020-12-2421:08andy.fingerhutUnderstood. Is there enough info available when that is detected to print which git lib caused the problem? Not a big deal for me on this issue, but I can imagine with larger dep trees that it would help to narrow down the source of the problem#2020-12-2422:05Alex Miller (Clojure team)It can be improved if you want to file a ticket#2020-12-2500:39andy.fingerhutDone: https://clojure.atlassian.net/browse/TDEPS-176#2020-12-2416:37andy.fingerhutAdding -Sverbose shows me install_dir, config_paths, etc. but no more detail on the final error message, which is the same as above. -Strace does not write any trace.edn file.#2020-12-2416:38hiredmanI forget the flags, but the dependency tree printing may help#2020-12-2416:38andy.fingerhutI am fairly sure I have a small enough issue I can track this down -- mainly curious whether the error message can be made more informative.#2020-12-2416:39andy.fingerhutIf you mean -X:deps tree, I tried adding that at the end of my command line, and it downloads a few extra deps, but then gives the same error message with no other output.#2020-12-2416:42hiredmanIt must be in the deps of your single git dep, or you have another version of that git dep in one of the other deps.edn files listed in config_paths#2020-12-2416:44andy.fingerhutAgreed that my little problem here should be straightforward to track down, since I can count the deps on the fingers of one hand. As I said, I'm mainly asking whether this is expected output, for cases where it isn't so few deps.#2020-12-2416:45andy.fingerhutI will get this fixed in this project, but in case you want to reproduce, a command line that should do it is, I believe: clj -Srepro -Sdeps '{:deps {cljol/cljol {:git/url "" :sha "36c45f741fc64db248020aeb155f1f0f24f95dd9"}}}'#2020-12-2417:16seancorfieldPer @alexmiller it's possible (probable?) that the clojure.basis system property will go away (which would make it impossible for a program to know how it was invoked). It's pretty straightforward to calculate the project basis (from the various deps.edn files) using tools.deps.alpha but you need to tell it what aliases you need merging (and whether you want the user deps.edn file or not -- equivalent to -Sforce).#2020-12-2514:48borkdudequick hack to get the same deps in project.clj from deps.edn (only works with mvn/version obviously):
project.clj:
(def deps-edn '{:deps {org.clojure/clojure {:mvn/version "1.10.2-rc1"}
medley/medley {:mvn/version "1.3.0"}}})
;; or:
;; (require '[clojure.edn :as edn])
;; (def deps-edn (edn/read-string (slurp "deps.edn")))
(def project
`{:version "0.0.1"
:dependencies ~(mapv (fn [[lib version]]
[lib (:mvn/version version)])
(:deps deps-edn))})#2020-12-2514:48borkdude(TIL: you can just def a map named project in project.clj)#2020-12-2520:14dominicmThere's a lein-tools-deps if you don't know :)#2020-12-2520:30borkdudedoes it (still) work?#2020-12-2522:14dominicmNo reason to believe otherwise :)#2020-12-2607:52mkvlrhttps://github.com/RickMoynihan/lein-tools-deps/pull/93 sounds like it doesnt#2020-12-2617:22dharriganI'm using clojure -Sdeps "${DEPS}" -X:uberjar and I've noticed that -Sdeps is not honoured anymore (I previously was using -M:uberjar). The var ${DEPS} contains a custom mvn/repos.... configuration to set a local company proxy to be the source for central and clojars. Am I missing something to now use -Sdeps?#2020-12-2617:43seancorfieldThe default behavior is as-if -Srepro per the docs for 2.0.#2020-12-2617:45seancorfieldYou can specify :repro false if you need the user deps included. But the command line deps are not part of the project basis.#2020-12-2617:46seancorfieldYou need to put that in an alias and then tell depstar about it via :aliases '[...]'#2020-12-2617:46dharriganhmmm#2020-12-2617:46dharriganthat's wont work for custom overrides#2020-12-2617:47dharriganfor example, when building in a pipeline, the repo is different than building locally#2020-12-2617:47dharriganso, previously, in the pipeline (on gitlab), I was able to override the default ones that come with clj (maven and clojars)#2020-12-2617:48dharriganwithout having to touch the deps.edn in the project, which can be different for usecases#2020-12-2617:48seancorfieldOpen an issue to add some sort of :deps argument#2020-12-2617:49dharriganto depstar?#2020-12-2617:49seancorfieldYes#2020-12-2617:49dharriganI'm not sure this is a depstar issue#2020-12-2617:50dharriganThis is more on how deps are resolved, and permitting the user to override (or merge in a deps file) from the commandline, like -Sdeps did before#2020-12-2617:50seancorfieldThe -Sdeps CLI argument only applies to the runtime basis, not the computed project basis#2020-12-2617:51seancorfieldThis applies to all tools that explicitly compute the project basis#2020-12-2617:52dharriganOkay, so the "global" mvn/repos key - that can't be replaced with the :aliases '[...]' functionality in depstar?#2020-12-2617:52dharriganan alias can't override the global mvn/repos key?#2020-12-2617:56dharriganI think the project basis and runtime basis should both allow for the key for mvn/repos to be overridden to suit this - to replace where deps are found (esp when running in a container - for building)#2020-12-2617:58seancorfieldThe project basis is computed from the system, user, and project deps.edn files. The -Sdeps argument is only for the CLI script itself and that information is "gone" by the time the tool runs. depstar by default omits the user deps.edn (i.e., as-if -Srepro) but you can override that with depstar's :repro false argument so that it includes user deps.edn in the project basis computation.#2020-12-2617:59seancorfieldSo to override the :mvn/repos key, it either needs to be in the user deps.edn file (`:repro false`) or the project deps.edn file.#2020-12-2617:59dharriganSo, if I do :repro false and include a mvn/repos in my project deps.edn, it should work#2020-12-2618:00seancorfieldOr every tool needs a way to provide additional deps from the command-line like the CLI script allows.#2020-12-2618:00seancorfieldThis is why depstar (and several other tools) has an :aliases argument: the aliases used by the tool to compute the project basis are not the same as the aliases used by the CLI to compute the runtime basis needed to start the tool.#2020-12-2618:01seancorfieldHence my suggestion to create an issue against depstar to add some new command-line argument to allow additional deps.edn-style context to be provided for the computation of the project basis.#2020-12-2618:02seancorfield(this is also why I created the post on http://ask.clojure.org about a higher-level API for tools to use t.d.a for exactly this sort of computation)#2020-12-2618:02dharriganwould you support a procurers argument?#2020-12-2618:03seancorfieldI don't think that mechanism is open for extension in t.d.a right now?#2020-12-2618:03dharriganI'm not sure I fully grok this yet - but have to disppear for a bit - kid needs bed time!#2020-12-2618:03dharrigan🙂#2020-12-2618:45Alex Miller (Clojure team)procurers are open for extension if you're using the library - they're just multimethods#2020-12-2621:32seancorfield@alexmiller hard to use with :replace-deps unless you add it as part of the same tool deps, I guess?#2021-12-2711:52vemvCurious, can one configure clj to run some code as soon as a REPL is fired up?
Basically an equivalent to Lein's :init option#2021-12-2711:54delaguardoclj -M -e '(some-code)' -r#2021-12-2711:58vemvNice! Thanks to your answer I checked out https://github.com/practicalli/clojure-deps-edn/blob/65a48e1547d289a3d647d9f32bf3d2cc7fe8d5d3/deps.edn#L479
I wish though that those humongous strings could be expressed as nicely formatted sexprs instead#2021-12-2715:09vlaaadThis can be expressed as a 5 lines of code in a file that is executed by clj#2021-12-2715:10vlaaadAh, it was already mentioned in later discussion#2021-12-2711:59delaguardoyou can make a function for that somewhere in sourcepath#2021-12-2712:00vemvwdym?#2021-12-2712:03delaguardoin one of my projects I have dev/src/dev/utils.clj file with (ns dev.utils)
I also add :extra-paths ["dev/src"] to one of my alias for dev environment
in that namespace you can create a function
and then from console run it using requiring-resolve#2021-12-2712:06vemvAh nice! Similarly, my -e could be a single require and I do all the work in a proper .clj file
That way I'd skip the last step you describe#2021-12-2712:07delaguardoclj -A:dev -M -e '((requiring-resolve 'dev/init))' -r
that’s how it could look like in cli#2021-12-2712:08delaguardoand you can also use -i path to specify namespace to load as init#2021-12-2714:51Alex Miller (Clojure team)Or use -i instead of -e#2021-12-2714:51Alex Miller (Clojure team)Oh sorry, slack didn’t show me the last message there :)#2021-12-2800:09seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L93 shows how to load a dev.clj script from from your .clojure folder (I know this won't work with XDG systems) so you can have a dev script for all your projects, without needing code in every project.#2021-12-2800:10seancorfield(I should probably rename that so it doesn't conflict with any :dev alias someone actually has in a project deps.edn file)#2021-12-2715:05SantiagoHow can I access my project’s version at runtime? I’m using deps.edn#2021-12-2716:17borkdude@slack.jcpsantiago does deps.edn have version numbers?#2021-12-2716:20Santiago@borkdude Mine straight from clj-new doesn’t and it seems it’s not supposed to? There is a version in pom.xml <version>0.1.0-SNAPSHOT</version> , so I could read this file. I was looking for something that would work like environ and would read this automatically#2021-12-2723:02practicalli-john@slack.jcpsantiago if using environ the set an Environment variable or create a .properties file for a Java system property#2021-12-2723:02borkdudeI just inspected clojure.basis to see if my jvm-opts propagated correctly when composing aliases. It did. Neat.#2021-12-2723:03borkdudeBtw, I stumbled upon this blog https://insideclojure.org/2020/07/28/clj-exec/ but it seems the arguments have been renamed from :fn and :args to :exec-fn and :exec-args?#2021-12-2723:03borkdudeMaybe update the blog or is it immutable?#2021-12-2723:09practicalli-johnThat name change was covered in https://insideclojure.org/2020/09/04/clj-exec/#2021-12-2723:18seancorfieldThat blog is a great example of the history of Clojure features and how they evolve during prerelease work. Changing past posts would remove that evolution.#2021-12-2723:24Alex Miller (Clojure team)“Don’t believe everything you read on the internet” - Lincoln#2021-12-2723:31andy.fingerhutIt would make sense to add a footnote to an older blog post with a reference to a later update, if one was so inclined.#2021-12-2723:32Alex Miller (Clojure team)Well, I’m on vacation :)#2021-12-2723:33andy.fingerhutSorry, I should have mentioned that I'm not trying to nudge you or anyone to do that. I do agree that leaving older posts unchanged (except perhaps by addition of such forward-reference footnotes) makes a lot of sense, for anyone wanting to see the history.#2021-12-2800:09seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L93 shows how to load a dev.clj script from from your .clojure folder (I know this won't work with XDG systems) so you can have a dev script for all your projects, without needing code in every project.#2021-12-2810:11Eugenhi, I have a question regarding deps.edn and git dependencies.
Can I build a deps.edn dependency locally?
I have a project (clj-antlr) that has clojure code + java classes.
Can I make a clj-antlr/deps.edn so I can use clj-antlr from another project: my-project/deps.edn ? from git ?
My goal is depend on the git version to avoid publishing binary dependenices for clj-antrl.
clj-antlr does have some java classes that need to be compiled in order for it to be used.
Can I trigger a build step in clj-antlr/deps.edn when I use it in my-project/deps.edn ?#2021-12-2810:28borkdude@eugen.stan For gitlibs, this only works if you check the compiled Java into source control#2021-12-2810:29borkdudewhich is probably not so nice#2021-12-2810:29borkdudefor libs that need .class files, I usually make a release to clojars#2021-12-2810:30borkdudeFor sci I have one component which needs this, so I have isolated it into its own mini-library. The rest of sci can be used as a git lib#2021-12-2810:30EugenSo there is no life-cycle hooks that I can use in deps.edn to trigger pre/post processing for dependencies ..
thanks, in my case there are 3 classes so I will make a branch for that and once I am sure all things work I'll publish them.#2021-12-2810:31borkdudewell, you can also push the compilation phase to the user of the library, as part of the main opts, but so far I haven't seen anyone do this#2021-12-2810:32borkdudeit's not unreasonable since you can then treat Java as a source-based language as well#2021-12-2810:32borkdudeI'm hoping tools.build will go a bit into this direction#2021-12-2810:34Eugenthanks, I also have some ideas on how to provide this capability.
With any luck I will push something this week 🙂 .#2021-12-2810:42vlaaadAfaik it's possible to compile and load Java code in the JVM process used for development, maybe tools-build will help with that :D#2021-12-2810:43borkdudethat's what I was suggesting#2021-12-2900:41seancorfieldseancorfield/depstar {:mvn/version "2.0.165"} -- bugfix for AOT compilation on Windows, courtesy of @borkdude (thank you!) -- follow-up in #depstar#2021-12-2901:01seancorfieldseancorfield/clj-new {:mvn/version "1.1.234"} -- mostly just version bumps for various dependencies and improved examples using the -X CLI option -- follow-up in #clj-new#2021-12-2918:28nnicholsHas anyone here deployed a library to a private S3 bucket with tools,deps? The closest tool I’ve found so far is a stuck PR for deps-deploy, and I’d rather not dust off too much maven to get the job done#2021-01-0418:19nnicholsFor posterity, I created the two following aliases:
:jar {:extra-deps {seancorfield/depstar {:mvn/version "2.0.165"}}
:exec-fn hf.depstar/uberjar
:exec-args {:jar "my-lib.jar"
:aot true
:main-class my-lib.core}}
and
:deploy {:extra-deps {slipset/deps-deploy {:git/url ""
:sha "588e4b96b33820a6ee1e35663410450ccb876b30"}}
:override-deps {s3-wagon-private/s3-wagon-private {:mvn/version "1.3.2"}}
:exec-fn deps-deploy.deps-deploy/deploy
:exec-args {:repository {"releases" {:url ""}}
:artifact "my-lib.jar"
:installer :remote}}#2021-12-3017:12Alexis Vincent@nichols1991 I took an approach of using pack.alpha to generate a jar and then use maven cli to deploy to private repo, but im guessing you could use similar thing to deploy to s3#2021-12-3121:37Spencer AppleHappy NYE everyone!
I was curious if anyone has seen the following error when the clojure command is run:
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
I’m on clj version: 1.10.1.727
It seems to be intermittent but I’ve only seen it when clj is run with -Sdeps and -Sforce. I’ve seen this error thrown with both the -e and the -m flags used.#2021-12-3121:41seancorfield@splayemu I wonder if you might be hitting a race condition when it is trying to figure out dependencies in parallel for some reason? You can try adding -Sthreads 1 to force it to be single-threaded.#2021-12-3121:49Spencer Applethank you! It’s been popping up a bunch on our CI lately so I’ll give that a go and report back if it solves the problem.#2021-12-3121:56Spencer AppleDoes it force the whole clojure process to only use 1 thread, or does it just change the classpath generation / clojure initialization?#2021-12-3121:59seancorfieldJust the part that calculates/downloads dependencies. Once all the dependencies are figured out, the subsequent process that actually runs your code will run as normal.#2021-12-3121:59dominicmClasspath generation#2021-12-3122:00Spencer AppleGreat, thanks 🙏#2021-12-3122:01seancorfieldI'm a bit surprised folks don't run into this issue more often. I've never seen it happen -- not locally on my Mac, nor on our build/QA server -- but I have seen a handful of people run into it repeatedly.#2021-12-3122:09Spencer Applehmm yeah I wonder what makes it more likely to happen…
I haven’t seen it locally - only on our drone env#2021-12-3122:26Alex Miller (Clojure team)are you using any git deps or s3 maven repo deps?#2021-12-3122:30Spencer Appleyes we have our own maven repo in s3#2021-12-3122:30Spencer Appleand we also use git deps - though not used heavily#2021-12-3122:32Spencer AppleWe also use a lot of :local/root deps (not sure if that’s useful info for you)#2021-12-3122:42Alex Miller (Clojure team)yep, although that doesn't help narrow it down much :) s3 repos do dynamic loading of the s3 code so that's a known issue but your stack trace looks like something else. a lot of the maven model building stuff is not thread-safe.#2021-12-3122:26ghadi@splayemu is there more stacktrace youre omitting?#2021-12-3122:27Spencer Applesure, here is one:
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1882)
at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:1998)
at java.base/java.util.HashMap.treeifyBin(HashMap.java:767)
at java.base/java.util.HashMap.putVal(HashMap.java:639)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:847)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:659)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:583)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:368)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:494)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:292)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$eval816$fn__818.invoke(maven.clj:108)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__1237$fn__1239$fn__1240.invoke(alpha.clj:250)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__500.invoke(concurrent.clj:33)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
#2021-12-3122:27Spencer Applesure, here is one:
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1882)
at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:1998)
at java.base/java.util.HashMap.treeifyBin(HashMap.java:767)
at java.base/java.util.HashMap.putVal(HashMap.java:639)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:847)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:659)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:583)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:368)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:494)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:292)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$eval816$fn__818.invoke(maven.clj:108)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__1237$fn__1239$fn__1240.invoke(alpha.clj:250)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__500.invoke(concurrent.clj:33)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
#2021-01-0320:48vlaaadMe, trying to be smart and sneak in read-eval to deps.edn:
1. creates data_readers.clj with {clj/eval clojure.core/eval}
2. creates alias with :key #clj/eval (requiring-resolve my-fn) in :exec-args
clj_exec.clj: uses edn/read-string that ignores data_readers https://github.com/clojure/brew-install/blob/fde9db19bfe92a315b81b5f56d290a6d37a91d89/src/main/resources/clj_exec.clj#L39#2021-01-0320:48vlaaad¯\(ツ)/¯#2021-01-0320:50vlaaadI think clj-exec is limiting by only accepting edn data. I want to pass a function, I don't want to add work-arounds that accept and auto-resolve symbols where exec function they expect functions as arguments#2021-01-0320:56seancorfield@vlaaad Whilst I share some of that frustration (after hours poking at the internals of Figwheel, Rebel Readline, and the cljs compiler itself), I think it would pretty dangerous -- I think it is reasonable for the function you invoke to have to do the requiring-resolve because it's generally easy enough to write a wrapper that you can invoke (and have it invoke whatever you're trying to pass a function into).#2021-01-0320:59vlaaadI agree that read-eval is probably too dangerous for a problem I don't have a solution for, but passing function to a function should not be impossible with clj-exec IMO#2021-01-0321:04dpsuttonWhat’s wrong with passing the fully qualified name? Don’t prepls repls and socket servers behave this way?#2021-01-0321:05vlaaadclojure.main/repl expects functions, for example, so it can't be used from clj-exec#2021-01-0321:07seancorfieldYeah, that was similar to my use case: where I want a resolved symbol, which has to happen at runtime, hence a wrapper is needed.#2021-01-0321:12Alex Miller (Clojure team)We are working towards a solution to this greater problem#2021-01-0321:13Alex Miller (Clojure team)In particular, making vars autoresolve on read#2021-01-0321:13Alex Miller (Clojure team)Note, not symbols but vars#2021-01-0321:14Alex Miller (Clojure team)A release or two ago we changed var deserialization to do that#2021-01-0321:15Alex Miller (Clojure team)I took a pass at var resolution at that point as well but we decided to wait on it. That’s likely to be in scope for 1.11#2021-01-0321:18Alex Miller (Clojure team)https://ask.clojure.org/index.php/3810/clojure-var-tag-for-transmitting-var-identity?show=3810#2021-01-0321:19Alex Miller (Clojure team)Votes welcome :)#2021-01-0321:20vlaaadUpvoted! Thanks for detailed response#2021-01-0321:22Alex Miller (Clojure team)You’re not crazy :). We’ve been thinking about this for years #2021-01-0419:52hanDerPedercan I add :jvm-opts without adding it in an alias?#2021-01-0419:53borkdudeI don't think that works#2021-01-0419:54hanDerPederI don’t know the internals of tools-deps. Does that make sense? Coming from leiningen it seems like a strange omission.#2021-01-0420:26vlaaad-J#2021-01-0420:29vlaaadYou can add jvm opts at the point of invocation with clj -J... , but not in deps.edn file#2021-01-0420:44seancorfield@peder.refsnes You can vote for it here: https://ask.clojure.org/index.php/7834/support-default-jvm-opts#2021-01-0420:53Alex Miller (Clojure team)I continue to feel this is weirdly at odds with using tools#2021-01-0420:54Alex Miller (Clojure team)and I'm going to continue not implementing it until that makes sense :)#2021-01-0421:01seancorfieldYeah, I don't see much of a use case for it either. I have only ever needed to set JVM options for a specific task, which means a specific alias.#2021-01-0421:02seancorfieldBut people do keep asking for it, so hopefully someone will come up with a justification for it that is convincing... 🙂#2021-01-0421:04seancorfield(and "leiningen supports it" isn't at all convincing to me 🙂 leiningen does a lot of things that I don't think are necessarily good ideas)#2021-01-0421:13Alex Miller (Clojure team)I do actually get the use case, it's just that's not the only use case#2021-01-0421:17seancorfieldWhat sort of JVM options can you imagine folks wanting to set for all operations in a deps.edn project? I'm curious...#2021-01-0421:18Alex Miller (Clojure team)well the use case is where "all" is the two things you usually do#2021-01-0421:19Alex Miller (Clojure team)and you "always" want to do something like set a system property for your app or a particular garbage collector or whatever#2021-01-0421:20Alex Miller (Clojure team)I think in some cases "all" is really "when I run my repl I want default stuff to be set"#2021-01-0421:21Alex Miller (Clojure team)that is in particular an interesting case where having a "default" alias might be useful#2021-01-0421:22Alex Miller (Clojure team)not too different than leiningen :default profile really#2021-01-0421:40hanDerPederMy current (and only) reason for wanting this is for playing around with Dragan Djurics ML libraries. Seems it needs a workaround for java 9+.
https://github.com/uncomplicate/neanderthal/blob/master/examples/hello-world/project.clj#L11-L12#2021-01-0421:44Alex Miller (Clojure team)Which you could put in an alias, you just need to remember to use it then#2021-01-0421:46hanDerPederIndeed, but it’s annoying when you forget 🙂 I’m all for features carrying their weight, just thought I’d ask.#2021-01-0421:54seancorfieldI think I would have a :run alias for running the app so it's just clojure -M:run and perhaps a :dev alias for running a REPL and just live with the duplication of :jvm-opts...#2021-01-0421:56vlaaadI find lein's default profiles confusing, it's always hard to tell what's enabled/on the classpath#2021-01-0421:57seancorfieldIndeed. Leiningen's "helpfulness" is one of the things I always liked the least about it.#2021-01-0421:59hanDerPedercouldn’t you use that same reasoning to argue against having a default`:deps` key?#2021-01-0421:59seancorfieldAlthough some folks think this is a bit much clj -M:repl:reflect:jedi-time:reveal:j14:classes:add-libs:dev+test 🙂#2021-01-0422:00seancorfield@peder.refsnes having a base set of dependencies that almost every use case builds on is very useful tho'... same with a base set of :paths...#2021-01-0422:02jjttjjI just have a :dev alias and have my editor always start a repl with that alias. It's pretty easy in emacs on both cider and inf-clojure. Do other editors offer something like this?#2021-01-0422:03seancorfieldI always start my REPL separately (and leave it running for ages -- my current REPL was started on December 24th!) and then connect one or more editors or other tools to it.#2021-01-0422:07jjttjjThat's impressive! I can't seem to go a day without some catastrophic crash that I brought on myself#2021-01-0422:11vlaaadhow do you ensure your code is live when e.g. switching between branches?#2021-01-0422:26seancorfield@U47G49KHQ We rarely have long-lived branches and for the most part the code changes across multiple branches are accretive, rather than contradictory. I almost never need to require ... :reload-all (but I do have it bound to a hot key in VS Code). I occasionally need to remove a namespace (and then load it back in), if I'm changing aliases around (and, again, I have remove-ns bound to a hot key as well).#2021-01-0422:28seancorfield(and I tend to develop against multiple server processes all running from inside the REPL -- we have an "everything" subproject in our monorepo that has :local/root dependencies on all the other subprojects so we can start a REPL there and have all source/test code and all dependencies available in a single REPL)#2021-01-0422:05hanDerPeder@seancorfield of course, just being pedantic 🙂#2021-01-0714:58pinkfrogI am using belman, so the dep setup is:
{:aliases
:rebel
{:extra-deps {com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
:main-opts ["-m" "rebel-readline.main"]}
} ;
#2021-01-0714:58pinkfrogI want in every ns, I can access doc, and dir (which are from the namespace repl)#2021-01-0715:01Alex Miller (Clojure team)there isn't really any way to do this without making a customized repl#2021-01-0715:02Alex Miller (Clojure team)(which is not really that hard, but might be more than you want to do)#2021-01-0715:07borkdudeMaybe evaluating things from within user is an alternative approach?#2021-01-0715:08borkdude(I typically just require 'clojure.repl if I need it)#2021-01-0715:11pinkfrogthought the repl should support such a feature.#2021-01-0715:11Alex Miller (Clojure team)yeah, I just use if I'm working at the repl#2021-01-0715:12Alex Miller (Clojure team)but I'm usually working in an editor-connected repl and leave the repl window in user, while sending expressions from whatever namespace I'm working in#2021-01-0715:12Alex Miller (Clojure team)and using editor tools for doc support etc#2021-01-0715:14pinkfrogYou don’t drop in the repl window and switch to the current active buffer namespace for adhoc editting?#2021-01-0715:15Alex Miller (Clojure team)no, I write code in my editor and eval it#2021-01-0715:16Alex Miller (Clojure team)the main thing I usually use the repl window for is just (pst *e) :)#2021-01-0715:13Alex Miller (Clojure team)I dropped an example of making a repl to do this somewhere#2021-01-0715:14Alex Miller (Clojure team)oh hey, I even wrote it down... https://insideclojure.org/2020/02/11/custom-repl/#2021-01-0715:15pinkfrogIt is about clojure.main, so might not applicable to rebel#2021-01-0715:20borkdude@alexmiller hah, I just tried this with bb and it also works there:
$ bb -e "(require '[clojure.main :as m]) \
(m/repl :init #(apply require m/repl-requires) \
:eval (fn [form] (do (require '[clojure.pprint :refer [pprint]]) \
(eval form))))"
user=> (ns foo)
nil
foo=> (pprint {:a 1})
{:a 1}
nil
(only pp isn't available yet)#2021-01-0715:23Alex Miller (Clojure team)nice#2021-01-0715:24borkdude@i You should probably read the README of rebel read-line:
https://github.com/bhauman/rebel-readline#quick-usage
> The main way to utilize this readline editor is to replace the clojure.main/repl-read behavior in clojure.main/repl.#2021-01-0715:24borkdudeIt seems to offer integration with clojure.main#2021-01-0715:33pinkfrogwhich part are you referring to ?#2021-01-0715:34borkdudeunder quick usage#2021-01-0716:24practicalli-john@i I think I have missed something. If using rebel readline, you have docs and eldoc (function signatures) in every namespace as part of the default Rebel readline UI.
Its easy to run other commands from the repl namespace by using the fully qualified name, which rebel readline keeps in its history so you only have to type once and even then TAB will auto-complete the name of namespaces and functions, so you dont even need to type it in full once.
Eg. (clojure.repl/dir clojure.string)#2021-01-0716:27practicalli-johnI did a short video of using REBL for a very simple project, showing eldoc and docs and tab completion of namespaces and function names
https://practicalli.github.io/clojure/clojure-tools/repl/#2021-01-0721:04Alex Miller (Clojure team)for any of you toolsmiths looking for a project, it would be cool to see a tool that compared the APIs from two versions of a Clojure lib at the source level (like https://abi-laboratory.pro/?view=timeline&lang=java&l=clojure does for Java compatibility)#2021-01-0721:06Alex Miller (Clojure team)even knowing surface level add/remove/modify functions and add/modify arities would be a big help in detecting things (deeper level that understood keys in maps or coll types or whatever would be a harder task)#2021-01-0721:07borkdudeThis is very easy to make using clj-kondo analysis.
@lee didn't you make this kind of tools once for clj-doc?#2021-01-0721:08borkdudeFor anyone who wants to have a stab a it: https://github.com/borkdude/clj-kondo/blob/master/analysis/README.md#2021-01-0721:11lreadWe were talking about such a thing for cljdoc but never got to it.
I did hack together something to compare rewrite-clj/rewrite-cljs/rewrite-cljc.
It serves this purpose and but I am certainly not proud of its quality https://github.com/lread/diff-apis#2021-01-0721:13borkdudeI can whip up something in an hour or so#2021-01-0721:15lread@borkdude, I think you might live in a different space/time continuum than me. simple_smile Or there are several of you “We are borkdude, resistance is futile”.#2021-01-0721:17lreadWill your clj-kondo solution pick up potemkin and potemkin-like imports?#2021-01-0721:30borkdude@lee clj-kondo knows how to handle potemkin itself, but not sure about potemkin-likes#2021-01-0722:07vlaaadIsn't it something Rich worked on? Codeq?#2021-01-0722:27Alex Miller (Clojure team)codeq is actually in this ballpark, but is actually more fine-grained#2021-01-0722:28Alex Miller (Clojure team)I think it's both more and less than what I'm saying although it could be a good substrate#2021-01-0722:17borkdude@alexmiller Here's a first stab at it: https://gist.github.com/borkdude/2b963db1582654ec28bfd40b4dc35748#2021-01-0722:17borkdudeinvokable as JVM script#2021-01-0722:19borkdudeUsage: api_diff.clj lib1.jar lib2.jar (or api_diff.clj foo/src bar/src)#2021-01-0722:20borkdudeThere is also :private information in this analysis, I didn't take that into account, but small change.#2021-01-0722:26borkdudeThere might be other diffing libs out there that give a format that's better suited for what you want, this is just a start#2021-01-0722:27Alex Miller (Clojure team)pretty cool. would love to see something like this integrated that also used tools.deps so you could tell it a lib and two versions to compare#2021-01-0722:29Alex Miller (Clojure team)ultimately I wonder if you could say that a version has a breaking api change#2021-01-0722:39borkdude@alexmiller tda returns this for lib-location: {:base "/Users/borkdude/.m2/repository", :path "org/clojure/clojure/1.8.0", :classifier nil, :type :mvn}, is there anything I could use to resolve the full path to the jar?#2021-01-0722:40Alex Miller (Clojure team)I wouldn't use lib-location at all#2021-01-0722:40Alex Miller (Clojure team)better to resolve-deps or calc-basis and get the paths out of the result#2021-01-0722:41borkdudeI just want the location for this one thing, that's why I thought lib-location would be fitting#2021-01-0722:42Alex Miller (Clojure team)lib-location won't actually download it#2021-01-0722:42borkdudeAh I see:
#:org.clojure{clojure {:paths ["/Users/borkdude/.m2/repository/org/clojure/clojure/1.8.0/clojure-1.8.0.jar"], :deps/manifest :mvn, :mvn/version "1.8.0", :parents #{[]}}}
I can work with that#2021-01-0722:42borkdudeyeah, I did the downloading with resolve-deps already#2021-01-0722:43Alex Miller (Clojure team)for a maven dep, :paths should always have a single entry#2021-01-0722:43Alex Miller (Clojure team)(at least for now, maybe some future case to consider)#2021-01-0722:44Alex Miller (Clojure team)alternately, you could call org.clojure.tools.deps.alpha.extensions/coord-paths directly I guess#2021-01-0722:45Alex Miller (Clojure team)(ext/coord-paths 'org.clojure/clojure {:mvn/version "1.10.0"} :mvn {:mvn/repos maven/standard-repos})#2021-01-0722:46Alex Miller (Clojure team)probably the most direct but still most generic call you can make in the public api#2021-01-0722:46borkdudeGist updated, works for mvn deps:
api_diff.clj org.clojure/clojure "1.8.0" "1.10.1" > /tmp/diff.txt
#2021-01-0722:46borkdudeDetecting breaking changes is not something I will include today, it's getting too late ;)#2021-01-0722:47borkdudeI guess the algorithm should look at things that were in v1 but were removed in v2, or arities that were removed in v2#2021-01-0722:48borkdudeafk now, sleep#2021-01-0722:51Alex Miller (Clojure team)pretty sweet, nice hack#2021-01-0722:58Alex Miller (Clojure team)@borkdude there is one bug there - need to provide the standard maven repos if you don't have stuff locally#2021-01-0722:58Alex Miller (Clojure team)(require '[clojure.tools.deps.alpha.util.maven :as mvn])
#2021-01-0722:58Alex Miller (Clojure team)then
(-> (tda/resolve-deps `{:deps {~lib {:mvn/version ~v}} :mvn/repos ~mvn/standard-repos} {})#2021-01-0723:04borkdude@alexmiller Thanks, updated. Also swapped out deep-diff with editscript, this prints things like:
[[[clojure.core into] :fixed-arities 0] :+ 0]
[[[clojure.core into] :fixed-arities 1] :+ 1]
This makes it pretty easy to grep for :- for things that have disappeared.#2021-01-0723:05borkdudeMy grep for 1.8.0 ~ 1.10.1:
[[[clojure.pprint format-simple-number] :fixed-arities] :-]
[[[clojure.core emit-method-builder] :fixed-arities 4] :-]
[[[clojure.core.reducers compile-if]] :-]
#2021-01-0723:08borkdude(some of these could be due to subpar analysis by clj-kondo)#2021-01-0723:09borkdudeah for emit-method-builder it correctly detected an API change: https://github.com/clojure/clojure/commit/2cc37bb56a9125a1829c73c505e32995e663059a
but this function is private, so no problem#2021-01-0723:11borkdudeand indeed, compile-if disappeared (also private)#2021-01-0723:11borkdudeso the only false positive seems to be the one in pprint#2021-01-0723:11borkdudegnite#2021-01-0723:14Alex Miller (Clojure team)thx!#2021-01-0723:14Alex Miller (Clojure team)just trying to help here man - you're already a week into the new year without publishing any new tools#2021-01-0723:15borkdudethanks ;)#2021-01-0809:16rickmoynihanwoah that’s pretty cool!#2021-01-0809:32hkjelsI do concur; well done!#2021-01-0809:57borkdude@alexmiller Now updated the gist, it prints warnings like a linter:
https://gist.github.com/borkdude/2b963db1582654ec28bfd40b4dc35748
clojure/pprint/cl_format.clj:168:1: warning: Arity 1 of clojure.pprint/format-simple-number was removed.
clojure/core_deftype.clj:572:1: warning: Arity 4 of clojure.core/emit-method-builder was removed.
clojure/core/reducers.clj:24:1: warning: clojure.core.reducers/compile-if was removed.
It will print warning when there a change to a private function and error when there is public breakage.
So no breakage detected between Clojure 1.8 and 1.10.1 ;)#2021-01-0809:58borkdudeIt uses the locations of the the older version, since you cannot refer to the location of something that is removed.#2021-01-0810:01borkdudeJust as an example, suppose 1.9.0 was newer than 1.10.1, then you would expect the tool to print errors for some vars:
$ /tmp/api_diff.clj org.clojure/clojure "1.10.1" "1.9.0"
clojure/core.clj:6082:1: error: clojure.core/requiring-resolve was removed.
clojure/instant.clj:281:1: error: Arity 1 of clojure.instant/read-instant-calendar was removed.
clojure/core.clj:6073:1: warning: clojure.core/serialized-require was removed.
...
#2021-01-0810:06hkjelsNow we need a next_version tool that bumps the current version according to semantic versioning and these diffs 😏#2021-01-0812:22nnicholsI just wrote a GitHub action to do a SemVer version bump against a pom based on commit messages. If you want to steal the bash for that, here's a link: https://www.github.com/nnichols/maven-version-bump-action/tree/master/version-bump.sh
If I recall correctly, leiningen has its own tool/command for version bumps against project.clj files. I may have an example of that tucked away somewhere, but I'll have to do some digging to find it#2021-01-0814:39nnichols@U0B1SDL67 here is the same script, but using leiningen’s native command to do the version bumping. I’ll create a similar GitHub action for lein-based projects#2021-01-0814:40nnicholshttps://github.com/Wall-Brew-Co/common-beer-format/blob/master/bin/verchg#2021-01-0810:14borkdudeAlso made a babashka version now. 75 lines of "clojure" with fast startup, that downloads all the deps for you and prints break warnings.
https://gist.github.com/borkdude/ba372c8cee311e31020b04063d88e1be#2021-01-0814:53Alex Miller (Clojure team)@borkdude you can add org.slf4j/slf4j-nop {:mvn/version \"1.7.30\"} to get rid of the slf4j warnings#2021-01-0815:06borkdude@alexmiller added#2021-01-0920:04idasi'd like to create a project where i'd like to use a mix of java and clojure code, where the clojure code will be calling relevant custom java classes in the project. is there an example project using tools deps that illustrates the best way to go about doing that? i've come across https://github.com/EwenG/badigeon for possibly custom compiling the java classes via tools deps. is this a recommended tool to use?#2021-01-0920:12mike_ananev@idas I recommend to use https://github.com/seancorfield/depstar + make utility script. e.g. make javac will compile java sources, and make jar will compile Clojure code using depstar.
Also, you may use my wrapper over Badigeon https://github.com/redstarssystems/pbuilder which can make basic Leiningen tasks. Also, see lib and app templates on pbuilder page.#2021-01-0920:16idasthank you @mike1452! i'll take a look at this!#2021-01-1112:49martinklepsch@alexmiller @borkdude there’s more discussion around adding version diffing to http://cljdoc.org in this issue, feel free to subscribe if you want to stay in the loop: https://github.com/cljdoc/cljdoc/issues/336#2021-01-1112:51borkdudeThanks, I commented there now, so I'll be in the loop#2021-01-1301:17deactivateduserNot sure if this is the correct channel, but following on from a discussion during @seancorfield’s talk today for the London Clojurians, I whipped up a prototype that will determine the namespaces for one or more tools.deps dependencies.
Example usage:
(namespaces-for-deps! {'http-kit/http-kit {:mvn/version "2.5.0"}
'ring/ring {:mvn/version "1.8.2"}})
==> {http-kit/http-kit {:mvn/version "2.5.0", #snip#, :nses [org.httpkit.timer org.httpkit.client org.httpkit.encode org.httpkit.server org.httpkit.sni-client]}, ring/ring {:mvn/version "1.8.2", #snip#, :nses []}}
Code is here: https://gist.github.com/pmonks/982ce90dac11a4cefe45378c0db3d555 - feedback welcome, especially regarding whether this is worth turning into something else (CLI app, tools.deps plugin, wotnot).#2021-01-1301:21Alex Miller (Clojure team)I actually have an impl of a couple things like this related to add-libs (buried in the add-libs3 branch atm)#2021-01-1301:22Alex Miller (Clojure team)well I did, can't remember where that branch is at now#2021-01-1301:22Alex Miller (Clojure team)I mean, it's trivial to do with combo of tools.deps/tools.namespace#2021-01-1301:28deactivateduserYep, and that’s all this code does. Definitely seems useful enough to want to surface - it was one of the few “holes” in Sean’s demo flow in today’s talk - “how do you determine the namespaces in a dependency at the REPL?“.#2021-01-1301:29deactivateduserhttps://github.com/hagmonk/find-deps also adds some nice ergonomics.#2021-01-1301:31deactivateduserFWIW I used tda’s resolve-deps fn to pull down the requested dependencies, but that felt a bit heavyweight since it brings down the transitive graph of dependencies as well (which I don’t think tools.namespace needs to do its thing). Though I guess short-cutting the graph might cause problems with other tda-based tooling...#2021-01-1301:33Alex Miller (Clojure team)yeah, you can pretty easily use tools.deps to pull just a single lib - it's not exactly part of the main api but you can do it#2021-01-1301:34deactivateduserYeah cool - not a big problem to pull down the entire graph for this kind of “exploration time” logic, imvho.#2021-01-1301:36Alex Miller (Clojure team)it's really just crafting a call to (c.t.d.a.extensions/coord-paths 'the/lib {:mvn/version "1.2.3"} deps-edn)#2021-01-1301:36Alex Miller (Clojure team)which should download and return a vector with one path which is the jar#2021-01-1301:38Alex Miller (Clojure team)you do need to require c.t.d.a just to cause the multimethod extensions to all get loaded#2021-01-1416:02ChaseI've been seeing folks use like clj -M:new and clj -X:new or whatever but if I put in anything but -A I get an error. What are the other symbols doing and what should I do to get access to them if desired?#2021-01-1416:03andy.fingerhutDo you see a version number near the beginning of the output of clj -h , and if so what is it?#2021-01-1416:04ChaseVersion: 1.10.1.536#2021-01-1416:04andy.fingerhutI don't know what minimum version is required to support the command line options you see, but the one I currently have is 1.10.1.763#2021-01-1416:04andy.fingerhutIf you use homebrew on Linux or macOS, there are brew commands for both upgrading and downgrading the version of the clojure CLI tools. So far I've only ever upgraded.#2021-01-1416:05Alexis VincentI seem to remember having to install an alpha version of clojure from brew#2021-01-1416:05andy.fingerhutI haven't kept track of all of the details of versions and command line option changes for Clojure CLI tools, but I see discussion flying by about it on #tools-deps#2021-01-1416:05ChaseI'm on linux and used the script installer. Can I just repeat those steps with the latest one? Would that override my current one. I would rather not have multiple versions if that is going to trip me up down the line#2021-01-1416:05Alexis Vincentbut that was a while ago. Might be stable release now#2021-01-1416:07andy.fingerhut1.10.1.763 is the current version mentioned here: https://clojure.org/guides/getting_started#2021-01-1416:08andy.fingerhutOn Linux, you can rerun the shell script installer and it will overwrite the version you installed earlier, no problems.#2021-01-1416:08ChasePerfect, exactly what I was wondering.#2021-01-1416:08andy.fingerhutclj -h version number is your friend if you are ever unsure what version you are getting.#2021-01-1416:09Alex Miller (Clojure team)tool version history can be found here: https://clojure.org/releases/tools#2021-01-1416:09Alex Miller (Clojure team)536 does pre-date the -M/-X support you mentioned in first post#2021-01-1416:10andy.fingerhut2020 was a big year for Clojure CLI tools!#2021-01-1416:11Alex Miller (Clojure team)well, 2019 was as well, this history doesn't go back that far (but see the more detailed changelogs for that)#2021-01-1417:09seancorfieldThe clj-new README specifies you need at least 1.10.1.727 (as does depstar's README).#2021-01-1618:16practicalli-john@U9J50BY4C Version 1.10.1.697 is the magic number when the flags change... but Alex put some nice fixes in since then so use the latest version from http://Clojure.org Getting Started page.
Here is a crib sheet of tasks I use and where their definitions come from
https://practicalli.github.io/clojure/clojure-tools/using-clojure-tools.html#common-tasks-for-clojure-development#2021-01-1618:17practicalli-johnIf you get problems, delete any .cache directories#2021-01-1618:18practicalli-johnThe Linux installer will install which ever version you install to the same place, so you only ever have one version installed (the last one you installed with the linux install script)
clojure -Sdescribe to see what version is being used#2021-01-1619:26seancorfieldInstead of deleting .cpcache (not .cache), it's better to use -Sforce as a one-off to tell the CLI to recompute the cache for that specified set of aliases and dependencies etc.#2021-01-1506:02didibusI understand -X lets you call a specific function from some random namespace and pass it arguments. So you can configure an alias like:
:new {:extra-deps {seancorfield/clj-new {:mvn/version "RELEASE"}}
:exec-fn clj-new/create
:exec-args {:template lib}}
And then call clj -X:new and clj will run the create function from clj-new namespace and pass it as argument {:template lib}#2021-01-1506:02didibusIt seems -M is just the new -A where your alias had a :main-opts [] defined.#2021-01-1506:02didibusAnd I don't know what has become of -A#2021-01-1506:06seancorfield-A will stop running :main-opts at some point and will be used just for starting a REPL (with various aliases). At least, that's what I understand @alexmiller to have said about it.#2021-01-1506:07seancorfield(and you'll get deprecation warnings right now if you try to use -A with any main opts -- either :main-opts or via the command line)#2021-01-1506:10seancorfield@didibus ^ does that answer your "question"? 🙂#2021-01-1506:11seancorfield(and, to be clear, -M has expanded from respecting just :main-opts to respecting "everything")#2021-01-1506:35didibusI see, so -A will be if you just want to like bring more or less deps in your repl sa you start it?#2021-01-1506:35didibusOr like on anything, since I assume you can combo like -M and -A#2021-01-1506:36didibusOh, so -M will be smart? So like you can use -M for alias that define an exec-fn ? So I guess only when its ambiguous you need to explicitly use -X ?#2021-01-1506:37seancorfieldWhat?#2021-01-1506:37seancorfield-X invokes a function. -M invokes a main.#2021-01-1506:38seancorfieldSo -X looks for :exec-fn or a function on the command-line. -M runs :main-opts (or main options on the command-line).#2021-01-1506:38seancorfieldThere's no overlap there.#2021-01-1506:39seancorfieldRe: -A, yes, my understanding, based on what Alex has said, is that -A will at some point simply stop running :main-opts and will instead always start a REPL (with all the dependencies and classpath stuff set up per whatever aliases you specify).#2021-01-1506:41seancorfieldOnce -A stops running main options, there will be no overlap there either:
* -X -- invoke a function with a single hash map argument
* -M -- invoke clojure.main with various options (`-m`, -e presumably)
* -A -- run a REPL#2021-01-1506:42didibusOk, maybe I didn't get what you meant by: (and, to be clear, -M has expanded from respecting just :main-opts to respecting "everything")#2021-01-1506:43didibusWhat is everything?#2021-01-1506:43seancorfield(at the risk of sounding like a broken record, I still think expanding -R for the REPL option and just plain deprecating -A would have been better but...)#2021-01-1506:43seancorfield"everything" = dependencies and class paths and jvm options etc#2021-01-1506:44seancorfield-R and -C and -O used to be individual options for resolve args, classpath args, and jvm options.#2021-01-1506:44didibusOh I see, ok I just assumed that haha, I guess I missed the time when -M did not respect those#2021-01-1506:45didibusDidn't -A respect all those as well?#2021-01-1506:45seancorfieldIt used to be, prior to this big change, that -A was "all things" and all the other options just dealt with a specific thing (and there was no -X but there was a -M).#2021-01-1506:46seancorfield-A still does respect all of those.#2021-01-1506:46seancorfieldAt some point it will ignore main opts.#2021-01-1506:48didibusAh ok I see. I feel like -A should become: run any alias fully. If the alias has a :main-opts run that, if it doesn't but has an exec run that, if not just start a repl with the extra-deps and all#2021-01-1506:49seancorfieldWell, no, that's not what it is and not what it's going to be. That ship has sailed too 🙂#2021-01-1506:49didibusIts strange to provide in your README like: add this alias, run with -X, or add this alias but this one remember to run with -M#2021-01-1506:49didibusLike what's the point of configuring anything in the deps.edn file, if I still need to remember some of it at the command line 😛#2021-01-1506:50seancorfieldI suspect, over time, most tools will change to -X execution because it is more powerful/more convenient.#2021-01-1506:51seancorfieldclj-new switched, depstar switched. I see deps-deploy switched. When tools.build arrives, I bet it's going to be all -X style stuff.#2021-01-1506:52didibusIt depends though, passing args to -X is tricky, and not unix like at all, so if you make a CLI, like take clj-kondo for example, if I want to use it from clj -X then the argument I would pass it need to be formatted differently than when I'd call it with clj-kondo executable#2021-01-1506:52seancorfield-X is "native" Clojure. -M is artificial. The latter is all strings that need to be parsed and turned into Clojure data. The former already is Clojure data.#2021-01-1513:57Alex Miller (Clojure team)bleh on those words. -X is function execution, -M is clojure.main. They are both Clojure.#2021-01-1517:35seancorfieldI just meant in terms of -X calling a Clojure function directly -- and being idiomatic in terms of the arg being a single hash map with keys and values from the command-line -- whereas -M is like a traditional JVM process, via -main, and passing strings that Clojure needs to parse. Hence "native" in quotes 🙂#2021-01-1506:54seancorfieldWith -main (and -M) everyone needs to write their own ad hoc argument parsing (even if they build on tools.cli). There's no consistency there at all. With -X, it's absolutely consistent and all tools will work the same way: they'll call a Clojure function with a hash map built as EDN.#2021-01-1506:54didibusI guess I see pros/cons for each. While -X is native Clojure, it is not native bash or powershell, so now that side of the interface needs to do the mapping.#2021-01-1506:54seancorfieldYou seem to presume there's a real standard in command-line arguments already... which there really isn't...#2021-01-1506:55didibusTrue, but I'm talking at least the formatting#2021-01-1506:55seancorfieldHow do you pass a vector of integers to a command-line program?#2021-01-1506:56seancorfieldDead easy with -X. Not easy with -M, even with tools.cli (which, remember, I maintain these days!).#2021-01-1506:56didibusRight, but then, a very common use case is to pass a string, and with -X you need to do this shenanigan: '"hi there"'#2021-01-1506:57seancorfieldI think that's actually less common that you might imagine. And for a lot of "string" arguments, it's actually fine to pass a symbol and have the function decide to handle symbols as strings if necessary.#2021-01-1506:58didibusFor tooling, I think -X makes more sense, but for distribution I feel -M might make more sense. Like if you want to use tools.deps to distribute your command line tools to users.#2021-01-1506:59seancorfieldHaving used this new stuff for a while I don't agree. I think you will change your mind in due course.#2021-01-1506:59seancorfieldRight now, it's just "different" and you're resisting that change.#2021-01-1507:01didibusI personally prefer -X a lot. I do feel now though I keep having to open my deps.edn to remember if that particular alias I have to call with -M or -X though 😛, that's my biggest problem. But I was wondering for -X what it mean for CLIs, it still seems to me if you're puporsely making a CLI tool, you wouldn't want it to take EDN as input, that would throw of most users. Like say I made a grep in Clojure. And then you'd probably use -M with it.#2021-01-1507:02seancorfieldHaving maintained tools.cli for a while and having written and maintained a lot of command-line stuff over the decades, I hate having to deal with strings and ad hoc parsing and inventing semantics for all that stuff over and over and over again...#2021-01-1507:03didibusSo you'd go as far as saying that if someone were to use Clojure for a CLI, it might be a good idea they just take EDN as args?#2021-01-1507:03seancorfieldYes, very much so.#2021-01-1507:04seancorfieldIt's consistent. It's provides access to all of Clojure's data types. You can provide structured data as arguments really easily.#2021-01-1507:05didibusHow easy is that? Is it just:
(defn -main
[& {:keys [:option1 :option2]}]
...)
Would this mimic exactly clj -X ?#2021-01-1507:05seancorfieldAlex was saying that they're looking at a way to pass Vars directly, so you don't have to then resolve symbols, which will make this even more powerful.#2021-01-1507:05seancorfield-main can't mimic -X -- it must take a sequence of strings because that's what Java does.#2021-01-1507:06seancorfieldpublic static void main( String[] );#2021-01-1507:06seancorfield(or whatever nasty syntax Java requires)#2021-01-1507:06didibusSo its not possible to make a CLI with Clojure that takes the same argument format as when launched with -X ? I mean, its got to be no since clj does it 😛#2021-01-1513:58Alex Miller (Clojure team)of course it is - that's what -X calls#2021-01-1507:07seancorfieldI don't understand your question.#2021-01-1507:07seancorfieldclj is a shell script.#2021-01-1507:09didibusLike say I'm writing a Command Line Application. Lets say an implementation of ls but in Clojure. And I'm going to distribute this to people as say a Graal Native Binary. Or as some UbrJar that I wrap a bash script around to launch it.
Now say I would like it the arguments to this Clojure based ls would be similar to when called with -X. I want it to be EDN. So I want to do: ls :verbose true for example#2021-01-1507:10didibusI guess I'm asking... Will tools.deps expose the parser it uses for -X so I can use it in place of tools.cli or something like that?#2021-01-1507:11seancorfieldI can't answer that, I'm not part of the Clojure core team.#2021-01-1507:11vlaaadThe parser is clojure.edn/read-string#2021-01-1507:13seancorfieldIt's more than that: there's a clojure.run.exec ns (currently) that strings all of that together including the function invocation.#2021-01-1507:13didibusLike, (read-string (reduce str args)) ?#2021-01-1507:18vlaaadMore like (apply hash-map (map edn/read-string args))#2021-01-1507:19vlaaadEvery string arg at CLI is parsed to edn separately#2021-01-1507:19didibusYa, I mean that's not a bad idea, even if not exactly what tools.deps does for -X (I don't know if it is or not), but for my next CLI such a simple parsing of args could make sense, I might try it out#2021-01-1507:23vlaaad-X is a bit more complicated since it has to deal with defaults and overrides, but I think this approach is very nice because you are able to have the same api both for -X and -M invocations#2021-01-1507:28didibusYa, it quite neat#2021-01-1507:29didibusAt first I was thinking, ok but users might find :option ... weird, but Sean is right, CLIs are already pretty inconsistent here, some take -o --option --o and all that#2021-01-1507:31didibusAlso I guess in theory, the EDX parsing can even return a map of symbol to value, so you can probably even use this to parse -o which I think is a valid symbol#2021-01-1507:33vlaaadhaha yeah#2021-01-1507:34vlaaadso is -h , -? and --help 🙂#2021-01-1507:35vlaaadI experimented a bit with it some time before the -X got released: https://vlaaad.github.io/tools-cli-in-10-lines-of-code#2021-01-1507:35didibusNice lol. So the only annoying bit is if you need to take a string as the value, it has to be: app -option '"the long string"'#2021-01-1507:35vlaaadyeah#2021-01-1507:37didibusOh neat, I will definitely bookmark and read this blog, thanks#2021-01-1507:37didibusOff to bed now though, gn#2021-01-1507:13vlaaadhttps://github.com/clojure/brew-install/blob/fde9db19bfe92a315b81b5f56d290a6d37a91d89/src/main/resources/clj_exec.clj#2021-01-1507:14vlaaadFound it!#2021-01-1507:15seancorfieldThat's very much an implementation detail right now -- it's some pretty sketchy code 😐#2021-01-1507:15didibusWell, either them releasing it separately, or someone replicating it in a lib would be neat. Then you'd be able to make a CLI and the UX would be the same either used with -X or from the command line directly.#2021-01-1507:16borkdudeBtw for some CLIs I support an —opts argument where you can pass all args as one EDN map. I find this more ergonomic in some cases than passing separate args with their own paths into the map which then also need individual quoting #2021-01-1507:16seancorfieldSince -X specifies which function to call (either directly or via :exec-fn in aliases) it is pretty much integral to how these arguments are handled.#2021-01-1507:17seancorfieldI mean, basically, -X is tied to deps.edn etc so the question you're asking about GraalVM binaries doesn't really make sense.#2021-01-1507:20seancorfieldIf you "fix" the values of :exec-fn, :exec-args and do not accept a function to invoke, then yes you could have a standardized command-line based on keywords and EDN values (and I think there's code in t.d.a directly that handles that -- the clj-exec thing is just a copy/variant specifically to support CLI invocation via -X).#2021-01-1507:20vlaaadYeah, if the target audience is not clojurists, it might be better to use something like tools.cli#2021-01-1507:21borkdudeRight. If you want to support the CLI for JVM invocation via deps.edn you can also add an -X one apart from the Unix style #2021-01-1507:22didibusLike taking an -X that then takes an EDN string?#2021-01-1507:23didibusOh you mean, in the CLI, have a -main and another function for -X ?#2021-01-1507:23seancorfield-X :keyword some-edn :another-keyword more-edn#2021-01-1507:24borkdudeYes, both #2021-01-1507:25borkdudeAnd please don’t deprecate one over the other wink wink, there is more than one build tool#2021-01-1507:25didibusYa, you can do that, extra work though. It be nice if you could just only have the -X one, and use it even for -main somehow#2021-01-1507:21borkdudeIt’s not an either/or#2021-01-1507:22didibusThat's what I mean. -X is starting a new convention for Clojure CLI applications in how their command line args are specified. But I would find it weird if a command line app takes different style of arguments based on how it is invoked. So if I'm to make one and use -X, it be nice if other ways to start it I can also just pass it -X style arguments.#2021-01-1507:24seancorfieldI'd be quite happy if all Clojure tooling worked like -X 🙂#2021-01-1507:25seancorfield(beer's empty, battery's nearly dead, and I have to get up and go have my brain swabbed in the morning to see whether or not I have COVID, so I'll bid y'all adieu)#2021-01-1507:34didibusI'm off as well, nice day everyone#2021-01-1514:01Alex Miller (Clojure team)clojure.run.exec should currently be considered a changing implementation detail#2021-01-1514:01Alex Miller (Clojure team)(and it has been actively and regularly changing)#2021-01-1514:05Alex Miller (Clojure team)@didibus @seancorfield I do think that ultimately most of the code there should be cleaned up and included in clojure (like clojure.main). right now, clj has to sneak that into your classpath and I'd prefer for it to rely on getting via your clojure dep. But the current "api" is pretty dirty and is doing half of clj's parsing job. until that's cleaner, not ready to do that yet.#2021-01-1514:07borkdudeI would consider a same kind of API for babashka scripts when this is considered done. I'm not 100% convinced of the way "nested" args are passed in, since that results in a lot of command line quotes for basic things like strings. Passing in an entire config map feels more natural to me, but maybe the design choice was: we don't want to decide how nested args should be merged, we only support assoc-in?#2021-01-1514:08borkdudeHow would you for example add an element to a vector inside an arg map? {:a {:foos [1 2]}} and now you want to add 3?#2021-01-1514:08Alex Miller (Clojure team)not something we were trying to support#2021-01-1514:09Alex Miller (Clojure team)the idea is that you're mostly assembling maps, not updating them#2021-01-1514:10borkdude{:some-tool {:scan-paths ["src"]}}
now add "test" to :scan-paths, this is maybe a better example#2021-01-1514:10Alex Miller (Clojure team)I await your ask clojure request :)#2021-01-1514:14borkdudefirst one: https://ask.clojure.org/index.php/10059/x-args-add-things-to-vector#2021-01-1514:17borkdudesecond one: https://ask.clojure.org/index.php/10060/x-args-pass-in-entire-map-merge-strategy#2021-01-1915:23vemvHow to prevent Ctrl-C from quitting clj?
I'm too used to repl-y's meaning of Ctrl-C (which interrupts eval OR discards current unsubmitted input but never quits the repl)#2021-01-1915:25Alex Miller (Clojure team)I think that's the default behavior for a Java process (default signal handler)#2021-01-1915:25Alex Miller (Clojure team)so you actually would need to install a new signal handler#2021-01-1915:25vemvthanks, I'll do that!#2021-01-1915:26Alex Miller (Clojure team)there is actually a thing in clojure.repl I think that does that#2021-01-1915:27Alex Miller (Clojure team)clojure.repl/set-break-handler!#2021-01-1915:28Alex Miller (Clojure team)at the very least, that's something to look at#2021-01-1915:56Alex Miller (Clojure team)the signal handler stuff is a little weird in that it's a very old sun.misc interface. Java made a big push to eradicate all of those a while back but at the time they had no replacement way to do this. afaik, they still don't.#2021-01-1916:05vemv:)
thanks for all the info
(clojure.repl/set-break-handler! identity) was quite satisfactory#2021-01-2119:09Eamonn SullivanHi all, I'm trying to build a tools.deps project at work using AWS CodeBuild. One huge obstacle I've had in getting Clojure into work is that deps.edn cannot resolve our internal libs (long story, but ssl cert-based auth), so I thought I'd try this kind of container-based building with :git/url refs instead and a personal access token. This works to check out the top-level repo, but when Clojure tries to resolve dependencies:
Error building classpath. : Authentication is required but no CredentialsProvider has been registered
org.eclipse.jgit.api.errors.TransportException: : Authentication is required but no CredentialsProvider has been registered
These references work fine locally, but I'm not sure how, so I'm not sure how to cajole CodeBuild into doing it...
Apologies if this the wrong channel. I seem to be unclear on the role of tools-deps. (And there's a ticket already about providing cert-related Java opts to Clojure.)#2021-01-2119:19pavlosmelissinosAccording to this: https://clojure.org/reference/deps_and_cli#_git
for private git repos it's better to use the ssh protocol under :git/url and set up the ssh-agent separately, so you might want to try it out (unless someone more knowledgeable chimes in with a better suggestion, which is pretty probable 😄).#2021-01-2206:44Eamonn SullivanThank you! That put me on the right track, I think. Will give it a go. #2021-01-2119:38Alex Miller (Clojure team)we don't currently have support for git https with auth#2021-01-2119:38Alex Miller (Clojure team)if that's what you're trying to do#2021-01-2206:43Eamonn SullivanNo, but I think I figured out what I need to do. It works locally because of the ssh config set up. I will need to set up a docker container configured the same way, save that to an ECR and then specify that docker in the build. 🤞 #2021-01-2212:52Jakub Holý (HolyJak)Hi! How to solve
> Cloning: https://github.com/borkdude/clj-http-lite
> Error building classpath. Destination path "clj-http-lite" already exists and is not an empty directory
> org.eclipse.jgit.api.errors.JGitInternalException: Destination path "clj-http-lite" already exists and is not an empty directory
> at org.eclipse.jgit.api.CloneCommand.verifyDirectories(CloneCommand.java:257)
caused by a borkdude/clj-http-lite {:git/url "" ...} dependency? 🙏
i.e. where does it exist, so that I can delete it?#2021-01-2212:54borkdude@holyjak rm -rf ~/.gitlibs#2021-01-2213:34Alex Miller (Clojure team)well you don't have to rm the whole thing :)#2021-01-2213:34Alex Miller (Clojure team)but yes, that usually means there is a partial download in the ~/.gitlibs dir#2021-01-2213:35Alex Miller (Clojure team)if you're running into this often, consider using -Sthreads 1 on clj#2021-01-2213:44borkdudeI usually do remove the whole thing because there are two things inside .gitlibs, repositories and some other things, so just to make sure, I bluntly get rid of it all ;)#2021-01-2213:54Eamonn SullivanI didn't know it was supposed to work without that error message. :thinking_face: Happens to me 100% of the time, so I do the same: just blow it all away. I didn't think to try single-threading.
Also, I just recently discovered why I was getting cryptic error messages about an invalid privatekey, unless I removed my .ssh/config every time. My key format was too old. A regen (`ssh-keygen -p -m pem -f ...`) finally got rid of that for me, making this feature much more useful.#2021-01-2213:55Alex Miller (Clojure team)if that doesn't help, I'd be curious to hear about it. my belief right now is that this is a concurrency issue where parallel downloads happen and one is stomping on the other in the same dir#2021-01-2213:56Eamonn SullivanI will definitely try this next time. I'm currently trying to get this all to happen in a docker container, so I'll probably have lots of opportunities...#2021-01-2214:05nnichols@UR71VR71S Forwarding the SSH agent for a dockerized build took me a while to grok. It requires the buildkit tools,
DOCKER_BUILDKIT=1 docker build --ssh default --tag ....
and some of the experimental options:
# syntax=docker/dockerfile:1.0.0-experimental
### -----------------------------------------------------------------------------
### PHASE 1:
### Build the application within a Docker image
FROM clojure:openjdk-8-tools-deps as builder
WORKDIR /home/app
COPY . .
### Install git so we can pull dependencies from GitHub repositories
RUN apt-get update -y
RUN apt-get install git
### Download public RSA keys for
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan >> ~/.ssh/known_hosts
### Use seancorfield/depstar to create a POM and build the app into an uberjar
### This fetches dependencies AOT, so we don't need to fetch anything when the app starts
### Make sure your app specifies a (:gen-class) directive in the main namespace
### This will AOT compile the main namespace, and anything required in that namespace
RUN --mount=type=ssh clojure -Spom && \
clojure -Sdeps '{:deps {seancorfield/depstar {:mvn/version "0.5.2"}}}' \
-m hf.depstar.uberjar my-app.jar -C -m my-app.main
#2021-01-2214:44Eamonn SullivanThank you very much, @UCZ5ZDJKF! You probably saved me hours...#2021-01-2214:46nnicholsAnytime! It wasn’t a fun discovery process lol#2021-01-2214:57Alex Miller (Clojure team)you should write that down somewhere :)#2021-01-2214:58Alex Miller (Clojure team)feel free to make a wiki page at https://github.com/clojure/tools.deps.alpha/wiki if you need a place#2021-01-2216:02nnichols@alexmiller I added a more in-depth explanation + sample here: https://github.com/clojure/tools.deps.alpha/wiki/SSH-Authentication-in-Docker#2021-01-2216:02Alex Miller (Clojure team)cool#2021-01-2215:12danboykisdoes anyone know if proxy username/password in ~/.m2/settings.xml are respected when fetching dependencies? From what I can tell the answer is no.#2021-01-2215:21Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_maven_proxies is supported but I'm not familiar with any user/pw thing for that#2021-01-2215:22Alex Miller (Clojure team)let me glance at the code#2021-01-2215:23Alex Miller (Clojure team)there is code there with that intent (can't say I've tested that personally)#2021-01-2215:24Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L98-L100#2021-01-2215:35borkdudeI'm not sure if this is relevant here, but deps.clj forwards proxy environment variables to the tools.deps JVM's java properties:
https://github.com/borkdude/deps.clj#proxy-environment-variables#2021-01-2215:43danboykisthanks @alexmiller i'll play with it and see if I come up with something more concrete#2021-01-2215:44Alex Miller (Clojure team)happy to look at fixes if something isn't working there (but I'm somewhat limited in ability to test)#2021-01-2500:13p-himikIs the WARNING: When invoking clojure.main, use -M warning supposed to be printed even when executing just clj -e 1?#2021-01-2500:19p-himikOr when I'm running my own -main, like in clj -m my.proj.#2021-01-2501:03Alex Miller (Clojure team)yes, and yes#2021-01-2501:04Alex Miller (Clojure team)-M (now) means "run clojure.main" so should be used with -m or -e or calling a script.clj etc#2021-01-2501:12p-himikAh, I see. I thought it was only for aliases but now I can see that the list of aliases is optional. And I didn't realize that clojure.main is called even when I specify -m. Thanks!#2021-01-2509:35lepistaneHello
this is super basic question.
I create simple command line program which i would like to execute. I used https://www.clojure.org/guides/deps_and_cli#_writing_a_program
as a reference but i have a problem.
i have `deps.edn` , `src/data.clj` that has fn `foo`, and when i do `clj -X data/foo` i get this error
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
-X (No such file or directory)
Full report at:...
what am i doing wrong?#2021-01-2509:39borkdude@lepistane Probably a too old version of clj#2021-01-2509:48lepistaneanother basic question :S
how do i remove clojure cli from my machine?#2021-01-2512:17lepistanethis worked!
i just repeated installation steps thanks!!#2021-01-2518:42seancorfield@lepistane I've found it helpful to use brew to manage my Clojure CLI install even on Linux, so that I can choose to install a specific version and also make it easy to keep the CLI up-to-date.#2021-01-2612:03CarnunMPHi all. Quick question - I hope this is the right place to ask it! :))
Lets say I have a user namespace in the same directory as my deps.edn, containing a start function. Why is this ⬇️ happening?
Clojure 1.10.1
user=> (start)
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: start in this context
user=> (require 'user)
Execution error (FileNotFoundException) at user/eval2 (REPL:1).
Could not locate user__init.class, user.clj or user.cljc on classpath.
#2021-01-2612:05borkdude@carnunmp you should probably put the user.clj on your classpath, e.g. in src ?#2021-01-2612:11CarnunMPI think that's done it. Cheers!#2021-01-2613:09dpsuttonI'm trying to ensure that the {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[\"cider.nrepl/cider-middleware\"]"]} main opts for cider startup are run in case there are main opts in the aliases used. The documentation states that "If multiple maps with these keys are activated, only the last one will be used". My attempt to make nrepl.cmdline the "last" is the following:
/usr/local/bin/clojure -A:test -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.8.3"} cider/cider-nrepl {:mvn/version "0.25.7"}} :aliases {:cider/nrepl {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[\"cider.nrepl/cider-middleware\"]"]}}}' -M:cider/nrepl
(where -A:test includes a main runner for kaocha). My assumption was that putting -M:cider/nrepl last in the string would ensure that it was last for the purposes of main arg combining but this doesn't seem to work. It still starts the kaocha runner instead of the nrepl server. Is there a way to achieve this? Have I misunderstood what last is in how the aliases are combined?#2021-01-2613:16Alex Miller (Clojure team)The -A are always pulled in last#2021-01-2613:16Alex Miller (Clojure team)Is there some reason you’re not doing both aliases in -M#2021-01-2613:17Alex Miller (Clojure team)-M:cider/nrepl:test#2021-01-2613:22dpsuttonyes. this is CIDER's startup command which the user can add their own aliases to like a template#2021-01-2613:23dpsuttonthe template being clojure [optional user aliases] -Sdeps {...} -M:cider/nrepl but it seems this is not a good template?#2021-01-2613:56Alex Miller (Clojure team)I do not think that is a good template - all of the aliases should be under -M imo#2021-01-2613:24dpsuttoni suppose clojure -Sdeps {...} -M:[optional user aliases]:cider/nrepl is a way to ensure cider/nrepl is the last alias?#2021-01-2613:26dpsuttonand if so, are there any implications to tucking what someone thought would run under -A:foo instead under -M:foo:cider/nrepl besides the desired suppression of the possible main opts?#2021-01-2613:30dpsutton#2021-01-2613:33dpsuttonreading clj --help
exec-opts:
-Aaliases Use concatenated aliases to modify classpath
-X[aliases] Use concatenated aliases to modify classpath or supply exec fn/args
-M[aliases] Use concatenated aliases to modify classpath or supply main opts
-P Prepare deps - download libs, cache classpath, but don't exec
it seems perhaps the current invocation will be correct under a future version? Reading -A and -M to differ by the -M allowing for main opts and therefore -A to not propagate those keys?#2021-01-2613:56Alex Miller (Clojure team)I do not think that is a good template - all of the aliases should be under -M imo#2021-01-2613:56Alex Miller (Clojure team)-A does currently provide :main-opts, but long-term, it will not#2021-01-2613:57Alex Miller (Clojure team)does :cider/nrepl have :main-opts?#2021-01-2613:57Alex Miller (Clojure team)I guess so#2021-01-2613:58Alex Miller (Clojure team)going back to your original question, why are you trying to use both cider and run test?#2021-01-2613:58Alex Miller (Clojure team)seems like you should be doing one or the other but not both#2021-01-2614:03dpsuttonyes :cider/nrepl specifies the {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[\"cider.nrepl/cider-middleware\"]"]} so that the nrepl server is correctly started up for CIDER.
This bug came up because someone had an alias which included kaocha, its main runner as main-opts, and extra test paths. Kaocha does have a useful repl mode, and the test paths are desirable as well. So it was hoped that the alias could be used and the main ignored in favor of CIDER's requirements.#2021-01-2614:04dpsuttonSo the hope was to structure the startup command in a way that we naturally eclipsed all other main args in favor of our own.#2021-01-2614:20Alex Miller (Clojure team)ah, ok. so the intent was to NOT get kaocha main when starting cider main (I was thinking it was the opposite). Indeed, once -A changes to not support :main-opts, what you had would have worked (but does not currently).#2021-01-2614:22dpsuttoncorrect. sorry i was unclear in desired behavior. I've made the change with the note that we can go back at some point in the future. thanks for helping confirm i'm doing things the right way. much appreciated#2021-01-2614:26Alex Miller (Clojure team)well I was still half asleep when I first read it so prob me :)#2021-01-2614:05dpsuttoni've already made the change for the template string to be -M[optional-args]:cider/nrepl so thanks for your help. Its always appreciated and the clojure cli as well 🙂#2021-01-2614:17tvaughanSorry, I'm not sure which is the best channel to ask this (I don't see an antq channel), but is anyone using antq to keep dependencies from a password protected maven repo up-to-date in deps.edn? Everything is setup correctly and works with tools.deps. Only antq seems to have a problem. For example, with Datomic dev-local:
| :file | :name | :version | :latest-version |
|----------+-----------------------+----------+-----------------|
| deps.edn | com.datomic/dev-local | 0.9.225 | Failed to fetch |
| pom.xml | com.datomic/dev-local | 0.9.225 | Failed to fetch |
Searching the antq github repo didn't turn up any relevant issues, nor do I see a verbose or debug option to get additional details#2021-01-2614:21Alex Miller (Clojure team)link for antq?#2021-01-2614:21borkdudehttps://github.com/liquidz/antq#2021-01-2614:24Alex Miller (Clojure team)seems like a bug in antq so I'd file an issue there#2021-01-2614:24Alex Miller (Clojure team)weirdly antq is using tools.deps.alpha for parts of its maven code, but not using it to read deps from deps.edn files#2021-01-2615:33dominicmI think that's intentional, it's reading the literally written deps in order to make decisions based on those. It's not trying to resolve the location of those deps nor include system or user deps.#2021-01-2615:34Alex Miller (Clojure team)I get that, but even so it could be using more of tools.deps#2021-01-2615:35dominicmI haven't checked the source, but there might also be a desire to use tools.reader to modify the deps.edn without losing comments and such. Not sure.#2021-01-2618:33lreadJust in case @UBL24PLE6 (the author of antq) is not browsing, this convo, I’ve roped them in!#2021-01-2622:09uochanOh, thanks for your information!
I should read tools.deps source more and more.#2021-01-2614:25tvaughanGreat! Thank you @alexmiller I'll file a bug#2021-01-2614:40tvaughanhttps://github.com/liquidz/antq/issues/54#2021-01-2615:33dominicmI think that's intentional, it's reading the literally written deps in order to make decisions based on those. It's not trying to resolve the location of those deps nor include system or user deps.#2021-01-2914:56mpenetwhat's the uberjar alternative "du jour" ?#2021-01-2914:56mpenetsomething rock solid#2021-01-2914:57mpenetI mean a clj plugin that would replace lein uberjar#2021-01-2914:57dharriganI use seancorfield/depstar in my deps.edn#2021-01-2914:57mpenetthere are a lot of them https://github.com/clojure/tools.deps.alpha/wiki/Tools#packaging#2021-01-2914:57dharrigani.e.,#2021-01-2914:58dharrigan:uberjar {:replace-deps {seancorfield/depstar {:mvn/version "2.0.165"}}
:jvm-opts ["-Dclojure.compiler.direct-linking=true"]
:exec-fn hf.depstar/uberjar
:exec-args {:jar foo.jar
:group-id bar
:artifact-id foo
:version "0.0.0"
:main-class foo.bar.main
:aot true
:sync-pom true}}}#2021-01-2914:58mpenet@dharrigan ah right. they also uses that extensively I think, so it's a good bet I guess#2021-01-2914:58dharrigannp 🙂#2021-01-2914:58mpenetoh it's supports aot even#2021-01-2916:29borkdudefor deps.edn depstar is my thing now too#2021-01-2918:30seancorfieldAnd there's a #depstar channel for questions/problems. I'm working on a new release today (that will remove the pom.xml requirement for folks who want to ignore/omit that, amongst other things).#2021-01-2919:34KevinKI know theres a way to add dependencies with add-lib3 branch of tools.deps.alpha. but is there a way to change a dependency in a running repl? e.g. change a dependency from being a Maven provider to a local provider and then reload the namespacees so you can use the local code instead? might not even be possible on the jvm but I don't know for sure. Thanks!#2021-01-2919:37seancorfield@kevin.krausse Assuming it is Clojure source code, you could probably do it manually by editing up the various source files in that local repo and evaluating them into your REPL (but you can't do it via dependency management AFAIK).#2021-01-2919:37Alex Miller (Clojure team)not possible#2021-01-2919:37borkdudeSounds like a hack to me that's more trouble than its worth :)#2021-01-2919:39borkdudeWhat Seancorfield describes is something I've been doing regularly pre deps.edn, when e.g. debugging libs from clojars. Just copy paste the file from the jar and REPL on.#2021-01-2919:41KevinKawesome, thanks a lot!#2021-01-2921:53JasonIs there a way to include an external deps file from, say, a git submodule in a project's deps file? I'm trying to give other devs access to clojure-deps-edn without either
1. making them modify their dot clojure since this is project-level from their perspective
2. hand-editing the project's deps
I can add 'clojure-deps-edn' as a git submodule but how best to wire it at the project level?#2021-01-2921:57borkdude@jasonhlogic if you use the git submodule as a local/root dep, the top-level deps.edn will also pull in the deps of the submodule deps, right?#2021-01-2921:58Alex Miller (Clojure team)I think we have a pending jira about submodules#2021-01-2921:58Alex Miller (Clojure team)in that they don't work#2021-01-2921:59Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-126#2021-01-2921:59borkdudedeps.edn {local/root: submodule}
submodule/deps.edn {cheshire, transit, paths: [src]}
now deps.edn will also include cheshire, transit and the submodule src path?#2021-01-2922:02Jason@borkdude I have an additional complicating factor: This is a fullstack app so the top-level has no project. There are several subprojects under /projects with their own deps files. I'm certainly willing to edit each deps file, but if I can only depend on filesystem hierarchy to get to submodules then i'm still out of luck#2021-01-2922:05borkdude@jasonhlogic Your goal is to include a set of libraries in each of the submodules? Or something else?#2021-01-2922:07Alex Miller (Clojure team)is this polylith?#2021-01-2922:08Alex Miller (Clojure team)if so, I think you should file an issue with them as I've been having similar discussions about this structure#2021-01-2922:26seancorfieldAh, that might explain why @tengstrand (I think) reached out to me via DM with questions about our monorepo setup with deps...#2021-01-2922:10Jason@borkdude My goal is to provide access to practicalli/clojure-deps-edn , which while technically including other libraries, is more like "additional tooling" than "additional libraries". I was hoping for something like an :include in deps files#2021-01-2922:11borkdudeah#2021-01-2922:13borkdudeThe way I would solve this right now is very bluntly, like this:
- make one template edn, like https://github.com/babashka/pod-babashka-aws/blob/main/deps.template.edn
- write a script: https://github.com/babashka/pod-babashka-aws/blob/main/script/update-deps.clj
- update deps.edn with tooling using script.
There is no support in tools-deps for this right now afaik.#2021-01-2922:13Jason@borkdude Thanks, will do#2021-01-2922:26seancorfieldAh, that might explain why @tengstrand (I think) reached out to me via DM with questions about our monorepo setup with deps...#2021-01-2922:29seancorfield@jasonhlogic For a while, we were using the CLJ_CONFIG "hack" to select a "user" deps.edn file for combination with a subproject's deps.edn file in our monorepo. But that's problematic because then developers can't have their own tooling setup. So eventually we caved and went to a deps-generation approach. Each subproject has <subproject>-deps.edn and we have a small Clojure script that reads that and the "common" deps template and merges them and spits out deps.edn. We keep .md5 checksums of all the template files and we automatically regenerate them in our build script (which wraps clojure) if either the common or subproject template has changed.#2021-01-3003:40tengstrandYes, it was me that reached out to you @seancorfield, and here is the issue (https://github.com/polyfy/polylith/issues/66) including a discussion with Alex about the #polylith workspace structure.#2021-01-3005:31seancorfieldYeah, every time I've looked at Polylith it's just looked like overkill formalism -- and I think the rigidity you've adopted is part of what's causing you problems with deps.edn and tooling. I agree with Alex's comment that you're aiming at an "easy" solution instead of "simple" as well and that's also causing you headaches.#2021-01-3006:10seancorfieldBTW @tengstrand I notice that your #polylith channel here does not have either @U055W814A or @UFNE73UF4 as members so none of that conversation is being logged anywhere, and therefore isn't searchable. You might want to add both of those bots to improve the discoverability of your discussions.#2021-02-0121:35tengstrandWe agree that the #polylith way of structuring code is opinionated and we respect that you (@seancorfield) think it’s overkill formalism, but we don’t agree. Polylith may look alien to people, in the same way Clojure looks like a bunch of parenthesis, added for no apparent reason. But the parenthesis are there to solve fundamental problems, and the same is true for Polylith. It is no coincidence that a component lives in its own src directory, that it’s just “plain code”, that it has an interface, that it’s only allowed to depend on interfaces (and libraries) and that it lives in a monorepo. All those things are there to decouple our system(s) into small Lego-like bricks in a way that they can be shared and put together in a meaningful way. Components are very similar to functions in that way, but operate on a higher level. A function is composable, easy to reason about, has a well defined interface and is fundamentally simple. A component is also composable, easy to reason about, has a well defined interface and is fundamentally simple.
Teasing things apart so that they can be composed together is often a sign of good design. It has been as hard to convince people that Polylith is a good idea as it has been to convince them that Clojure is. Our experience though, is when they start using either of them, they will soon discover how simple they are, how fun they are to work with and how productive you get, and that’s why we love Clojure and Polylith.#2021-01-2922:30seancorfieldSince we were already generating an "everything" deps.edn -- so we could easily start a REPL with all of our monorepo's code and all the dependencies for our dev work.#2021-01-2922:30seancorfieldBasically, it's like Polylith without the rigid formalized module interfaces.#2021-01-2922:31borkdudeThe CLJ_CONFIG hack doesn't work for his case, since he wants to include basically aliases from another deps.edn file#2021-01-2922:31seancorfieldCLJ_CONFIG=path/to/that/deps clojure -M:aliases:from:that:deps:and:other:aliases works though.#2021-01-2922:31borkdudeyeah, it kind of does, I guess#2021-01-2922:32seancorfieldBut it excludes your own user deps.edn.#2021-01-2922:32seancorfieldIt's how we worked for a year or so.#2021-01-2922:32borkdudeI was thinking of the nil version in combination with CLJ_CONFIG, which is what we are using it for#2021-01-2922:32seancorfieldWe use {} for the version and :override-deps in the master deps file.#2021-01-2922:32borkdudebut I'm thinking of using a scripting + template approach instead, feels more robust#2021-01-2922:33borkdudeat least the deps.edn you see is the deps.edn you get#2021-01-2922:33seancorfieldWe had a :defaults alias in the common deps file. Well, we still do, but that's in the common deps template now 🙂#2021-01-2922:35seancorfieldYup, local deps getting updated still breaks stuff though -- but that's a known issue with the CLI/t.d.a already.#2021-01-2922:35seancorfieldHaving the ability to add an extra deps.edn file into clojure's invocation, between user and project, would solve this.#2021-01-2922:36seancorfield@alexmiller I gather folks are still asking for that "fourth file" in various guises?#2021-01-2922:56Alex Miller (Clojure team)I haven’t forgotten it :)#2021-01-2922:49Jason@seancorfield I can make that work. Thanks for the suggestion. Happily, we already use https://github.com/casey/just (non-clojure but useful) to package commands for devs and devops, so I can use that to add the environment hack you suggest.#2021-02-0103:06andy.fingerhutTo use add-lib3 branch as Sean Corfield said he is using in his recent REPL-Driven Development talk to London Clojurians group, is it necessary to run any git clone, checkout add-lib3 branch, build, etc. steps to use that? Has Sean or someone written up instructions on any installation steps required to use that branch?#2021-02-0103:25seancorfield@andy.fingerhut It's in my dot-clojure repo (as I said in that talk 🙂 )#2021-02-0103:26seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L192-L215#2021-02-0103:28seancorfieldThe only caveat with a Socket REPL is you need to ensure a DCL between starting it up. Hence the weird -e stuff in my dot-clojure file -- and also in the dev.clj file in that repo.#2021-02-0103:28seancorfieldI believe that'll be fixed in Clojure 1.11 based on what Alex has said.#2021-02-0103:30andy.fingerhutYou did, and thanks for that talk and all the stuff you release! Never having used it, I was assuming that there might be some mvn install step somewhere, but it looks like no? It is overriding the default version of tools.deps.alpha with a particular git sha, likely the latest commit on the add-lib3 branch, and that is enough. Cool. Will give it a go.#2021-02-0103:31seancorfieldYeah, just a git dep, since it's on a branch.#2021-02-0103:31andy.fingerhutI sometimes guess that things will be extra work to use than they actually are, and am pleasantly surprised otherwise.#2021-02-0103:37seancorfieldI've gotten very used to my setup now -- works on both macOS and Windows (WSL2/VcXsrv) -- Socket REPL, Reveal, add-libs, VS Code. It's a really nice Clojure dev env.#2021-02-0103:41seancorfieldI often have multiple Socket/Reveal setups active -- SOCKET_REPL_PORT=... clojure -M:rebel:reveal:test:add-libs:dev/repl -- and just connect to whichever one I'm currently working on.#2021-02-0103:52Alex Miller (Clojure team)one caveat is that using it as a git dep means you don't get the compiled Java class that provides access to s3 maven repos so that one feature will not work with add-lib3 branch#2021-02-0104:01seancorfieldGood point! Thanks for the reminder, yes! I should probably add that to the comment in my deps.edn file.#2021-02-0104:05seancorfield(added)#2021-02-0112:26mpenet@seancorfield Didn't you write a blog post about how you work with deps on a monorepo?#2021-02-0112:26mpenetI can't seem to find it, if I didn't just dream it#2021-02-0112:36borkdude@mpenet since Sean is asleep probably: if I recall correctly he first used the CLJ_CONFIG hack to merge deps.edn from a common dir, he then stepped away from this and now use deps.edn generation using some scripting.
https://clojurians.slack.com/archives/C6QH853H8/p1611959382106000
I can't recall that he ever blogged about this.#2021-02-0112:37mpenetseems like a sound solution#2021-02-0112:37mpenetthanks for the pointer#2021-02-0112:38mpenetthat could be a deps plugin btw 🙂#2021-02-0112:39borkdude@mpenet FWIW babashka contains a function to merge deps files:
$ bb -e "(babashka.deps/merge-deps ['{:aliases {foo {}}} '{:aliases {bar {}}}])"
{:aliases {bar {}, foo {}}}#2021-02-0112:40mpenetI'd just need the part that factors out the deps coordinates/versions for all "sub-modules" really#2021-02-0112:40mpenetso it's an easy little clj "script" that can be used via an alias potentially#2021-02-0112:43borkdude@mpenet There is also this undocumented function:
$ bb -e "(babashka.deps/merge-defaults '{:deps {foo/bar nil}} '{foo/bar {:mvn/version \"1.10.2\"}})"
{:deps {foo/bar {:mvn/version "1.10.2"}}}#2021-02-0112:43borkdudeI wasn't sure about the API and name yet, this is why it's undocumented#2021-02-0112:49borkdudeThe code for this is public though, if you want to steal it#2021-02-0112:58mpenetI'd prefer to only rely on clj, but I ll have a look, thx#2021-02-0112:59borkdudeyes, this is just messing with maps, nothing bb specific#2021-02-0113:00borkdude@mpenet https://github.com/babashka/babashka/blob/23e12a11ee5c42b3c0ab3ffb39777409c9451dbe/src/babashka/impl/deps.clj#L10-L48#2021-02-0117:50seancorfield@mpenet Yeah, I should write all that up as a blog post some time... I'm not very happy with either "solution": CLJ_CONFIG was convenient and worked well except devs could not have their own tooling; generating deps.edn from a pair of template EDN files works but has edge cases around transitive deps changes (t.d.a already has issues with :local/root deps in that respect -- and trying to keep the generated deps.edn files up to date automatically makes that a bit worse).#2021-02-0308:27mpenetwouldn't it be good to have more deps.edn sources to load from ? (to recap now it's root/user/project/config(cli))#2021-02-0308:27mpenetI am still thinking about the monorepo problem#2021-02-0308:28mpenetit would be the missing piece to avoid having either to hijack the user level conf or do deps.edn generation, or pass something to the cli as string I think#2021-02-0308:29mpenetrelated: https://clojure.org/reference/deps_and_cli#_deps_edn_sources#2021-02-0308:29mpenetI suspect i am not the first one asking that question#2021-02-0308:31mpenetI guess https://clojure.atlassian.net/jira/software/c/projects/TDEPS/issues/TDEPS-174 would be that (kinda)#2021-02-0308:33mpenetbasically an option (or env var) to specify a comma separated list of deps.edn files at project level could be it. But maybe there are better ideas out there#2021-02-0308:33mpenet./cc @alexmiller#2021-02-0309:07borkdudeaka "the fourth file" https://clojurians.slack.com/archives/C6QH853H8/p1611959766114500#2021-02-0309:17mpenet🙂 indeed#2021-02-0309:25mpenetI would make it "N files" so that people don't ask for the "fifth file" right after it's patched 🙂#2021-02-0310:02borkdudeyeah. in clj-kondo I support something like :config-paths ["foo" ...] so it will merge in foo/clj-kondo.edn and hooks, which in turn can include :config-paths relative to itself. You can set up an entire chain of configs. Basically a classpath of configs.#2021-02-0310:04mpenetyeah that's a good way to solve the issue for good#2021-02-0313:24Alex Miller (Clojure team)I’m not yet convinced that this is the answer. I also think it’s important to separate the problems of shared deps mgmt from shared tooling config#2021-02-0313:25Alex Miller (Clojure team)I do not think N configs is a good answer#2021-02-0313:26Alex Miller (Clojure team)And I think it’s pretty crummy to need to pass stuff explicitly#2021-02-0313:29mpenetdo you already have ideas for potential alternatives?#2021-02-0313:29mpenetI mean, solutions to that problem#2021-02-0313:29Alex Miller (Clojure team)No#2021-02-0313:32Alex Miller (Clojure team)I haven’t really sat down to think about it yet#2021-02-0317:02mpenetMaybe something like juxt/aero #include tag could help compose a deps file from multiple sources otherwise. But that's static (no merging), even tho another tag could do that.#2021-02-0317:03mpenetBut that feels a bit odd now that I wrote it down#2021-02-0314:37arohnerIs there an “official” way to run clj with a java that isn’t on the path?#2021-02-0314:38delaguardoJAVA_HOME="path to java home" clj …#2021-02-0314:39delaguardoactually it should be
JAVA_CMD="path to java executable" clj …#2021-02-0315:00Alex Miller (Clojure team)no, the latter won't work#2021-02-0315:03Alex Miller (Clojure team)but JAVA_HOME will#2021-02-0315:06delaguardobut only if java executable is not present in PATH, isn’t it?#2021-02-0315:06Alex Miller (Clojure team)yes#2021-02-0315:07Alex Miller (Clojure team)but that was the precondition in the original question#2021-02-0315:07delaguardoright, sorry, misread the original question#2021-02-0316:46mbjarland@alexmiller (or anybody else for that matter) a follow up question on my maven resolution question. It seems to me after reading up on this that maven poms can define extra repositories under the repositories element and that, if specified, the maven behavior is to use those repositories to resolve the dependencies for that pom instead of the one used to resolve the pom. Two questions really: a) I failed to find a definitive answer. This seems to be the maven behavior, but it does not seem to be well documented. Is there even a spec for this or is the implementation of maven the spec? b) does deps alpha use this mechanic of changing the resolution repository?#2021-02-0317:02mbjarlandAs a side note, it's interesting how maven discussions about resolution strategy (such as this one: -P, my understanding was it would essentially merge all aliases and :deps :extra-deps and resolve them all, without having to do stuff like: clojure -R:all:the:aliases -e "(println :resolved)"#2021-02-0517:18rickmoynihanIf I run clojure -P it appears to do nothing#2021-02-0517:24Alex Miller (Clojure team)-P will "prepare" by downloading all necessary deps and caching classpath etc, then not execute#2021-02-0517:24Alex Miller (Clojure team)so it's intended to be used in combination with other exec options (-A / -M / -X)#2021-02-0517:25rickmoynihanah ok#2021-02-0517:29rickmoynihanwhen I run clojure -P -A:all:my:aliases it doesn’t appear to resolve the deps (after previously having removed them from my .m2)#2021-02-0517:45seancorfield@rickmoynihan What version of the CLI do you have? clojure -Sdescribe#2021-02-0517:49rickmoynihanit’s ok it does seem to work — I made a mistake with the dep I chose to delete#2021-02-0517:52seancorfieldAh, OK.#2021-02-0714:44pyrIf you like clj but miss the group dynamics that leiningen provided (with known names bound to actions), I whipped up a small joker script that provides support for common cases: https://gist.github.com/pyr/d5e17af9c572b681a57de52895437298. It allows doing klein {deps,run,repl,jar,uberjar,test} and supports inferring artifact versions from the command line. I'll see about turning it into a real repo, if there's any interest#2021-02-0715:28borkdude@U06V097TP This is awesome!
I made a babashka fork here. The changes are fairly minimal.
https://gist.github.com/borkdude/c34e8e44eb5b4a6ca735bf8a86ff64fa#2021-02-0717:36pyrreally nice. I will play with the two and compare output sizes/times thanks!#2021-02-0714:46pyrIt's missing a number of things (`new`, compile, javac, and support for submodules which we use extensively).#2021-02-0715:38tengstrandWould it be possible to support giving the name of the config file that should be used when executing a command (if it's something else than deps.edn) so that e.g. the 'poly' project config file could also live at the root along with 'deps.edn', and e.g. be named 'poly-deps.edn'. That would solve our problems we have in #polylith because then we could put all our project files at the root and refer to the source directories without using relative paths (all paths would then look like e.g. "components/mycomp/src"). I have started the work to support the new project structure where each component has its own deps.edn. I would like the different project deps.edn files to handle the components consistently, so that we either specify :paths and :extra-paths or :deps and :extra-deps for all projects. Mixing the two ways doesn't make sense, but if we could put all the project files at the workspace root, then we could more or less keep it as it is without referring to paths outside the projects.#2021-02-0715:42borkdudeIt's not the official clojure deps runner, but my deps.clj project supports this with -Sdeps-file:
https://github.com/borkdude/deps.clj#2021-02-0716:16tengstrandOkay, interesting. But it needs to work for Datomic cloud also. I just talked with @U2BDZ9JG3 and he said that we can't use :deps or :extra-deps for our single development environment, the deps.edn at the root "The reason we can’t use local/root approach in the root dep is that the REPL doesn’t work. You’ll loose the single dev environment and you can’t run tests from the REPL in that case.". So if we can't have all our project files at the root, we need to have a mix where we specify the paths for the development project and dependencies for all other projects. With the support that you have added, we could keep it consistent and simpler and it would make much more sense also, because there are no difference between the projects except that they contain a different set of building blocks. Maybe we could add an issue together.#2021-02-0716:28borkdudeThere has been some discussion about a fourth deps.edn file (system, user, project, fourth) or n deps.edn files. cc @U04V70XH6 and @U064X3EF3#2021-02-0716:47Alex Miller (Clojure team)I do not currently expect that is where we will end up#2021-02-0717:05tengstrandOkay, good to know.#2021-02-0718:29seancorfield@U064X3EF3 is there any guidance you can offer at this point on what is likely to be the direction?#2021-02-0718:45Alex Miller (Clojure team)No, not trying to be coy. We are having ongoing discussions about combinations of features, problems, future directions, etc. It’s just hard to discuss without hours of context.#2021-02-0718:50seancorfieldI look forward to several posts on Inside Clojure summarizing the possible solutions you discarded and why they wouldn't work 🙂#2021-02-0719:05Alex Miller (Clojure team)unlikely to ever get that. It’s just too hard to recreate multidimensional design discussions between a bunch of people over weeks/months. #2021-02-0719:07seancorfield🙂 Then I'll just look forward to whatever solution y'all eventually decide on and adjust our monorepo accordingly...#2021-02-0719:08borkdudesince tools.deps is a tool/lib, I can imagine some scripting on top of this can make any solution you want already. it's just not the official blessed bash script.#2021-02-0719:09borkdudebut deps.edn generation is always a way out as well#2021-02-0719:11seancorfieldYeah, that is also a direction I've considered -- but now that we are not using CLJ_CONFIG and we generate our "compound" deps.edn file, it's opened up the possibility to use clojure directly instead of having to use our build shell script... so I'm not thrilled about the idea of forcing developers back to a custom script 😐#2021-02-0719:12seancorfieldEven with CLJ_CONFIG, we could use clojure directly: it was just rather verbose.#2021-02-0719:13borkdudeI now have custom clojure bash scripts in our mono-repo projects. These are also used by tools like clojure-lsp to calc the classpath#2021-02-0719:14borkdudeso instead of clojure -Sfoo you type ./clojure -Sfoo. Not custom as in a fork of the original, just something like CLJ_CONFIG=../base clojure -A:base. I guess what you had before#2021-02-0719:15seancorfieldYeah, we vendor clojure into our repo so we have a reliable, self-contained version, using a specific install path. But it's still "almost" identical to the official version. I'm trying not to fork it and diverge.#2021-02-0719:18seancorfield(! 856)-> diff clojure/bin/clj `which clj`
2c2
< vendor_path=/var/www/worldsingles/build/clojure
---
>
4c4
< exec rlwrap -r -q '\"' -b "(){}[],^%#@\";:'" ${vendor_path}/bin/clojure "
That's the sum total of our changes (compared to the brew install'd version).#2021-02-0801:33tengstrandOne more question to @U064X3EF3. Are you going to keep the support for absolute paths in :paths and :extra-paths? That could be a way out for us, to generate the deps.edn files for our projects, and instead of having relative paths, e.g. "../../components/command/src" as today, we could generate the paths, e.g. "/Users/tengstrand/source/polylith/components/command/src". We have tested this and it solves the issue we have for Datomic Cloud.#2021-02-0802:18seancorfield@U1G0HH87L And those generated files would be excluded from version control I assume? (we've had to do a similar thing)#2021-02-0802:19seancorfieldIt does make things a bit of a pain that you can't just check code out and run it -- the deps.edn files have to be generated first, so just using clojure/`clj` isn't enough.#2021-02-0802:21Alex Miller (Clojure team)@U1G0HH87L we are planning to disallow paths and extra-paths outside the project root. I expect to add a warning for this soon and eventually it will be an error#2021-02-0802:28Alex Miller (Clojure team)I think you’re going to have different but related problems with ions - this ever having worked with ions is more accident than intent. Ions rely on using shas as a way to differentiate versions. I don’t work on that stuff and defer to the Datomic team on the details.#2021-02-0802:31seancorfieldYeah, if something is outside the project, it should be a :local/root dep not a :paths thing.#2021-02-0803:09tengstrandOkay, then I know.#2021-02-1014:50jjttjjIs there a way to have an alias that uses the :extra-deps from one or more other aliases?#2021-02-1014:53jjttjjOr is the an obvious better way to organize this:
{:deps {
org.lwjgl/lwjgl {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl {:mvn/version "3.2.3"}
}
:mvn/repos {"space-maven"
{:url ""}}
:aliases
{
;; copy of windows
:dev
{:extra-deps
{org.jetbrains.skija/skija-windows {:mvn/version "0.89.8"}
org.lwjgl/lwjgl$natives-windows {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw$natives-windows {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl$natives-windows {:mvn/version "3.2.3"}}}
:windows
{:extra-deps
{org.jetbrains.skija/skija-windows {:mvn/version "0.89.8"}
org.lwjgl/lwjgl$natives-windows {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw$natives-windows {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl$natives-windows {:mvn/version "3.2.3"}}}
:macos
{:extra-deps
{org.jetbrains.skija/skija-macos-x64 {:mvn/version "0.89.8"}
org.lwjgl/lwjgl$natives-macos {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw$natives-macos {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl$natives-macos {:mvn/version "3.2.3"}}}
:macos-m1
{:extra-deps
{org.jetbrains.skija/skija-macos-arm64 {:mvn/version "0.89.8"}
org.lwjgl/lwjgl$natives-macos {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw$natives-macos {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl$natives-macos {:mvn/version "3.2.3"}}}
:linux
{:extar-deps
{org.jetbrains.skija/skija-linux {:mvn/version "0.89.8"}
org.lwjgl/lwjgl$natives-linux {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-glfw$natives-linux {:mvn/version "3.2.3"}
org.lwjgl/lwjgl-opengl$natives-linux {:mvn/version "3.2.3"}}}}}#2021-02-1015:04Alex Miller (Clojure team)in short, no. you can of course supply multiple aliases at the CLI, so the common stuff could be in its own alias#2021-02-1015:04Alex Miller (Clojure team)(also note you have a typo in extar-deps near the end)#2021-02-1015:09jjttjjgood catch thanks!#2021-02-1015:07mpenetmakes me wonder: was it ever considered to add tagged litterals like in aero for referencing values (#ref) or even including deps.edn fragments from other files?#2021-02-1015:07mpenetnot sure it's a good idea, it just came to mind, merely curious#2021-02-1015:11Alex Miller (Clojure team)I would not rule out ever having an answer to this question#2021-02-1015:12Alex Miller (Clojure team)(as an aside #ref is a bad tagged literal as it's not qualified and those are reserved for Clojure)#2021-02-1015:12Alex Miller (Clojure team)(and "#ref" is particularly likely to actually potentially be a thing Clojure might want to use)#2021-02-1015:12mpenetyou should go into politics 🙂#2021-02-1015:12mpenetagree about the naming#2021-02-1015:13rickmoynihanIndeed see this issue: https://github.com/juxt/aero/issues/44#2021-02-1015:13mpenet#include + #merge could maybe even solve the monorepo thing#2021-02-1015:13mpenetor some derivative/form of #merge#2021-02-1015:13Alex Miller (Clojure team)it's definitely a possible solution#2021-02-1015:17borkdudeI remember a conversation where this kind of feature was objected to since the EDN would need an external tool to be processed which makes it harder for tooling to reason about it.#2021-02-1015:19rickmoynihanyou can always use a :default reader — you’ll probably need tools.deps the library to reason about the file anyway no?#2021-02-1015:19mpeneti imagine tools.deps would give the means to have the final deps.edn form#2021-02-1015:19slipsetJust saw this https://www.bleepingcomputer.com/news/security/researcher-hacks-over-35-tech-firms-in-novel-supply-chain-attack/ how does tools.deps work in this scenario?#2021-02-1015:20nnicholsGit coordinates would be safe, as well as Maven central dependencies (Since maven requires DNS validation to register a group). However, clojars does not have this validation#2021-02-1015:22nnicholsYou’d need to know the effective GroupID, ArtifactID, and version of a package to execute it, and it’d depend on resolution order. IMO, you ought to register your org’s group on clojars just to be safe#2021-02-1015:22slipsetRight. I guess you could build this into tooling as well, right, either by specify a repo hierarchy, or specifying which repo you’d want to fetch stuff from?#2021-02-1015:23nnicholsYeah, if I have an artifact I know that should only ever be pulled from S3 or my .m2 for example, I’d need to specify that after the :mvn/version or, for certain dependencies just tag them with the :mvn/repos key?#2021-02-1015:23slipsetSomething like that#2021-02-1015:24nnicholsActually, debugging access to remote deps would be a lot easier then too. I always have to stop and think whenever tools.deps reports it can’t find a private dependency on central#2021-02-1015:29Alex Miller (Clojure team)tda/clj only uses the :mvn/repos at the top level, not in transitive deps.edn files, so you are in full control of which repos you look at#2021-02-1015:29Alex Miller (Clojure team)also, we guarantee that the first repos searched are central, then clojars#2021-02-1015:30Alex Miller (Clojure team)btw, maven central does not require dns validation, although there is a process with a human in the loop for verification of groupIds#2021-02-1015:32slipsetWhat happens if I publish clojure to clojars under version 1.12 and someone depends on latest Clojure#2021-02-1015:32Alex Miller (Clojure team)well first you shouldn't depend on "latest" clojure#2021-02-1015:33Alex Miller (Clojure team)but would depend on what clojars allows there#2021-02-1015:33Alex Miller (Clojure team)so good question for them#2021-02-1015:34Alex Miller (Clojure team)seems like they do have some old releases there https://clojars.org/org.clojure/clojure#2021-02-1015:36Alex Miller (Clojure team)maybe worth an issue for them, should really not allow uploads to org.clojure group there#2021-02-1112:27imre> we guarantee that the first repos searched are central, then clojars
Is this a desirable behavior when I want to pull in internal libs from an internal repo?
Also, would it ever be an option to lock a specific dep down to be retrieved from a specific set of repositories?
Say, I don't even want to allow clj to try central/clojars for my internal lib dep#2021-02-1113:13Alex Miller (Clojure team)We’re going through the Maven apis for that stuff and that’s just not how it works#2021-02-1113:14Alex Miller (Clojure team)So would be a large and unknown amount of work#2021-02-1113:15Alex Miller (Clojure team)The way most people handle this stuff at enterprise level is to use something like Nexus or Artifactory as a proxy in front of the internet#2021-02-1113:16Alex Miller (Clojure team)tools.deps supports that via the mirror/proxy settings in settings.xml#2021-02-1015:28borkdudeI'm reading back the logs from this channel on the zulip archive. I'm at Feb 2019 now (2 years ago) where Sean Corfield talks about his monorepo problem. LOL I didn't realize we've been talking about that for so long already (https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/tools-deps/near/157547436).#2021-02-1015:31borkdudeI now found the relevant part that I was looking for on 6th September 2020:
borkdude: @alexmiller I know tools.deps isn't going to expand ~, but I'd like to know the reason for this. Reason I'm asking is that I'm considering doing this in .clj-kondo/config.edn and would like to know if I'm missing some problems
Slack: alexmiller: Because then everyone reading deps,edn has to interpret the file
Slack: alexmiller: It’s no longer just edn/read, it’s that plus run magic code
Slack: borkdude: Why does it have to be interpreted by anything else than tools.deps which already does interpretation of this entire model?
Slack: borkdude: I mean, merge, assoc, etc still works for other tools in this case?
Slack: alexmiller: If you have alias data, anyone may be reading it via the basis (without tools.deps, which has a lot of deps)
Slack: alexmiller: There are a bunch of tools already reading and manipulating deps.edn. The less they have to know, the better
(https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/tools-deps/near/209221777)
I think the contextual beginning question is missing maybe due to some logging error.#2021-02-1015:36borkdudeoh yes, I think I remember what this was about: interpretation of env vars like HOME probably#2021-02-1015:56borkdudeFor completeness, I now also found the other message via the other archive at clojurians logs:
borkdude: @alexmiller I know tools.deps isn't going to expand ~, but I'd like to know the reason for this. Reason I'm asking is that I'm considering doing this in .clj-kondo/config.edn and would like to know if I'm missing some problems
#2021-02-1015:57borkdude(I haven't implemented this in clj-kondo either, solved it a different way)#2021-02-1015:58borkdudeSo the point was: the EDN should be readable by other tools without magic code / deps. I wonder if the opinion shifted on that (everyone is allowed to change their mind).#2021-02-1016:03Alex Miller (Clojure team)it was never an absolute statement. my general preference is - the less you need to know/do to read, the better#2021-02-1016:03Alex Miller (Clojure team)as always, it's a matter of tradeoffs#2021-02-1103:51Oliver GeorgeI'm struggling with a CI build. I have a public git repo dependency which seems to get converted from https -> ssh and ends up with an auth fail error.
Cloning:
Error building classpath. : Auth fail
org.eclipse.jgit.api.errors.TransportException: : Auth fail
https://gist.github.com/olivergeorge/f3b8b08721f736da74d9a21f097cd3e9#2021-02-1103:52Oliver GeorgeHoping for "Oh yeah, jgit might use ssh: for a https: repo if you have XXX set".#2021-02-1103:53Oliver George(this is a new repo but copies from an existing working one... we've upset something and I can't think how)#2021-02-1104:34seancorfieldCheck your local git config -- I bet it is swapping https for git? This is on CirceCI right @olivergeorge?#2021-02-1104:35seancorfieldFor some reason, they started using a default git config that rewrites the URLs -- the solution is to remove the config as part of your setup: https://github.com/seancorfield/honeysql/blob/v2/.circleci/config.yml#L11-L13#2021-02-1821:44Kira McLeanah this is brilliant! I worked around the issue by re-writing the ssh key to be one that jgit could handle.. wonder if this would be a simpler fix!#2021-02-1104:35seancorfieldLots of Clojure devs have been complaining about to the #circleci folks 😐#2021-02-1105:00Oliver GeorgeYep, on circleci#2021-02-1105:00Oliver GeorgeThanks I'll check that out#2021-02-1210:09vemvhow to run a random .clj file's -main that has no 3rd party dependencies or a deps.edn file?
I tried clojure -M some_folder/the_file.clj, it loads the file but it doesn't run its -main#2021-02-1210:19practicalli-johnEdit: sorry I read the question incorrectly. If you do add a deps.edn, then you can do this..
The -M flag specifies that the clojure command should call Clojure main in the project (as opposed to the -X flag that calls a qualified function)
-m is required to state which namespace to run -main, the namespace that includes (defn -main ,,,)
So from your example it would be
clojure -M -m some-folder.the-file#2021-02-1210:22vemvlooks like in that case I'd have to add some-folder to the classpath. How to do it in an inline fashion?#2021-02-1210:25practicalli-johnI was assuming that some_folder is under a src folder and you have a {:path ["src"]} in the deps.edn file
Edit: can the -i flag be used for this, it loads a resource?#2021-02-1210:26practicalli-johnI believe you can use . Instead of src if there is no directory structure, but t that still requires a deps.edn file.#2021-02-1210:44borkdude@U45T93RA6 this is a similar problem in babashka world. we invented this pattern for it:
https://book.babashka.org/#main_file
inspired by the Python way of doing things#2021-02-1210:46borkdudeSo you can check using that condition if you are directly invoking this file or if you are referring to that file using the classpath#2021-02-1210:47borkdudeBut for clj this doesn't work. Some people also solve this using an env var in babashka: REPL=true -> don't invoke main at the bottom#2021-02-1212:22vemvclojure -M -i ./some-folder/the_file.clj -m the-file worked.
Bit of an intrincate construction though :)
Thank you both!#2021-02-1216:56practicalli-johnThanks for sharing the solution that worked for you.#2021-02-1217:07borkdudeWhat's the point of having a -main btw if you are invoking this as a standalone file?#2021-02-1217:34vemvwe are all a curious bunch :)
This was a experimental CI-oriented script belonging to a big repo. Didn't seem to deserve its own deps.edn file.
Indeed I could have unwrapped the main but there's some remote possibility of the top-level side-effect being executed by tooling (e.g. Eastwood, refactor-nrepl or t.n (refresh))#2021-02-1214:51mbjarlandis there any way of using tools.deps code to retrieve the available list of versions for an artifact? This normally comes from the maven-metadata.xml file#2021-02-2300:17agilecreativityI build something like this once back in 2017.
It allow you to fetch the version from maven
https://github.com/agilecreativity/pom-versions
@U4VDXB2TU#2021-02-1214:53Alex Miller (Clojure team)nothing in the tools.deps api, no#2021-02-1214:53Alex Miller (Clojure team)you could use the lower parts of the tools.deps api as an access point into maven apis to answer that question though#2021-02-1215:05Alex Miller (Clojure team)you basically want to use the Maven RepositorySystem API to make VersionRequests or VersionRangeRequests etc#2021-02-1215:58mbjarland@alexmiller ok thanks, that's the route I was heading, just wanted to make sure I wasn't reinventing the wheel#2021-02-1410:32borkdude#2021-02-1414:01Alex Miller (Clojure team)wizardry#2021-02-1420:30cobyI'm a little fuzzy on the distinction between clj's -A and -M options. What exactly is a "main option" alias and how does it differ from normal aliases/"aliases of any kind"? (Side note: where did that diagram go with all the arrows labeled with the different CLI options? Could've sworn it was on the deps/cli reference page or the guide)#2021-02-1420:34borkdude@ctamayo -A is for invoking a REPL, -M is for invoking clojure.main (typically with a -main function with :main-args), -X is for executing a function, typically with pre-configured args in deps.edn.#2021-02-1420:40Alex Miller (Clojure team)the diagram was removed as it was mostly no longer correct#2021-02-1420:43cobyso -M:some-alias -m my.project a b c is like calling (my.project/-main a b c) right? And :some-alias presumably just adds extra stuff to the classpath that the my.project ns needs?#2021-02-1420:44borkdudecorrect#2021-02-1420:54cobyThanks! OK so I think something weird is happening. This has happened once or twice before - I forget where - but most recently on the datahike test suite: Commands that are documented with -M fail with a FileNotFoundException, but succeed when I use -A instead. So for example running unit tests exactly as they are invoked by the helper scripts (https://github.com/replikativ/datahike/blob/development/bin/kaocha via https://github.com/replikativ/datahike/blob/development/bin/run-unittests) gives me:
$ TIMBRE_LEVEL=':fatal' clojure -M:test -m kaocha.runner --focus :clj
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate kaocha/runner__init.class, kaocha/runner.clj or kaocha/runner.cljc on classpath.
Full report at:
/tmp/clojure-16371792498407115921.edn
But switching to -A:test works.#2021-02-1420:55seancorfield@ctamayo That indicates the version of clojure is too old.#2021-02-1420:56seancorfieldclojure -Sdescribe will tell you what version it is. You need at least 1.10.1.697 per https://clojure.org/releases/tools#2021-02-1420:57cobyah, I was looking for a --version option or something. Good to know, thank you!#2021-02-1420:57seancorfieldIn older versions of clojure the -M option did not respect :extra-deps so part of the alias would not be pulled in (the dependencies), just the :main-opts part.#2021-02-1420:58seancorfieldclojure -h displays the version (at the top of the help) as well as clojure -Sdescribe#2021-02-1420:59cobynot sure how I missed that at the top of clojure -h! Guess I need a break. 🧟#2021-02-1421:03seancorfieldIt scrolls off the top of the screen for me so I tend to recommend clojure -Sdescribe because the output is shorter.#2021-02-1421:06cobyI also had an embarrassingly old version (1.10.0.411) that didn't print that at the top. But I will remember that trick! And also take a break still. 😛#2021-02-1421:09seancorfieldAlso worth noting that the latest is 1.10.2.774 now that Clojure 1.10.2 is out (not 1.10.1.x).#2021-02-1421:11cobyyep, that is the one I just installed and it's working now!#2021-02-1423:24zaneIs there a way to find out which version of Jetty the installed Clojure tools are using?#2021-02-1423:37seancorfield@zane Jetty? I wouldn't expect the Clojure tools to be using a web server. Can you provide a bit more context?#2021-02-1423:50seancorfieldIf you mean "in a deps.edn project, where Jetty is a transitive dependency", you can use clojure -Stree#2021-02-1505:25zane@seancorfield I believe the Clojure tools use the HTTP client part of Jetty via com.cognitect/http-client.#2021-02-1505:30zane@seancorfield The context is that we’re trying to ship a Docker image to a partner. They ran a Twistlock security scan on it and it reported that the container contains a version of Jetty that has a vulnerability.#2021-02-1505:31seancorfieldInteresting. OK, so clojure -Stree -A:deps should do it.#2021-02-1505:31zaneIt doesn’t actually tell you where in the container it found the offending copy of Jetty, but we looked around and there appears to be a copy in /usr/local/lib/clojure/libexec/clojure-tools-1.10.2.774.jar.#2021-02-1505:31seancorfieldThat will pull in the version of tools.deps.alpha used by the tools and show the dependency tree.#2021-02-1505:31zaneIt’s not in our application’s dependencies, so clojure -Stree -A:deps shouldn’t show it, right?#2021-02-1505:32zane-Stree includes the tools’ dependencies as well?#2021-02-1505:32seancorfieldorg.eclipse.jetty/jetty-client 9.4.24.v20191120#2021-02-1505:32seancorfield-A:deps brings in t.d.a so it shows up in -Stree#2021-02-1505:32zaneAhhh.#2021-02-1505:32zaneThat’s what I was missing.#2021-02-1505:32zaneOkay, great. That matches what I found with find … | xargs jar --list --file.#2021-02-1505:33seancorfieldAnd, yes, that is a transitive dep of the cognitect http client. TIL!#2021-02-1505:33zaneYeah, it was news to me as well.#2021-02-1505:33zanehttps://nvd.nist.gov/vuln/detail/CVE-2020-27216#2021-02-1505:34zaneSeems like a vulnerability is with the server part of Jetty. Wish me luck trying to convince their security team that the Clojure CLI tools are benign. 😅#2021-02-1505:40seancorfieldFWIW, the latest com.cognitect/http-client is 0.1.105 (t.d.a. uses 0.1.104) and that later version uses org.eclipse.jetty/jetty-client 9.4.27.v20200227#2021-02-1505:42seancorfieldThat's still in the range affected (but, again, is the client not the server).#2021-02-1505:44seancorfield(this made me check what version of Jetty we're using at work -- we tend to update lib deps regularly -- and we're on 9.4.36 which is after that vuln was fixed!)#2021-02-1506:11hiredmanThat vuln is about the way jetty unpacks war files in the temp directory, if you don't use war files, no problem#2021-02-1506:11hiredmanIt effects jetty-runner#2021-02-1511:07borkdudeFYI, we now got rid of the CLJ_CONFIG hack in our mono-repo project by generating the deps.edns with babashka. It takes a map with default versions from another deps.end and merges those into the subproject deps.edn.
https://gist.github.com/borkdude/58f099b2694d206e6eec18daedc5077b#2021-02-1521:35seancorfieldI seem to recall that t.d.a checks Maven Central before it checks Clojars but what does it do about other :mvn/repos provided via deps.edn files?#2021-02-1521:41seancorfieldAnswering my own question: looks like it has code to turn the :mvn/repos hash map into a sequence that is always "central", "clojars", followed by any other repos you specify?#2021-02-1521:43nnicholshttps://clojurians.slack.com/archives/C6QH853H8/p1612970968242200?thread_ts=1612970371.234100&cid=C6QH853H8#2021-02-1521:45seancorfieldThanks. I thought it had been discussed recently and I had read part of that thread before but not all of it.#2021-02-1521:46seancorfieldSo it still seems like someone who knew that Acme Inc was pulling some/thing 1.2.3 from their private repo could upload a fake some/thing 1.2.3 to Clojars and get into Acme's code that way...?#2021-02-1521:48nnicholsThe access tokens are scoped to the groups each artifact falls beneath, which helps; however, if you org has no presence on clojars, there’s no DNS/ownership validation. So, it would be possible for a former employee of Acme to publish that dep + version to clojars#2021-02-1521:48seancorfieldYup, that was what I was thinking...#2021-02-1521:48nnicholsI really, really hope com / org / etc are restricted group names#2021-02-1521:50seancorfieldWell, it's not about that per se: if Acme Inc were using arbitrary group/artifact IDs in their internal private repo, such as department names and project names, then it's not even tied to their reverse domain.#2021-02-1521:50seancorfieldI mean, I know that's probably bad practice on their part, but I suspect that's common practice.#2021-02-1521:50nnicholsYou’re probably right on that one#2021-02-1521:51hiredmanmaven does allow for jar signing, and clojars even really pushes you to do it#2021-02-1521:51seancorfieldhttps://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L126-L131 -- It would seem to be safer if the ordering here was central, then user-supplied repos, then clojars to avoid the supply chain issue?#2021-02-1521:51seancorfieldClojars long ago dropped any coercion to sign JAR files.#2021-02-1521:52hiredmanyeah, no one is checking signatures, and it annoys everyone#2021-02-1521:52seancorfieldIt used to have a two tier system where only signed JARs could be "promoted" to the top tier, but that's all gone.#2021-02-1521:52nnicholsAny reason not to have user-supplied first? (Unless signature warnings are provided by the toolchain, I’m not sure how many devs would notice)#2021-02-1521:53seancorfieldI would say we'd want to discourage shadowing of central?#2021-02-1521:53seancorfieldBut shadowing Clojars with an in-house repo seems acceptable/safer maybe.#2021-02-1521:54seancorfield(I have no dog in this fight: we no longer have an in-house repo/proxy service so pushing something to Clojars cannot intercept a dependency from a custom repo any more for us)#2021-02-1521:54nnicholsCentral had a minor incident is the only reason I question it: https://securityboulevard.com/2021/01/sonatype-stops-software-supply-chain-attack-aimed-at-the-java-developer-community/#2021-02-1521:55hiredmanmoving the repos around doesn't really solve the issue though, if you put used defined repos before clojars, then some third party repo the user adds could be used to shadow some package in clojars#2021-02-1521:57nnicholsYeah, I guess I’m strictly assuming that the only other real mvn/repos addition in this case is something like a private/internal artifactory that’d be sparsely populated#2021-02-1521:57seancorfieldYeah, that was my thinking too.#2021-02-1521:57hiredmanpart of the reason the supply chain attack was so effective on other tooling is node and whatever python's package manager is both allow fetched packages to execute code as part of their build#2021-02-1522:00seancorfieldIn Central's "minor incident" that relied on devs searching Maven and mistaking the fake plugin artifact for the real one -- and stupidly adding a dependency on com.github.codingandcoding/something into their project at work. You can't protect against that sort of thing really.#2021-02-1522:01seancorfield(not that Central's massive range of group IDs isn't confusing at times with an artifact ID appearing under multiple group IDs due to not being unique enough or a project migrating from one org to another)#2021-02-1522:01hiredmanthe problem of using names assigned to the content of files instead of names derived from the contents of files#2021-02-1522:01seancorfieldAye.#2021-02-1521:43seancorfieldIt seems like that would allow malicious actors to upload to Clojars an artifact that shadowed something in your private repo and deliver it without you really noticing? (I was just reading the stuff about npm, PyPI etc where the researcher subverted private artifacts for a bunch of companies)#2021-02-1611:04borkdude#2021-02-1722:21Alex Miller (Clojure team)Clojure CLI tools 1.10.2.786 are now available in a prerelease
* https://clojure.atlassian.net/browse/TDEPS-56 - fix main-opts and jvm-opts splitting on space#2021-02-1722:22Alex Miller (Clojure team)This fixes the long-standing issue with space splitting necessitating the Corfield comma. I tested on both Mac and Windows, but I would really appreciate anyone kicking the tires on this.#2021-02-1722:29Alex Miller (Clojure team)the .jvm and .main cache files are now in a different format (arg per line instead of space-separated args in a single line), but simply installing a new version of the CLI installs a new root deps.edn, which is part of the cache staleness check, so all existing cached files should be considered stale and the recomputed cache file should just happen automatically. I re-tested that too across the install boundary but again, more eyes welcome.#2021-02-1722:29seancorfieldNice! I look forward to being regulated to the annals of history as a Clojure curiosity! :rolling_on_the_floor_laughing:#2021-02-1722:29Alex Miller (Clojure team)you never when the Corfield comma will strike again#2021-02-1722:30Alex Miller (Clojure team)the next conf you go to, you should just wear a shirt with a big comma on it#2021-02-1722:32seancorfieldImma stealin' that idea! 🙂#2021-02-1722:41andy.fingerhutPeople may think you have joined the Tres Comas club (Silicon Valley TV show reference)#2021-02-1722:41andy.fingerhutif you have 3 commas on the shirt#2021-02-1722:31ghadipour one out for a brilliant hack 🍷#2021-02-1722:32seancorfieldI put a Jira issue on our board to switch up to 1.10.2.786 but it probably won't happen until tomorrow. I'll report back.#2021-02-1722:33seancorfield(right now I'm dealing with an edge case I discovered today when we switched from the CLJ_CONFIG hack to the generate-`deps.edn` hack... annoyed I missed this when I switched over a while back!)#2021-02-1722:37Alex Miller (Clojure team)I am also deep into considering a major change in the git impl (thanks @ghadi) which may also "fix" a slew of the outstanding git/ssh issues, hoping to get more time on that on Friday#2021-02-1722:38Alex Miller (Clojure team)and of course, some new stuff :)#2021-02-1722:46borkdudeIs it the shelling out to git branch which ghadi made or something else?#2021-02-1722:56Alex Miller (Clojure team)yeah#2021-02-1722:57Alex Miller (Clojure team)I'd like to release the current set of changes as stable before I push that into the prerelease though#2021-02-1722:58borkdudecool. I used that in tools-deps-native as well, when I had problems compiling jgit, worked great. later on I managed to get it working as is though. just fyi.#2021-02-1723:39Alex Miller (Clojure team)There is a tools.gitlibs branch for it now that incorporates some additional changes. I’m going to do some more work on the error handling still#2021-02-1723:37Alex Miller (Clojure team)Good to know#2021-02-1723:40seancorfieldI'm working with t.d.a. 0.9.871 as a library, trying to resolve a specific set of deps and I'm getting this error:
{:clojure.main/message
"Execution error (ArtifactNotFoundException) at org.eclipse.aether.internal.impl.DefaultArtifactResolver/resolve (DefaultArtifactResolver.java:414).\nCould not find artifact org.apache:apache:pom:13\n",
:clojure.main/triage
{:clojure.error/class
org.eclipse.aether.transfer.ArtifactNotFoundException,
:clojure.error/line 414,
:clojure.error/cause
"Could not find artifact org.apache:apache:pom:13",
:clojure.error/symbol
org.eclipse.aether.internal.impl.DefaultArtifactResolver/resolve,
:clojure.error/source "DefaultArtifactResolver.java",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type org.eclipse.aether.resolution.ArtifactDescriptorException,
:message
"Failed to read artifact descriptor for org.apache.httpcomponents:httpcore:jar:4.4.5",
:at
[org.apache.maven.repository.internal.DefaultArtifactDescriptorReader
loadPom
"DefaultArtifactDescriptorReader.java"
301]}
{:type org.apache.maven.model.resolution.UnresolvableModelException,
:message "Could not find artifact org.apache:apache:pom:13",
:at
[org.apache.maven.repository.internal.DefaultModelResolver
resolveModel
"DefaultModelResolver.java"
176]}
I can depend on org.apache.httpcomponents/httpcore {:mvn/version "4.4.5"} just fine from the CLI and other contexts but I'm at a bit of a loss to know where to start digging for this error...#2021-02-1819:50seancorfieldI eventually figured out my problem this morning: in building the basis using t.d.a I had managed to "lose" the :mvn/repos key when I was combining deps hash maps and so it was working if the local ~/.m2 repo contained the artifact but not if the artifact needed to be pulled from Maven Central facepalm -- so it was behaving differently on different machines, depending on what was already downloaded 😞#2021-02-1723:56seancorfieldI can't resolve org.apache/apache "13" via the CLI tho' so at least that part checks out (I found it on Maven and looks like it might just be a pom but the error above says it can't resolve that either...)#2021-02-1819:50seancorfieldI eventually figured out my problem this morning: in building the basis using t.d.a I had managed to "lose" the :mvn/repos key when I was combining deps hash maps and so it was working if the local ~/.m2 repo contained the artifact but not if the artifact needed to be pulled from Maven Central facepalm -- so it was behaving differently on different machines, depending on what was already downloaded 😞#2021-02-1800:03seancorfieldInterestingly, if I just comment out the dependency on org.apache.httpcomponents/httpcore "4.4.5" everything works and it still comes in as a transitive dependency of something else... Weird.#2021-02-1800:13Alex Miller (Clojure team)it would be useful to know the original set of deps that failed#2021-02-1800:13Alex Miller (Clojure team)that UnresolvableModelException implies to me that we're running into trouble reading that pom file somehow#2021-02-1800:26seancorfieldorg.apache/apache "13" seems to be problematic, but I don't know why that would be requested when the httpcore dep is a top-level dep but not when it is a transitive dep of something else. This was kind of a weird edge case and it's hard to isolate to a repro case because it relies on some context from our monorepo setup. If I get time, I'll try to boil it down to something small but I have a workaround for now so it's not terribly important.#2021-02-1800:26seancorfield(when I tried to boil it down to a simple repro case, I couldn't -- it just started working)#2021-02-1801:58Alex Miller (Clojure team)There is some filtering in the transitive dep stuff, it’s possible it’s getting filtered out#2021-02-1809:56dominicmIf you're splitting on newline, won't that break for newlines instead of spaces? 😄 Are newlines encoded somehow then decoded before going to args?#2021-02-1814:00Alex Miller (Clojure team)well that didn't work before, and still doesn't work, but I think is a much less sensible thing to want to do#2021-02-1817:35seancorfieldWe're running into this warning at work:
WARNING: Specified path is external to project: ../activator/test
WARNING: Specified path is external to project: ../affiliate/test
WARNING: Specified path is external to project: ../api/test
WARNING: Specified path is external to project: ../auth/test
WARNING: Specified path is external to project: ../build/test
...
We have an "everything" subproject that we use for dev work (so we can have a single REPL with all source code and all test code available). Is there a recommended way around the warning @alexmiller?#2021-02-1915:21Alex Miller (Clojure team)the recommended way is to not do that :)#2021-02-1915:22Alex Miller (Clojure team)this is a step towards eventually making this an error#2021-02-1915:28Alex Miller (Clojure team)but I will take a note to think about this particular case of accessing the test src of a project - I've had past convos where we talked about ways to include a dep + activate an alias in the remote project as you interpret it which could be one way to do this with local deps instead (local dep + include a test alias to grab its test paths). but needs a lot more thought.#2021-02-1915:52seancorfieldGiven that several of these pain points seem to relate to mono repos, I vote that you be made to work on a mono repo for a few months 😁#2021-02-1916:02Alex Miller (Clojure team)or everyone else should stop working on mono repos... :)#2021-02-1916:03Alex Miller (Clojure team)I have worked on them in the past. did not like, would not mono repo again.#2021-02-1916:51seancorfieldSo, if you were forced to work in a monorepo and you could set up deps.edn however you wanted, how do you think you might approach it? I'm hoping for some insight that gets us closer to a "standard" CLI/`deps.edn` approach that isn't going to get pulled out from under us...#2021-02-1916:52seancorfieldHaving a single deps.edn in the top-level perhaps? With an alias for each subproject that brings in just that subproject's paths/deps?#2021-02-1917:00Alex Miller (Clojure team)yes, I think at top level#2021-02-1917:08seancorfieldOK, I might try switching us to that setup and see what pain points I run into. My immediate thought is that running a REPL in the "everything" context would then require a lot of aliases (one for each subproject -- about three dozen for us right now) or would require duplicating the content of aliases into the :everything alias.#2021-02-1919:07seancorfieldI don't think that will work with :local/root since each subproject would still need a deps.edn file for its own dependencies -- unless you duplicate everything up into the master deps file... and that's a lot of duplication...#2021-02-1919:14seancorfieldI started down this path to see what it might look like but ran into the :local/root issue pretty much immediately. Happy to screen share with you at some point to try to hammer some of this out.#2021-02-1817:38seancorfieldI guess we could move our "everything" subproject to effectively be the top-level folder above all those other subprojects, so all paths within them would be within the overall "project"? Not sure what other weirdness that would cause though since we rely on :local/root a lot and currently can guarantee that all subprojects can depend on each other by using `:local/root "../<subproject>" URLs...#2021-02-1817:40borkdudeYep, I saw this one too today#2021-02-1817:41borkdude"external to project, but within monorepo, so still okay?"#2021-02-1817:43borkdudeMy use case for this is to compensate for the fact that a generated pom.xml file does not include the source folders, so I'm adding add that manually using :paths#2021-02-1817:43seancorfieldFor us it's about adding test folders "manually" but, yeah.#2021-02-1817:44seancorfieldI understand why the warning is given -- having :paths outside the project should be a red flag in general -- but I'm not sure what the recommended approach should be for dev/test in a large monorepo...#2021-02-1817:47borkdudeLooking deeper into it, we now ported that lib to deps.edn as well, so we don't need the pom.xml anymore :) Just local/root works now (we did local/root pom + paths before)#2021-02-1819:12borkdudeCleaned that up, so problem solved for us#2021-02-1920:57Alex Miller (Clojure team)https://clojure.org/releases/tools#v1.10.2.790 is now available
• Add -version and --version options (same as java)
• https://clojure.atlassian.net/browse/TDEPS-56 - Fix main-opts and jvm-opts word splitting on spaces (goodbye Corfield comma!)
• https://clojure.atlassian.net/browse/TDEPS-125 - Use JAVA_CMD if set (thanks Gregor Middell!)
• Add warning if `:paths` or `:extra-paths` refers to a directory outside the project root (in the future will become an error)#2021-02-2020:12mike_ananevGot some problems in Idea 2020.3 & Cursive (stable or EAP) and latest clojure/tools//cdn-cgi/l/email-protection
Symptoms: can't see any modules in Clojure Deps tab in Idea. Also can't start remote repl.
Downgrading to clojure/tools//cdn-cgi/l/email-protection solves the problem.#2021-02-2020:16andy.fingerhutI don't know whether alexmiller uses Cursive or not, but if not, perhaps letting Cursive dev on #cursive know this might help get to root cause.#2021-02-2021:10Alex Miller (Clojure team)Are there messages in the IntelliJ error log?#2021-02-2021:12Alex Miller (Clojure team)It is possible that the cache file format change is not properly picked up in the cursive scenario - rm ing your .cpcache dir would help if that’s the case, but just guessing#2021-02-2105:23mike_ananevThere was some NPE in Cursive. I've downgraded deps cli so cannot reproduce.#2021-02-2105:25mike_ananevI made a report to Cursive via Idea about NPE. So Colin should receive its details.#2021-02-2206:29murtaza52hi, a colleague of mine is not able to get any aliases working for clj, clj repl works fine, this is the error report -
{:clojure.main/message
"Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).\n-X:test (No such file or directory)\n",
:clojure.main/triage
{:clojure.error/class java.io.FileNotFoundException,
:clojure.error/line -2,
:clojure.error/cause "-X:test (No such file or directory)",
:clojure.error/symbol java.io.FileInputStream/open0,
:clojure.error/source "FileInputStream.java",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.io.FileNotFoundException,
:message "-X:test (No such file or directory)",
:at [java.io.FileInputStream open0 "FileInputStream.java" -2]}],
:trace
[[java.io.FileInputStream open0 "FileInputStream.java" -2]
[java.io.FileInputStream open "FileInputStream.java" 195]
[java.io.FileInputStream <init> "FileInputStream.java" 138]
[java.io.FileInputStream <init> "FileInputStream.java" 93]
[clojure.lang.Compiler loadFile "Compiler.java" 7571]
[clojure.main$load_script invokeStatic "main.clj" 475]
[clojure.main$script_opt invokeStatic "main.clj" 535]
[clojure.main$script_opt invoke "main.clj" 530]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause "-X:test (No such file or directory)"}}
#2021-02-2206:47pavlosmelissinos1. First of all, run clj -Sdescribe
and upgrade if you're using an old version of clojure
2. It seems like you're running the command in two lines:
clj
-X:test
Probably not related to the error but just use a single line
3. -X runs a function. Is test a function? If not, and the test alias has a :main-opts, you should be using the -M (main) flag instead: clj -M:test#2021-02-2206:36hiredmanMy guess is the clj your colleague has is either a old version without -X support, or is not the tools.deps clj script#2021-02-2206:37hiredmanCalling it with -Sdescribe will tell you which#2021-02-2206:38hiredmanIf you get a similar looking error to the above then it is not the tools.deps clj script, otherwise you will get output that tells you the version of the script#2021-02-2217:49Jakub Holý (HolyJak)Hello! How do I use :git-url with :tag ? The official docs
> Optional key :tag is used only to indicate the semantics of the sha
is not very clear to me. Can I require a particular branch, or master, without hardcoding a sha? Thank you!#2021-02-2217:52borkdude@holyjak :git/url (with the slash) and I think you should run -Sresolve-tags to fill in the SHA#2021-02-2217:53borkdudeOh, this is now called: -X:deps git-resolve-tags#2021-02-2217:54borkdudeI wonder if this will preserve comments, etc in the deps.edn file :thinking_face:#2021-02-2217:58Jakub Holý (HolyJak)thank you!#2021-02-2218:06Alex Miller (Clojure team)It won’t #2021-02-2219:22Jakub Holý (HolyJak)I am sorry, was that a response to me? What won't what?#2021-02-2219:22Jakub Holý (HolyJak)Ah, you meant that comments will not be preserved in the file.#2021-02-2219:27Alex Miller (Clojure team)yes, that's what I meant#2021-02-2218:30Jakub Holý (HolyJak)It does not do anything, I get "No unresolved tags found." So how can I point the dep to "master" (and possibly get :sha auto-updated)? 🙏#2021-02-2219:27borkdudemaster isn't a tag right?#2021-02-2219:29Alex Miller (Clojure team)correct, it's a branch#2021-02-2219:30Alex Miller (Clojure team)you're basically fighting what's provided now (which is intentionally narrowly scoped)#2021-02-2219:33Alex Miller (Clojure team)it would not be that hard to make something that updated the deps.edn to use the latest sha tracking a branch (tools.gitlibs in fact supports resolution of any rev, including branches)#2021-02-2219:33Alex Miller (Clojure team)I believe there is now a lib to update a deps.edn while preserving comments etc as well#2021-02-2219:33borkdudeThere is nothing which can't be fixed with a lil' scripting and templating#2021-02-2219:34borkdudeThis is one lib that can do it (based on rewrite-clj): https://github.com/borkdude/rewrite-edn#2021-02-2219:34Alex Miller (Clojure team)yeah, that + tools.gitlibs is about 95% of what you need#2021-02-2219:35Alex Miller (Clojure team)you could add a :branch to the git coord, it will just be ignored by tools.deps#2021-02-2219:35Alex Miller (Clojure team)(he says, without confirming :)#2021-02-2219:35borkdudeThe approach that we are taking at work is having a deps.template.edn which we manipulate using a script which produces the final deps.edn. So comments are only preserved in the template, the deps.edn itself is just a pretty printed thing. You could just do this using pure git and no tools.deps even.#2021-02-2219:36Alex Miller (Clojure team)using tools.gitlibs will benefit you as it will cache the git dir and working tree shas in the same place and way as you'll get using them through tools.deps/clj#2021-02-2219:36borkdudeGetting the latest SHA doesn't require you to fetch the whole tree I think?#2021-02-2219:37Alex Miller (Clojure team)it won't#2021-02-2219:37Jakub Holý (HolyJak)I see, so branches do not work. I have pushed a tag, v1, and tried
{:git/url ""
:sha "fake"
:tag "v1"}
but still it does not do anything.#2021-02-2219:37Alex Miller (Clojure team)you can answer that from the git object dir, which is maintained independently from the sha working trees#2021-02-2219:37Alex Miller (Clojure team)did you push tags?#2021-02-2219:37Alex Miller (Clojure team)they are not pushed by default#2021-02-2219:38Jakub Holý (HolyJak)yes, I did, and can see it in GH.#2021-02-2219:39Jakub Holý (HolyJak)hm, maybe I should check if I have latest CLI ...#2021-02-2219:39Alex Miller (Clojure team)try it without :sha "fake"#2021-02-2219:42Alex Miller (Clojure team)I think it may not resolve the tag if it has a sha#2021-02-2219:45Jakub Holý (HolyJak)spot on! Thanks a lot!#2021-02-2219:46seancorfield@holyjak FYI, this shows how to resolve master to a SHA: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L207-L211#2021-02-2219:46Alex Miller (Clojure team)that seems like a bug, will fix it#2021-02-2219:47seancorfield(if you ignore the add-libs aspect of it -- I'm just talking about the gitlibs part)#2021-02-2314:31mpenet@alexmiller gitlibs will shell out to git instead of using jgit in the near future? (looking at these recent commits in the repo)#2021-02-2314:38Alex Miller (Clojure team)working towards that yes#2021-02-2314:38Alex Miller (Clojure team)all on a branch atm#2021-02-2314:38Alex Miller (Clojure team)but looking good so far#2021-02-2314:38mpenetgood to hear#2021-02-2322:00mike_ananev@seancorfield Hello. I've created new library template for clj-new and published it on http://Clojars.org
Source code is here https://github.com/redstarssystems/rssyslib
When I'm trying to use it by clojure -X:clj-new :template rssyslib :name then clj-new can't find it on Clojars.
My ~/.clojure/deps.edn contains:
:clj-new {:extra-deps {seancorfield/clj-new {:mvn/version "1.1.243"}}
:exec-fn clj-new/create
:main-opts ["-m" "clj-new.create"]}
#2021-02-2322:03mike_ananevWhat I'm doing wrong?
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "81ece8eae4c1ece3f1b1b3"}, :content ("[email protected]")}#2021-02-2322:08seancorfield@mike1452 clj-new only looks for releases -- until you tell it to look for snapshots (and your lib is only a snapshot right now, yes?)#2021-02-2322:09seancorfield(! 737)-> clojure -X:new :template rssyslib :name :snapshot true
Downloading: seancorfield/clj-new/maven-metadata.xml from clojars
Downloading: rssyslib/clj-template/maven-metadata.xml from clojars
Downloading: rssyslib/clj-template/0.1.0-SNAPSHOT/maven-metadata.xml from clojars
Generating project from library template
#2021-02-2322:11seancorfieldYour :clj-new alias doesn't need :main-opts BTW since you're invoking it with -X:
:new {:extra-deps {seancorfield/clj-new {:mvn/version "RELEASE"}}
:exec-fn clj-new/create
:exec-args {:template lib}}#2021-02-2322:30mike_ananevYes, my lib has snapshot version. How can I tell to look for snapshots?#2021-02-2322:31seancorfield@mike1452 I just showed you.#2021-02-2322:32seancorfield(it's also explained in the clj-new readme)#2021-02-2322:33mike_ananevOh! Now I see it! @seancorfield thank you!!#2021-02-2322:34seancorfieldI wanted to check the template actually worked (which it did!).#2021-02-2322:35seancorfieldIf you want to check a template without publishing it to Clojars, you should be able to run clojure -X:new :template rssyslib ... directly in your project's folder since the template will be on the classpath (via the src path and your existing deps.edn) and clj-new tries to require the namespaces locally before starting to search clojars for clj templates (then boot templates, then lein templates).#2021-02-2322:39mike_ananevYes. I've checked it locally. It works fine. Now, I will make latest command for bumping version and then publish release.#2021-02-2322:39mike_ananevThank you!!!#2021-02-2322:59hiredmanI saw a blog post about infering dependencies for python projects based on imports, and it got me wondering if there are any projects that will look at clojure code and spit out a deps.edn for it#2021-02-2323:00Alex Miller (Clojure team)something Rich and I have been talking about for years :)#2021-02-2323:02hiredmanI wrote a little function that does its best https://gist.github.com/15186e238dc365fd72e2e09c3eb7561a but I was wondering it was tackled more seriously#2021-02-2411:02borkdudeHere is my stab at it:
https://github.com/borkdude/deps-infer#2021-02-2411:39borkdudeI think this can be improved in various ways (as documented in the README). Having an up to date (daily) index (of namespaces to lib coords) of clojars (and mvn central) available is one of the best improvements. Not hard, just has to be maintained somewhere.#2021-02-2411:45borkdudeI think such an index will also help detect namespace clashes in the entire clojure ecosystem which will be pretty useful#2021-02-2412:38borkdudeThis tool can be used for e.g.:
- Making implicit deps explicit
- Checking which deps you are _not_ using by comparing the output with your existing `deps.edn`
- Porting a `lein` `project.clj` or `boot` `build.boot` to `deps.edn`
- Porting scripts (e.g. [babashka]()
scripts) to a `deps.edn` project.
- Migrating from git deps to mvn deps
- Checking if there is a newer version available
#2021-02-2412:43pavlosmelissinosI know where I'm posting this question and I'm not looking to start a flame war, but I've noticed that beginners tend to gravitate towards lein, not cli tools.
1. Is there a reason for that?
2. I suppose that part of the reason could be due to outdated tutorials on the web (lein was obviously more prevalent in the past). However, given that tools.deps is endorsed by nubank/cognitect, comes with clojure and is used by an increasing number of libraries, would you ever recommend lein over cli tools for new projects in 2021? If yes, why?
edit: Apologies if this comes up too often 🙂#2021-02-2412:49borkdudeProbably UX is part of the equation: lein comes with a test runner, uberjar, etc, built-in#2021-02-2412:50borkdudetools.deps doesn't try hard to provide a full tool for most needs: you have to assemble it yourself#2021-02-2412:51borkdudethere is a project in development called tools.build which might perhaps make this more beginner friendly, but this remains to be seen#2021-02-2412:58borkdudeThere was a thread on Reddit (I believe) recently about this, if you want to read comments from the community.
I like both tools for what they are. Tools.deps / Clojure CLI allows you to only use what you need, which is generally better for startup, etc.#2021-02-2413:42pavlosmelissinosAh, good points, thanks @borkdude. Are you referring to this thread? https://reddit.com/r/Clojure/comments/jailzq/aleksandar_simic_from_lein_to_deps_and_back_lndclj/#2021-02-2413:42borkdudeNo, it was a different one#2021-02-2413:45borkdude@UEQPKG7HQ It was related to this blog post:
https://quanttype.net/posts/2021-01-14-clojure-project-management-tool-of-my-dreams.html#2021-02-2413:46borkdudeHere is the Reddit thread: https://www.reddit.com/r/Clojure/comments/l3a2iz/clojure_project_automation_tool_of_my_dreams/#2021-02-2416:02mike_ananevI did my own project workflow for tools.deps which has basic Leiningen functionality.
https://github.com/redstarssystems/rssyslib#2021-02-2416:08borkdudeThat's cool. Another attempt was done by @U06V097TP as a joker script which I ported to babashka here:
https://gist.github.com/borkdude/c34e8e44eb5b4a6ca735bf8a86ff64fa (original you can see via the forked link)
This can even become its own GraalVM compiled standalone binary perhaps :)#2021-02-2416:08borkdudeBut I guess a template is more flexible because you can tweak it later and won't require others to install the tool#2021-02-2603:11mpenetJust + deps is a really good match#2021-02-2603:11mpenetJust gives you a lot for free#2021-02-2414:27Alex Miller (Clojure team)definitely more to come... from the world of tools.deps :)#2021-02-2414:39jjttjjdeps-infer + add-lib would be an absolute dream#2021-02-2414:42Alex Miller (Clojure team)we've actually spent some time on this a couple times, but actually going further with integrating it into core#2021-02-2414:42Alex Miller (Clojure team)still trying to figure out where to draw the boundaries#2021-02-2414:44Alex Miller (Clojure team)we'd love to give you a ton of power at the repl and maybe in ad hoc scripting, but not encourage you to make a mess of your actual programs :)#2021-02-2414:45Alex Miller (Clojure team)we are working on some stuff right now that is taking another step or two in this direction#2021-02-2414:49borkdude@jjttjj you can use deps-infer today :) It actually works quite well, I've solved a couple of small issues already#2021-02-2414:49borkdudeAnd I'm co-ordinating now with @tcrawley to fetch deps info from clojars 😍#2021-02-2415:36borkdudeThe rewrite-edn lib was referenced here a couple of days ago and I thought I'd make a video on it:
https://youtu.be/bPoe2N7J5yg#2021-02-2418:08seancorfieldIn case, any folks aren't in #news-and-articles#2021-02-2418:13borkdude@seancorfield So you basically have one top-level project to rule them all. But this will be forbidden in the future, unless I have misunderstood the warning:
worldsingles/worldsingles {:local/root "../worldsingles"}#2021-02-2418:16hiredmanI believe the warning is for outside paths, not for local roots#2021-02-2418:16borkdudeah, you are correct sir#2021-02-2418:17borkdudeSo why did we have to discuss this for 2 years before this solution emerged? It makes sense to me at first glance ;)#2021-02-2418:19borkdudeWe have other scripts and Makefiles etc within the subprojects, so being in the subproject is a convenience maybe#2021-02-2418:19borkdudeBut it's good to have alternatives. Thanks for the write-up Sean!#2021-02-2418:31seancorfield@borkdude I think @hiredman had the same response when he read over a draft of that article last night 🙂#2021-02-2418:32borkdudeI wonder how one top-level project solves the local/root caching issues you were seeing with the scripting approach. I've seen this problem in "normal" usages of local/root as well#2021-02-2418:32seancorfieldI think I just kept holding out, hoping that the CLI would magically sprout a "fourth" deps.edn file in there... but I've given up on that now 🙂#2021-02-2418:34seancorfieldYeah, we're still at the mercy of :local/root transitive deps but it's much less of a problem now that we're not trying to auto-generate the deps.edn files: and now we only have a single .cpcache folder to worry about instead of one in every subproject, which means -Sforce should be sufficient in all cases now.#2021-02-2418:34borkdudeAh, transitive local/roots, yeah, that we don't have yet#2021-02-2418:35borkdudeWe only have direct ones for now#2021-02-2418:35borkdudeBut this might change...#2021-02-2418:35seancorfieldI still have a couple of tickets at work around cleaning up the last few vestiges of the "big shift" we just did last week to arrive at this new setup.#2021-02-2418:36borkdudeWhat I like about the scripting approach is that it doesn't rely on any features that I don't use often, just a simple "what you see is what you get" deps.edn#2021-02-2418:36borkdudeBut it's good to know that this works in case the scripting approach has clear negatives#2021-02-2418:36seancorfieldOf course, the CLI / t.d.a may change again in the future and necessitate yet another (big) shift but I get the sense that we're much closer to the core team's thinking about how deps.edn should be used now than we were in either of our previous incarnations.#2021-02-2508:03flowthingIs there a way to run -P for all aliases in a deps.edn file? My use case is that we have CI build that runs steps in parallel, so we'd like to download every dep that every alias needs into the local Maven cache before running the CI steps to prevent concurrent access to ~/.m2.
We could list all aliases manually, but if there's a new alias, then we need to remember to add it into the -P step.#2021-02-2508:10borkdudeWrite a script :)#2021-02-2508:18flowthingYeah, that's one option. Was wondering whether we could leverage the tools.deps API and -X to do something like that. Will look into it.#2021-02-2509:16borkdude@flowthing Getting all aliases from a deps.edn is quite easy.
This is a bb script:
(require '[clojure.edn :as edn]
'[clojure.string :as str])
(def edn (edn/read-string (slurp "deps.edn")))
(def aliases (keys (:aliases edn)))
(require '[babashka.deps :as deps])
(def cmd ["-P" (str "-A" (str/join aliases))])
(println "Downloading deps using:" cmd)
(deps/clojure cmd)#2021-02-2509:17flowthingThat looks simple enough, thanks!#2021-02-2509:20borkdudeAdded it here:
https://github.com/babashka/babashka/blob/master/examples/download-aliases.clj#2021-02-2509:41flowthingGolfing a bit here, but I guess this'd also do it:
{:aliases {:foo {:extra-deps {criterium/criterium {:mvn/version "0.4.6"}}}
:bar {:extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}}}
:baz {:extra-deps {cheshire/cheshire {:mvn/version "5.10.0"}}}}}
(require
'[ :as io]
'[clojure.tools.deps.alpha :as deps]
'[clojure.tools.deps.alpha.util.maven :as mvn])
(as->
(assoc (deps/slurp-deps (io/file "deps.edn")) :mvn/repos mvn/standard-repos)
deps
(deps/resolve-deps deps (deps/combine-aliases deps (->> deps :aliases keys))))#2021-03-0106:20Eugenhow can I use tool-deps to download a jar to a specific location?
I need to install a JVM agent that should not be on the classpath but I would like to manage it via deps.edn.
I hcecked https://clojure.github.io/tools.deps.alpha/clojure.tools.deps.alpha-api.html and it does not say how the dependencies are downloaded#2021-03-0106:24Alex Miller (Clojure team)Tools.deps uses Maven which only downloads jars to the local repo#2021-03-0106:26Alex Miller (Clojure team)You can specify a :mvn/local-repo key to affect that but seems like it would be cumbersome in combination with managing your regular class path#2021-03-0106:27Alex Miller (Clojure team)I guess you could create an alias with just that dep on it, use that to download, and then control what you’re doing with the result somehow#2021-03-0221:23seancorfieldcom.github.seancorfield/depstar {:mvn/version "2.0.193"} -- A JAR builder for the Clojure CLI -- https://github.com/seancorfield/depstar/releases/tag/v2.0.193 -- adds :manifest option to populate MANIFEST.MF with additional entries; warns if :group-id does not look like a reverse domain name per Clojars policy change to Verified Group Names; changes the group ID of depstar to a VGN (releases will be double-published to seancorfield/depstar for a while). Follow-up in #depstar for the library, follow-up in #clojars for the policy#2021-03-0313:43borkdudeJust fyi, I think moving to shell git instead of jsch will probably alleviate this issue as well:
https://github.com/oracle/graal/issues/3251#2021-03-0313:43borkdudeUnless the maven stuff also depends on jsch. If not, then an all native dependency resolution/download might be near.#2021-03-0313:52borkdudeIf you are curious about this, I talk about tools-deps-native here: https://youtu.be/P09GZVqiDdM?t=2240 (start from m37s20)#2021-03-0314:06Alex Miller (Clojure team)maven does not depend on jsch#2021-03-0314:07Alex Miller (Clojure team)moving to git shell will actually drop about 1/4 of the transitive lib weight for clj#2021-03-0315:55borkdudefyi, updated to git shell branch of gitlibs project:
https://github.com/borkdude/tools-deps-native-experiment/tree/shell#shell-branch#2021-03-0317:14dnolenanybody figured out workarounds for git deps and Apple M1?#2021-03-0317:38borkdudeWhat's the problem on m1?#2021-03-0317:39borkdudeMaybe the git shell branch is a workaround :)
But maybe running under Rosetta is also one?#2021-03-0317:53Spencer AppleUpgrading from 1.10.2.774 to 1.10.2.790 broke this for us:
GRAALVM_HOME=/path/graalvm-ce-19.2.0.1/Contents/Home clojure -A:native-image
our deps.edn
:aliases {:native-image
{:main-opts ["-m clj.native-image monorepo.core"
"--initialize-at-build-time"
"--no-fallback"
"--report-unsupported-elements-at-runtime"
"-H:+PrintClassInitialization"]
:jvm-opts ["-Dclojure.compiler.direct-linking=true"]
:extra-deps
{clj.native-image/clj.native-image
{:git/url ""
:sha "f3e40672d5c543b80a2019c1f07b2d3fe785962c"
:exclusions [org.slf4j/slf4j-nop]}}}}
error:
{:clojure.main/message
"Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).\n-m clj.native-image monorepo.core (No such file or directory)\n",
:clojure.main/triage
{:clojure.error/class java.io.FileNotFoundException,
:clojure.error/line -2,
:clojure.error/cause
"-m clj.native-image monorepo.core (No such file or directory)",
:clojure.error/symbol java.io.FileInputStream/open0,
:clojure.error/source "FileInputStream.java",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.io.FileNotFoundException,
:message
"-m clj.native-image monorepo.core (No such file or directory)",
:at [java.io.FileInputStream open0 "FileInputStream.java" -2]}],
:trace
[[java.io.FileInputStream open0 "FileInputStream.java" -2]
[java.io.FileInputStream open "FileInputStream.java" 219]
[java.io.FileInputStream <init> "FileInputStream.java" 157]
[java.io.FileInputStream <init> "FileInputStream.java" 112]
[clojure.lang.Compiler loadFile "Compiler.java" 7571]
[clojure.main$load_script invokeStatic "main.clj" 475]
[clojure.main$script_opt invokeStatic "main.clj" 535]
[clojure.main$script_opt invoke "main.clj" 530]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause
"-m clj.native-image monorepo.core (No such file or directory)"}}#2021-03-0317:56seancorfield@splayemu "-m clj.native-image monorepo.core" should be three separate strings: "-m" "clj.native-image" "monorepo.core"#2021-03-0317:57seancorfieldI'm a bit surprised it used to work in earlier versions, to be honest.#2021-03-0318:00Spencer AppleWow thanks! I swear I had tried that but it appears to be working.#2021-03-0318:33Alex Miller (Clojure team)I'm not surprised :)#2021-03-0318:41seancorfieldI had never tried :main-opts strings with multiple command-line arguments in because it just never occurred to me that it would work -- all the examples show a separate string for each argument, I believe -- but I'm not surprised it no longer works 🙂#2021-03-0318:50borkdudeI made a parser for command line strings, in case anyone needs it ;)
https://github.com/babashka/pod-babashka-aws/blob/e119d08f738ed4fc6bb533a99288ccbd7794c18f/src/pod/babashka/aws/impl/aws/credentials.clj#L70#2021-03-0318:51borkdudee.g. it splits echo 'foo bar' in "echo" and "foobar"#2021-03-0322:15Alex Miller (Clojure team)I have done a prerelease of the Clojure CLI (1.10.2.801) that switches git/ssh to shell out rather than using jgit/jsch. this potentially fixes many existing git/ssh issues in that it is just shelling out to git, so if it works there, it should work through clj. I would love to see any and all tire-kicking on this:
• public https repos
• https repos with user/pw auth (not previously supported, subject to whatever credentials you have set up through git)
• private ssh git repos
• newer key exchange methods like ecdsa-sha2-nistp256
• .ssh/config options that didn't work before
• ed25519 identity keys
• concurrency issues in parallel downloads (I think this is improved but I never had a working repro for it)
• doesn't work on M1 macs
• Windows
• CI systems like Travis/Circle/etc
when you're testing, don't forget that a) git dirs and working trees are cached in ~/.gitlibs - consider selectively clearing/moving that while testing), and b) classpaths are cached in .cpcache dirs - use clj -Sforce to force that to override#2021-03-0322:21Alex Miller (Clojure team)there are still a couple more things I need to do here. currently, tools.gitlibs defaults to non-interactive behavior so it will not prompt you if, for example, https://github.com hosts are not in known_hosts, it will just fail. there is a flag in the API to support that, but I have not pulled that through the whole stack yet - it can be selectively interactive (and should be if you're just sitting at the terminal I think)#2021-03-0322:55borkdudeExciting! Bumped at https://github.com/borkdude/tools-deps-native-experiment. Native compilation only takes 1m40s on CI which is really good: an indication that this project is now much more light weight.#2021-03-0323:00Alex Miller (Clojure team)yeah, the brew tarball went from like 19M to 15M#2021-03-0402:04seancorfieldcom.github.seancorfield/clj-new {:mvn/version "1.1.264"} -- Create new applications and libraries with the Clojure CLI -- https://github.com/seancorfield/clj-new/releases/tag/v1.1.264 -- adjusts all generated projects so they follow the Clojars policy change to Verified Group Names; changes the group ID of clj-new to a VGN (releases will be double-published to seancorfield/clj-new for a while). Follow-up in #clj-new for the library, follow-up in #clojars for the policy#2021-03-0414:29mkvlrI’m seeing this
/t/deps ❯❯❯ clojure -version
Clojure CLI version 1.10.2.801
/t/deps ❯❯❯ cat deps.edn
{:deps {org.clojure/clojure {:mvn/version "1.10.2"}
org.clojure/clojurescript {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}}}
/t/deps ❯❯❯ clojure -Stree
Error building classpath. Manifest type not detected when finding deps for org.clojure/clojurescript in coordinate {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}#2021-03-0414:43Alex Miller (Clojure team)I can't reproduce that - can you try with -Sforce ?#2021-03-0414:44Alex Miller (Clojure team)(git urls typically end in .git btw, although github doesn't seem to require it)#2021-03-0414:45mkvlr@alexmiller same:
/t/deps ❯❯❯ clojure -Sforce
Error building classpath. Manifest type not detected when finding deps for org.clojure/clojurescript in coordinate {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}#2021-03-0503:02seancorfieldFWIW, I ran into a similar situation later today @alexmiller but I wasn't able to nail down a repro. In the end, I blew away ~/.gitlibs and that fixed it. Here's the output from one of my attempts:
Checking out: at ee2badaaf4b26754f903ed92e77fd4e82cf2ed70
Error building classpath. Manifest type not detected when finding deps for com.github.seancorfield/depstar in coordinate {:git/url "", :sha "ee2badaaf4b26754f903ed92e77fd4e82cf2ed70"}
And that checked out folder was empty.#2021-03-0503:03seancorfieldAnd with a different SHA -- same result:
(! 806)-> clojure -X:jar
Checking out: at a8cf78c9e09e3504e64fc77bcb133a7ada39a68f
Error building classpath. Manifest type not detected when finding deps for com.github.seancorfield/depstar in coordinate {:git/url "", :sha "a8cf78c9e09e3504e64fc77bcb133a7ada39a68f"}#2021-03-0503:04seancorfieldThe only odd thing is that this line was missing when it failed:
Cloning:
but present when it worked.#2021-03-0504:30seancorfieldJust a follow-up note: in the main channel, you indicated that the problem might be due to prior parallel jgit downloads causing corruption but these SHAs were new since I moved to 1.10.2.801 -- are you saying that the jgit impl could have messed things up in the ~/.gitlibs folder such that it causes problems for the new git-sh impl @alexmiller?#2021-03-0512:41Alex Miller (Clojure team)Yes. Or there was some logic that detected this and recloned and I removed it. Going to look at it today.#2021-03-0516:36seancorfieldHappy to test more and do any debugging that might assist.#2021-03-0414:45Alex Miller (Clojure team)that... does not make any sense to me#2021-03-0414:45mkvlr(also if I append .git)#2021-03-0414:45Alex Miller (Clojure team)the manifest type is pulled from the :git/url ns attribute#2021-03-0414:46Alex Miller (Clojure team)(I'm not saying you're not seeing it, just saying I don't understand why)#2021-03-0414:47mkvlrI’m also a bit puzzled, works for another git dep#2021-03-0414:48Alex Miller (Clojure team)anything in your ~/.clojure/deps.edn? Can you try with -Srepro (that will omit that one)#2021-03-0414:49Alex Miller (Clojure team)can you attach the actual deps.edn file here#2021-03-0414:50mkvlrthere you go, still reproduces with clojure -Sforce -Srepro for me#2021-03-0414:50borkdude$ clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.2"}
org.clojure/clojurescript {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}}}'
Works for me locally#2021-03-0414:52mkvlrclojure -Srepro -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.10.2"}
org.clojure/clojurescript {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}}}'
Error building classpath. Manifest type not detected when finding deps for org.clojure/clojurescript in coordinate {:git/url "", :sha "715cdc07dbb1d595af91ea12affb6faf0b615d4b"}#2021-03-0414:53borkdude@mkvlr try rm -rf ~/.gitlibs; rm -rf .cpcache#2021-03-0414:53Alex Miller (Clojure team)neither of those should be in play here#2021-03-0414:56mkvlr@borkdude moving ~/.gitlibs away fixed it#2021-03-0414:56borkdude🤷#2021-03-0414:56Alex Miller (Clojure team)hrm#2021-03-0414:58Alex Miller (Clojure team)oh, I'm misreading the error message#2021-03-0414:58mkvlr@alexmiller I had previously our fork at https://github.com/nextjournal/clojurescript.git of ClojureScript with the same sha in the deps.edn . Let me know if I should try to isolate the issue with ~/.gitlibs and send you a minimal version#2021-03-0414:58Alex Miller (Clojure team)this is not detecting a manifest type in the git working tree#2021-03-0414:59Alex Miller (Clojure team)it's looking for either deps.edn or pom.xml in the dir#2021-03-0415:00Alex Miller (Clojure team)so curious what you see in ~/.gitlibs/libs/org.clojure/clojurescript/715cdc07dbb1d595af91ea12affb6faf0b615d4b#2021-03-0415:00Alex Miller (Clojure team)is that maybe an empty dir?#2021-03-0415:05mkvlr@alexmiller yes, it’s empty#2021-03-0415:07Alex Miller (Clojure team)ok, well I understand the error then :)#2021-03-0415:08Alex Miller (Clojure team)that (I think) is a side effect of parallel download conflicts from the prior jgit impl#2021-03-0415:08Alex Miller (Clojure team)I was never able to repro that so I can't say that it won't happen in the new one (but I think it won't)#2021-03-0415:10borkdudeMaybe treat an empty dir as non-existing?#2021-03-0415:10borkdudeThat would cover up the cause maybe too much#2021-03-0415:19Alex Miller (Clojure team)right now we use the existence of the dir as a signal, but I want to look at this more#2021-03-0415:20Alex Miller (Clojure team)anyhow, enqueued :)#2021-03-0415:21Alex Miller (Clojure team)thx for trying it and reporting#2021-03-0415:27mkvlrthank you!#2021-03-0422:26seancorfield@alexmiller Is there a documented, supported way to get hold of the injected basis for a t.d.a based program? The system property went away, right? I know how to calculate the project basis but that's not necessarily the same.#2021-03-0423:51seancorfieldI guess (t/merge-edns ((juxt :root-edn :user-edn :project-edn) (t/find-edn-maps))) is really all I need (since I'm just trying to look up available aliases, not deps at this point).#2021-03-0504:57sofraHi, I just upgraded to the latest version.
I used to setup my test runner like this
:runner {:extra-paths ["test"]
:extra-deps {lambdaisland/kaocha {:mvn/version "1.0.641"}}
:main-opts ["-m" "kaocha.runner"]}
this now gives my deprecation warnings
❯ clj -A:test:runner --focus unit
WARNING: When invoking clojure.main, use -M
but after reading the docs I am not sure how to address them#2021-03-0505:02seancorfield@sofra Use clj -M:test:runner --focus unit#2021-03-0505:04seancorfieldWith the introduction of -X for function execution, -M has been expanded from just :main-opts to include deps/paths as well and is how you should run anything relying on clojure.main (`:main-opts`). -A is going to be reserved for starting a REPL in due course (but still functions as before, except with that warning).#2021-03-0505:07sofrathanks @seancorfield for the explanation#2021-03-0505:09sofraI had tried with -M instead of -A but it seems to look for the kaocha.runner as a file on the file system then
❯ clj -M:test:runner --focus unit
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
kaocha.runner (No such file or directory)
#2021-03-0505:13dpsuttonI think kaocha is looking for a config file?#2021-03-0505:13seancorfieldNo.#2021-03-0505:14seancorfield@sofra I just created a brand new project and put this in it:
:aliases
{:test {:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}}}
:runner {:extra-paths ["test"]
:extra-deps {lambdaisland/kaocha {:mvn/version "1.0.641"}}
:main-opts ["-m" "kaocha.runner"]}
...}
and it works fine for me:
$ clojure -M:test:runner --focus unit
[(F)]
Randomized with --seed 150177492
FAIL in sofra.example-test/a-test (example_test.clj:7)
FIXME, I fail.
Expected:
0
Actual:
-0 +1
1 tests, 1 assertions, 1 failures.
#2021-03-0505:14seancorfieldI think perhaps you changed your :main-opts?#2021-03-0505:16seancorfieldThat's with the latest prerelease:
$ clojure --version
Clojure CLI version 1.10.2.801
but I can try it with the latest stable version...#2021-03-0505:17sofrahmmm, thanks … my bad, it does seem to work now, not sure what it was#2021-03-0505:18sofrayeah strange, sorry for the noise, it is working 🙂#2021-03-0505:18seancorfield(I just double-checked it works with 1.10.2.790 as well)#2021-03-0505:18sofracool#2021-03-0615:34roklenarcicAre key values for -X not automatically strings? I’m doing the call:
clj -X:deploy :artifact "cljc-shuffle.jar"
And I’ve noticed that slipset/deps-deploy i am running is explicitly converting :artifact value to string, via (str artifact)#2021-03-0615:39borkdudeNo, this is very annoying, you have to write '"foo"'#2021-03-0616:49roklenarcicso what does foo produce in the program? A symbol foo?#2021-03-0617:08Alex Miller (Clojure team)yes#2021-03-0617:08Alex Miller (Clojure team)all args are read as edn strings#2021-03-0617:09roklenarcicHow do cmd line args and :exec-args combine? I’ve had this:
:deploy {:extra-deps {slipset/deps-deploy {:mvn/version "0.1.5"}}
:exec-fn deps-deploy.deps-deploy/deploy
:exec-args {:installer :remote :artifact "cljc-shuffle.jar"
:sign-releases? true}}
If I run, clj -X:deploy it worked. Then I tried to specify artifact on cmd. So I removed it from deps and ran:
clj -X:deploy :artifact "'cljc-shuffle.jar'" and then I get
Execution error (IllegalArgumentException) at java.lang.reflect.Array/set (Array.java:-2).
array element type mismatch
#2021-03-0617:09roklenarcicdo command line arguments replace :exec-args in full?#2021-03-0617:43seancorfieldNo they are additive. I would have expected that to work#2021-03-0617:44seancorfieldOh wait, no. Your arg is wrong.#2021-03-0617:44seancorfieldUse '"string"'#2021-03-0617:45seancorfieldStrings require double quotes. Then you need single quotes around that for the shell#2021-03-0617:46seancorfield@roklenarcic ^#2021-03-0618:09seancorfieldFYI to tool authors, such as @slipset since deps-deploy was mentioned, for convenience of use I have been allowing symbols where strings are expected so that if users could write the argument without quotes, they are able to do so. For a local JAR file such as cljc-shuffle.jar you could just pass a symbol in and the code could convert the symbol to a string. Obviously, if you're passing in a path, that's generally not going to be a legal symbol and you are going to have to quote ' and double-quote " it.#2021-03-0619:53roklenarcicAh yes, the symbol fix isn’t released yet for deps-deploy#2021-03-0813:38tianshuLooks like there are more and more wonderful libraries in Clojure, most of them provide a main function that can start with -M . I wonder whether it's possible to start multiple main functions in a single repl, or I have to write a .clj script to combine them.#2021-03-0814:00Alex Miller (Clojure team)no, the Clojure runner runs one program, so if you want to start multiple you need to wrap them in something else#2021-03-0814:15tianshu@U064X3EF3 Thank you for the quick answer!#2021-03-0813:46teodorluCould you call the -main function directly? I guess the main function might be private, but there are ways around that.#2021-03-0813:59Alex Miller (Clojure team)the -main function can't be private (if you want Java to be able to call it)#2021-03-0814:16tianshuI can call main functions in a script.#2021-03-0816:59seancorfield@doglooksgood More and more libraries intended for use with the CLI are getting functions that you can call via -X these days, and those are also more amenable to calling directly from Clojure code, making it easier to write a short Clojure script that calls those functions one after the other. I wouldn't be surprised if we adopted a standardized approach for such pipelines of functions fairly soon...#2021-03-0819:16borkdudeI'm running with the latest clojure CLI and I've had several occasions now that there's something weird with gitlibs.
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "167479647d72637273565b54462426272f"}, :content ("[email protected]")}
#2021-03-0819:18borkdudemoving ~/.gitlibs aside fixes it once again#2021-03-0819:24Alex Miller (Clojure team)so it's a partial checkout?#2021-03-0819:24borkdudeseems like it yes#2021-03-0819:25Alex Miller (Clojure team)can you share the deps.edn you're using?#2021-03-0819:25borkdudesure:
{:aliases {:json {:extra-deps {cheshire/cheshire {:mvn/version "5.10.0"}}}
:test {:extra-paths ["test"]
:extra-deps {cheshire./cheshire {:mvn/version "5.10.0"}
cognitect-labs/test-runner
{:git/url ""
:sha "b6b3193fcc42659d7e46ecd1884a228993441182"}}
:main-opts ["-m" "cognitect.test-runner"]}}}#2021-03-0819:25borkdude(oh I see a typo in the cheshire name, but that wasn't the partial checkout)#2021-03-0819:26Alex Miller (Clojure team)so just the one git lib#2021-03-0819:26borkdudeyes#2021-03-0819:28borkdudeit is a gitlib of which I have two different checkouts as you can see#2021-03-0819:28Alex Miller (Clojure team)sure#2021-03-0819:28Alex Miller (Clojure team)I assume you didn't like ctrl-c a previous test run#2021-03-0819:30borkdudeI can't rule that out necessarily#2021-03-0819:30borkdudedon't remember#2021-03-0819:30borkdudemaybe a shutdown hook should do some clean up if that is the case?#2021-03-0819:32Alex Miller (Clojure team)what clean up could it do?#2021-03-0819:34mkvlrI ran into a similar issue today, the symptom was it couldn’t find a namespace. I removed offending folder from ~/.gitlibs but it would only refetch it after running clojure with -Sforce#2021-03-0819:35Alex Miller (Clojure team)I think it's probably better to think about it from perspective of only making it visible when it's done#2021-03-0819:37Alex Miller (Clojure team)git itself is using some locking protocols so there is some benefits from cloning into the same dir so I will need to look at this more closely#2021-03-0819:37Alex Miller (Clojure team)(cloning re git dir, not checking out re working trees)#2021-03-0819:38borkdudeyes, my problem was also fixed using -Sforce#2021-03-0819:39borkdudeI meant cleaning like removing the entire folder if the checking out was interrupted#2021-03-0819:39borkdudemaybe some atomic move like construct could also work#2021-03-0819:39Alex Miller (Clojure team)yes, that makes sense#2021-03-0819:40Alex Miller (Clojure team)same as corrupted maven downloads#2021-03-0911:21pinkfrogDo you know how to auth with
with deps without setting the $HOME/.maven config ?#2021-03-0913:37Alex Miller (Clojure team)I don’t think you can#2021-03-0914:21borkdudeJust some feedback: I've had the cognitect test-runner problem several times again today, and I'm pretty sure I didn't interrupt anything#2021-03-0914:35Alex Miller (Clojure team)noted, update coming soon#2021-03-0914:36Alex Miller (Clojure team)I am also wondering if there is some issue with the whole shelling out process not running to completion or something#2021-03-0914:37Alex Miller (Clojure team)are you running this in some way such that ~/.gitlibs doesn't exist each time?#2021-03-0914:42borkdudeI am sometimes just rm -rf ~/.gitlibs (because of problems) and then when I try again, everything works like expected. but if I go to another project which uses a different SHA of the test runner, I run into this problem#2021-03-0916:59zaneI recall there’s a way to get clj / clojure to include its own dependencies in -Stree, but I don’t remember what it was. Could someone remind me?#2021-03-0917:16seancorfield@zane If you use -Sdeps, they'll be included in -Stree I believe. If you use -X:deps tree they won't, as I recall.#2021-03-0917:31zaneHmm. Like so? All I’m seeing is spec.
❯ clojure -Stree -Sdeps '{}'
org.clojure/clojure 1.10.2
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56#2021-03-0917:36seancorfieldOh, I misread your question. clj/`clojure` are shell scripts so they do not have dependencies, but what I think you mean is you want to see the tools.deps.alpha stuff?
(! 843)-> clojure -Stree -A:deps
org.clojure/clojure 1.10.2
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56
org.clojure/tools.deps.alpha 0.10.889
. org.apache.maven.resolver/maven-resolver-api 1.6.1
. org.apache.maven.resolver/maven-resolver-spi 1.6.1
. org.apache.maven.resolver/maven-resolver-api 1.6.1
. org.apache.maven.resolver/maven-resolver-impl 1.6.1
. org.apache.maven.resolver/maven-resolver-api 1.6.1
. org.apache.maven.resolver/maven-resolver-spi 1.6.1
. org.apache.maven.resolver/maven-resolver-util 1.6.1
. org.apache.commons/commons-lang3 3.8.1
. org.slf4j/slf4j-api 1.7.30 :newer-version
. org.apache.maven.resolver/maven-resolver-util 1.6.1
... and a screenful more ...#2021-03-0917:48zaneProbably! This is that issue I mentioned before where a third-party security scan is flagging Jetty as having vulnerabilities.#2021-03-0917:51zaneI’m trying to figure out which version of Jetty winds up being included in the .jars the clj and clojure shell scripts rely on.#2021-03-0917:51zaneI think what you posted above does what I want.#2021-03-0917:54zaneThe vulnerability they’re worried about was fixed in org.eclipse.jetty_jetty-io 9.4.36, but alas it looks like tools.deps still relies on org.eclipse.jetty/jetty-io 9.4.24.v20191120.#2021-03-0918:02Alex Miller (Clojure team)the chain of deps down to that can be upgraded to address that, just hasn't yet#2021-03-0918:02Alex Miller (Clojure team)or that is, it has been, but tools.deps isn't using it yet#2021-03-0918:02Alex Miller (Clojure team)but I will get that in the next build#2021-03-0918:04seancorfield@zane That vulnerability is in the server or the client portion of that library? ISTR that t.d.a uses only the non-vulnerable part of Jetty (we had this discussion already, didn't we?)#2021-03-0918:08Alex Miller (Clojure team)I don't think it's an issue, but upgrading regardless#2021-03-0918:09Alex Miller (Clojure team)committed, will be in next build#2021-03-0918:09zaneIt’s definitely not a real issue, but thank you all the same @U064X3EF3!#2021-03-0918:10zane@U04V70XH6
> That vulnerability is in the server or the client portion of that library?
Server.
> we had this discussion already, didn’t we?
We sure did.
Our partner’s security team keeps flagging this issue even though in the past they’ve accepted our explanation that our software doesn’t use Jetty’s web server. 😩#2021-03-0918:11seancorfieldSorry 😞 Some of those security folks can be... a bit intransigent...#2021-03-0918:15zaneYeah, I mean, I understand their position but it’s definitely a hassle.#2021-03-1007:16Alex Miller (Clojure team)Clojure CLI 1.10.2.805 prerelease is now available:
• Fixed git checkout implementation (issues when using multiple commits from the same repo). If you have been using prerelease 1.10.2.801 for git deps, you should update and
• Resolution of commits now only fetches if it can't resolve, which reduces network use and improves performance
• When using the CLI, if your git command is not on the path as git for some reason, you can set GITLIBS_COMMAND env var
◦ If using tools.deps.alpha / tools.gitlibs directly, you can also use the Java system properties clojure.gitlibs.dir and clojure.gitlibs.command (these take precedence over the env vars).
• Fix reflection error in tools.gitlibs
• Updated tools.cli, and com.cognitect.aws libs (notably moves transitive deps org.eclipse.jetty/jetty-* to 9.4.36.v20210114)#2021-03-1007:42raspasovIs the brew version still pending?
brew install clojure/tools/clojure
… still fetches 1.10.2.796 (I believe)#2021-03-1007:43raspasovActually managed to get it via:
brew install clojure/tools/#2021-03-1017:53seancorfield@alexmiller Given that 1.10.3 is available, should the CLI move to 1.10.3.x or will that happen just with the next stable CLI version?#2021-03-1018:50Alex Miller (Clojure team)I will get there when this stabilizes#2021-03-1008:12dharriganIs there a link?#2021-03-1008:18dharriganAh, it's still showing as pre-release on github and Cloning:
Checking out: at cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b
Cloning:
Checking out: at a033bccaf0024b279480f2a7ac4b446621629b73
Error building classpath. Manifest type not detected when finding deps for cognitect/test-runner in coordinate {:git/url "", :sha "cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b"}
I'm now logged in with ssh. Yay CircleCI. I can debug.#2021-03-1013:02borkdude$ ls ~/.gitlibs/_repos/github.com/cognitect-labs/test-runner/
HEAD branches config description hooks info objects packed-refs refs
#2021-03-1013:02borkdudeJust tell me what you want to know @alexmiller#2021-03-1013:03borkdudeThis is a build which does not cache gitlibs, so it's cloned from afresh.#2021-03-1013:04borkdude-Sforce doesn't help either:
$ clojure -Sforce -M:test
Checking out: at cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b
Checking out: at a033bccaf0024b279480f2a7ac4b446621629b73
Error building classpath. Manifest type not detected when finding deps for cognitect/test-runner in coordinate {:git/url "", :sha "cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b"}#2021-03-1013:05borkdudeMoving gitlibs aside didn't work either, same error.#2021-03-1013:12Alex Miller (Clojure team)What’s in the libs dirs?#2021-03-1013:14borkdude{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "1c7f756e7f70797f755c2b2c2e7d7a7a7d792a7a247f"}, :content ("[email protected]")}#2021-03-1013:14borkdude#2021-03-1013:17Alex Miller (Clojure team)I mean under .gitlibs/libs#2021-03-1013:17borkdudeNothing, it doesn't exist#2021-03-1013:17Alex Miller (Clojure team)Well that seems bad :)#2021-03-1013:18borkdudeMaybe it is related to that circleci git config. I tried to move that aside as well, but to no avail.#2021-03-1013:18borkdudeI can add your public key to this build if you want, then you can check yourself#2021-03-1013:20borkdudeor you can clone my project + branch and build it yourself#2021-03-1013:21Alex Miller (Clojure team)Probably easiest to start there#2021-03-1013:22Alex Miller (Clojure team)Can you check the git version in ci#2021-03-1013:22borkdude$ git --version
git version 2.11.0
#2021-03-1013:23borkdudeis that too old?#2021-03-1013:23borkdudeI can upgrade if that helps#2021-03-1013:25borkdudeI am using the circleci/clojure:lein-2.8.1 image#2021-03-1013:26Alex Miller (Clojure team)I need to expose a couple of the things I’ve been using to debug, should have done that in this release#2021-03-1013:30borkdudeI think you should be able to reproduce this with the mentioned Docker image.#2021-03-1013:31borkdudeLet me try to make a small repro#2021-03-1013:38borkdudeRepro:
https://gist.github.com/borkdude/932cdc1362359e4f0d7c83ffce3e4c50#2021-03-1013:40borkdudeBtw, I would be happy to upgrade my builds if needed to newer images, I just didn't do so because of laziness#2021-03-1014:04Alex Miller (Clojure team)no need, just wanted to know#2021-03-1014:05Alex Miller (Clojure team)thx for the repro, will take a look#2021-03-1013:13borkdudefwiw, I checked with 796 in a different branch and builds ok over there.#2021-03-1013:16borkdudefwiw, on my local system (macOS) it builds fine with 805#2021-03-1013:38borkdudeRepro:
https://gist.github.com/borkdude/932cdc1362359e4f0d7c83ffce3e4c50#2021-03-1015:26Alex Miller (Clojure team)@borkdude turns out, it is related to git version. I'm using a --quiet option that was added in git 2.19 (I think), but I've removed it for the next release as it doesn't matter, and with that change your existing repro works. I think our minimum git version is probably 2.5.#2021-03-1015:28Alex Miller (Clojure team)I will get a new release out at some point today, want to hook up the debugging stuff and include that too so that's available#2021-03-1015:28borkdude@alexmiller maybe propagate the git stderr to the parent process' stderr so you can see this?#2021-03-1015:29Alex Miller (Clojure team)yeah, that's part of it#2021-03-1015:29borkdudecool, thanks for looking into it#2021-03-1015:29Alex Miller (Clojure team)just trying to hook this stuff up to tools.deps etc in a good way#2021-03-1018:27markaddlemanShould ~/.gitlibs directory be listed as part of clj -Sdescribe ? I recently ran into a problem where the downloaded library was somehow corrupted. Deleting ~/.gitlibs fixed the problem but it took me a while to discover that the directory existed in the first place.#2021-03-1020:40Alex Miller (Clojure team)could do#2021-03-1020:40Alex Miller (Clojure team)Clojure CLI 1.10.3.810 prerelease is now available
• Removed use of git option only available on newer gits to work with broader version range (git ≥ 2.5)
• Check exit code on git dep checkout and throw with the git error
• Overhauled configuration and option implementations for gitlibs, will write this up more for stable release, stuff from last release all still works
• Use Clojure 1.10.3 as default Clojure version (as always, you can use any version of Clojure with any version of the Clojure CLI)
#2021-03-1020:51seancorfieldThank you! I'll update my dev env as soon as my current test suite run finishes 🙂#2021-03-1021:10seancorfieldSeems to work fine for my relatively simple use case (although, after rm -rf ~/.gitlibs, I forgot I needed -Sforce to refresh the cache since .cpcache still referred to the now-missing .gitlibs path).#2021-03-1021:20Alex Miller (Clojure team)hey, don't forget to -Sforce! :)#2021-03-1021:21Alex Miller (Clojure team)maybe it's time to actually solve that problem :)#2021-03-1112:58imreIs there a way to launch clj completely ignoring the local deps.edn but using an alias from the user deps.edn?#2021-03-1113:01imreI suppose I could use replace-deps/paths in said alias...#2021-03-1113:02borkdudeYes, for tools unrelated to your main deps you should use this#2021-03-1113:03imreKinda clunky when I have a tool that I usually use together with my main deps but every once in a while I want to use it on its own#2021-03-1113:04imreStill doable#2021-03-1113:41Alex Miller (Clojure team)Working on such a thing now actually#2021-03-1113:50imreooh nice one!#2021-03-1120:33phronmophobicI started using clojure cli a few months ago and I'm already sold on switching all my projects to work with deps.edn. I still feel clumsy trying to consume 3rd party tools like deps-deploy and depstar and was wondering if my mental model of how the cli is supposed to work is off.
I organized my thoughts in this gist, https://gist.github.com/phronmophobic/5ae932b8769b6d03e5749e9a135b6374. Maybe it's useful as an experience report. Based on learning how to use the cli recently, I would be happy to suggest edits to the cli documentation, but I'm not sure if that would be helpful.
Anyway, I know there's been lots of work and progress on the cli tools and the clojure cli has already improved my workflows. Thanks!#2021-03-1120:47Alex Miller (Clojure team)Yeah, I have some definite thoughts in this area, but it’s queued up behind other work#2021-03-1120:47Alex Miller (Clojure team)More to do for sure!#2021-03-1120:58seancorfield@smith.adriane That explain option is an interesting idea. Being able to ask the CLI script to display what the various aliases resolve to and what the ultimate command execution would be could be very helpful for folks new to the CLI or even just new to a particular CLI tool.#2021-03-1120:59seancorfieldIt's partly why I added a :query true option to clj-new so it can tell you what it would do if you executed it for real.#2021-03-1121:02phronmophobicyes seems generally useful
• provides the data the cli “sees” which helps you form a mental model by flattening all of the different options and params into one piece of data
• makes it easier to debug other’s cli commands since they can just provide the explain#2021-03-1121:05phronmophobicI think having a built in way to get the help text for a cli tool is a bigger win though since otherwise there’s no standard way to see examples or see the available options other than source code and github readmes (afaik)#2021-03-1121:35Alex Miller (Clojure team)I would really like to have something like:
• a command that tells you help for a namespace (probably the ns doc string)
• a command that tells you the functions in a namespace you can invoke (all public? all with particular meta? all doc'ed? filter to 1-arity? lots of open questions)
• a command to give you help about a particular function (presumably the fn doc string)
and then some of the debugging aids you mention would probably be helpful too#2021-03-1122:01phronmophobic> a command that tells you the functions in a namespace you can invoke (all public? all with particular meta? all doc'ed? filter to 1-arity? lots of open questions)
Conceptually, I view choosing a function and specifying the arguments as orthogonal. The arglist is either a map for a function designed with cli -X in mind or a sequence in the general sense. It is a tricky problem since at some point you're better off just writing the command into a script instead of typing it at the command line.
> a command that tells you help for a namespace (probably the ns doc string)
I've used several of the cli "toolbelts" (aws, heroku, git) and started toolbelt projects at a few of the places I've worked. One of the reasons I think the clojure cli is particularly exciting is that it's potentially an open toolbelt. Maybe it would be better as a 3rd party library, but the idea of being able to find cli tools using the cli tool itself sounds appealing (eg. cli find linter to find linter tools). The syntax isn't consistent, but hopefully the idea makes sense.
I also like the idea of being able to copy seancorfield's deps.edn into my .clojure/deps.edn and then run a command like cli ls to list all of the aliases and associated commands.
These suggestions gloss over all the details of how the cli tool would be able to discover cli tools on the internet or from a deps.edn file, but adding discoverability makes it easier to find the right tool as a user and have your tools be found as an author.#2021-03-1122:13seancorfield@smith.adriane
(! 847)-> clj -A:deps
user=> (require '[clojure.tools.deps.alpha :as t])
nil
user=> (->> (t/find-edn-maps) ((juxt :root-edn :user-edn :project-edn)) (t/merge-edns) :aliases (keys) (sort))
(:1.0 :1.1 :1.10 :1.10.0 :1.10.1 :1.10.2 :1.2 :1.3 :1.4 :1.5 :1.6 :1.7 :1.8 :1.9 :add-libs :bench :check :classes :closh :comp :decompile :deploy :deps :depstar :dev :eastwood :expect :graph :install :j14 :jar :jedi-time :liquid :master :measure :new :nrepl :outdated :pne :portal :prepl :readme :rebel :rebl :reflect :reveal :runner :socket :socket-rebl :socket-zero :spec2 :test :uberjar :dev/repl)
#2021-03-1122:14seancorfieldJust FYI, that's how to find all the available aliases in any given project or non-project context.#2021-03-1122:15phronmophobicMy example is more of a hypothetical for someone picking up deps.edn. ie. How do I get up and running with a reasonable set of tools at my disposal?#2021-03-1122:16seancorfieldAlso, not sure if you're aware but :exec-args can be a keyword, and it is looked up as an alias to get the data structure that is the actual :exec-args map -- and tools can do the same thing (as depstar does and I believe deps-deploy soon will too).#2021-03-1122:18seancorfieldYeah, the absolute "getting started" (with deps) part is still a bit fiddly. My comment there is if you at least start with clj-new and create a new project that way, it'll already have the aliases you need for testing, building a JAR, deploying to Clojars etc.#2021-03-1122:18seancorfieldBut of course clj-new is opinionated about how to do those things and it's encoded in the built-in templates, and other new-project templates will be similarly opinionated in a different direction.#2021-03-1122:19seancorfieldAnd of course there's also tools.build on the horizon.#2021-03-1122:19phronmophobicYep, I use clj-new, but I have to either reuse the clj-new command that's in my bash history or go to the clj-new readme every time I use it#2021-03-1122:19phronmophobicI have the clj-new aliases from your deps.edn, but the clj-new docs don't actually use those#2021-03-1122:19seancorfieldWhich part(s) of the invocation are hard to remember?#2021-03-1122:20phronmophobicI don't create new projects that often and when I do, it's usually after copy and pasting the command from the github readme, so I never remember which keys are required#2021-03-1122:21seancorfield; my .clojure/deps.edn:
:new {:extra-deps {com.github.seancorfield/clj-new {:mvn/version "1.1.264"}}
:exec-fn clj-new/create
:exec-args {:template lib}}
; the clj-new readme:
:new {:extra-deps {com.github.seancorfield/clj-new
{:mvn/version "1.1.264"}}
:exec-fn clj-new/create
:exec-args {:template "app"}}
so the only difference is defaulting to app in the clj-new readme and lib in the .clojure file.#2021-03-1122:22seancorfieldI could sync those up if I knew which was the more likely to be used 🙂#2021-03-1122:24phronmophobicfwiw, I think the docs for your dot-clojure, clj-new, and depstar are really good. It's mostly just that
• there's no 0 arg version that makes sense
• I use the tools infrequently enough that I don't remember the keyword names that are required
• there's no way to figure out which options are available without going to the github readme page#2021-03-1122:24phronmophobicI'm not sure how cli tool authors provide a way to display options and examples except github readmes.#2021-03-1122:25phronmophobicthe best work around that I can think of is for tools like clj-new to provide a 0 args version that prompts for any required/common options#2021-03-1122:26phronmophobicthen I can just do clj -X:new and it can prompt me for the project name and template#2021-03-1122:27phronmophobicit's also totally possible that I'm doing it wrong. My intention was to use it like the aws cli toolbelt, but that might just be the wrong way.#2021-03-1122:28seancorfield(! 848)-> clj -X:new
Usage:
clojure -m clj-new.create template-name project-name options
or:
clojure -X clj-new/create :template template-name :name project-name options
Any additional arguments are passed directly to the template.
The template-name may be:
* app - create a new application based on deps.edn
* lib - create a new library based on deps.edn
* template - create a new clj template based on deps.edn
#2021-03-1122:28seancorfieldI agree that isn't quite as consistent and helpful as it could be.#2021-03-1122:29seancorfield(part of the problem there is not being able to tell how something was invoked since that information is gone by the time the tool is invoked)#2021-03-1122:30seancorfieldSo clj-new would need to introspect the available aliases, see if :new was available and what it was aliased to, and then somehow weave that into the help.#2021-03-1122:30seancorfieldSince you could be doing clojure -Sdeps '...' -X clj-new.create ... and not even have an alias available.#2021-03-1122:31phronmophobicactually, the help text for clj -X:new is exactly what I need. I guess I just never tried it. 😳#2021-03-1122:34phronmophobichmm, actually, neither of the example commands work when I copy and paste them into the terminal. I guess that's since the cli tool doesn't know how the command was invoked as you were saying. It does solve my specific problem for clj-new though since I now have a way to look up the options without going to the github readme.#2021-03-1122:39seancorfieldI will update the help so it's a bit "smarter" about things...#2021-03-1122:42seancorfieldhttps://github.com/seancorfield/clj-new/issues/55 -- feel free to add more notes there.#2021-03-1208:22vlaaad> A command that tells you the functions in a namespace you can invoke (all public? all with particular meta? all doc’ed? filter to 1-arity? lots of open questions)
I’ve been thinking about this as well in the context of cli entry point for Reveal. I wanted to have a single entry point ns accessible both from code and from cli, but some functions that are definitely a public API don’t make sense to use from cli (if they are e.g. have to be composed with other functions). Given that, it made sense to use particular meta.
There is also another question that’s not asked — how to show a short description of a function vs a full documentation. Given an ns, it makes sense to get some quick overview of what functions are there before diving into details of a particular function. There were a couple of options, and the most pragmatic to me seemed treating very first line of cli docstring as this short description.
The whole code for that is here: https://github.com/vlaaad/reveal/blob/master/src/vlaaad/reveal.clj#L159-L182#2021-03-1317:02eval2020@U47G49KHQ that’s similar to the approach I took for inclined:
clj -Sdeps '{:deps {eval/inclined {:git/url "" :sha "a40aeeba2d471f19e83eeeceb3ebbc700f23185f"}}}' -M -m inclined.main -h#2021-03-1121:47Alex Miller (Clojure team)@borkdude did you ever re-try with latest cli?#2021-03-1121:51borkdude@alexmiller Yes, no problems so far, but I haven't worked on my own projects a lot today with which I had the problems. More on the weekend#2021-03-1121:58Alex Miller (Clojure team)thx, I'll probably hold off till next week to release stable anyways, so no rush#2021-03-1123:59souenzzoCan clj -Spath add an option emit all paths as canonical-path ?
It will allow to create a classpath and invoke the java command from anydir#2021-03-1200:03seancorfield@souenzzo That wouldn't work for code that expects to read files (from the current directory).#2021-03-1200:04seancorfield(and it's only stuff like src/`test`/`resources` that are relative in path?)#2021-03-1200:04souenzzo@seancorfield i understand that limitation, but I still think that this transformation is useful#2021-03-1200:05souenzzocurrently all deps looks canonical, only the :paths items (src resources) are "relative"#2021-03-1200:07seancorfieldHere's a trick to add the :paths to the path as canonical versions:
clojure -Sdeps '{:deps {my/app {:local/root "."}}}' -Spath#2021-03-1200:08seancorfieldYou'll still get src etc but you'll also get them as full paths.#2021-03-1200:08souenzzoclojure -Sdeps '{:deps {my/app {:local/root "."}}}' -Spath | tr ':' '\n'
src
resources
service
😞#2021-03-1200:08seancorfieldIt's not quite what you want but it seems like a good workaround...?#2021-03-1200:08Alex Miller (Clojure team)If you want to file a question on ask.Clojure I can track it there#2021-03-1200:09seancorfield@souenzzo what version of the CLI are you using? That's not what I see when I run that.#2021-03-1200:10souenzzo1.10.2.796#2021-03-1200:12seancorfieldHmm,
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "83f0e6e2ede0c3c7c6d0c8d7ccd3aeb0b3cac0c2b4b5"}, :content ("[email protected]")}
#2021-03-1200:16souenzzoOk It's only duped. I will work with that local/root hack#2021-03-1204:05dnolencan confirm the pre-release fixed up my issues on M1 hardware#2021-03-1213:17Alex Miller (Clojure team)Thanks! Good to know#2021-03-1215:28Alex Miller (Clojure team)anyone else here with feedback on the CLI prereleases that shell out to git?#2021-03-1215:47raspasov1.10.2.805? Seems to work on my machine…#2021-03-1215:47raspasovOh wait… there’s a new one.#2021-03-1215:47Alex Miller (Clojure team)latest is 1.10.3.810#2021-03-1215:47raspasovI’ll try.#2021-03-1215:49Alex Miller (Clojure team)thx!#2021-03-1215:56flowthingI've used it pretty lightly (macOS Big Sur) -- no problems so far.#2021-03-1215:57raspasovGot it:
% clj --version
Clojure CLI version 1.10.3.810#2021-03-1216:00raspasovNuked
rm -rf ~/.gitlibs
after installing 1.10.3.810
Seems to be downloading everything correctly when building projects. (macOS Big Sur)#2021-03-1216:14Alex Miller (Clojure team):thumbsup: thx#2021-03-1216:58seancorfieldWorked fine in my basic tests, but I don’t have too many git deps. Vendored 1.10.3.810 into the repo at work and everything still works in our dev/test/build pipeline (only one git dep there though, on Cognitect’s test-runner).#2021-03-1219:28dnolen@alexmiller hrm in work project if I change the sha it says it can't resolve#2021-03-1219:28dnolenas in it worked the very first I tried it - but bumping a sha results in#2021-03-1219:28dnolenChecking out:
#2021-03-1219:29dnolenfwiw, it doesn't matter if it's an https or git style url#2021-03-1219:29dnolendeleting ~/.gitlibs allows me to proceed#2021-03-1219:50Alex Miller (Clojure team)what version are you using?#2021-03-1219:52Alex Miller (Clojure team)1.10.2.801 was definitely broken when using multiple shas from the same repo, but that was fixed in 1.10.2.805 and latest 1.10.3.810#2021-03-1219:54Alex Miller (Clojure team)but what that sounds like as I think about it more is a missed fetch - trying to resolve using a new sha that has been created since the initial clone#2021-03-1219:54Alex Miller (Clojure team)I'll see if I can repro#2021-03-1220:17dnolen1.10.3.810#2021-03-1220:17dnolenyes this is fetching a new sha after the initial clone#2021-03-1220:49Alex Miller (Clojure team)@dnolen bug found, and fixed, new release in the works, thx#2021-03-1220:49dnolenthanks!#2021-03-1221:35Alex Miller (Clojure team)Clojure CLI prerelease 1.10.3.814 is now available:
• fix issue with detecting when git fetch is needed for git deps#2021-03-1221:35Alex Miller (Clojure team)@dnolen ^^#2021-03-1222:13borkdudetools.gitlibs is now so "pure Clojure" because of the shelling out, that it can be run with babashka from source :)
$ bb -cp src/main/clojure -e "(require '[clojure.tools.gitlibs :as gl]) (gl/procure \"\" 'org.clojure/tools.gitlibs \"11fc77496f013871c8af3514bbba03de0af28061\")"
Cloning:
Checking out: at 11fc77496f013871c8af3514bbba03de0af28061
"/Users/borkdude/.gitlibs/libs/org.clojure/tools.gitlibs/11fc77496f013871c8af3514bbba03de0af28061"
(second time: 37ms)#2021-03-1222:18Alex Miller (Clojure team)metal#2021-03-1513:31Darin Douglassi'm quite new to using clj and i'm attempting to get my feet under me with regards to how all the aliases/deps/paths keys work together.
is there any functional difference between -Stree and -X:deps tree? the docs mention them together and make no distinction between the two other than -X:deps "has more options".
i'm finding with a super simple deps.edn file -Stree and -X:deps tree return different trees:
;; deps.edn (i've also tried this same config using `-Sdeps '...'`)
{:aliases {:1.10 {:override-deps {org.clojure/clojure {:mvn/version "1.10.0"}}}}}
➜ clj --version
Clojure CLI version 1.10.3.814
➜ clj -Srepro -Sforce -A:1.10 -Stree
org.clojure/clojure 1.10.0
. org.clojure/spec.alpha 0.2.176
. org.clojure/core.specs.alpha 0.2.44
➜ clj -Srepro -Sforce -A:1.10 -X:deps tree
org.clojure/clojure 1.10.3
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56
the -Stree result is the tree i would expect to see given my aliases#2021-03-1515:09Alex Miller (Clojure team)Yes, these are different. -Stree shows you the tree for the classpath being run, including aliases or other modifiers on the command line#2021-03-1515:09Alex Miller (Clojure team)-X:deps tree is running a tree program that computes the tree for the project.edn (command line modifiers are not considered)#2021-03-1515:36Darin Douglassby project.edn do you mean <project>/deps.edn?#2021-03-1515:38Darin Douglassalso a blank -Stree 's output should be equivalent to -X:deps tree?#2021-03-1515:48Alex Miller (Clojure team)Sorry, yes I meant project deps.edn#2021-03-1515:48Alex Miller (Clojure team)And yes, should be the same#2021-03-1517:12Darin Douglassawesome thanks for the clarification#2021-03-1513:51dnolen814 appears to fix the git fetch issues, I don't see problems in Cursive or at the CLI now#2021-03-1600:00Joshua SuskaloHas there been any consideration to adding a key to a dependency map which has :deps/manifest be of type :deps to permit including a vector of aliases to include?
For example, I have a project intended to be used as a library which needs to include different natives jars depending on the OS, but without asking my dependents to include those native jars themselves I can't provide a good way to use only one set of them. I think it would be useful to be able to specify something like so:
{:git/url "some-git"
:sha "abc123"
:deps/manifest :deps
:deps/aliases [:windows-natives]}
and have this bring in all the appropriate deps, based on the keys which modify paths or dependencies.#2021-03-1600:03Alex Miller (Clojure team)it has been suggested, yes#2021-03-1600:04Alex Miller (Clojure team)at some point I will think more about it :)#2021-03-1600:04Joshua SuskaloSounds good! I was mostly curious since this seems to be something that would require running a modified version of tools.deps, it couldn't really be done with additional tooling.#2021-03-1608:22vlaaadFYI: the issue is https://clojure.atlassian.net/browse/TDEPS-116#2021-03-1610:22vlaaadBtw I think there might be another solution to this problem — specifying path in the git coordinate to the folder in the repo of where the actual dependency is located#2021-03-1612:27Alex Miller (Clojure team)You can do that now with :deps/root but I’m not sure how that does what you’re asking about#2021-03-1612:40vlaaadoh, forgot about :deps/root#2021-03-1610:23vlaaadThat way there is no need to figure out what aliases are public and what aliases are internal and may change#2021-03-1610:38mkvlrI really like the ability to set the `:deps/manifest` to :deps to consume a standard clojure lib from git without needing a `deps.edn` upstream as long as the source files are in `src/` . Is is even possible set options this way? If not and I’d like to see it should I put it on http://ask.clojure.org?#2021-03-1612:26Alex Miller (Clojure team)I don’t understand what you’re looking for - what options?#2021-03-1612:31borkdudeMy guess, mkvlr means: an option to set :deps/manifest :edn as the default if no pom.xml or deps.edn is found#2021-03-1612:32souenzzoNot sure how it scale, but you can control "profiles" with deps/root:
deps.edn
windows-extras/deps.edn
linux-extras/deps.edn
Where
windows-extras/deps.edn
{:deps {lib/window-dll {...}}}
Then the consumer:
:deps {app/core {:git ...}
app/core+windows-extras {:git ... :git/root "windows-extras"}}
#2021-03-1612:40mkvlr@alexmiller @borkdude say setting another :paths or :deps in case the project requires it. I normally fork the project and make a fix and then I have another branch where I add a deps.edn because I only want to open a PR with the fix (and consume it locally) but not add the deps.edn upstream. A workaround for setting :deps is to add them to my root, but then it’s not clear what to remove later. Does that make sense?#2021-03-1612:41borkdude@mkvlr you don't have to fork, you can add :deps/manifest :deps in your deps.edn for that lib#2021-03-1612:42mkvlr@borkdude and can I provide a different :paths and :deps for that lib in that case? Within the same map?#2021-03-1612:42borkdudeah, so when the dep doesn't use the standard src and has other deps? hmm, right#2021-03-1612:43mkvlryes, the first part I learned https://clojurians.slack.com/archives/C6QH853H8/p1615891133005300 and like it a lot#2021-03-1612:43mkvlrbut might be useful to extend it?#2021-03-1612:44mkvlrwell I’d find it useful anyway 😼#2021-03-1612:44borkdude:deps/manifest {:type :deps :paths ["src2"]}, something like that?#2021-03-1612:45mkvlryeah#2021-03-1612:48mkvlr:deps/manifest {:paths ["src2"]} not sure if :type could even be omitted or if overloading :deps/manifest is right. It currently stands for a manifest type.#2021-03-1612:53borkdude:deps/manifest currently indicates either :pom or :deps#2021-03-1612:54mkvlryes, which is the type of project (from the docs: Default is to auto-detect the project type (currently either `:deps` or `:pom`))#2021-03-1612:58Alex Miller (Clojure team)I don't think it makes sense for an external consumer of a project to determine the paths to use in the project, seems like the project creator should be the one defining that#2021-03-1612:59Alex Miller (Clojure team)but it does make sense to me for a project creator to potentially provide standard ways to modify the classpath (presumably via aliases)#2021-03-1613:00mkvlrmy use case is having this be a temporary measure for a lib to which I’m contributing a fix. Temporary until its merged upstream.#2021-03-1613:01mkvlrin some way :deps/manifest :deps is also an external consumer of a project defining an (empty) deps.edn#2021-03-1613:01Alex Miller (Clojure team)there are already features in deps.edn for that - namely :classpath-overrides#2021-03-1613:02Alex Miller (Clojure team)or :override-deps, depending what you're doing#2021-03-1613:08mkvlryes, I can add the deps of the lib somewhere else (to the root or an alias or whatever) but the intention is less clear. At that point I’d rather fork the repo and add a deps.edn.#2021-03-1616:15Alex Miller (Clojure team)Clojure CLI https://clojure.org/releases/tools#v1.10.3.814 is now available!
• git deps: switch from using jgit to shelling out to git (must be git >= 2.5) - big thanks to @ghadi for all the initial work on this!
◦ New env vars for control:
▪︎ GITLIBS_COMMAND - command to invoke when shelling out to git, default = git
▪︎ GITLIBS_DEBUG - set to true to print git commands and output to stderr, default = false
• Made git fetch only when shas can’t be resolved to improve performance
• Bump dep versions for tools.cli and aws api to latest (in particular, moves transitive jetty deps to latest, past a CVE)
• Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.11.905
In general, this should resolve a large set of differences between using git deps through clj vs git at the command line, since they are now the same. Some additional benefits include a smaller deps footprint, and better performance.#2021-03-1616:24borkdudeAwesome!#2021-03-1620:39millettjonIs there a way in to override the location of ~/.m2/settings.xml? I am trying to work around a limitation of our CI build system. This issue seems to be related https://clojure.atlassian.net/browse/TDEPS-99.#2021-03-1620:45Alex Miller (Clojure team)No, not currently#2021-03-1620:47millettjonOk. Thanks.#2021-03-1620:50Alex Miller (Clojure team)if you want to vote for this, please do so here: https://ask.clojure.org/index.php/7865/allow-specify-maven-settings-provide-credentials-directly#2021-03-1712:51dev-hartmann@borkdude just a quick follow up question, if you don’t mind:#2021-03-1712:52dev-hartmannI added the Java project with :root/local “absolute path to root containing Pom”#2021-03-1712:53dev-hartmannBut when I try to import an object from the dependency inside my namespace#2021-03-1712:53borkdude:local/root#2021-03-1712:53dev-hartmannSure, sry have it that way #2021-03-1712:54dev-hartmannStill get a class not found exception#2021-03-1712:55dev-hartmannSeems like the project is loaded correctly, as I had to add my private nexus mvn repo #2021-03-1712:56borkdudeTry clojure -Spath and check if the directory with your classes is in there#2021-03-1712:56dev-hartmannSure#2021-03-1712:57dev-hartmannYes, it’s there#2021-03-1712:58borkdudeok, then it should work, or the class isn't there somehow#2021-03-1712:58dev-hartmannHm#2021-03-1712:58dev-hartmannThat’s strange #2021-03-1712:59dev-hartmannDo I need to add more dirs to the paths vector into my deps.edn to make it work?#2021-03-1713:00borkdudedirs in :paths are just the first dirs on the clojure -Spath, the rest come from your dependencies#2021-03-1713:00dev-hartmannOk, I see#2021-03-1713:06dev-hartmannHm, I’ll try pointing to a .jar file of the project#2021-03-1713:11dev-hartmannStill class not found #2021-03-1713:12dev-hartmannI need to point to the root of the project containing the Pom.xml, correct?#2021-03-1713:15borkdudeyes. but if your pom.xml doesn't include your generated classes on the classpath, then they won't be found#2021-03-1713:15dev-hartmannI see#2021-03-1713:16borkdudethis is why I asked you to inspect clojure -Spath. that should contain the dir with your generated classes#2021-03-1713:17dev-hartmannAh, ok#2021-03-1713:17dev-hartmannThat it didn’t #2021-03-1713:17dev-hartmannThen I need to include them#2021-03-1713:17dev-hartmannI just saw it was including the base dir of my code#2021-03-1713:18dev-hartmann.../src/java...#2021-03-1713:19borkdudewhat is not on the classpath, cannot be found, that's how the classpath works :)#2021-03-1713:20dev-hartmannSure, makes sense #2021-03-1713:20dev-hartmannI thought since it had the root of my project it could check all the files under that dir#2021-03-1713:20dev-hartmannTbh I thought that the .java files were enough#2021-03-1713:21dev-hartmannAs you might see not a java specialist#2021-03-1713:21borkdudeclojure does not do any compilation for .java files, you need .class files#2021-03-1713:21dev-hartmannI can see that building the java application generates a target/classes folder #2021-03-1713:21dev-hartmannMakes perfect sense#2021-03-1713:22dev-hartmannHow do I make mvn aware there is a target/class folder?#2021-03-1713:25borkdudeI'm not a mvn expert either, but if you need to add target/classes to the clojure classpath you could just add a deps.edn in this other project containing {:paths ["target/classes"]}#2021-03-1713:26dev-hartmannI see#2021-03-1713:26borkdudeand then you use that other project using :local/root and it should pick up the target/classes dir#2021-03-1713:26dev-hartmannWill try that, thx again and sry for nagging. #2021-03-1713:50dev-hartmannok, with the additional deps.edn in the java project#2021-03-1713:51dev-hartmannand local/root with :deps/manifest :deps it worked#2021-03-1713:51dev-hartmanneven company autocompletion works and is suggesting the correct imports#2021-03-1713:51dev-hartmannbut when I eval the namespace#2021-03-1713:51dev-hartmannstill#2021-03-1713:51dev-hartmannClassNotFound#2021-03-1713:52dev-hartmannclojure -Spath clearly has the folder there#2021-03-1713:52dev-hartmannand the folder is not empty 🙂#2021-03-1713:55dev-hartmannI can even see the javadoc and the info on what it extends#2021-03-1714:08rickmoynihanSince upgrading tools.deps to the latest I’m seeing warnings like this:
WARNING: Specified path /Users/rick/.gitlibs/libs/swirrl/muttnik-dev-files-cogs/116624aa86f42b765d2eda3b83a64c3981746a8e/shared/resources is external to project /Users/rick/.gitlibs/libs/swirrl/muttnik-dev-files-cogs/116624aa86f42b765d2eda3b83a64c3981746a8e/cogs#2021-03-1714:09Alex Miller (Clojure team)latest = ?#2021-03-1714:09Alex Miller (Clojure team)clj --version#2021-03-1714:10rickmoynihansorry Clojure CLI version 1.10.3.814#2021-03-1714:10Alex Miller (Clojure team)that error looks like a relative path in muttnik-dev-files-cogs that goes outside the project#2021-03-1714:11rickmoynihanyes, but those deps are pulled in like this:
swirrl/muttnik-dev-files-cogs {:git/url "#2021-03-1714:12rickmoynihanoh wait I think see what you’re saying… 👀#2021-03-1714:12rickmoynihanah yeah ok#2021-03-1714:12rickmoynihanthe deps edn file in the cogs folder there looks like this:
{:paths ["resources", "../shared/resources"]}#2021-03-1714:13Alex Miller (Clojure team)This warning was added in https://clojure.org/releases/tools#v1.10.2.790 and eventually external references not be allowed#2021-03-1714:13rickmoynihanok so I guess I need to pull in shared as an extra dep#2021-03-1714:15rickmoynihanoh actually I see that the shared directory has been deleted anyway. I just need to clear out the cruft. Thanks.#2021-03-1714:17Alex Miller (Clojure team)yeah, local deps would be an ok way to accomplish the same#2021-03-1714:20rickmoynihandoes that work for transitive deps?
Sorry the dev-files repo (above) is consumed in another project as a direct dependency, and the dev-files repo was the one with "../shared/resources" in it. Would a local dep resolve properly in the consuming project?#2021-03-1714:20Alex Miller (Clojure team)yes#2021-03-1714:21rickmoynihanawesome that’s good to know#2021-03-1714:21rickmoynihanthough looks like I don’t have this problem anymore 🙂#2021-03-1714:25rickmoynihanPresumably the move to using git proper, also means issues where JGit had a 100mb max file size limit are now also fixed?#2021-03-1714:34Alex Miller (Clojure team)I wasn't aware of that one but it will match whatever you get with git#2021-03-1714:34Alex Miller (Clojure team)since it is git :)#2021-03-1714:40borkdude@dev-hartmann try just clj and then type the classname you want to find directly in the REPL. Post the output here along with ls the-other-project/target/classes/<the-class>.class#2021-03-1714:41borkdudealso post the snippet where that directory occurs on your classpath as output with clojure -Spath#2021-03-1714:46dev-hartmann@borkdude you mean open a repl with clj and try to import the classname, right?#2021-03-1714:46dev-hartmannsure sec.#2021-03-1714:46borkdudenot even import, just try the fully qualified class name#2021-03-1714:48dev-hartmannit works now#2021-03-1714:48dev-hartmann!!!#2021-03-1714:48dev-hartmannI feel so stupid#2021-03-1714:48dev-hartmannwas my error#2021-03-1714:49dev-hartmannthe class with package is net.company.plugin.PluginFactory#2021-03-1714:49dev-hartmannand I tried to import it like this:#2021-03-1714:49dev-hartmannimport [net.company.plugin.PluginFactory PluginFactory]#2021-03-1714:49dev-hartmannthx for all the help, works now!#2021-03-1715:24Chaseso if I want to run a random function like clj -X practice.scratch/my-fn how am I supposed to feed in the arguments to that function? I'm getting a key is missing value error so I assume it has to be a map but my attempts at making it a map are failing too#2021-03-1715:24ChaseI specifically want to feed that function a file name that is in my resources folder#2021-03-1715:25Alex Miller (Clojure team)-X expects to invoke a function that takes a map#2021-03-1715:25Alex Miller (Clojure team)the args are key/value pairs#2021-03-1715:26Alex Miller (Clojure team)which will go in the map passed to the function#2021-03-1715:26Alex Miller (Clojure team)so like clj -X practice.scratch/my-fn :file '"foo.edn"'#2021-03-1715:27ChaseAhh, I had everything right except for the single quotes surrounding the file name#2021-03-1715:27Alex Miller (Clojure team)will invoke (practice.scratch/my-fn {:file "foo.edn"})#2021-03-1715:27Alex Miller (Clojure team)yeah, more details at https://clojure.org/reference/deps_and_cli#quoting#2021-03-1715:28Chaseahh, thank you. I was looking at the guide page, not the reference page#2021-03-1715:35dev-hartmann@borkdude just wanted to say thanks again, I could get it to work with the .jar file and it's working now even after creating an uberjar!#2021-03-1720:37pithylessTIL that :extra-paths can be a vector of aliases. Is there a way to make an alias that just brings in a group of aliases? The idea being to make a custom alias (new name) for a group of aliases (actual extra-deps, extra-paths, etc.) for the purpose of an easier to remember -X on the CLI.#2021-03-1720:38seancorfieldThat’s not possible at the moment @pithyless but it is something folks keep asking for 🙂#2021-03-1720:42pithylessOK, thanks 🙂 Right now, I'm often solving this problem via a Makefile/Justfile. Could this perhaps be tackled with a custom function and some fancy parsing of basis?#2021-03-1720:51seancorfieldPossibly. It would depend on exactly what you’re trying to achieve and in which context. Tooling is already at liberty to treat keywords as aliases and look up the values in the (calculated project) basis — depstar does this with almost all its :exec-args, for example (on develop — I haven’t cut a release containing that yet).#2021-03-1720:51seancorfieldI believe that :exec-args itself may have a keyword value and the CLI looks that up in the basis when you use -X.#2021-03-1720:52seancorfieldI just tend to string a whole bunch of aliases together — but at work we do have a shell script (called build) that provides some shortcuts for common CLI commands and alias combinations.#2021-03-1800:34seancorfieldcom.github.seancorfield/depstar {:mvn/version "2.0.206"} is available — new :paths-only option to exclude local/git deps from library JARs if needed; exec args can now be alias names (keywords) that are looked up in the basis (that includes user deps.edn) so that you can be more DRY in your deps.edn file by using aliases to define shared/common argument values (e.g., deps-deploy currently has a PR open to add similar functionality, so you would be able to define your JAR filename in an alias and then reuse it between both tools!). Follow-up in #depstar#2021-03-1803:48andy.fingerhutSo is it expected for brew install clojure/tools/clojure to install some version of a JDK via Homebrew now? I thought that one of the motivations for switching to clojure/tools/clojure from the former clojure Homebrew package was to avoid having a dependency on any JDKs?#2021-03-1803:51Alex Miller (Clojure team)nothing has changed (from clojure formula perspective), but brew works really hard to break stuff so something could have changed#2021-03-1803:53Alex Miller (Clojure team)you can see in the formula https://github.com/clojure/homebrew-tools/blob/master/Formula/clojure.rb that it does not explicitly depend on java#2021-03-1803:54Alex Miller (Clojure team)(vs https://github.com/Homebrew/homebrew-core/blob/master/Formula/clojure.rb in homebrew-core)#2021-03-1804:06andy.fingerhutI tried wiping homebrew off a Linux system where I had it before, and reinstall from scratch, and brew install clojure/tools/clojure did not install a JDK this time. Probably something left over from my previous brew installation#2021-03-1804:10Alex Miller (Clojure team)by default, brew will upgrade stuff, so if you had previously installed, it may have just upgraded#2021-03-1804:20seancorfieldbrew is annoying (because of the strong opinions of the core maintainer team) but it is also very convenient. I long resisted installing it on my Mac (and macports was also problematic) but now that I've given in -- because of Clojure! -- I'm happy to use it on my WSL2/Ubuntu setup as well to manage my Clojure CLI. So it's all your fault! 🙂#2021-03-1915:19cap10morganWe keep running into issues w/ pom.xml generation / deps-syncing via clojure -Spom combined with pom.xml also being the source of truth for things like project name and version. With tools like make, it's really easy for pom.xml's modified time to be newer than deps.edn's (b/c e.g. you just bumped the version number) but deps.edn still contains newer deps data that needs to be synced over. This seems like an example of the type of problem you run into when you have two sources of truth that need to be kept in sync with each other (or even just one way like it is here). Have other folks run into this? What are people doing to work around it reliably?#2021-03-1915:20cap10morganI get the impression that there is an aversion to putting something like project version in deps.edn but I wonder if something like a project.edn could make sense and then you generate everything in pom.xml (clobbering the whole file each time) from deps.edn and project.edn (or other similar approach that makes pom.xml just a representation of truth from other sources).#2021-03-1915:21Alex Miller (Clojure team)there is an aversion to doing more on this atm as it's going to be superseded by tools.build and other stuff, hopefully "soon"#2021-03-1915:21cap10morganah, cool. that rings a bell now that you mention it. 🙂#2021-03-1916:52seancorfield@cap10morgan Also, if you’re using depstar, you can just say :sync-pom true and it will always update the pom.xml for you. And I tend use :version '"x.y.z"' via depstar too for updating the pom.xml file.#2021-03-1916:53seancorfield(under the hood, depstar does both the equivalent of clojure -Spom — using the same parts of t.d.a. — and some other editing so that the SCM tag is also updated when you change the <version> tag)#2021-03-1916:53seancorfieldI documented my deployment process for my libraries in the depstar README recently.#2021-03-1918:05dcj@cap10morgan I share your concern about having more than one source of truth in a project. I feel that a pom.xml file ought to be able to be generated dynamically when needed, ideally from the contents of deps.edn and the SCM status of the project. Recent work by @seancorfield on depstar and my (currently unmerged) PR to deps-deploy enable me to specify enough in deps.edn to build a pom.xml dynamically when needed for jar creation and maven deployments. For example, here are the relevant portions of a deps.edn:
:mvn/repos {"acn-snapshots" {:url ""}}
:deps { ... }
:aliases
{:mvn/artifact-id util
:mvn/group-id org.aircraft-noise
:mvn/version "0.1.4-SNAPSHOT"
:jar/file-name "util.jar"
:jar {:replace-deps
{com.github.seancorfield/depstar {:mvn/version "2.0.206"}}
:exec-fn hf.depstar/jar
:exec-args {:jar :jar/file-name
:artifact-id :mvn/artifact-id
:group-id :mvn/group-id
:version :mvn/version
:sync-pom true}}
:deploy {:extra-deps {com.dcj/deps-deploy {:mvn/version "2.0.999-SNAPSHOT"}}
:exec-fn deps-deploy.deps-deploy/deploy
:exec-args {:installer :remote
:sign-releases? false
:artifact :jar/file-name
:repository "acn-snapshots"}}}
Note that the project version is defined as an alias, and then it is used by depstar to create the pom.xml and jar file.#2021-03-1920:38dcjFurther note that the value of :repository in the :exec-args of the :deploy alias is a string, which is used to find the actual respository location from the :mvn/repos map elsewhere in deps.edn, and then the credentials for that repo are obtained via ~/.m2/settings.xml#2021-03-2021:57borkdudeIn the future, will we be able to invoke -A:example for getting a REPL with extra deps from :example, even though it contains :main-opts as well?
{:aliases {:example {;; -A:example (for REPL invocation with extra deps)
:extra-deps {medley/medley {:mvn/version "1.3.0"}}
;; -X:example
:exec-fn clojure.core/prn
:exec-args {:my 1}
;; -M:example
:main-opts [-e "(+ 1 2 3)"]}}}
#2021-03-2022:02seancorfieldThat's what I've understood Alex to say about it, yes.#2021-03-2022:05seancorfieldBut I also got the impression there's likely to be more to it than just "not executing main opts".#2021-03-2113:29orestisI was happy to see that Netlify includes clojure as a default CLI tool but it seems like the version there doesn't include the recent additions of -M and -X which is annoying because now I get local warnings about -M ...#2021-03-2114:21borkdudeIs there a technical reason -m foo/bar doesn't work when you have (ns foo) (defn bar [& args])? Maybe the JVM background where -main is compiled to a Java main function?#2021-03-2114:23thhellerhttps://clojure.atlassian.net/browse/CLJ-2316#2021-03-2114:25thhellergiven the recent work on the deps.edn execute stuff I doubt this will ever make it in 😛#2021-03-2114:25borkdudeI want to support this in babashka, -f already means: execute file there.#2021-03-2114:26borkdudeIf there are no technical or philosophical objections, I will support this#2021-03-2114:26thhellergiven that many other tools already support using namespaced symbols in -m I'd say go for it#2021-03-2114:27borkdudethanks#2021-03-2114:22borkdudeBut this should probably not matter for :main-opts invocations which usually happen from source?#2021-03-2213:46imreShould gitlibs be nuked when upgrading from clojure tools 1.10.2.796 to 1.10.3.814?#2021-03-2213:58borkdudeIt's always safe to nuke those, so if in doubt, go ahead I would say#2021-03-2214:19imreI know it's safe 🙂 just wanted to know if it is necessary or recommended. I'd prefer to skip doing it on all the build agents in case it isn't needed#2021-03-2215:31seancorfieldI think if you went through the incremental prerelease builds you could get into a bad state but 814 fixed that. Or at least fixed part of that. I would upgrade to 814 and see if a build fails. It's pretty obvious if the gitlibs cache is broken.#2021-03-2217:39borkdudeI'm trying to get deps.clj up to date with the latest and greatest tools.deps / clojure bash. I tried coming up with a test that would have failed under the old clj and works with the newest clj with regards to the corfield comma / whitespace problem.
So I tried this:
clj -Srepro -Sdeps '{:aliases {:space {:jvm-opts ["-Dfoo=\"foo bar\""] }}}' -M:space -e "(prn (System/getProperty \"foo\"))"
The string prints as "\"foo bar\"".
1. Should it?
2. Is this a representative test?#2021-03-2218:51borkdudeYou can also do, deps.edn:
{:aliases {:space {:jvm-opts ["-Dfoo=foo bar"] }}}
Not sure if that's how it should be done. Just looking for a good test case.#2021-03-2218:53borkdudeBoth seemed to fail with 1.10.2.781#2021-03-2219:42borkdudeI think I caught up with 814 now. Thanks! duckie#2021-03-2221:13grzmGood afternoon, all! What’s the syntax for specifying multiple Java properties at the command line using -J For example, I want to specify a custom truststore using -.ssl.trustStore=/path/to/truststore and -.ssl.trustStorePassword=changeme. Space-separated like this?
clojure -J'-Djavax.net.ssl.trustStore=/path/to/truststore -Djavax.net.ssl.trustStorePassword=changeme' -M http-test.clj
#2021-03-2221:14thheller@grzm a -J for each I believe, -J-Dfoo -J-Dbar#2021-03-2313:15borkdudeNew version of deps.clj, a port of the clojure bash script to clojure:
https://github.com/borkdude/deps.clj/releases/tag/v0.0.12
- catch up with 1.10.3.814#2021-03-2316:06Lone Rangermessing around with this neat tool I found (https://gist.github.com/ericnormand/6bb4562c4bc578ef223182e3bb1e72c5) to make .clj files into scripts. Curious if it's possible to use either -X or -m or
something to cherry pick a function from the script.
#!/bin/sh
#_(
#_DEPS is same format as deps.edn. Multiline is okay.
DEPS='
{:deps {}}
'
#_You can put other options here
OPTS='
-J-Xms256m -J-Xmx256m -J-client
'
exec clojure $OPTS -A:strapped -Sdeps "$DEPS" "$0" "
#2021-03-2316:07Lone RangerI've tried ./test.clj -X test, ./test.clj -M -m test, ./test.clj -m, ./test.clj -m test, ./test.clj -M, ./test.clj -M -m , ./test -X test/test#2021-03-2316:14seancorfield@goomba Since it invokes the script directly via the clojure command, you cannot use other main- or exec- options with it, as it stands.#2021-03-2316:15Lone Rangerohhhh#2021-03-2316:15Lone Rangerwould require clj?#2021-03-2316:15seancorfieldNo, clj is just a wrapper around clojure that adds rlwrap.#2021-03-2316:15seancorfield(`clj`/`clojure` are otherwise interchangeable)#2021-03-2316:15Lone Rangerah#2021-03-2316:18seancorfieldYou’d need to have the directory containing the script as a :local/root dependency and then you could invoke functions in it via -X but that’s a bit hacky.#2021-03-2316:18Lone RangerI'm also a bit hacky haha#2021-03-2316:25seancorfield@goomba Something like:
(! 962)-> ./unbalanced.clj -X unbalanced/test :foo '"bar"'
WARNING: test already refers to: #'clojure.core/test in namespace: unbalanced, being replaced by: #'unbalanced/test
[+] test successful {:foo bar}
(! 963)-> cat unbalanced.clj
#!/bin/sh
#_(
#_DEPS is same format as deps.edn. Multiline is okay.
DEPS='
{:deps {} :paths ["."]}
'
#_You can put other options here
OPTS='
-J-Xms256m -J-Xmx256m -J-client
'
exec clojure $OPTS -Sdeps "$DEPS" "#2021-03-2316:25seancorfield:paths should really be dynamically constructed to be the folder for the script.#2021-03-2316:26seancorfieldNotes: needs a ns form for -X invocation (so it has a defined namespace); removed "$0" since we don’t want to execute the script directly; added an arg to test so it can be called via -X.#2021-03-2316:33Lone RangerI must be missing an assumption#2021-03-2316:33Lone Ranger#2021-03-2316:34Lone Rangerusing
:deps {
org.clojure/clojure {:mvn/version "1.10.2-alpha4"}
}
in my ~/.clojure/deps.edn#2021-03-2316:34Lone Rangerdoes it require a deps.edn in the same dir?#2021-03-2316:37seancorfieldWhat is your clojure version? clojure -Sdescribe#2021-03-2316:37Lone Rangeroh, fascinating#2021-03-2316:37Lone Ranger{:version "1.10.0.411"
:config-files ["/usr/local/lib/clojure/deps.edn" "/home/jay/.clojure/deps.edn" "deps.edn" ]
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/jay/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
#2021-03-2316:38seancorfieldThat is very old (and does not support -X). The current version is 1.10.3.814#2021-03-2316:38Lone Rangerwow, great catch, had no idea. I thought my ~/.clojure/deps.edn would take precedence, but I suppose not 😮#2021-03-2316:39seancorfieldNot sure what you’re asking about. That’s the user deps.edn and, yes, it factors in.#2021-03-2316:39seancorfield(unless you pass -Srepro)#2021-03-2316:40Lone Rangerthis file /usr/local/lib/clojure/deps.edn has a different clojure version than my ~/.clojure/deps.edn , I think. Sorry for confusiion#2021-03-2316:41dpsuttonits not the version of the jvm hosted language that's an issue. its the version of the command line program clojure which assembles a classpath and calls an entrypoint. That command line clojure is too old to understand the -X command line argument#2021-03-2316:41Lone Rangerahhhh#2021-03-2316:42Lone Rangertime to upgrade!#2021-03-2316:43seancorfieldClojure CLI version != Clojure language version.#2021-03-2316:43Lone Rangerworks like a charm!!!!!!!#2021-03-2316:43Lone Rangergo team!!!! 🔥 🎆#2021-03-2316:43Lone Rangerthank you!!!#2021-03-2316:44seancorfieldThe prefix portion of the CLI version indicates the default version of Clojure it would use (1.10.0, 1.10.1, 1.10.2, 1.10.3) and the last segment is the commit count for that release.#2021-03-2316:45Lone Rangeraha! working great now 🙏#2021-03-2319:21borkdudeIs there any precedent of tools that need tools.deps.alpha for downloading deps and calc classpath that also put their non-deps-related tool config in deps.edn?
E.g. shadow-cljs has a config file which allows :deps true to resolve deps from deps.edn, but it does not store its own config in deps.edn. Are there any tools who do? Is this encouraged / not recommended / neutral?#2021-03-2321:14Alex Miller (Clojure team)it is discouraged to add top-level keys to deps.edn#2021-03-2321:15Alex Miller (Clojure team)doing so could break in the future, and would be unsupported#2021-03-2321:15Alex Miller (Clojure team)putting named data in :aliases is ok#2021-03-2321:15borkdudeunderstood. and what about an alias convention, e.g. tool X always reads from :aliases :tool-x ?#2021-03-2321:17Alex Miller (Clojure team)why not use a ns?#2021-03-2319:28seancorfield@borkdude My feeling is that given the alias-as-data thing that has crept into deps.edn, we will start to see tooling be passed one or more keyword-valued exec arguments and it will be common for the tool to then lookup those keywords as aliases in the project basis and use the data structures those aliases point to. depstar already does this, deps-deploy has an open PR to add that.#2021-03-2319:29seancorfieldThat allows users to name additional tool config in deps.edn however they want and then communicate that name to the tool via a keyword-valued exec arg.#2021-03-2319:30borkdudeok, but depstar is a dependency-centric tool, so it's more obvious there probably#2021-03-2319:30seancorfieldYou can also specify a keyword for the whole :exec-args thing and that is looked up as an alias by the CLI -X machinery.#2021-03-2319:32seancorfieldI think any tooling that you run via CLI can reasonably assume it could look up aliases in the basis to get configuration data.#2021-03-2319:32seancorfieldBut I strongly believe it should be via an exec arg passing a keyword and not a hard-code alias name.#2021-03-2319:32borkdudewhat if the tool isn't invoked via the CLI but maybe via a binary.#2021-03-2319:33seancorfieldIf you pass the tool a keyword argument that tells it what alias in the basis to use, I think that’s reasonable.#2021-03-2320:29borkdudeOn the other hand, if all kinds of tools start saving their config in deps.edn, this file could become pretty big and tedious to parse on every tool invocation#2021-03-2321:06seancorfieldI think that depends on just how much config various tools have. Having some sort of “build” config in deps.edn seems reasonable but maybe a tool that has a very open-minded config (clj-kondo) should not use deps.edn for that and should keep it separate?#2021-03-2321:08borkdudeIn this case I'm thinking of a task runner (like make, npm scripts, just, etc) in babashka that you configure in EDN. It is not being invoked through the clojure CLI but it can use tools.deps.alpha for itself. This will capture commonly invoked commands, including invocations of the clojure CLI itself, so you don't have memorize those or write bash scripts to wrap them.#2021-03-2321:10borkdudeSo it leans on tools.deps.alpha but I don't think it will read from a basis file using an alias name for example.#2021-03-2321:14borkdudeBut clj-kondo is also a good example: clj-kondo could read from your deps.edn, but it doesn't necessarily belong inside deps.edn#2021-03-2321:16Alex Miller (Clojure team)alias data good, novel top-level keys bad#2021-03-2321:16Alex Miller (Clojure team)alias data could be the name of a config edn file to read if you want to centralize definition but offload the contents#2021-03-2321:16borkdudeyeah, I think it's better to split it#2021-03-2321:17borkdudeI think I'll adopt the way shadow-cljs did it#2021-03-2321:27thhellerFWIW deps.edn didn't exist when I started with shadow-cljs.edn but I would make the same choice again. I prefer having targeted configuration files over the all-in-one gigantic configs#2021-03-2321:46seancorfieldGlad to have Alex confirm what I was saying there 🙂#2021-03-2321:55borkdudeFWIW I wasn't planning on adding top level keys to deps.edn, but if I went the alias way it would probably have been a fully qualified keyword unique to the tool. But I agree with thheller's remark. Thanks for the useful exchange.#2021-03-2413:11Ronny LøvtangenHi! Paths can be an alias name that refer to a path vector in the alias data:
{:paths [:clj-paths :resource-paths]
:aliases
{:clj-paths ["src/clj" "src/cljc"]
:resource-paths ["resources"]}}
Is there some similar support for deps (:deps, :extra-deps etc)?#2021-03-2416:42Alex Miller (Clojure team)no#2021-03-2416:42Alex Miller (Clojure team)maybe some day#2021-03-2414:05delaguardoNew release of setup-clojure github action - https://github.com/DeLaGuardo/setup-clojure/releases/tag/3.2
it comes with windows support for clojure cli#2021-03-2416:46seancorfield@delaguardo Is the recommendation still to reference that @master to always get the latest?#2021-03-2416:48delaguardoI have stopped suggesting that because github recently introduce dependabot alerts about new releases for github actions.#2021-03-2416:51seancorfieldAh, good to know. I think antq has started picking up Clojure CLI versions in GH actions too recently…#2021-03-2416:51nnicholsiirc, antq and transitively clojure-dependency-update-action report out of date GitHub actions too#2021-03-2511:11imreI found that ~/.gitlibs/_repos/<repo-path>/worktrees/<sha>/gitdir files contain absolute paths to subfolders under ~/.gitlibs/libs/. This seems to result in conflicts when I want to share my ~/.gitlibs between my physical machine (mac) where I conduct most of my dev work and a docker container (linux, different user etc.) which is necessary for a special project I work on.
Is there any nice way around this? For now I just let git deps be downloaded every time the container starts. Perhaps I can set up another local gitlibs folder which would be container-only?#2021-03-2512:01imreI might have just rubber duckie -ed myself using this channel. I can mount different, image/project-specific folders into ~/.gitlibs and .cpcache#2021-03-2512:43Alex Miller (Clojure team)You can set the GITLIBS env var to say where the root is#2021-03-2512:44imreThat's good to know, thank you.#2021-03-2512:48imreThere isn't one to override the location of .cpcache though, am I right?#2021-03-2512:48imrecpcache also has absolute paths#2021-03-2610:10rickmoynihanI just ran into a small issue with org.clojure/tools.namespace, where when used in a deps.edn project would over-zealously by default load namespaces found in ~/.gitlibs. Presumably because prior to tools.deps it was assumed deps found on the filesystem were most likely to be source code in your project and suitable for loading.
Would it make sense for tns to now by default ignore any paths found in ~/.gitlibs?#2021-03-2610:25Alex Miller (Clojure team)I don’t understand - can you explain a repro in an ask.clojure question?#2021-03-2614:47rickmoynihanhttps://ask.clojure.org/index.php/10381/should-tools-namespace-repl-refresh-exclude-found-gitlibs#2021-03-2610:11rickmoynihanIncidentally this is easily fixed by explicitly setting tnsrepl/set-refresh-dirs, but it did cause a colleague to get a very misleading stacktrace about a missing class from a namespace that his project wasn’t explicitly requiring.#2021-03-2610:13rickmoynihanI should add that they likely weren’t even fully aware they were using tns, because they were using integrant.repl/reset (which uses it)#2021-03-2616:46seancorfieldCognitect’s test-runner seems to be accumulating issues and PRs without getting any feedback from Cognitect. Is it just considered “done” or just very low priority right now I wonder? I was thinking it would be really nice to have a -X entry point so that folks could leverage :exec-args and, in particular, their ability to merge across aliases so you could specify generic options via your :test alias (or :runner alias, in my case) and then specific options in other aliases. This would clean up a bit of boilerplate in our monorepo scenario where subprojects all have aliases but we still need to specify --dir via the command-line (since :main-opts don’t merge — just last-one-wins).#2021-03-2616:48seancorfieldLuckily the test function in cognitect.test-runner is public so writing a tiny :exec-fn wrapper outside the runner is easy to do for now.#2021-03-2617:01Alex Miller (Clojure team)Just waiting for a cycle of attention#2021-03-2617:02seancorfieldI’ll write the wrapper in our codebase for now and maybe submit a PR if/when test-runner gets an attention cycle.#2021-03-2617:43Alex Miller (Clojure team)Go for it!#2021-03-2618:30borkdudeThe test runner is working well for me#2021-03-2618:32borkdude@seancorfield I'd love to see an example invocation that is "better" with -X than with the -main function#2021-03-2618:34seancorfieldWe have a test alias for each subproject that brings in the test code and dependencies for that subproject. Via -X we can specify :exec-args {:dir #{"subproject/test"}} which will merge with our :runner alias which already has :exec-fn and the default :exec-args.#2021-03-2618:35seancorfieldWith -M, :main-opts do not merge so we either have to repeat the whole :main-opts for every subproject -test alias with just -d changed, or we have to specify -d on the command-line.#2021-03-2618:36borkdudeso now you have to write another alias for every -d change?#2021-03-2618:36seancorfieldBefore:
$ clojure -M:defaults:subproj:subproj-test:test:runner -d subproj/test
After:
$ clojure -X:defaults:subproj:subproj-test:test:runner
#2021-03-2618:37borkdude@seancorfield Do you store these long invocations in some separate bash script or Makefile? Or do you remember them from the top of your head?#2021-03-2618:38borkdudeI'm currently considering a task runner which is supposed to solve this problem (of not having to remember these things)#2021-03-2618:38borkdudeThis is why I am curious about these use cases#2021-03-2618:38seancorfieldWe have a build shell script that turns build test subproj into the above command. But now that we have a primary deps.edn file at the top of our monorepo, it’s easy enough to run most things with the CLI directly (for single command invocation).#2021-03-2618:39seancorfieldThe build script supports multiple commands so we could say build tests subproj1 subproj2 and it runs two clojure commands.#2021-03-2618:40borkdudeThe example after -X is still pretty long, too long for me to remember without a bash history probably#2021-03-2619:46hiredmanWhy would you ever not have a history?#2021-03-2619:47borkdudeThat can happen when I switch systems yes, I have multiple laptops. But even then, fetching from history comes up with similar invocations and not always the one I'm looking for right away, unless I remember it more or less correctly. Hence, something like a task runner (something like make) could be nice#2021-03-2619:49borkdudeBut also for making these invocations known to colleagues, contributors, etc, you don't want to depend on history#2021-03-2620:03hiredmanmaybe; I am skeptical of lot of this additional tooling stuff. shells already support most of these things#2021-03-2620:03hiredmaneverything goes in the history#2021-03-2620:03hiredmansupport running all kinds of things, background jobs, etc#2021-03-2620:04borkdudeyou don't write any docs for your colleagues how to invoke the tests like Sean showed? I would have a hard time figuring that out by only looking at deps.edn#2021-03-2620:04hiredmanit is already the case that I almost never type a command in full into my shell, I pull it out of history and maybe edit it#2021-03-2620:04hiredmanif you write docs then you already may as well just share the long command#2021-03-2620:05hiredman"paste this into your shell, and it will run the tests"#2021-03-2620:05borkdudeif you do document this, you might as well make those docs executable somehow#2021-03-2620:05borkdudeyeah, that's usually how I've done it so far#2021-03-2620:05hiredmanI mean, I work with sean#2021-03-2620:07borkdudeLucky you ;)#2021-03-2620:08hiredmanyeah, I mean, he does a lot of work on the tooling, so documentation wise it is way more likely to be written for me then by me :)#2021-03-2720:44borkdudeJust as an example:
https://github.com/borkdude/antq/blob/bb.edn/bb.edn
(ported from this Makefile: https://github.com/borkdude/antq/blob/bb.edn/Makefile)#2021-03-2723:27hiredmanThis makefile is bad#2021-03-2723:28hiredmanLike, the whole deal about makefiles is it tracks dependencies between files and has tasks that generate files#2021-03-2723:29hiredmanThis make file has a target named 'pom' that is used to generate the file pom.xml#2021-03-2723:30hiredmanWhich has no dependencies (like maybe on deps.edn?)#2021-03-2723:30hiredmanGross#2021-03-2723:35hiredmanThis task dsl is gross too. To impose a restrictive dsl without getting anything out of it (like dependency tracking) is silly. If you aren't doing extra stuff (dependency tracking, staleness tracking, rebuilding, etc) the just use normal clojure functions#2021-03-2723:36hiredmanYou want give some bit of functionality a name, and call it, we have defn for that#2021-03-2723:40borkdudeThis isn't my makefile btw, I just scouted some makefiles in the wild, and what I mostly see is that people just use it as a way to quickly invoke it from the command line. It's clear that you don't see the value of this, that was already clear in our earlier conversation. Thanks for the feedback.#2021-03-2723:41borkdudeI do want to consider tracking, etc, like make does, it's done done yet#2021-03-2810:27borkdudeNow ported the tasks to normal functions:
https://github.com/borkdude/antq/blob/bb.edn/script/tasks.clj
The code longer, but more flexible, so that may the way to go indeed.
Tasks can be discovered using (maybe warrants a bb dir CLI option) :
$ bb -e "(require 'tasks) (clojure.repl/dir tasks)"
clean
coverage
deploy
docker
docker-test
install
jar
jar-file
lint
outdated
pom
repl
standalone-jar-file
tests
uberjar
Tasks can be invoked using:
bb -m runner/coverage
Docs:
$ bb doc tasks/coverage
-------------------------
tasks/coverage
([])
Run test coverage.
#2021-03-2618:40seancorfieldWe’re not going to invest more work in pipelining this stuff until we’ve seen tools.build released.#2021-03-2618:41seancorfieldI’m anticipating being able to replace our build shell script with a tools.build invocation 🤞:skin-tone-2: 😸#2021-03-2618:42borkdudeI don't know what tools.build is, nor spec2 ;)#2021-03-2618:43borkdudeBut I hope they come out soon ;) 🔮#2021-03-2618:47seancorfieldBased on what Alex has said publicly about tools.build, I would expect it to include “task runner” functionality (and some “standard” tasks). ISTR he’s already mentioned that it will add Java compilation as a task?#2021-03-2710:16dominicmhttps://github.com/juxt/edge/issues/132 ran into this on the latest cli. I can't figure out how I repro'd it initially, but I cannot now. Bit confused. Putting it in the "record" in cae it comes up again.#2021-03-2716:09seancorfield@dominicm Were you testing against prerelease versions at any time recently? There were some glitches in the new git implementation that could bork local ~/.gitlibs in some situations. Several of us testing against those versions had to rm -rf ~/.gitlibs` to fix it.#2021-03-2716:11seancorfield(I think everyone who went from stable to stable, without testing any prerelease versions, was OK?)#2021-03-2716:19dominicm@seancorfield nope, I went straight to 814.#2021-03-2718:26Alex Miller (Clojure team)It was possible to get this on the old stable, I think due to concurrent downloads (I never reproed it)#2021-03-2718:26Alex Miller (Clojure team)I believe it should be harder for that to occur in new impl#2021-03-2720:17dominicmI'm sure you'll be glad when you find a stable repro :D#2021-03-2721:22Alex Miller (Clojure team)I did try to make it happen in the new version and could not#2021-03-2809:20pinkfrogis there a deps.local.edn which is not tracked by git but allows for per-project customization?#2021-03-2809:36borkdudeI don't believe so#2021-03-2810:07pinkfrogThen how do you customize the project setting without hindering other developers?#2021-03-2810:16borkdudewhat project setting would you customize?#2021-03-2810:18borkdudeif you mean personal developer tools, you can put those in your ~/.clojure/deps.edn#2021-03-2810:44pinkfroglike project specific different alias#2021-03-2810:49borkdudecan you be more specific? why don't you just add your alias to the deps.edn?#2021-03-2812:00pinkfrogbecause that file is tracked in git. and shared by other devs.#2021-03-2812:03borkdudedo your colleagues mind if you put a personal fully qualified alias in there?#2021-03-2813:17favilaThere is not, but this would be really useful for monorepos#2021-03-2813:20favilaThere’s a tools deps lib function that merges deps maps (same thing used to merge personal +project deps.edn). We generate the project deps.edn by calling that on the repo + project deps edn with make #2021-03-2814:06borkdudewe do the same using a babashka script, but the resulting deps.edn lives in our git, so it doesn't contain "personal" config let's say#2021-03-2818:26seancorfieldAfter pushing for an extra deps file for a long time for monorepo use, I eventually realized we could do everything we needed by approaching the structure a bit differently: https://corfield.org/blog/2021/02/23/deps-edn-monorepo/#2021-03-2914:48erwinrooijakkersIs there a way to automatically upgrade dependencies, like lein ancient upgrade for Leiningen?#2021-03-2914:51nnicholsantq is the most actively supported tool for this:
https://github.com/liquidz/antq#--upgrade for the cli form
https://github.com/nnichols/clojure-dependency-update-action for a github action form#2021-03-2919:24practicalli-johnhttps://github.com/practicalli/clojure-deps-edn#common-development-tasks includes a range of common tools for Clojure CLI#2021-03-3014:20gphilippWhat is the recommended approach to override transitive dependencies, a bit like Leiningen’s :managed-dependencies? From what I’ve seen, you cannot use :override-deps at the top level alongside :deps, it has to live under an alias, which is a bit cumbersome because people now have to remember to add the alias for all clj commands. The other option is to add the override version directly to the :deps vector but I’m not fond of that because it now looks like you project depends on it directly.#2021-03-3014:24borkdude@gphilipp I would go for the last option, because in that case you are stating the truth: your app DOES depend on that specific version to function correctly#2021-03-3014:24borkdudeyou could add a comment to it to explain why you added it#2021-03-3014:27gphilippYeah, this is what I do currently (adding an explicit comment). The other downside that I forgot to mention is that it lifts the dependency to the top of the deps tree (`clj -Stree)` and you don’t have the entry anymore under the lib that pulled that dep initially.#2021-03-3014:27Alex Miller (Clojure team)usually I would add an :exclusions and add the dep in :deps#2021-03-3014:30gphilippI usually skip the :exclusion part due to the way tools.deps resolve dependency conflicts (usually we want the higher version). But now that I’m think of it, IIRC it doesn’t work across maven repositories or across different vendors of the same GAV, am I right?#2021-03-3014:34Alex Miller (Clojure team)not sure what you mean#2021-03-3014:34Alex Miller (Clojure team)those seem like orthogonal concerns#2021-03-3015:07gphilippI’ll write down a small example to clarify. #2021-03-3021:34JasonIf one were to add keys to a deps maps like so:
{:extra-deps {thheller/shadow-cljs {:mvn/version "2.11.22"
:bazel/hash "hash goes here"} ; <= new map entry
1. would anything break, and
2. is there a way to resolve them along with the maven coordinates?#2021-03-3021:43seancorfield@jasonhlogic There’s a “procurer” mechanism built into tools.deps.alpha that could leverage that but it is not currently exposed as an extension point. @alexmiller might have more to say about that.#2021-03-3021:45Alex Miller (Clojure team)in this particular case, you're adding keys to an existing procurer coordinate and that code may not pull them along (although generally the existing code does)#2021-03-3021:46Alex Miller (Clojure team)if you are making your own custom procurer (some qualifier other than :mvn), it is possible to install that and have tools.deps use it and do anything you like#2021-03-3021:47Alex Miller (Clojure team)installing custom procurer extensions is not currently possible when using the clj CLI (no one has needed it yet) but that is possible#2021-03-3022:07JasonIn theory, what I would like to do is let maven do the procuring as it does now, but then to get a link to the local jar and the value of the key i have added so an external tool can hash the jar and compare for its own purposes. (the external tool, surprise, is <Https://bazel.build|Bazel>).
My wishcast is that some magical function would appear which
1. resolves deps
2. downloads the artifact via whatever procuring mechanism is in place
3. adds the url of the downloaded artifact to the map I added the new key to above
4. returns the new map
Given that resolve-deps exists, 1 and 2 are covered, so I guess I'm asking whether there is a way to know the local url of each procured artifact per dep for reasoning purposes 🙂#2021-03-3022:11seancorfieldThe computed project basis would give you the path to the actual JAR (as well as all the rest of that data — although as Alex says I don’t think you can guarantee that your additional keys will always get passed through).#2021-03-3022:11seancorfieldTake a look at how depstar does it: https://github.com/seancorfield/depstar/blob/develop/src/hf/depstar/uberjar.clj#L314-L333#2021-03-3022:12seancorfieldIf you play with that in the REPL, you should be able to find the data you are after and verify whether your keys survive the round-trip too.#2021-03-3022:12seancorfield(there’s very similar code in t.d.a itself — I just knew I could find this code faster)#2021-03-3022:12Alex Miller (Clojure team)there actually is an api method for answering this question#2021-03-3022:16JasonFantastic. thanks very much, will investigate.
Confirmed that nothing breaks when I add the extra key, so there's that.#2021-03-3023:09Alex Miller (Clojure team)there are no future guarantees about that - you are firmly in "undefined" land#2021-03-3023:10Alex Miller (Clojure team)re "I'm asking whether there is a way to know the local url of each procured artifact per dep", see https://clojure.github.io/tools.deps.alpha/clojure.tools.deps.alpha-api.html#clojure.tools.deps.alpha/lib-location#2021-03-3023:12Alex Miller (Clojure team)and just fyi, Maven will also download the sha1 hash of the jar if it's in the maven repo (which it will be for things in Maven central, but not necessarily for clojars or other maven repos)#2021-03-3100:16Jasonthanks for looking that up. much appreciated#2021-03-3102:07Alex Miller (Clojure team)oh, I didn't look it up, I wrote it :)#2021-03-3112:24pavlosmelissinosI just tried using :replace-deps without specifying org.clojure/clojure and it seems like clj -A:some-alias tries to bring in the top-level deps as well (including the private git repo).
Is this behaviour intended/known?
:deps {org.clojure/clojure {:mvn/version "1.10.1"}
some-private-github-repo {:git/url <git url>
:sha <some-sha>}
:aliases {:some-alias {:replace-deps {org.clojure/tools.gitlibs {:mvn/version "0.2.64"}}
I don't have an clean repro case but I can try to make one if necessary.
This is from a CI job that uses the circleci/clojure:openjdk-11-tools-deps image. Not sure which version of tools-deps that is but it shouldn't be too far behind...#2021-03-3112:28pavlosmelissinosThe thing about the private git repo is not irrelevant; I might have a follow up question about that...#2021-03-3112:54Alex Miller (Clojure team)would be helpful to know the clj version (either with clj -Sdescribe or clj --version if it's new enough#2021-03-3112:55Alex Miller (Clojure team)if you happen to be on a version before :replace-deps existed, then what you describe would make sense#2021-03-3112:59pavlosmelissinosI think it's >1.10.2 but I'll confirm in a bit and get back to you, thanks 🙂#2021-03-3113:00Alex Miller (Clojure team)assuming it's new enough to have :replace-deps, don't know, would need a repro#2021-04-0109:57kirill.salykinis it possible to include some java.class with deps.edn? I found example with .jar , but is it possible to include just one .class file?#2021-04-0109:58thhelleryou can take it out of the .jar and just put in some path you add via :paths#2021-04-0110:05kirill.salykinshould I specify the full path to a class file?
eg
:paths ["lib/some.class"]
or to root?
:patsh ["lib"]
#2021-04-0110:05kirill.salykinthanks#2021-04-0110:14thhellerthe class needs to have the same path that is has in the jar#2021-04-0110:14thhellerso foo/bar/thing.class you put into lib/foo/bar/thing.class with :paths ["lib"]#2021-04-0110:15kirill.salykinthanks
giving it a try#2021-04-0110:22kirill.salykinthanks a lot, works#2021-04-0110:00kirill.salykinnice, thanks!#2021-04-0117:13seancorfield@alexmiller I’m running into that hard-to-repro failure with git deps and I’m beginning to think it has to do with :deps/root. I had the following in my deps.edn aliases:
:poly {:main-opts ["-m" "polylith.clj.core.poly-cli.core"]
:extra-deps {polyfy/polylith
{:git/url ""
:sha "2d89676724e98d4c1b858c28c452cf2a64fc1b1c"
:deps/root "projects/poly"}}}
I updated the SHA to 499401d4d2c5ac206f00a35c2cd8e569e4e6c3a8 and when I re-ran that alias I got
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate polylith/clj/core/poly_cli/core__init.class, polylith/clj/core/poly_cli/core.clj or polylith/clj/core/poly_cli/core.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.
It isn’t happening every time, but after nuking ~/.gitlibs it re-cloned and things worked. Then I updated the SHA to 1ae83891e5910fd1c2e130527a0571419ff3d257 and it failed again. Nuking ~/.gitlibs solved it again.#2021-04-0117:14seancorfieldAnd this is purely on
Clojure CLI version 1.10.3.814
#2021-04-0117:18seancorfieldAlso, something that seemed very strange to me: I checked out that repo locally and switched the (working) :git/url to a :local/root pointing to the folder where I checked it out. I verified the HEAD SHA in the local checkout matched the :sha I had been using, and commented out the :git/url and :sha keys — leaving just :local/root and :deps/root — and that fails with
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate polylith/clj/core/poly_cli/core__init.class, polylith/clj/core/poly_cli/core.clj or polylith/clj/core/poly_cli/core.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.
I would have expected :local/root / :deps/root and :git/url / :sha / :deps/root to behave identically if the local checkout is at the same SHA. Am I misunderstanding something about local root deps?#2021-04-0117:20seancorfield(switching the :local/root to point to <repo>/projects/poly directly works — I thought :deps/root worked with :local/root but I guess it doesn’t?)#2021-04-0117:23Alex Miller (Clojure team)when this happens, can you check the sha dir? I assume it's empty#2021-04-0117:24Alex Miller (Clojure team)local/root to the sha dir in gitlibs should be comparable (both from that point use the identical code to read the manifest, deps, paths etc), although there could be differences in dir resolution related to absolute/relative#2021-04-0117:26Alex Miller (Clojure team)if you get into the first scenario, and can save off your gitlibs dir, would be interested in looking at it more#2021-04-0117:27Alex Miller (Clojure team)in particular, I'm curious about the timeline of the sha update, whether it's a case like:
• use git dep at sha X
• project pushes new sha
• you switch git dep to new sha Y (not included in the original clone when getting X)
#2021-04-0117:28Alex Miller (Clojure team)in that case, it should try to resolve Y, fail, fetch to update the git dir, then re-resolve (successfully)#2021-04-0117:29Alex Miller (Clojure team)also, there are known issues with gitlibs and git submodules and that doesn't work right now (I doubt you're doing that but just in case)#2021-04-0117:57seancorfieldAh, maybe because these SHAs are on a branch, not on the default?#2021-04-0118:03seancorfield(and, yes, the sha dir was empty)#2021-04-0118:04seancorfieldAnd also, yes, this is when I’ve been working against a SHA (on a non-default branch) and the maintainer pushes an update on that branch and I switch to the newer SHA on that branch.#2021-04-0118:24Alex Miller (Clojure team)branch should be unimportant#2021-04-0118:25Alex Miller (Clojure team)next time you do it, you can also set GITLIBS_DEBUG=true and it will print all the git commands it's running - you should see the fetch happen#2021-04-0223:17seancorfieldDarn! I forgot to add that var when I updated again. It broke again:
Error building classpath. Manifest type not detected when finding deps for polyfy/polylith in coordinate {:git/url "", :sha "783916018b20a76f9054af8086e09e8f3335a3ae", :deps/root "projects/poly"}#2021-04-0223:17seancorfieldI saved the poly-related ~/.gitlibs pieces for you in a tar file after it broke.#2021-04-0223:20seancorfield(! 1176)-> ls libs/polyfy/polylith/
1ae83891e5910fd1c2e130527a0571419ff3d257/
2d89676724e98d4c1b858c28c452cf2a64fc1b1c/
Those are the two earlier SHAs. There’s no corresponding folder for the new SHA (above).#2021-04-0223:37seancorfieldPretty sure I’ll get another chance at this, maybe tomorrow.#2021-04-0304:02seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "bbc8dedad5d8fbfffee8f0eff4eb96888bf2f8fa8c8d"}, :content ("[email protected]")}#2021-04-0312:04Alex Miller (Clojure team)thx, that's exactly what I needed and the fact that it was on a branch was important!#2021-04-0312:05Alex Miller (Clojure team)new prerelease of clj 1.10.3.822 available for you to test - you will need to rm your ~/.gitlibs (or at least the _repos) to have the fix take affect as it is both a change in the initial clone and a change in the working tree checkout#2021-04-0319:52seancorfieldThanks. I'll update both systems now and blow away ~/.gitlibs. I'll let you know if I run into any more issues.#2021-04-0518:16seancorfieldHere’s the debug output when I updated my SHA this morning (and it worked — thank you!):
(! 997)-> GITLIBS_DEBUG=true clojure -M:poly libs
git --git-dir /Users/sean/.gitlibs/_repos/github.com/polyfy/polylith rev-parse 65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8^{commit}
fatal: ambiguous argument '65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8^{commit}': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
git --git-dir /Users/sean/.gitlibs/_repos/github.com/polyfy/polylith fetch --quiet --all --tags --prune --prune-tags
git --git-dir /Users/sean/.gitlibs/_repos/github.com/polyfy/polylith rev-parse 65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8^{commit}
Checking out: at 65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8
git --git-dir /Users/sean/.gitlibs/_repos/github.com/polyfy/polylith worktree add --force --detach /Users/sean/.gitlibs/libs/polyfy/polylith/65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8 65a7918bb6de886cda7a9b8a5fe7dbdf10996bf8#2021-04-0518:17seancorfieldAlthough that fatal error sounds like a bug?#2021-04-0518:43Alex Miller (Clojure team)No, that’s expected#2021-04-0518:43Alex Miller (Clojure team)It’s trying to resolve the commit and when it fails, that’s what triggers the fetch#2021-04-0518:45Alex Miller (Clojure team)Avoiding the unnecessary fetch when it can be resolved is a big help perf wise#2021-04-0519:10seancorfieldGotcha. Better to try and fail to make the happy path faster.#2021-04-0118:28seancorfieldOK, will try that. Thanks.#2021-04-0118:34Alex Miller (Clojure team)I did run tests around scenarios like this, and I believe it should be working, but it seems like the problem area#2021-04-0118:34seancorfieldI should get a chance to test it later today I expect 🙂#2021-04-0119:27Alex Miller (Clojure team)this was broken prior to 814, and old prerelease stuff in your gitlibs dir would maybe fail if you had that#2021-04-0119:30seancorfieldI completely blew away ~/.gitlibs when I updated to 814 (and I’ve actually blown it away a couple of times since).#2021-04-0213:28jjttjjI tweeted at the https://mvnrepository.com maintainer requesting support for the deps.edn syntax, if anyone want's to ❤️ it 🙂 https://twitter.com/jjttjj/status/1377975687705935878 (not sure if anyone's tried contacting him before)#2021-04-0215:26raspasovJust did.#2021-04-0301:42souenzzoWhere we can find the https://mvnrepository.com src?!#2021-04-0315:20mokrHi, is there any plans to allow some sugar (short names / shortcuts) that would allow one to hide the sometimes long, not so sexy, commands behind a short name? Like Leiningen does with “aliases” (!= deps aliases)?
So that e.g. clj graph could execute clj -X:graph graph :deps '"mydeps.edn"' :trace true :output '"trace"
I know tools deps is not intended to replace Leiningen, but it seem to have fragmented the already small Clojure(script) community and intended or not, my impression is that it’s used as a replacement as is, with no other tool wrapping it.
I do not mean to be rude and I’m very grateful for all the hard work that Cognitect and others puts into the tech I benefit from. I’m just a bit concerned in regards to how it looks to outsiders and newcomers and that you now need to remember, wrap in a script or copy paste long commands to get things done. Sure you can still use lein, but enough people seem to have transitioned that it’s pretty much mandatory to know both.
PS: If I missed something obvious here, then please enlighten me, nothing would be more welcome.#2021-04-0315:25vlaaadSounds like aliases with :exec-fn and :exec-args#2021-04-0315:45andy.fingerhutI believe it is semi-common to create short bash scripts and/or Makefile kind of things for this, too.#2021-04-0315:55Alex Miller (Clojure team)the best place to file requests is at https://ask.clojure.org (and this is not something that has an existing question specifically for it already afaik) - asking there lets this accumulate votes, which we factor into prioritization#2021-04-0315:58Alex Miller (Clojure team)as @U47G49KHQ says, you can get part of the way there by creating an alias with :exec-fn and :exec-args keys and then invoking the alias. for this example, you'd create an alias like:
:trace {:exec-fn graph
:exec-args {:deps "mydeps.edn"
:trace true
:output "trace"}}
and then execute this in combination with the graph tool like:
clj -X:trace:graph
#2021-04-0315:58Alex Miller (Clojure team)or you could fold the :graph alias stuff into that same alias to get down to just clj -X:trace#2021-04-0315:59Alex Miller (Clojure team)there is also some additional work coming soon around tool installation#2021-04-0316:40mokrThanks, I knew this was close to a feature request, but I wanted to see the reaction to it 🙂
For some reason I keep forgetting that http://ask.clojure.org exists. If that’s the desired channel, then I will consider writing up something around this. Clojure itself and many of the libraries are very elegant and concise and I have the same hopes for the tooling.#2021-04-0317:14borkdude@U1S4F3M4M I am making something like "npm scripts" for babashka:
https://github.com/babashka/babashka/discussions/765#discussioncomment-564820
Feel free to give feedback#2021-04-0317:15borkdudeIt's already possibly today to "hide" this behind a function and call it like bb -m tasks/my-clojure-invocation#2021-04-0317:17borkdudelike this: https://github.com/djblue/portal/commit/39144dbe520783e18ec13440c70844b3c207d9b5#2021-04-0317:17borkdudeAlso made a short video about it:
https://youtu.be/_zr1dicJs-E#2021-04-0317:32mokr@U04V15CAJ Thanks, I think I have had a look at babashka previously, but after a quick read now I see that I should have another look as it might actually fit well in my toolbox. That said, I think my initial question/request should be solved preferably in tools-deps itself to keep things simple. Especially for beginners that will soon enough have to navigate the library landscape in search of pieces that fit together.#2021-04-0317:48borkdudeRight. It would be useful if deps.edn had something like lein aliases. I wished this existed so I'm implementing it for bb. But I want to support something broader than lein aliases (more something in between npm run and make) and I don't expect deps.edn will support this use case. There is always the startup of the JVM too.#2021-04-0317:49borkdudeBut only having "shortcuts" for specific clojure invocations would already be useful.#2021-04-0406:53practicalli-johnI just use my shell history, I never have to type in a full command more than once. I have zsh (presto) with fish shell completion.
A lot simpler than any other approach.#2021-04-0415:08mokrI’ve not filed a request at https://ask.clojure.org/index.php/10400/support-for-shorthand-commands-in-tools-deps#2021-04-0417:24mpenetJust (Justfile) is also a good companion to deps.edn, it's quite easy to replicate most of lein with it +deps.edn#2021-04-0417:25mpenethttps://github.com/casey/just#2021-04-0417:25mpenetThere are many runners like this that can be used for now#2021-04-0416:41hugodWith the latest tools.deps, is it possible to use -M with --report stderr? I’m trying clojure -M:kaocha --report stderr, which used to work, and kaocha complains ‘Unknown option: “--report”’#2021-04-0416:45Alex Miller (Clojure team)That should work#2021-04-0416:46Alex Miller (Clojure team)What’s the :kaocha alias?#2021-04-0416:46Alex Miller (Clojure team)I mean that’s an option to clojure.main, not kaocha#2021-04-0416:46Alex Miller (Clojure team)The error looks like it’s coming from kaocha, I think#2021-04-0416:56hugodThe alias is
{:kaocha {:extra-deps
{lambdaisland/kaocha {:mvn/version "1.0.641"}}
:main-opts ["-m" "kaocha.runner"]}}
#2021-04-0416:57hugodkaocha shouldn’t be seeing the --report iiuc#2021-04-0417:13seancorfield@hugod Everything after -m gets passed to that -main function.#2021-04-0417:14seancorfieldIf you want --report to work, you need to invoke clojure.main (and have it invoke the koacha.runner)#2021-04-0417:14seancorfield:main-opts ["--report" "stderr" "-m" "kaocha.runner"] — try that?#2021-04-0417:16hugodUm, that’s unfortunate. Also not what clj --help implies.#2021-04-0417:19Alex Miller (Clojure team)Which part?#2021-04-0417:21hugodIt seems like you should be able to chose a report option when running a main with -M.#2021-04-0417:21borkdudeI would expect: clj --report stderr -M:kaocha kaocha-args... here: everything after -M gets passed to clojure.main / the subtask, everything before gets parsed by the clojure CLI (but this is not according to the help)#2021-04-0417:28borkdudeor is --report an option to clojure.main as well?#2021-04-0417:29borkdudeoh I see, it is. yeah, I guess you can't really put stuff in between the main args in your deps.edn alias from the command line#2021-04-0417:51Alex Miller (Clojure team)You can also activate that property with a Java system prop, that might combine better here#2021-04-0417:53Alex Miller (Clojure team)-J-Dclojure.main.report=stderr#2021-04-0417:53Alex Miller (Clojure team)^ add to clj command line#2021-04-0417:54Alex Miller (Clojure team)clj -J-Dclojure.main.report=stderr -M:kaocha#2021-04-0417:57hugodok, that looks like it works as a consistent way (over all possible invocation types) of specifying it. It would still be nice to be able to specify it somehow with --report though.#2021-04-0418:00borkdudeas this is probably a user preference, maybe it could be a setting in $HOME/.clojure/deps.edn?#2021-04-0418:01borkdudeor you could make an alias with jvm opts and combine it -M:report:kaocha ?#2021-04-0418:01borkdudenot sure if jvm opts concatenate when you combine aliases, do they?#2021-04-0418:24seancorfieldYes, they do.#2021-04-0418:24seancorfield:main-opts do not, most everything else does.#2021-04-0713:04ingesolHi! Is there a known pattern in tools.deps for loading/running multiple processes in parallel, like CLJS-build, SASS build, reloadable CLJ-system?#2021-04-0713:14tvaughanPerhaps your looking for something like https://devcenter.heroku.com/articles/procfile? #2021-04-0713:53ingesolThanks! That is one option, yes. I was mostly wondering if this was a common problem with well known patterns for people who use tools.deps.#2021-04-0713:10Alex Miller (Clojure team)the Clojure CLI runs one program#2021-04-0713:10Alex Miller (Clojure team)if you want to run multiple things, then you need to make something that does that#2021-04-0713:57ingesolRight, that’s what I suspected. I also was hoping that the community had established some patterns for solving this kind of problems. My first idea was to just create a CLJ-script that I could invoke from the CLI, that would simply start those processes. @borkdude provided a bb-technique below, I suppose that is preferable to a JVM script as it leans on OS features for shutdown.#2021-04-0713:58ingesolor at least it seems that the bb-script saves me from having to think about graceful shutdown.#2021-04-0713:34borkdude@ingesol We use this (bb) script at work for launching these three things at once:
#!/usr/bin/env bb
(ns dev)
(require '[babashka.process :refer [destroy-tree process]])
(def opts {:out :inherit
:err :inherit
:shutdown destroy-tree})
(defn cljs []
(process ["clojure" "-Sforce" "-M:frontend:cljs/dev"] opts))
(defn less []
(process ["clojure" "-Sforce" "-M:frontend:less/dev"] opts))
(def platform-alias
(case (System/getProperty "os.name")
"Linux"
"backend/linux"
"Windows"
"backend/windows"
"Mac OS X"
"backend/macosx"))
(defn clojure []
(process ["clojure" "-Sforce"
(str "-A:backend:backend/dev:" platform-alias)
"-X" "dre.standalone/start"]
(assoc opts :in :inherit)))
(cljs)
(less)
(-> @(clojure) :exit (System/exit))#2021-04-0713:53ingesolThat is lovely, thanks#2021-04-0715:36Jeff Evansdoes anyone have an example of using -J to pass JVM options from the command line in conjunction with -M ? I can get it to work if I put :jvm-opts into deps.edn directly but I’d rather these be specified by people at the command line, and can’t figure out how to do that#2021-04-0715:39Alex Miller (Clojure team)you just -J-Dfoo=bar on the command line#2021-04-0715:39Alex Miller (Clojure team)what did you try?#2021-04-0715:41seancorfield-J must come before -M (otherwise it will be treated as a command-line arg to what you are running).#2021-04-0715:42Jeff Evansaha, that was it, and I guess multiple options need -J multiple times#2021-04-0715:42Jeff Evanstired putting everything into one quoted string and that wasn’t working#2021-04-0715:42Jeff Evansas the docs state 😆#2021-04-0715:44Jeff Evansanyways, thanks to both of you#2021-04-0718:35borkdudeOur projects at work are now fully deps.edn-ized, no more boot. Migration complete.#2021-04-0722:05Lone Rangeryou work on top of all your opensource stuff!? I figured you'd be FT open source by now!!#2021-04-0722:09borkdudeif I got paid to do full time open source, I maybe would. but currently I get sponsored for about 1/5th - 1/4th of what I would need to make that jump. even then, I do like to be grounded in "reality" to get a feel for what the real problems are.#2021-04-0718:40seancorfieldWelcome to the future @borkdude! 🙂#2021-04-0918:33seancorfieldI’m wondering: how many folks are using :override-deps (or :default-deps — but more specifically :override-deps)?#2021-04-0918:39tvaughanWe have small-ish monorepo with some libraries, and fulcro front and back ends. 0 uses of override-deps#2021-04-0918:44seancorfieldThanks. Yes, I should have mentioned that “Folks using a monorepo setup and not using :override-deps is also a very useful data point for me”.#2021-04-0918:48nnicholsI only use it in my personal deps.edn, and mostly for quickly testing version compatibility. I have yet to see it in a project edps.edn file#2021-04-0918:58favilaUsing both in a monorepo setup, or at least trying to. We keep encountering odd rough edges where the version replacement doesn’t seem to happen correctly#2021-04-0918:59favilawe use :default-deps to synchronize direct dependencies among monorepo modules. none of them are supposed to use version coordinates, but inherit from default-deps, which is in an alias that must always be active#2021-04-0918:59favilawe use :override-deps to address version-pinning problems for transient deps, e.g. to address CVEs on things we don’t depend on directly#2021-04-0919:00seancorfield@U09R86PA4 I’m curious how you specify the lib deps where :default-deps plays a role? some/lib nil?#2021-04-0919:00favilayes#2021-04-0919:02seancorfieldWe use some/lib {} with :override-deps. I seem to recall some tooling having a hard time with nil but maybe that’s just in the past. I’m just wondering how much value it is really adding to our setup, since it forces every invocation to specify an alias to pick those overrides up.#2021-04-0919:03seancorfieldIf we were using it solely to address transitive deps/CVEs, I wouldn’t be so averse to just duplicating the lib specs — I think Jackson is about the only thing where we have to force an older version due to incompatibilities with some other lib we use…#2021-04-0918:35seancorfield(background: we use it heavily at work but I’ve been looking at Polylith a lot recently and there doesn’t appear to be anywhere to “hang” a single alias with all of the “pinned” deps for a project — and I know our heavy use of :override-deps is somewhat core to our struggles to “fit” the CLI/`deps.edn` world — so I’m thinking that :override-deps is very rarely used in the wild)#2021-04-0918:44seancorfieldThanks. Yes, I should have mentioned that “Folks using a monorepo setup and not using :override-deps is also a very useful data point for me”.#2021-04-1010:20henrik42Hi! I'm doing my first experiments with clj and noticed that I can put this into my deps.edn:
:aliases {:foo {:main-opts ["-e" (let [x :foo] (println x))]}
and then use -Mfoo to run. Is this ok? Can I use any form in there? I found https://clojurians-log.clojureverse.org/tools-deps/2018-06-11 and I'm wondering, what came out of it.#2021-04-1010:43practicalli-johnclojure -M:foo should run that expression. Unless you really use an expression a great deal or need it for some kind of work-around, I don't see the value myself.
It seems simpler to use the -e flag on the command line.#2021-04-1010:45practicalli-johnThere are a whole range of example aliases in https://github.com/practicalli/clojure-deps-edn#2021-04-1010:49henrik42I'd like to put clojure forms into the aliases as examples. Putting them on the command line would make them platform dependent. So I just wanted to check that putting them there is fine.#2021-04-1012:21Alex Miller (Clojure team)That embedded Clojure form there is not valid syntax in deps.edn#2021-04-1012:22Alex Miller (Clojure team)So putting that there is not fine#2021-04-1012:23Alex Miller (Clojure team):main-opts is a vector of strings#2021-04-1012:51henrik42@U064X3EF3 thanks for the clarification#2021-04-1019:15zalkyHi all, wondering if there's a way to retrieve the resolved deps environment, specifically the fully resolved :paths attribute after all the aliases have been merged and a main function has been launched? Thanks!#2021-04-1019:17seancorfield@zalky Not really. By the time your code is actually running, a lot of that information has been merged away.#2021-04-1019:18seancorfieldYou can use tools.deps.alpha as a library to compute the “project basis” which is a superset of everything in the root deps.edn, user deps.edn, and project deps.edn files but you can’t tell which aliases were used to invoke your code — only what different paths/deps combinations would be computed from any given aliases.#2021-04-1019:18seancorfieldYou can get the classpath though — which might be enough for your needs — via a JVM system property.#2021-04-1019:20seancorfield(System/getProperty "java.class.path")
will produce a very long string of JAR files and directories separated by (System/getProperty "path.separator") (which is different on Windows vs macOS/Linux).#2021-04-1019:23zalkyThanks @seancorfield, this is very helpful. If I filter that list for directories, would that be equivalent to the fully resolved paths? Might there be anything else thrown in? My motivation is that I'm doing some work with clojure.tools.namespace and I need to know all the directories I might have source in.#2021-04-1019:24seancorfieldYou’ll get directories for the equivalents of src, resources, and maybe test plus any git deps or local/root deps in play.#2021-04-1019:27seancorfieldYou could probably filter out any absolute paths and what is left should be just your own project’s stuff (depending on how you have your project set up).#2021-04-1019:31zalkyGotcha, thanks!#2021-04-1019:47Alex Miller (Clojure team)what's your actual goal? project paths?#2021-04-1020:23zalkyHi @U064X3EF3, I have a clojure.tools.namespace based reloaded workflow, which requires as input a set of directories to watch for changes. While I can supply a list of hard-coded directories in :main-opts , my understanding is that project paths is dynamically computed from all the active aliases.#2021-04-1020:24Alex Miller (Clojure team)it is, but there is not an explicit place to get just the paths#2021-04-1020:25Alex Miller (Clojure team)the classpath computed by tools.deps will put all paths at the front of the classpath though#2021-04-1020:26Alex Miller (Clojure team)So you could just walk the (System/getProperty "java.class.path") while it's directories. To be more righter, could probably only take the paths in your project (local or git deps could also be dirs and happen to be at the front)#2021-04-1020:30zalkyThanks, I'll give the classpath approach a go.#2021-04-1020:40seancorfieldWhy are you using c.t.n / reloaded? I've never felt the need for that sort of thing (and my REPLs run for weeks, sometimes months).#2021-04-1020:41seancorfieldI'm always curious when I find folks using this workflow and -- as in your case -- jumping through hoops to make it work...?#2021-04-1020:53Alex Miller (Clojure team)(+1)#2021-04-1114:19zalky@seancorfield, I am working with a component based system, and the particular reloaded workflow I am using automatically restarts the running application when reloading a namespace would otherwise leave stale objects to gum up the works. c.t.n is the thing that determines when such a namespace has changed. Do you have any alternatives workflows that you recommend for working with a stateful application? I'm not sure I immediately see a way around it when working with component. At least the reloading task is fairly straightforward.
The bigger picture is that we're in the process of migrating our build framework from boot to deps, which has been great in many respects. But because deps is much narrower in scope, the work has been figuring out how to stitch our previous build and development workflows back together.#2021-04-1116:44seancorfieldWe use Component very heavily at work - we also migrated from Boot to the CLI - we do not use any sort of reloaded workflow. We evaluate every change we make into the running program with no restarts.#2021-04-1117:11zalky@seancorfield, interesting, I think most of the justification for the reloaded workflow comes down to stale records not implementing newer instances of a protocol. Do you just take special care to make sure protocol namespaces are only ever loaded once?#2021-04-1117:13seancorfieldWe don’t change records/protocols much after the initial work on them. We also tend to use protocol implementation via metadata these days (since Component supports that).#2021-04-1117:14seancorfieldIf you find you are changing protocols so much that you are forced to remove + reload namespaces a lot, I would take a harder look at how you’re using protocols (or records) and see if you can optimize how you design/build them.#2021-04-1117:27zalkyI agree, it's like you say: they don't change too much after the initial design, and the need for them is not very common to begin with, but it was nice to have even that edge case covered. Maybe it's worth revisiting at some point during our build migration. Thanks again this was very helpful.#2021-04-1117:51seancorfieldI mean, we do have RCFs (“Rich Comment Forms”) with component/start / component/stop forms in them so we can eval code to start and stop systems when we need to but mostly I start the REPL (outside the editor — because it lives longer than my editor, in general) and just leave it running for weeks, and then eval the main ns for an app and then eval a component/start form and just work on a feature until I’m done, evaluating every single change I make (often without even bothering to save files). If I need to switch to another app in our monorepo, I may stop the first app before starting the second app, but I often just have multiple apps running in my REPL (I tend to make sure tests start/stop components as needed and use different ports to the dev port).#2021-04-1218:28dominicm@zalky#2021-04-1218:28dominicmugh, sorry. I have code for this I can share#2021-04-1218:30dominicmhttps://git.sr.ht/~severeoverfl0w/wedge/tree/master/item/src/io/dominic/wedge/user.clj#L48-86
I intentionally leave in :local/root deps in case you are doing development on them, but you can use exclude-libs-refresh to remove those libraries manually.#2021-04-1314:11zalkyThanks @U09LZR36F for sharing, that's a useful example!#2021-04-1110:52golanweissHi,
lein has a flag java-source-paths to compile Java files in a clojure project - see https://github.com/technomancy/leiningen/blob/master/src/leiningen/javac.clj#L137.
Is there an equivalent way of doing this in CLI + tools.deps ?
Should I use badigeon as in https://github.com/EwenG/badigeon/blob/master/sample/badigeon/sample.clj#L37?
thanks.#2021-04-1110:54borkdudeyou can also just use java and set the classpath with $(clojure -Spath ...) if it's only a few classes
EDIT: I meant javac#2021-04-1111:08golanweissDo you mean without compiling it before running?#2021-04-1111:10borkdudeSorry, I meant javac#2021-04-1111:34golanweissindeed this works.#2021-04-1111:34golanweissthanks!#2021-04-1208:28pinkfrogIs it possible to eval some clojure code inside deps.edn? The idea is, if some ENV variable is set to true, modify the mvn repo directory.#2021-04-1210:27borkdudeMaybe you can write a script, if -Sdeps allows you to override the mvn repo dir#2021-04-1210:31vlaaadYou can also use tools.deps as a library and add custom pre/post processing there#2021-04-1210:42borkdudetrue!#2021-04-1211:51p-himikThe page at https://clojure.org/reference/deps_and_cli#_classpath_caching makes it clear that it's impossible to override the path to .cpcache when deps.edn is present.
I wonder whether it would be useful to have such an ability.
Consider a situation where you need to give someone a read-only access to some src-based project, just so that they can run the clj command with some useful aliases.
It will require the .cpcache dir to be writable for that user. And, I think, that opens a possibility for injections, when that user changes the cache in some malevolent way and some other user with higher privileges ends up using that cached classpath.
If it were possible to override path to .cpcache when deps.edn is present, a user with low privileges can be instructed to just use some personal directory.
In the scenario above, maybe completely disabling such a cache for such users would be useful.#2021-04-1212:00borkdude@p-himik The way the clojure CLI currently works is that the cache files contain a return value from tools.deps. Those are always read from the file system or when not available, created by invoking tools.deps. So you can't run without the cache, currently. But I agree that making the cache dir configurable/overridable would be useful for the above scenario.#2021-04-1212:05p-himikThanks!
In that case, disabling the cache can implicitly create an auto-generated temp directory, to prevent two users from using the same temp cache directory accidentally, like /tmp/my-proj/cpcache.
But perhaps a better term than "disabling the cache" could be used in the documentation, to avoid any misunderstandings.#2021-04-1212:06borkdudeThat could be a workaround indeed#2021-04-1212:18Alex Miller (Clojure team)We do have this captured in TDEPS-119 and I agree there needs to be some solution for it#2021-04-1215:45borkdudeWorking on a task runner. One goal is to create shortcuts for various clojure invocations, but another thing falling out of the feature to run deps in parallel is to create a dev env that runs multiple processes in the background. Just sharing this for early feedback. This is my version from work. (we have more tasks like uberjar, but I've only included those that I invoke from the command line regularly today)#2021-04-1215:58erwinrooijakkersIs there a way to upgrade dependencies automatically? A Leiningen’s lein ancient upgrade for tools.deps?#2021-04-1216:00borkdude@erwinrooijakkers That tool is called antq. There are multiple but that's the most actively maintained and complete one I believe#2021-04-1216:01erwinrooijakkersgreat! started upgrading manually so just in time 🙂#2021-04-1216:08erwinrooijakkersthanks! works like a charm
:aliases {:upgrade-deps {:extra-deps {com.github.liquidz/antq {:mvn/version "0.12.4"}}
:main-opts ["-m" "antq.core" "--upgrade" "--force"]}}#2021-04-1216:30tvaughanThis https://github.com/practicalli/clojure-deps-edn/ may also be a helpful resource#2021-04-1216:46zalkyHi all, I'm trying to use clojure -X:my-alias:deps tree to inspect the dependencies of my application but none of the dependencies that are being brought in by :my-alias via :extra-deps are showing up anywhere in the output tree. I can confirm that :my-alias bings in the relevant dependencies in the running application. Is this the correct usage? Is -X:deps tree meant to work with other aliases?#2021-04-1216:54seancorfieldYou can use clojure -A:my-alias -Stree for the time being. There’s currently no way to tell -X:deps tree to use aliases when it is computing the project basis.#2021-04-1216:56zalkyThanks @U04V70XH6, worked like a charm!#2021-04-1216:50lreadHiya! Just noticed that https://github.com/clojure/tools.deps.alpha/blob/9bf5778dc26dd5018dbf04fc8e7dbb32ddc4036c/src/main/clojure/clojure/tools/deps/alpha.clj#L611 but the https://github.com/clojure/tools.deps.alpha/blob/master/API.md.#2021-04-1217:33Alex Miller (Clojure team)Thanks, I’m not at a computer right now but if you could file an issue at https://github.com/clojure/clojure-site/issues that would be helpful for me to remember#2021-04-1217:48lreadThanks, done: https://github.com/clojure/clojure-site/issues/521#2021-04-1218:33Alex Miller (Clojure team)Thx much#2021-04-1217:40lilactownwhere's the docs for what options clj / clojure take? e.g. if I want to add a dependency via the CLI#2021-04-1218:33Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli also see -h and man clj)#2021-04-1220:09lilactownty! -h i think is what i wanted#2021-04-1217:40lilactownI'm skimming https://clojure.org/reference/deps_and_cli and don't see a list or a link to one#2021-04-1217:42dharriganThe file itself is a wrapper, so if you can examine the contents, you'll find the list of options#2021-04-1217:42dharriganFor example, on Arch linux, it's in /usr/bin/clojure#2021-04-1217:42dharrigan(which aliases to clj)#2021-04-1217:43dharriganIt's also in the source repo $ clj -Sdeps '{org.clojure/clojurescript {:mvn/version "1.10.844"}}' -m cljs.main --repl-env node
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate cljs/main__init.class, cljs/main.clj or cljs/main.cljc on classpath.#2021-04-1217:57lilactownI've also been trying to write an alias to put in my ~/.clojure/deps.edn and I cannot figure out how to specify the -m cljs.main part#2021-04-1218:58seancorfieldUnder :aliases:
:node-repl
{:extra-deps {org.clojure/clojurescript {:mvn/version "1.10.844"}}
:main-opts ["-m" "cljs.main" "--repl-env" "node"]}
and then clj -M:node-repl#2021-04-1219:01seancorfield(! 1208)-> cat deps.edn
{:aliases
{:node-repl
{:extra-deps {org.clojure/clojurescript {:mvn/version "1.10.844"}}
:main-opts ["-m" "cljs.main" "--repl-env" "node"]}
}}
(! 1209)-> clj -M:node-repl
ClojureScript 1.10.844
cljs.user=> (* 1 2 3)
6
cljs.user=>
#2021-04-1220:09lilactownthat's great, thanks!#2021-04-1217:59dharriganMissing :deps#2021-04-1217:59lilactownack I just figured that out facepalm#2021-04-1217:59dharrigannp 🙂 have fun! 🙂#2021-04-1219:38Wesley MatsonIs there a way to override the searched locations for a maven settings.xml?
I’m trying to get a reasonable tools-deps jenkins pipeline working (using docker agents to minimize the need for installing things on jenkins), but I’m having trouble getting tools-deps to access a private maven repo from within docker.
The projects that use maven get their credentials through a fancifully-provided settings file and include a flag on each maven call to specify it as the one to use (e.g. mvn -s ./provided/settings.xml clean install). Docker doesn’t seem to like me copying things into ~/.m2, so I’m wondering if there’s something like the -s mvn flag for specifying the maven settings to use in clj.#2021-04-1219:39Wesley MatsonRelevant part of my docker file#2021-04-1219:40Wesley MatsonRelevant buildkit/docker build secrets documentation: https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information#2021-04-1219:46dharriganThis is something I have used in the past...#2021-04-1219:46dharrigan(I use gitlab, but should be adaptable)#2021-04-1219:46dharriganvariables:
DOCKER_HOST: ""
DEPS: '{:mvn/repos {"central" {:url ""} "clojars" {:url ""}}}'#2021-04-1219:47dharrigantest:
extends: .cache
stage: test
script:
- clojure -Sdeps "${DEPS}" -A:test-runner
cache:
policy: push#2021-04-1219:47dharrigannexus in the example above, is my own private repo.#2021-04-1219:51Wesley MatsonI have private repos working locally on my machine outside docker, but getting the credentials into docker is proving funky (I don’t see where that’s happening above)#2021-04-1219:54dharriganI have anonymous access to pull from the repos#2021-04-1219:55dharriganno need for authentication to pull 🙂#2021-04-1219:55Wesley MatsonThat makes sense in most cases, this is for grabbing a company-internal Java library that’d be kind of hard to recreate/vendor in properly#2021-04-1219:57dharriganNot sure then, our company has anon access to pull artifacts#2021-04-1219:57dharrigansoz ;(#2021-04-1219:58dharriganMaybe someone else can help out 🙂#2021-04-1219:59Wesley MatsonThanks for trying 🙂#2021-04-1220:13nnichols@U9LRMUCAF Are you using S3 as your internal repo?#2021-04-1220:14Wesley MatsonNo, using nexus#2021-04-1220:56Wesley MatsonI tested some more locally, including clearing my local m2 cache to find that the settings-security.xml (part of a way to handle credentials for maven without storing passwords in plaintext), wasn’t being handled at all in tools-deps, but using a plaintext-passworded settings.xml did not fix the issue inside docker (it worked locally/outside-docker).#2021-04-1220:57Wesley MatsonNewest relevant section of Dockerfile, tried some more experimentation and confirmed that RUN cat ~/.m2/settings.xml inside docker reveals the correct file that works when using tools-deps outside docker in the “same” position#2021-04-1220:56Wesley MatsonI tested some more locally, including clearing my local m2 cache to find that the settings-security.xml (part of a way to handle credentials for maven without storing passwords in plaintext), wasn’t being handled at all in tools-deps, but using a plaintext-passworded settings.xml did not fix the issue inside docker (it worked locally/outside-docker).#2021-04-1311:47golanweissI’m moving from a lein project to tools deps.edn and unfortunately I had some “injections” in my project.clj
Is there a way to “inject” (i.e. require) some namespace when using the https://github.com/cognitect-labs/test-runner?#2021-04-1312:09borkdude@golan1w you can use -e "(require 'foo.bar)" -m "test-runner.main"#2021-04-1312:35golanweissthanks,
The problem is that when using multiple aliases only the last main-opts in is kept. So I don’t see how can I add this to my deps.edn to be permanent.
For example a good place for such thing would have been under alias called
:my-lib-test#2021-04-1312:37borkdudecreate a new alias with other main opts?#2021-04-1313:34golanweissyeah but then I can’t use my :runner alias for all tests.
:runner {:extra-deps ...
:main-opts ["-m" "cognitect.test-runner"]}
and I’ll need to create a different alias for each library that needs this “injection”. This will also be a bit more complicated because I’m also building the CI integration and trying to have some code re-use.
Eventually I forkedhttps://github.com/localizedev/test-runner and added support as another command line argument…#2021-04-1313:35pinkfrogI don’t understand why deps.edn doesn’t allow to dynamically evaluate some clj expression. This quite limits the flexibility of deps.#2021-04-1313:35Alex Miller (Clojure team)isn't that what -M -e does?#2021-04-1313:36Alex Miller (Clojure team)if not, what do you mean?#2021-04-1313:38pinkfrogI’ve a deps.edn file, I want to set the :mvn/local-repo to be :mvn/local-repo=$SOMEPATH, where $SOMEPATH is an environment variable#2021-04-1313:51borkdudenot possible without scripting#2021-04-1313:53pinkfrogAny scripting example on this subject?#2021-04-1313:56pinkfrogI can do things like reading a template file and output a deps.edn. Just found it is a little bit unnecessary as deps doesn’t support these functionalities.#2021-04-1313:57borkdudeCan you accomplish this with -Sdeps perhaps?#2021-04-1314:16pinkfrogEh.. A little bit complicated.#2021-04-1314:16pinkfrogUse
@npx shadow-cljs watch mobile
shadow will invoke clojure#2021-04-1314:16pinkfrogno way to pass in -Sdeps#2021-04-1313:51borkdudemaybe possible with -Sdeps#2021-04-1313:58Alex Miller (Clojure team)can you do it with stdin?#2021-04-1313:59Alex Miller (Clojure team)oh sorry, you're talking about in deps.edn#2021-04-1313:59Alex Miller (Clojure team)you can pass that particular case on the command line with -Sdeps#2021-04-1314:02pinkfrogI think I might hack the PATH and use a wrapped verison of clj which automatically adds -Sdeps.#2021-04-1314:53Ronny LøvtangenAnyone knows if it is possible to add -e "(require 'sc.api)" to IntelliJ’s Run Configuration?#2021-04-1315:25golanweisscan’t you add it in an additional alias?#2021-04-1316:51Ronny LøvtangenIdeally, I would follow scope-capture’s recommendation and keep it outside the project. (https://github.com/vvvvalvalval/scope-capture)
But if that’s not possible, I could add an alias.
I tried to add it to my server alias, which already has a :main-opts :
:main-opts ["-m" "dvp.clj.server"]
Tried with
:main-opts ["-m" "dvp.clj.server" "-e" "(require 'sc.api)"]
but to no success.
But IntelliJ won’t even start the main function with my current alias, so maybe I need to figure out that first.#2021-04-1606:33Ronny LøvtangenIs the syntax correct, btw?#2021-04-1813:01golanweissI think “-e” should come before.
Check out https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#2021-04-1817:21Ronny LøvtangenThanks 👍#2021-04-1315:53golanweissIt seems clojure -Spom doesn’t add :local/root dependencies to the final pom.xml file.
Does anyone knows if this is a known issue?#2021-04-1315:55delaguardodoes maven supports arbitrary path as a dependency?#2021-04-1315:56delaguardothe same happens with git-based dependencies#2021-04-1320:10golanweissI see now..
The behavior I was expecting is to go into the path of the :local/root and check if there’s a pom.xml file and use its version. maybe a feature request? 🙂#2021-04-1320:16seancorfieldI’m not sure how that even makes sense? :local/root and :git/url are source-only dependencies that don’t fit Maven’s group/artifact/version world — and pom.xml doesn’t contain any information about transitive dependencies anyway.#2021-04-1320:27golanweissindeed, my usage is wrong.
actually this behavior makes sense. this is not a Spom problem.#2021-04-1315:55Alex Miller (Clojure team)there is no way to do so?#2021-04-1315:58pinkfrogHow to tell deps to ignore the deps.edn file?#2021-04-1315:59borkdudeYou can't. deps.clj (unofficial) does support this: https://github.com/borkdude/deps.clj.#2021-04-1316:00Alex Miller (Clojure team)why do you want to ignore it?#2021-04-1316:00pinkfrogA temporary fix would be -Sdeps {:deps nil}. I will use it for now.#2021-04-1316:02borkdudeThis still won't ignore your local deps.edn, it will just merge it#2021-04-1316:00borkdudeif you have babashka installed, you can invoke it with bb clojure -Sdeps-file=other.edn#2021-04-1316:01pinkfrogbb looks promising#2021-04-1412:02pinkfrogI finally go with bb to preprocess the deps.edn file. Blazingly fast.#2021-04-1316:01pinkfrogI am writing a simple script to transform a deps.edn file, basically add some entries.#2021-04-1316:01pinkfrog#!/bin/sh
#_stub ; -*- mode: clojure; -*-
#_(
DEPS='
{:mvn/local-repo "", :deps {}}
'
exec clojure -Sdeps "$DEPS" -M "$0" "#2021-04-1316:03Alex Miller (Clojure team)I don't see why that requires you to ignore the deps.edn#2021-04-1316:04pinkfrogbecause this script doesn’t need to install the dependencies from deps.edn#2021-04-1316:05Alex Miller (Clojure team)I think you'd really want something wonkier like clj -Sdeps '{:aliases {:TMP {:replace-deps {} :replace-paths []}}}' -A:TMP#2021-04-1316:45pinkfrogYou are right.#2021-04-1320:10golanweissI see now..
The behavior I was expecting is to go into the path of the :local/root and check if there’s a pom.xml file and use its version. maybe a feature request? 🙂#2021-04-1412:45otfrom👋
Is there a way to point at the head of master/main/other branch on https://github.com in deps.edn? Sort of like pointing at a snapshot in a maven repo, or :mvn/version "LATEST"?#2021-04-1413:18delaguardoyou can add :tag to coordinates map and then use -X:deps git-resolve-tags to update :sha#2021-04-1413:19delaguardo{:git/url "git://..."
:sha "some-sha"
:tag "master"}
#2021-04-1415:11otfromthat sounds funky. thx!#2021-04-1417:26seancorfieldThe :sha value can be anywhere on any branch — it doesn’t have to be the head of a branch or even a specific tag.#2021-04-1417:58otfrom@U04V70XH6 and that is great for repeatability. Just a bit frustrating when tracking fast moving changes in a library being developed between multiple team members. It isn't too much trouble and we can always point at it locally.#2021-04-1418:00seancorfieldYou can also use tools.gitlibs programmatically to find the current SHA for a given tag/branch. I have an example in my dot-clojure deps.edn file.#2021-04-1418:01otfromslash me goes to look at your deps.edn again#2021-04-1412:47ghadino. In fact I think support of LATEST in mvn is unintentional. deps.edn wants values (SHAs/versions), not references (branches / LATEST)#2021-04-1412:49otfrom🆒 thx#2021-04-1417:26seancorfieldThe :sha value can be anywhere on any branch — it doesn’t have to be the head of a branch or even a specific tag.#2021-04-1716:35pinkfrogHi. I wonder what is allowed to put under an :alias. It looks to me putting :mvn/repos under an alias has no effect.#2021-04-1716:36pinkfrogBut the reference is vague on what I can do. https://clojure.org/reference/deps_and_cli#2021-04-1716:48practicalli-john@i
:mvn/repos is a top level key in deps.edn, along with :deps , :path and :aliases, so I wouldn't expect it to be used within one of the other top level keys.
I assume any other names than those of top level keys are usable within :aliases.
I use namespaced keywords to add context to the aliases and hopefully avoid clashing with any additional top level aliases that may be introduced in future (although things have been quite stable for a while)
https://github.com/practicalli/clojure-deps-edn#2021-04-1802:01pinkfrogTake :mvn/repos for example, no way I could know it cound’t be put into aliases, unless I put it and find there would be an error.#2021-04-1802:04pinkfrogAn extra question, do you know where to specify the .rebel-history file. Currently, it is always created under my working directory.#2021-04-1802:43seancorfield@i It makes sense that it would be created wherever you run a REPL using Rebel Readline — that’s exactly what nREPL does with its .nrepl-history file.#2021-04-1802:44seancorfieldSame as the Clojure CLI creating .cpcache in the working directory.#2021-04-1802:49practicalli-johnI agree with Sean about the history as the large majority of expressions will be project specific.
There is a global config file for rebel, but I don't see an option for setting a global history file.
https://practicalli.github.io/clojure/clojure-tools/repl/customize-rebel.html
If there really is a case for a global history file, then I suggest raising an issue on the rebel readline project.#2021-04-1802:50seancorfieldThe docs (for deps.edn) are not as explicit as they could be. deps.edn is described “with top-level keys :deps, :paths, and :aliases, plus provider-specific keys for configuring dependency sources” but you have to read the Procurers section much further on to find mention of the :mvn/repos (top-level procurer-specific key). And I don’t see :mvn/local-repo mentioned there (it’s another top-level procurer-specific key, to change the default "~/.m2" folder path).#2021-04-2008:24flowthingI have a transitive dependency to an artifact with a reported security vulnerability. With Leiningen, I'd add a :managed-dependencies entry for a version of that artifact where that vulnerability has been fixed. What's the "right" way to do this with deps.edn? Just add a :deps entry?#2021-04-2009:00borkdude@flowthing I think so yes, tools.deps will pick the newest version possible that is required in the deps tree#2021-04-2009:08flowthingAll right, thanks. :thumbsup::skin-tone-2:#2021-04-2009:43Mark WardleHi all. Is there a way of referencing an alias (or indeed multiple aliases) in a git coordinate in a deps.edn file? The background is https://clojurians.slack.com/archives/C03S1KBA2/p1618904583308100 but essentially, I’d argue this approach might foster modularity and composability - permitting declaration of faceted dependencies. An example might be a repository aligned to a business domain with core code that can be used as a library, but an alias providing, say, ring handlers or graph resolvers that clients can explicitly opt-in or out of, to bundle the functionality into a server application. Possible with maven - by building different artifacts - and with a multiple repository model. Am I barking up wrong tree?#2021-04-2312:02Mark WardleIt looks as if the polylith type approach might work for this - although I do still think git coordinates that could include specific alias(es) and nested directories would do much to configure the exact building blocks to be used to build artefacts from a menu of common libraries / components.#2021-04-2010:09borkdude@mark354 I had something like this with boot:
https://github.com/borkdude/boot-bundle
But nowadays we just use some EDN to list the deps + fixed versions we need and use a babashka script to update our deps.edn. This is very fast and only needs to happen when we update a dependency.#2021-04-2010:10borkdudeAnother approach might be to just make a library which depends on other libraries and use that as the managed dependency, tools.deps will always use the newest versions of a lib in a dependency tree, I think#2021-04-2011:42Mark WardleThanks! I never used boot as I migrated very quickly from lein to deps when I started with clojure last year. A small standalone library would be fine too except for the potential explosion of repositories named xxx-library xxx-server xxx-graph-api etc…#2021-04-2012:09borkdudeWe have left boot behind us now too#2021-04-2010:25borkdude(off topic, but slightly related:
Found another good use for fs/modified-since (a new function in babashka.fs):
https://gist.github.com/borkdude/35bc0a20bd4c112dec2c5645f67250e3#file-1-bb-edn-L2-L5
We rebuild our deps.edn from a template when any relevant files on which it depends changes, when invoking any task.
)#2021-04-2312:02Mark WardleIt looks as if the polylith type approach might work for this - although I do still think git coordinates that could include specific alias(es) and nested directories would do much to configure the exact building blocks to be used to build artefacts from a menu of common libraries / components.#2021-04-2112:22dvingoWhen I execute the following in a shell:
clojure -Sdeps '{:deps {com.github.liquidz/antq {:mvn/version "RELEASE"}}}' -m antq.core
I see this warning:
WARNING: When invoking clojure.main, use -M
Is there a way to remove this warning? or is there a problem with the above invocation?#2021-04-2112:23delaguardoadd -M in the front of -m
-M -m antq.core#2021-04-2112:26delaguardoor call antq.core/-main directly using -X
-X antq.core/-main#2021-04-2112:32borkdude-X is not intended to be used with -main functions since the arg parsing is processed differently#2021-04-2112:33dvingothanks @U04V4KLKC
clojure -Sdeps '{:deps {com.github.liquidz/antq {:mvn/version "RELEASE"}}}' -M -m antq.core
this worked#2021-04-2115:15seancorfield@U04V4KLKC Just to follow on from Michiel’s point and perhaps clarify for @U051V5LLP: -X invokes a function passing a single hash map as an argument but -main expects a sequence of zero or more strings. As long as -main doesn’t do anything with its & args, you can get away with calling it via -X but that’s an accident of implementation rather than an intended usage.#2021-04-2115:08zalkyHi all, I was hoping to use node foreman launch a couple of simultaneous clojure processes in development, but it seems there's some concurrency issues when working with a project with remote git coordinates. If two clojure processes try to clone/checkout the same git coordinate concurrently, they run into problems. The workaround is to resolve concurrency by manually launching processes, giving one process a chance to locally clone git first. My question is: is this a reasonable expectation of the clojure deps tooling? Or might this be considered an issue?#2021-04-2115:09Alex Miller (Clojure team)are you using the latest version of clojure?#2021-04-2115:11Alex Miller (Clojure team)prior to 1.10.3.814, we were using jgit. as of 1.10.3.814, we're shelling out to git, so these are completely different tool sets with different behaviors. when shelling out to git, you're really asking, can git support multiple concurrent clones/checkouts and the answer is generally yes due to how git uses lock files#2021-04-2115:12zalkyI'm using Clojure CLI version 1.10.3.822#2021-04-2115:13Alex Miller (Clojure team)ok, so my follow up would be more precisely you are seeing the issue - is it in git clone of the repo or getting the working tree (it actually doesn't use git checkout, but rather git worktree)#2021-04-2115:14Alex Miller (Clojure team)and also what git --version you're on#2021-04-2115:14zalkygit version 2.20.1 (Apple Git-117)
#2021-04-2115:14Alex Miller (Clojure team)you can set GITLIBS_DEBUG=true to see all the commands being run#2021-04-2115:14Alex Miller (Clojure team)if that helps narrow it down#2021-04-2115:16Alex Miller (Clojure team)when you see "problems", what does that look like?#2021-04-2115:16zalkyThanks @alexmiller, I believe I've seen more than one related error message, sometimes related to a git lock file, other times a error: cannot lock ref#2021-04-2115:17zalkyOther times a files already exists message#2021-04-2115:17Alex Miller (Clojure team)would love to see those. might be something I can detect/retry or otherwise work with#2021-04-2115:18zalkyOk, I can try to reproduce a number of those messages with the GITLIBS_DEBUG set, is this place to follow up or would you prefer somewhere else?#2021-04-2115:36Alex Miller (Clojure team)https://ask.clojure.org is probably the best to file that#2021-04-2117:38zalky@alexmiller, thanks, I've followed up here:
https://ask.clojure.org/index.php/10508/concurrency-issues-launching-multiple-process-coordinates#2021-04-2117:54Alex Miller (Clojure team)yep, thx#2021-04-2209:37borkdudeIn some situations like when using nil :classpath-overrides, the classpath can end up having empty paths in between like src::/home/russ/russmatney/ralphie/src:::/home/russ/.m2/repository/cheshire/cheshire/5.10.0/cheshire-5.10.0.jar.
This has the effect that the local dir is added to the classpath. Demo:
echo '(ns foo) (prn :foo)' > foo.clj
clj -Scp ":$(clojure -Spath)" -M -e "(require 'foo)"
:foo
Is this a feature, or a bug?#2021-04-2212:24Alex Miller (Clojure team)Bug#2021-04-2212:27borkdudeShall I put this on ask?#2021-04-2212:29Alex Miller (Clojure team)Sure#2021-04-2212:31borkdudehttps://ask.clojure.org/index.php/10512/current-directory-added-classpath-with-classpath-overrides#2021-04-2212:34vlaaadthat’s not really a demo though, you set empty folder on the classpath yourself and not using :classpath-overrides#2021-04-2212:35borkdudeI changed the title#2021-04-2212:35borkdudeWhen adding the current dir you should imo add . to the classpath#2021-04-2212:35vlaaadah#2021-04-2212:35vlaaadisn’t it java behavior?#2021-04-2212:36borkdudebut maybe this is a behavior in Java indeed, it's nonetheless confusing#2021-04-2212:36borkdudeMaybe it should only be fixed on the level of classpath overrides etc though#2021-04-2212:37borkdudeIn bb I made a workaround for this that just ignores all empty paths when searching on the classpath, when getting back such a classpath from the clojure tooling#2021-04-2212:38borkdudeThe issue was that an uberjar built with bb that had such a classpath would add everything in the current dir to it#2021-04-2212:40borkdudeI added this note:
Maybe the empty paths returned from tools.deps.alpha should be fixed.
The empty path on the classpath behavior might be JVM behavior.
#2021-04-2212:44borkdudeok, edited the issue once more. please have another look#2021-04-2323:06didibusHow does tools.deps transitive dependencies work for Git deps? For Maven, I'm guessing it pulls the artifact from Maven, and that should always contain a POM.xml file which can then parse to continue the transitive dependency resolution. But what about for git deps? Does it rely on deps.edn files to do so?#2021-04-2323:15hiredmanyes#2021-04-2323:18hiredmanthe code eventually calls https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions.clj#L22-L30 so it will look for deps.edn or pom.xml in the git dep#2021-04-2323:29didibusOh cool, so it can also work if the deps has a pom.xml only.#2021-04-2323:40hiredmanwell, pretty good chance if there is a pom.xml it will require compiling java, which won't work#2021-04-2323:41Alex Miller (Clojure team)....... yet#2021-04-2412:03vlaaad@U064X3EF3 stop, I can only get so hyped for tools-build!#2021-04-2323:40hiredmanbut it'll try#2021-04-2705:59didibusI was thinking for lein projects, lein has a task to make a compatible pom.xml, but not to make a deps.edn I believe#2021-04-2711:28golanweissif you mean how to convert lein -> deps I used https://github.com/hagmonk/depify#2021-04-2712:07borkdudeWhat is the proper way of invoking:
clojure -Sforce -A:backend:backend/prod:backend/build -X:backend/build
Can this be squashed into one -X:...? Of which alias is the exec fn chosen?#2021-04-2712:37Alex Miller (Clojure team)You can combine, will use the last one iirc#2021-04-2712:23borkdudeWhat was the option again to specify a different local mvn .m2 dir?#2021-04-2712:36Alex Miller (Clojure team):mvn/local-repo#2021-04-2712:53borkdude@alexmiller Is this a documented option? I couldn't find it#2021-04-2712:54Alex Miller (Clojure team)It’s part of the deps.edn, not a clj option#2021-04-2712:56simongrayIf I run clj -M -m .main.ns lots of dependencies are downloaded. I thought running clojure -Spath beforehand would fetch those dependencies, but it doesn’t seem to do so?#2021-04-2712:56borkdudeI know, but any reason this is not part of guides or references for deps_and_cli?#2021-04-2712:57Alex Miller (Clojure team)No, I think it’s just an oversight#2021-04-2712:57Alex Miller (Clojure team)I can add it#2021-04-2716:03seancorfieldI was trying to find that the other day in response to someone’s new-to-tools-deps question. It would be nice if the docs were also a little clearer that :mvn/repos and :mvn/local-repo can only be used at top-level keys and not inside aliases — that seems to be a fairly common confusion for some folks.#2021-04-2716:22Alex Miller (Clojure team):thumbsup: #2021-04-2712:58borkdude@simongray I think it would need to fetch those deps in order to calc the classpath, no?#2021-04-2712:58simongrayYou would think so, but it doesn’t seem to do that. The clojure -Spath runs inside a Docker container, so perhaps something is amiss#2021-04-2713:23simongraySolved: I didn’t set the WORKDIR in the Dockerfile before running clojure -Spath so there was no deps.edn to fetch download depencies according to.#2021-04-2712:59borkduderepro or it didn't happen :)#2021-04-2713:00simongraykinda in the middle of a bunch of changes :S#2021-04-2713:02simongrayAlso, maybe someone can tell me why you need to do -M -m ? Or should I do -X now instead (the docs seem to use that now)?#2021-04-2716:05seancorfieldBecause -M means “run clojure.main” and -m is an option to clojure.main itself (as are -i, -e, and -r).#2021-04-2716:51Darin Douglasswas the decision to provide only short-form arguments for clj deliberate? having better-reading commands would, IMO, alleviate a good portion of the -M v -X v -A mixups.
long-form arguments are definitely not magic pill for this style problem, but after a cursory think, i can't really think of a downside to having them#2021-04-2713:03borkdudeOr was it -X -x ? :thinking_face: troll#2021-04-2713:04simongrayOr maybe -M -x 😛#2021-04-2713:04borkdudeM x of course!#2021-04-2713:05simongrayall paths lead to emacs#2021-04-2713:06borkdude{:emacs {:replace-paths ["emacs"]}}
clojure -Spath -A:emacs
M x clojure
#2021-04-2713:06simongrayhah#2021-04-2713:13borkdudehttps://imgflip.com/i/57awpv ;)#2021-04-2716:05seancorfieldBecause -M means “run clojure.main” and -m is an option to clojure.main itself (as are -i, -e, and -r).#2021-04-2716:51Darin Douglasswas the decision to provide only short-form arguments for clj deliberate? having better-reading commands would, IMO, alleviate a good portion of the -M v -X v -A mixups.
long-form arguments are definitely not magic pill for this style problem, but after a cursory think, i can't really think of a downside to having them#2021-04-2717:04borkdude@ddouglass Another option could be having subcommands instead of dashed options.
clojure -A:foo:bar exec <exec-fn>
clojure -A:foo:bar main <foo.bar>
clojure -A:foo:bar path ;; print classpath
clojure -A:foo:bar run <tool>
clojure -A:foo:bar pom ...
#2021-04-2717:04borkdudeso just one option to "compose" the classpath and subcommands that handle what should be done (with their own arg parsing and help output)#2021-04-2717:07Darin Douglassya, i'm more preferential to that option b/c i think the current -A|M|X options are a bit overloaded which bring up questions (like your's earlier) like "which alias does exec-opts come from", etc.
same thing for -S*: some are commands (`-Stree`) while some are options (`-Scp cp`)#2021-04-2717:08borkdudeIn the case of
clojure -A:foo:bar exec <exec-fn>
you would still have the ambiguity of where the exec-fn could come from (the last alias?)#2021-04-2717:10Darin Douglassmmhmm#2021-04-2717:55Alex Miller (Clojure team)It’s not ambiguous - there is a merge strategy defined for each#2021-04-2721:06borkdudeI mean, from the point of the ignorant consumer of this CLI#2021-04-2721:20Alex Miller (Clojure team)oh sure, it might be inscrutable :)#2021-04-2721:21Alex Miller (Clojure team)but generally I think <hand-wave> last one wins <hand-wave/> is what you expect most#2021-04-2723:22Alexis VincentI’m a bit confused about something. I have a deps.edn with :paths ["src"] , but clojure.tools.namespace.repl/reset doesn’t work unless I (clojure.tools.namespace.repl/set-refresh-dirs "src") . Checking (clojure.java.classpath/classpath-directories) returns '(). I guess my expectation was that this would be set using :paths ?#2021-04-2723:26Alexis Vincentrepl is started using cider via /usr/local/bin/clojure -A:dev -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.8.3"} refactor-nrepl/refactor-nrepl {:mvn/version "2.5.1"} cider/cider-nrepl {:mvn/version "0.25.9"}} :aliases {:cider/nrepl {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[refactor-nrepl.middleware/wrap-refactor,cider.nrepl/cider-middleware]"]}}}' -M:cider/nrepl#2021-04-2723:29Alexis Vincentclojure -Spath prints src:resources:<...rest>#2021-04-2723:42seancorfield@mail024 My advice would be to try to ween yourself off the t.n.r reset/reload workflow and develop a “better” REPL workflow that doesn’t need that sort of thing, i.e., eval’ing every change you make as you go so the REPL state is always fresh. I see a lot of people get into all sorts of problems with “reloaded” style workflows 😞 and folks like Eric Normand, Stu Halloway, Rich Hickey, myself, and many others avoid this sort of tooling.#2021-04-2723:44seancorfield(but it does sound like a but in java.classpath since src and resources are on the classpath and are directories — you could create a post on http://ask.clojure.org tagged with the java.classpath library and see what the library maintainers say)#2021-04-2723:45seancorfieldI would also recommend trying to repro in a plain REPL started yourself — without all that nREPL/CIDER stuff — and see if the problem persists: it’s entirely possible that something in nREPL/CIDER is causing this problem…#2021-04-2723:51Alexis Vincent@seancorfield Thanks. Appreciate the advice. Guess it’s an old habit. Will give the suggested approach a spin, been meaning to try life without ns refresh for a while now#2021-04-2723:53Alexis VincentI’m thinking it might actually be a cider thing. plain repl has the expected entries in classpath-directories#2021-04-2723:58seancorfieldI haven’t used nREPL or CIDER for a very long time so I would only point you to #cider at this point 🙂#2021-04-2723:59seancorfield(I use a plain Socket REPL with no additional dependencies)#2021-04-2800:00Alexis Vincentno additional deps for the repl, or started on launch?#2021-04-2800:00Alexis VincentAssuming deps.edn to form classpath and deps right?#2021-04-2800:02Alexis VincentCulprit is nrepl#2021-04-2800:06seancorfieldFor non-local processes, I just provide a JVM option when starting the JAR or when running clojure.#2021-04-2800:07seancorfieldFor my local dev process, I have a Clojure script that starts Rebel Readline, Reveal, and a Socket REPL (in a DynamicClassLoader), and that also sends a custom view into Reveal (that makes it more like REBL). But I could just run clojure to start a REPL with a JVM option.#2021-04-2800:09seancorfieldhttps://github.com/seancorfield/dot-clojure is my ~/.clojure/deps.edn file and the dev.clj I start my REPL with, but on QA/production, we just start JAR files with a JVM option to start the Socket REPL — yup, we run REPLs in production.#2021-04-2800:11Alexis VincentAwesome! Thanks. Will check out the setup. Been meaning to revamp. Had the same process now for years.#2021-04-2800:15seancorfieldI use VS Code + Clover for Clojure evaluation. I have Calva installed for everything else Clojure-related (but I have its nREPL features turned off). My VS Code / Clover set, with all the tap> extensions to work with Reveal, is also on GitHub https://github.com/seancorfield/vscode-clover-setup#2021-04-2800:15seancorfield(I used Emacs on and off for twenty years but I was happy to move off it — initially to Atom in 2015 and then to VS Code last year)#2021-04-2800:19Alexis VincentFair. Going to be setting up some proper dev infra later this year. Will checkout the socket stuff. Emacs has worked well for me for a while. Think if I moved away I’d go to a slimmed down IDEA. But relearning key bindings for a whole new ide is going to be painful.#2021-04-2800:21Alexis VincentVSCode is nice. But if I move I expect it will be to gain static analysis and broader language support. Also concerned about VSCode performance#2021-04-2800:23Alexis VincentThanks for the feedback. Appreciate the pointers. Also, a general thanks for being so active on here and in the broader ecosystem. Helps us all#2021-04-2800:26seancorfieldRe: VS Code performance — I work with 113K lines of Clojure at work all-day, every-day in VS Code and it’s fine. Calva also has LSP under the hood and clj-kondo so you can get a lot of static analysis for Clojure, even without a REPL running.#2021-04-2812:04borkdudeI am trying to run a project locally inside a project which has a conflicting dep:
$ clojure -M:babashka/dev recent-clj-kondo
Error building classpath. Unable to compare versions for borkdude/sci: {:local/root "/Users/borkdude/Dropbox/dev/clojure/babashka/sci", :deps/manifest :deps, :deps/root "/Users/borkdude/Dropbox/dev/clojure/babashka/sci"} and {:mvn/version "0.2.4", :deps/manifest :mvn}
I would say: always prioritize the local/root?#2021-04-2812:07borkdudeI don't understand where it gets 0.2.4 from and I can't see this with clojure -Stree#2021-04-2812:08borkdudemy bad, I added a dep in my home .clojure/deps.edn for testing something and this pulled in 0.2.4#2021-04-2812:18dvingoRequest to improve developer experience:
Basic sanity checks of deps.edn keys.
The one I've run into a few times is adding :jvm-opts as a top level key, but it can only be supplied within an alias. It would be nice to get an error or at least a warning if this key is present in the top level map.#2021-04-2812:21borkdude@danvingo clj-kondo already does a couple of these sanity checks. feel free to request more#2021-04-2812:21borkdudein an issue#2021-04-2812:24dvingothanks borkdude! good to know, will try it now#2021-04-3016:06amaris there a way to determine what version of tools deps is being used by clj ?#2021-04-3016:15seancorfieldclj -version if you’re on a recent version, clj -Sdescribe otherwise.#2021-04-3016:16seancorfieldThen look at the https://clojure.org/releases/tools page.#2021-04-3016:16amarthanks @seancorfield#2021-05-0318:16dvingoIs there any way to enable an alias in a dependent deps.edn based project?
Use case is:
I have one deps.edn based projects which depends on others which are all internal to the same organization. During development I want to enable the dev and test aliases (and possibly others) in each of those dependent projects.
Pretty sure this is a feature request an not something that is supported..
What I'm imagining is:
my-org/my-lib {:git/url "#2021-05-0318:24Alex Miller (Clojure team)not currently, no#2021-05-0318:25Alex Miller (Clojure team)you can vote at https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects or comment if you want to add support for that#2021-05-0318:35dvingothanks - voted!#2021-05-0407:45yuhanDoes anyone know of a worked example of integrating Java sources into a tools.deps project like https://github.com/puredanger/clojure-from-java, ideally with support for iterative development/reloading of the Java files?
I understand there's no equivalent of :java-source-paths in deps.edn and we have to do some sort of jar compilation step manually#2021-05-0407:53delaguardohttps://github.com/xapix-io/axel-f/tree/master/src/axel_f/buddy/util
small example
to compile - javac $(find . -name '*.java' -type f)#2021-05-0408:36flowthingNot sure whether this is the best channel for this question, but: I've inherited a codebase that makes heavy use of :pre and :post. The project currently uses Leiningen, where it's possible to disable assertions when building an uberjar by adding :global-vars {*assert* false} into project.clj. Is there a way to disable assertions with tools.deps?#2021-05-0408:37borkdudeHow does leiningen implement this feature?#2021-05-0409:19flowthingNo idea. Could look into it. Might be it just does alter-var-root on *assert* at the start.#2021-05-0409:27flowthinghttps://github.com/technomancy/leiningen/blob/4d8ee78018158c05d69b250e7851f9d7c3a44fac/leiningen-core/src/leiningen/core/eval.clj#L358-L374#2021-05-0409:29flowthingIt'd be nice if there were a system property to disable assertions.#2021-05-0409:30borkdudeI still don't see how leiningen sets the dynamic var for each namespace, or does it establish thread bindings for the entire project, I guess so#2021-05-0409:31flowthingYeah, not sure.#2021-05-0409:31flowthinghttps://ask.clojure.org/index.php/1529/debug-builds#2021-05-0409:35flowthingLooks like my best bet is to bite the bullet and migrate to spec/assert (most of the pre/post conditions are spec assertions).#2021-05-0409:35flowthing(Or fdef.)#2021-05-0409:36borkdudeor wait for spec2 ;)#2021-05-0409:36flowthingI'm afraid I'll need to deliver before that. 😛#2021-05-0416:14seancorfieldWhy do you want to disable the assertions? That would mean that if your code tripped over those conditions in production, it would continue on with bad data and potentially cause all sorts of corruption, instead of failing fast.#2021-05-0416:14seancorfieldI’ve never understood why folks want to disable assertions in production…#2021-05-0416:15borkdudeIt depends if you use those assertions for dev purposes or domain purposes. We use asserts for dev purposes e.g. to verify if our system has the right components in certain function calls#2021-05-0416:19flowthing@U04V70XH6 This codebase is absolutely chock full of pre/post-conditions. Most every function has one. If we enabled them in production, the application would be slow as molasses. It would be a different matter if there were assertions only at the boundaries of the system.#2021-05-0416:24seancorfieldUgh! That’s a poor use of asserts IMO. Doesn’t surprise me though, really. Sorry.#2021-05-0416:25flowthing100% agreed.#2021-05-0408:39pinkfrogHow to let deps list all the direct and indirect jar dependencies of a project ?#2021-05-0409:20borkdude@i
$ bb -e '(run! println (babashka.classpath/split-classpath (with-out-str (babashka.deps/clojure ["-Spath"]))))'
src
/Users/borkdude/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar
/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar
#2021-05-0409:22thhellerclj -Stree?#2021-05-0409:47pinkfrogThanks for the -Stree. Didn’t find that in man clojure on mac. But that option exists in clj -h.#2021-05-0411:11borkdudeSince you mentioned list I thought you were looking for a flat list instead of -Stree :)#2021-05-0414:10souenzzoclj -Stree | tr ':' '\n'#2021-05-1116:05Alex Miller (Clojure team)New dev release of Clojure CLI 1.10.3.829, just sweeping up some minor changes:
• https://clojure.atlassian.net/browse/TDEPS-177 - Fix Maven mirrors to look up by id, not name
• Remove flag when fetching git deps so that older git versions can work
• Tweak some warning messages
#2021-05-1116:05Alex Miller (Clojure team)If no one sees any issues, I'll release to stable soon#2021-05-1203:30practicalli-johnNo problems so far, thank you.#2021-05-1215:56Joshua SuskaloWhen calling resolve-deps, are all the aliases included in the deps-map assumed to be active? That seems like that'd be the only reasonable semantic to me, but I'm not sure.#2021-05-1215:57delaguardoaliases might have overlapping deps, so the answer is - no, resolve-deps expect you to specify which aliases to use#2021-05-1216:00Joshua SuskaloHow's that done? The resolve-deps function takes a deps map with an :aliases key, but doesn't have any way to specify which of the aliases are active in the args-map or other arguments.#2021-05-1216:06delaguardohttps://github.com/clojure/tools.deps.alpha/blob/444b99b5d6b18e308d9a5817c8f07dc88a0bde2f/src/main/clojure/clojure/tools/deps/alpha.clj#L492
take a look. this function is looking only at :deps key. So I think resolve-deps function is just a part of resolution procedure which is taking care about used aliases.#2021-05-1216:11delaguardohttps://github.com/clojure/tools.deps.alpha/blob/9bf5778dc26dd5018dbf04fc8e7dbb32ddc4036c/src/main/clojure/clojure/tools/deps/alpha.clj#L184-L190
so I think before using resolve-deps you shoul use combine-aliases function to build second argument for resolve-deps#2021-05-1216:12Alex Miller (Clojure team)that is correct#2021-05-1216:13Alex Miller (Clojure team)which deps.edns to merge and which aliases to use will depend on the scenario so this is outside calc-basis/resolve-deps#2021-05-1216:29Joshua SuskaloOh, thanks! That helps out a lot!#2021-05-1216:44Joshua SuskaloI see make-classpath is deprecated. Is there something preferred to be used?#2021-05-1216:44Joshua SuskaloJust calc-basis and getting the classpath off of it?#2021-05-1217:26Alex Miller (Clojure team)make-classpath-map is the replacement#2021-05-1217:27Alex Miller (Clojure team)but also, calc-basis returns both a map of classpath data and just the cp#2021-05-1217:27Alex Miller (Clojure team)so kind of depends what you need#2021-05-1217:40Joshua SuskaloAlright. I definitely just need the classpath because I'm producing something to pass to the java compiler, but if calc-basis is preferred, I can definitely use that.#2021-05-1218:02Alex Miller (Clojure team)if calc-basis serves your needs, I would use that over the fine-grained functions#2021-05-1219:06Alex Miller (Clojure team)New release of Clojure CLI https://clojure.org/releases/tools#v1.10.3.833, just some relatively minor changes:
• https://clojure.atlassian.net/browse/TDEPS-177 - Fix Maven mirrors to look up by id, not name
• Remove flag when fetching git deps so that older git versions work
• Tweak some warning messages
• Clean up scripts to simplify variable replacement
#2021-05-1220:09Alex Miller (Clojure team)apologies, that was broken for linux - 1.10.3.839 is now available#2021-05-1220:17seancorfieldAh, I guess I didn’t notice because I use brew on Linux to manage my Clojure CLI stuff?#2021-05-1220:18Alex Miller (Clojure team)yeah#2021-05-1223:03practicalli-johnI always use the Linux install, but I rarely use the clj wrapper script. I'll try remember to test a pre-release with clj as well next time#2021-05-1223:16Alex Miller (Clojure team)well, I should have tested it better :)#2021-05-1313:53otfromthere, but for the grace of <insert deity here> go I#2021-05-1318:41jimmyWhen running -X:deps mvn-pom is there a way to specify a version it should use? I tried a :version key on my deps.edn and that didn't seem to be picked up.#2021-05-1319:21Alex Miller (Clojure team)not right now - you'll need to set it in the pom.xml either before (if syncing) or after (if gen'ing)#2021-05-1319:24seancorfield@jimmy What are you using to build your JAR file? If you’re using depstar, it has an option to update version in the pom.xml file.#2021-05-1319:24seancorfield(it can also generate a pom file, synchronize the deps, and update group ID and artifact ID)#2021-05-1319:29jimmyI am using depstar. Just got that working. Thanks 🙂#2021-05-1602:15andy.fingerhutHuh. A recent question on the Clojure Google group is fun in its implications for Clojure and its tools: Github and Maven projects whose names begin with a digit like this one: https://github.com/2captcha/2captcha-java https://search.maven.org/artifact/com.github.2captcha/2captcha-java#2021-05-1602:17andy.fingerhutQuickest workaround I can think of is to create a Java wrapper lib around it that avoids Clojure needing to see any names starting with a digit#2021-05-1602:38seancorfieldIt works just fine with Leiningen…#2021-05-1602:43andy.fingerhutHmm. I should have tried out more things before mentioning this. Sorry.#2021-05-1602:44seancorfieldIt doesn’t work with deps.edn because the EDN is invalid with the leading 2.#2021-05-1602:44Alex Miller (Clojure team)you can use whatever lib name you like in deps.edn#2021-05-1602:44Alex Miller (Clojure team)oh, this is maven lib#2021-05-1602:45seancorfieldIf it was a :git/url or :local/root you mean @U064X3EF3?#2021-05-1602:46Alex Miller (Clojure team)yeah#2021-05-1602:47Alex Miller (Clojure team)well, happy to get an ask.clojure question to think about it#2021-05-1603:11Alex Miller (Clojure team)a wrapper wouldn't help as you'd still get it as a transitive dep#2021-05-1603:14Alex Miller (Clojure team)maven central doesn't have any groupIds that start with a number, but there are probably ~100 artifactIds that do#2021-05-1603:14Alex Miller (Clojure team)(which is a tiny percentage of maven central of course)#2021-05-1603:16Alex Miller (Clojure team)my initial thought would be to allow string lib names that canonicalize to a symbol#2021-05-1603:16Alex Miller (Clojure team)or support some escape character that's ignored#2021-05-1609:22borkdudeI vote for the string lib name#2021-05-1920:29dpsuttonwas there a way to specify other aliases to use inside of an alias? I feel like i've both seen that and also heard that's explicitly not a thing#2021-05-1920:31seancorfieldYou mean for “aliases as data”?#2021-05-1920:33seancorfieldThe :paths/`:extra-paths` can use aliases instead of strings as I recall. I think you can also specify :exec-args as an alias (which is defined elsewhere as a hash map). And some tooling allows for specific exec args to be aliases that get looked up (`depstar` does, I think deps-deploy does too).#2021-05-1920:33dpsuttonkinda copying from a coworker. it's more an inheritance thing.#2021-05-1920:34dpsutton> With deps.edn is there a way top make an alias inherit from another alias? For example in Leiningen we have the :eastwood profile which includes :linters-common which in turn includes :include-all-drivers, :ee, and :test-common.
> I know you can give it a list of aliases in the CLI but I wouldn't want to have to either type
clojure -M:include-all-drivers:ee:test-common:eastwood
> or just have the :eastwood alias duplicate everything in the :include-all-drivers, :ee, and :test-common aliases#2021-05-1920:37seancorfieldNo, there’s no such feature.#2021-05-1920:37seancorfield(but it’s commonly requested 🙂 )#2021-05-1920:38dpsuttonyeah. perhaps on june 5th 🙂#2021-05-1921:00borkdude@dpsutton FWIW, I made a task runner in babashka that solves these kinds of issues for me (typing long combinations of aliases or hard-to-remember combinations)#2021-05-1921:00dpsuttonyeah been eyeing that for sure.#2021-05-1921:02seancorfield@borkdude How does your task runner solve that issue?#2021-05-1921:04borkdude@seancorfield Here is a brief example:
https://github.com/borkdude/jayfu/blob/main/bb.edn#L6-L10
When you type bb tasks you get an overview of all available tasks in the project:
https://github.com/borkdude/jayfu#tasks#2021-05-1921:05dpsuttonand you would string join some def'ed common aliases?#2021-05-1921:05borkdudesure, why not#2021-05-1921:06borkdudeFull docs are here: https://book.babashka.org/#tasks#2021-05-1921:13seancorfieldAh, so you’d have a task that does apply clojure and the list of aliases. That’s kinda side-stepping the issue by using something outside the Clojure CLI though.#2021-05-1921:15seancorfieldThat’s essentially what our build shell script does: you say build test subproject and it calls clojure with a bunch of aliases because it knows how to expand test in relation to subproject.#2021-05-1921:16borkdudeCorrect. At work most of our projects have already been ported to the bb task runner so we have a unified way of running things#2021-05-1921:16seancorfieldbuild test lowlevel
=>
/var/www/worldsingles/build/clojure/bin/clojure -J-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory -X:lowlevel:test:lowlevel-test:runner
#2021-05-1922:29vemvwhat should be used instead of the deprecated t.d.alpha/make-classpath?
the defns that its impl uses are not deprecated. is it a good idea to simply copy defn make-classpath?#2021-05-1922:30vemvoh whoops, it's asked right above. didn't expect that ^^#2021-05-1922:44seancorfield@vemv I’ve just been using calc-basis since that produces “all” the information you might need for pretty much any t.d.a operation.#2021-05-2000:44Alex Miller (Clojure team)make-classpath-map#2021-05-2018:27Alex Miller (Clojure team)A dev version of the Clojure CLI 1.10.3.849 is now available
• Adds support for a trailing map in -X calls (similar to 1.11 trailing map to vararg calls)
• Updates all Maven deps to latest (maven-resolver 1.7.0, maven core 1.8.3) - https://maven.apache.org/docs/3.8.1/release-notes.html
◦ CVE-2020-13956 - bumps deps on Apache HttpClient (unlikely this would actually be affecting you)
◦ CVE-2021-26291 - potential problems accessing Maven repos, changes of note:
▪︎ http repo access is now blocked by default (to avoid possible MITM). tools.deps has always used https repos in the default repository list (central and clojars) but you could see some impact here if you are currently specifying a http Maven repo (but don't do that)
▪︎ concerns about repository urls getting "hijacked" by transitive pom deps (or their super poms) that declare their own different urls. afaict they made no changes here, but did clarify how repos are resolved on https://maven.apache.org/guides/mini/guide-multiple-repositories.html#repository-order. From a deps perspective, we only use repositories declared in the top-level deps.edn (if transitive deps need a custom repo, you will need to add it at top-level too). For tools.deps use of pom dependencies, we are providing the repos of the top deps.edn file (which should always put Maven Central and Clojars first), then deferring to Maven for the rest.#2021-05-2018:27Alex Miller (Clojure team)Anyhow, would appreciate anyone trying this, particularly if you use repositories other than central and clojars.#2021-05-2018:34borkdude> Adds support for a trailing map in -X calls (similar to 1.11 trailing map to vararg calls)
Can you give an example of what this looks like on the command line?
-X:foo :a 1 :b 2 '{:c 3}' ?#2021-05-2018:34Alex Miller (Clojure team)yep, that#2021-05-2018:35borkdudecool#2021-05-2018:35Alex Miller (Clojure team)haven't added the doc yet as this is not a stable release#2021-05-2018:36Alex Miller (Clojure team)maybe I can share that#2021-05-2018:36Alex Miller (Clojure team)https://github.com/clojure/clojure-site/compare/master...fogus:trail-cli?short_path=8bd3a80#diff-8bd3a80ddf7d25123e1a04abb7bfed88ffb3b35cadcd62945d1491c7aae43ac0#2021-05-2018:36Alex Miller (Clojure team)still a couple things to fix there#2021-05-2018:37Alex Miller (Clojure team)and of course, trailing can be the only arg#2021-05-2018:37Alex Miller (Clojure team)clj -X clojure.core/prn '{:a 1 :b 2}'#2021-05-2214:51dominicmHow can you enable http maven repos again? I've worked at places where there's privately hosted in-VPN http-based repos which is perfectly fine in the context.#2021-05-2215:07Alex Miller (Clojure team)I believe you can change the default blocked repo settings in the m2 settings.xml but I don’t have a link handy#2021-05-2215:18dominicmIs this is a direct result of upgrading maven, rather than a conscious tdeps choice? (as I interpreted it)#2021-05-2215:24dominicmYeah, I think that's the case. I didn't realize you were replicating maven's release notes 🙂#2021-05-2322:51cflemingI’m working on a bug in Cursive where I get the “Manifest type not detected” bug when using :local/root in :override-deps. The bug is basically because Cursive doesn’t call into deps with the CWD set to the project directory. I do set *the-dir* when reading the deps maps with modern deps versions, but perhaps the override is applied later during deps calculation? I’m not sure. I’ve had to fix several bugs of this type, and I think the most robust solution would be to absolutise all the paths in the deps map before passing it to t.d.a. I think the following places in the deps map are where I would have to do it, are there any that I’ve missed?
:local/root "path"
:paths ["src"]
:extra-paths ["test" "resources"]
:replace-paths ["foo" "bar"]
:classpath-overrides {org.clojure/clojure "/my/clojure/target"}#2021-05-2323:13seancorfield@cfleming and account for paths including aliases that are expanded to lists of strings.#2021-05-2323:14cfleming@seancorfield Oh, you mean that the paths include a keyword alias ref? Good catch, thanks.#2021-05-2323:14seancorfieldLike this https://clojure.org/reference/deps_and_cli#_paths#2021-05-2323:32cflemingRight, I remember that now.#2021-05-2520:31Joshua SuskaloThis is referencing something back a good ways, but for typing out long commands, this is exactly the usecase of https://github.com/casey/just , and if you need to do something slightly more complicated like making a deploy task that'll auto-build a jar you might want to look into GNU Make. A lot of people don't consider make much these days, but it fits really well into the part of the ecosystem around dealing with the build process of using the clojure cli.#2021-05-2520:31Joshua Suskalo@dpsutton#2021-05-2520:56borkdude@suskeyhose babashka has a task runner which is inspired by just#2021-05-2520:56Joshua SuskaloOh cool! I saw you talking about a babashka task runner, but I'll admit I haven't really looked into babashka just yet.#2021-05-2520:57Joshua SuskaloOr at least not since a long time ago, back when I was starting with Clojure.#2021-05-2520:57borkdudeIt's explained here, in case you want to have a look: https://book.babashka.org/#tasks#2021-05-2520:59Joshua SuskaloThanks!#2021-05-2712:01simongrayI can’t seem to pick up any :extra-deps inside an alias when running
clojure -A:backend -Stree
where :backend is the alias in question. I wonder why? I thought that -A was the option to use for aliases.#2021-05-2712:02Alex Miller (Clojure team)should work, can you share deps.edn#2021-05-2712:02simongray{:paths ["src" "resources"]
:deps {org.clojure/clojure {:mvn/version "1.10.3"}
org.clojure/data.json {:mvn/version "2.3.1"}
org.clojure/data.csv {:mvn/version "1.0.0"}
semantic-csv/semantic-csv {:mvn/version "0.2.1-alpha1"}
time-literals/time-literals {:mvn/version "0.1.5"}
io.pedestal/pedestal.service {:mvn/version "0.5.9"}
io.pedestal/pedestal.route {:mvn/version "0.5.9"}
io.pedestal/pedestal.jetty {:mvn/version "0.5.9"}
luminus/ring-ttl-session {:mvn/version "0.3.3"}
org.slf4j/slf4j-simple {:mvn/version "1.7.30"}
hiccup {:mvn/version "1.0.5"}
;; Using the same version of Jetty as Pedestal, since upgrading will
;; create an incompatibility with newer versions of Jetty:
;;
org.eclipse.jetty/jetty-servlets {:mvn/version "9.4.35.v20201120"}
;kuhumcst/cuphic {:local/root "/Users/rqf595/Code/cuphic"}
;kuhumcst/rescope {:local/root "/Users/rqf595/Code/rescope"}
;kuhumcst/recap {:local/root "/Users/rqf595/Code/recap"}}
kuhumcst/cuphic {:git/url ""
:sha "5122b24da9e1e723f4d1496b1acc98bc699d9d38"}
kuhumcst/rescope {:git/url ""
:sha "901879bdfa489a70b26a1e9fdd381475207130ca"}
kuhumcst/recap {:git/url ""
:sha "2668835572a5523bc5a9f0f5543dd85ab1c31681"}}
;; For now, the backend alias only contains metabase/saml20-clj since it
;; brought in a Google Guava dependency that conflicted with shadow-cljs.
;; Using a separate alias for backend-only deps solves this.
:aliases {:backend {:extra-deps {metabase/saml20-clj {:mvn/version "2.0.0"}}}
:frontend {:extra-paths ["dev"]
:extra-deps {org.clojure/clojurescript {:mvn/version "1.10.866"}
thheller/shadow-cljs {:mvn/version "2.14.0"}
reagent {:mvn/version "1.0.0"}}}
:test {:extra-paths ["test"]}}}#2021-05-2712:04Alex Miller (Clojure team)I see it when I use that command#2021-05-2712:04simongraythat’s strange#2021-05-2712:05Alex Miller (Clojure team)maybe try adding -Sforce to make sure you don't have something weird in cpcache?#2021-05-2712:05simongrayThat did the trick”!#2021-05-2712:05simongraythanks Alex#2021-05-2712:06Alex Miller (Clojure team)did you copy that file in or do anything that might cause the deps.edn to have an old timestamp?#2021-05-2712:06Alex Miller (Clojure team)the script checks timestamp of cached cp vs deps.edn#2021-05-2712:07simongraynope#2021-05-2712:07simongrayOrdinarily, I use Cursive to fetch deps and so on, but in this case I needed to test stuff on the command line since my build was failing in production#2021-05-2712:08simongraymaybe that has something to do with it#2021-05-2712:27Alex Miller (Clojure team)probably - I don't think Cursive uses the cpcache so could be put something weird in combination there#2021-05-2721:51cflemingRight, Cursive doesn’t use the cpcache.#2021-05-2811:06otfromIs .cpcache only used in the local directory, not from transitive dirs#2021-05-2811:06otfromjust wondering what I need to clean up#2021-05-2811:09borkdude@otfrom fwiw -Sforce ignores the local .cpcache. When not using that, the CLI calculates a hash of the used deps.edn + options and looks if there is already a file with that hash. All only in the local directory.#2021-05-2811:10otfromthx!#2021-05-2811:12borkdudeFor reference:
https://github.com/borkdude/deps.clj/blob/master/src/borkdude/deps.clj#L453-L458#2021-05-2811:12borkdude(this is a faithful translation from the bash/Powershell code to Clojure)#2021-05-3113:27Nicolas EstradaHi everyone, first time poster. I'm trying to use tools.cli from within a babashka script. So I'm not sure if my question should be oriented there or here but here it goes#2021-05-3113:28Nicolas EstradaDoes anyone know how to enforce required options (ie. not missing) with tools.cli?#2021-05-3113:29Nicolas EstradaI've tried `:validate` and `:validate-fn` but they don't seem to be invoked for the value specified by the `:default` key (which as was suggested was ::missing and the :validate fn was :validate [(fn [x] (println "XXXXX:" x) (not= ::missing)) "Missing required option: --gitlab-token"]#2021-05-3113:30borkdude@nicolas.estrada938 You could just do (assert (:required (:options parsed)) "required is required")#2021-05-3113:31Nicolas EstradaShould I do this after I invoke parse-opts or in a :validate key?#2021-05-3113:31borkdudeafter#2021-05-3113:33Nicolas Estradaoh ok... that's all I needed to know 😉 Thanks! I do feel that tools.cli could benefit with a :mandatory flag the way https://github.com/grammarly/omniconf does (too bad it doesn't work with babashka 😅 )#2021-05-3113:33borkdudeI think required options are contextual so it depends on how you invoked the tool. E.g. tool --help doesn't require any other options, but tool --foo might#2021-05-3113:35borkdudeWhere does omniconf fail? We could look to make it work with bb if it's within reasonable distance. Let's talk in #babashka#2021-05-3113:36Nicolas EstradaOf course they are contextual, but if tools.cli performs validation I believe checking for existance falls into that category#2021-05-3113:36borkdudePerhaps #clojure is the best channel to discuss new features for tools.cli.#2021-06-0110:15pavlosmelissinosI have the following deps.edn file:
{:deps {org.clojure/clojure {:mvn/version "1.10.3"}
org.apache.httpcomponents/httpclient {:mvn/version "4.5.13"}}
:aliases {:nvd {:extra-deps {lein-nvd/lein-nvd {:git/url "
If I run clj -M:nvd, I'm getting an error about not being able to find artifact org.apache:apache:pom:21 (I'm including the full report in a conversation thread)
If I remove the dependency on org.apache.httpcomponents/httpclient, I get no errors.
What could be the issue here?
Apologies if it's off-topic 🙂#2021-06-0110:16pavlosmelissinos#2021-06-0110:56pavlosmelissinosHuh, it seems to be fixed if I manually add the maven repositories:
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}
I thought both of these were used by default :thinking_face:#2021-06-0112:44Alex Miller (Clojure team)They are. It’s possible changing the file just affected the staleness check and forced a cache recompute#2021-06-0112:45Alex Miller (Clojure team)If you remove those, does it still work?#2021-06-0112:48pavlosmelissinosI does not, unfortunately! 😞
> Execution error (ArtifactNotFoundException) at org.eclipse.aether.internal.impl.DefaultArtifactResolver/resolve (DefaultArtifactResolver.java:414).
> Could not find artifact org.apache:apache:pom:21
I'm on Clojure CLI version 1.10.3.855 btw#2021-06-0112:50Alex Miller (Clojure team)I’ll look at it in a little bit then#2021-06-0112:50Alex Miller (Clojure team)Might have to do with it being a pom dep#2021-06-0112:51pavlosmelissinosThanks! Let me know how I can help 🙂#2021-06-0203:04λustin f(n)I am trying to use lein-tools-deps to build an uberjar that includes a :git/url dep. It works on my local machine, but not when part of a CircleCi build. Does anyone know if there is a setting or something I am missing that would fix it?#2021-06-0203:04λustin f(n)Cloning:
org.eclipse.jgit.api.errors.TransportException: : Auth fail
at org.eclipse.jgit.api.FetchCommand.call (FetchCommand.java:248)
#2021-06-0203:05λustin f(n)My deps.edn looks like:#2021-06-0203:05λustin f(n){:deps {org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/tools.deps.alpha {:mvn/version "0.8.695"}
com.wsscode/pathom3 {:git/url ""
:sha "354574f6a4fbd30e54c85fa41d5bf2eb7de59a39"}}}
#2021-06-0203:05λustin f(n)And project.clj:#2021-06-0203:06λustin f(n)(defproject sandbox "0.1.0-SNAPSHOT"
:main ^:skip-aot banzai.sandbox.core
:plugins [[duct/lein-duct "0.12.1"]
[lein-tools-deps "0.4.5"]]
:middleware [lein-duct.plugin/middleware
lein-tools-deps.plugin/resolve-dependencies-with-deps-edn]
:lein-tools-deps/config {:config-files [:install :user :project]}
:profiles {:uberjar {:aot :all}})
#2021-06-0203:33Alex Miller (Clojure team)Circleci has a setting in their .git config that forces all https urls to ssh, which won’t work unless you install an ssh key etc#2021-06-0203:34Alex Miller (Clojure team)But you can modify your git config to stop doing that#2021-06-0210:46lreadHere’s the CircleCI fix many of us now use: https://clojurians-log.clojureverse.org/circleci/2021-01-17#2021-06-0415:11dwhite44I am having the most bizarre issue on my machine. I am on a Mac with the clojure cli installed via homebrew. If I launch an app or repl from cli, all network calls fail immediately with a socket error “Operation Not Permitted” and “Connection Refused”. Doing the exact same thing using leiningen works fine. Everything else is exactly the same. I’m not able to connect to databases or make http requests. I have the same issue when launching a server from figwheel (both legacy and figwheel-main). Occasionally I can get it to work for a single connection usually after uninstalling and re-installing Clojure in homebrew. I suspect this could be due to security software as my company has forced a few new scanners on our computers lately, but I’m just wondering what could possibly be different about the calls being made when launching via cli vs lein. Here is the simplest example of what fails for me. The equivalent with a project.clj and using lein repl works just fine. Any ideas what could possibly be different?#2021-06-0415:20Alex Miller (Clojure team)well if you want to get clj out of the debugging, you could clj -Spath and then java -cp <that-path> clojure.main#2021-06-0415:22Alex Miller (Clojure team)I'd wonder about Java version, and whether you have granted that Java version access to do network things#2021-06-0415:22Alex Miller (Clojure team)it will pop up a prompt for that - possible that's getting buried somehow through clj. just guessing#2021-06-0415:22dwhite44well i have tried multiple java versions. currently on adoptopenjdk 1.8. and using same version whether cli or lein#2021-06-0415:25borkdudePerhaps you could also try -Srepro which ignores config in your user folder#2021-06-0415:27Alex Miller (Clojure team)I'd rather get clj completely out of the way first and see if it still fails#2021-06-0415:28dwhite44so yes it does work when doing the java -cp and still fails when doing the -Srepro#2021-06-0415:28Alex Miller (Clojure team)does clj / clojure differ?#2021-06-0415:32dwhite44they both fail#2021-06-0415:33Alex Miller (Clojure team)the clojure bash script ultimately basically does java -cp cp clojure.main#2021-06-0415:34Alex Miller (Clojure team)what if you just wrap the call to java in a simple bash script?#2021-06-0415:35dwhite44i’ll give it a try#2021-06-0415:36Alex Miller (Clojure team)#!/usr/bin/env bash
exec java -cp ... clojure.main
#2021-06-0415:37dwhite44hmm. that fails. seems surprising#2021-06-0415:37Alex Miller (Clojure team)the plot thickens :)#2021-06-0415:37Alex Miller (Clojure team)bash?#2021-06-0415:38Alex Miller (Clojure team)lein is also bash that is#2021-06-0415:39Alex Miller (Clojure team)maybe look at perms for the clojure and lein scripts and their ownership / group?#2021-06-0415:40dwhite44ok. so yeah tried each method a few times and consistently calling java -cp … directly works and running the bash script with the same thing doesn’t#2021-06-0415:41Alex Miller (Clojure team)what if you remove the exec in the script?#2021-06-0415:41Alex Miller (Clojure team)that's a difference between lein script and clj script although seems hard to believe that's the difference#2021-06-0415:42dwhite44hmm. yes removing exec works#2021-06-0415:43Alex Miller (Clojure team)interesting#2021-06-0415:44dwhite44heh - i have some other words for it but yeah that one fits too 🙂#2021-06-0415:44delaguardomaybe bash is replaced by recent security tools update?#2021-06-0415:45Alex Miller (Clojure team)as we say at Nubank... fascinating#2021-06-0415:45dwhite44i am using zsh but have also tried running it via bash w/ same result. wouldn’t think that would make a difference. course i wouldn’t think exec and no exec would make a difference#2021-06-0415:46delaguardobash is used by clj/clojure scripts#2021-06-0415:46dwhite44looks like i’m using bash from homebrew not the system bash#2021-06-0415:47delaguardomac os X is no longer shipped with system bash )#2021-06-0415:48dwhite44ah. true. not sure if it helps at all but i’m on Catalina just upgraded last week from Mojave in an attempt to fix this issue#2021-06-0415:49dpsuttonI don't mean to derail but i had a colleague that would get different jvms when using lein and clj. Quite subtle and due to homebrew bringing in jdk15 for lein. You mentioned you were using the same 1.8 jvm but how sure are you on this? How did you check it at runtime. (sorry if this is irrelevant but just something i saw recently)#2021-06-0415:50dwhite44yeah i’ve been wondering about that too. is it possible to get the jvm from repl? i do have JAVA_HOME set and the clojure script seems to use that if set rather than the openjdk installed by homebrew#2021-06-0415:51Alex Miller (Clojure team)clj -M -e '(System/getProperty "java.version")'#2021-06-0415:51dwhite44“1.8.0_292”#2021-06-0415:51Alex Miller (Clojure team)I don't think the java is relevant here#2021-06-0415:51dwhite44same as when i do java -version#2021-06-0415:52Alex Miller (Clojure team)smells to me like there is some kind of prohibition re fork#2021-06-0415:53Alex Miller (Clojure team)I can imagine something like that from a security pov#2021-06-0415:53Alex Miller (Clojure team)I don't know what it is or how to check for it#2021-06-0415:53dwhite44what’s really baffling to me is when it will work 1 time. yesterday i uninstalled clojure, did brew cleanup, brew doctor, re-installed and it worked for several attempts. but then by the end of the day, it stopped working#2021-06-0415:55dwhite44also when i do have a working repl or application instance, it seems to work endlessly until that is broken. so definitely seems to be some sort of permissions thing on when it starts#2021-06-0416:09Alex Miller (Clojure team)do higher ports work? ports < 1024 require root privileges to open I believe#2021-06-0416:10dwhite44network calls to higher ports?#2021-06-0416:10ghadiprivilege for low ports is on listening sockets, not destinations#2021-06-0416:10Alex Miller (Clojure team)ah, nvm#2021-06-0416:11delaguardodoes it work with curl when it become broken with clj?#2021-06-0416:11ghadi> I suspect this could be due to security software as my company has forced a few new scanners on our computers lately
what's that about?#2021-06-0416:11ghadi(reading through scrollback)#2021-06-0416:12borkdudeit's always the virus scanners#2021-06-0416:13dwhite44have Carbon Black and Microsoft Defender. wouldn’t shock me if there is other than I haven’t noticed#2021-06-0416:13borkdudeare you on windows?#2021-06-0416:13dwhite44but none of my co-workers are having this issue#2021-06-0416:13dwhite44no. mac Catalina#2021-06-0416:14borkdudeI didn't know Microsoft had a virus scanner for mac#2021-06-0416:14dwhite44my company likes to install as much Microsoft as they can though lol#2021-06-0416:15Alex Miller (Clojure team)the works for a while and then stops seems like it would match that theory#2021-06-0416:15dwhite44works very briefly then stops. almost like the first attempt gets through but then it gets blocked. but what could possibly be different about these network requests?#2021-06-0416:15ghadican always check the diagnostic logs for rejected connection attempts#2021-06-0416:16dwhite44will say they fail immediately. don’t think they are actually making it to the network#2021-06-0416:16ghadi(AV tools, surveillance ware or macOS Console)#2021-06-0416:16ghadifrom the description, it doesn't seem like any network activity is happening#2021-06-0416:18Alex Miller (Clojure team)exec is going to replace the process so I can imagine a virus scanner seeing a process, replaced with another process space, then making network calls, as a possible virus#2021-06-0416:19ghadimost solid hypothesis so far ^#2021-06-0416:19Alex Miller (Clojure team)lein doesn't do the exec and works#2021-06-0416:19borkdudeso then clojure would work, but not clj?#2021-06-0416:20Alex Miller (Clojure team)well they both exec at the end#2021-06-0416:20borkdudeoh wait, clojure also does exec#2021-06-0416:20Alex Miller (Clojure team)presumably a virus scanner would have some way to see and make this ok#2021-06-0416:20borkdudeWell, if that theory holds, then @dwhite44 could try https://github.com/borkdude/deps.clj and see if that works, since it doesn't exec#2021-06-0416:20borkdude(just to confirm the theory more)#2021-06-0416:21borkdude@dwhite44 you could try this binary: https://github.com/borkdude/deps.clj/releases/download/v0.0.15/deps.clj-0.0.15-macos-amd64.zip
Instead of clojure you invoke deps. deps -Spath or whatever#2021-06-0416:21dwhite44i can give it a try. i think the sh file that works w/ no exec and fails with it is pretty good proof#2021-06-0416:22borkdudeBut it could be that the virus scanner also marks this binary as a virus of course#2021-06-0416:23dwhite44is anybody familiar with how figwheel works? assume it also creates a new process and is likely why that’s not working? ultimately, I can get around the clojure cli not working by using leiningen but I can’t really do what i need to do if I can’t get around the figwheel problem#2021-06-0416:23borkdudeyou could also try the uberjar if that is the case:
https://github.com/borkdude/deps.clj/releases/download/v0.0.15/deps.clj-0.0.15-standalone.jar
java -jar <uberjar>.jar -Spath
#2021-06-0416:24borkdudelet's take it one step at a time#2021-06-0416:24dwhite44right. just wanted to add the fighwheel as an additional hint. seems like that probably confirms the issue is the extra process created by exec#2021-06-0416:25dwhite44well maybe not confirms but more evidence#2021-06-0416:30dwhite44yes it does work using borkdude/deps#2021-06-0416:30Alex Miller (Clojure team)I mean, you can hack your clojure to remove the exec :)#2021-06-0416:36dwhite44yeah. i’ll give my project a try using that. i suspect though that figwheel will make it fail for the same reason#2021-06-0416:37dwhite44but at least knowing that difference is causing the problem gives me something to tell the IT people so they can look at the anti-virus software settings or something. I really appreciate everybody’s help on this!#2021-06-0417:25borkdude@dwhite44 @alexmiller possibly this also works as a workaround:
alias exec=''
#2021-06-0417:27borkdudeI would write a wrapper script called clojure2 / clj2 where I would apply this only locally and then invoke the normal tool (to not mess with other calls of exec in your system)#2021-06-0417:32dwhite44good idea!#2021-06-0417:40Alex Miller (Clojure team)that's a built-in - can you really subvert it like that?#2021-06-0417:40borkdudeI tested it and it seems to work#2021-06-0417:41borkdudeat least in zsh#2021-06-0417:41Alex Miller (Clojure team)seems terrible :)#2021-06-0417:41Alex Miller (Clojure team)that you can do that :)#2021-06-0417:41borkdudeit's a bit like clojure in this respect, I like it :P#2021-06-0417:43borkdudehmm, forget it, it doesn't seem to work#2021-06-0417:53borkdudeThis seems to work:
clojure2:
#!/usr/bin/env bash
exec() {
echo "exec replacement"
eval
#2021-06-0417:55borkdude(I don't give up)#2021-06-0418:34dwhite44hmm. using that script actually doesn’t work for me even though i do see the “exec replacement” being output#2021-06-0418:43borkdude@dwhite44 to be 100% certain it's exec triggering your error, make a copy of clojure and remove "exec" from it#2021-06-0418:44borkdudeif the error persists, bisect the script#2021-06-0418:45borkdudeor maybe even easier: change echo "exec replacement" into echo "exec replacement" #2021-06-0418:45borkdudeperhaps that will tell you something, not sure#2021-06-0418:48borkdudejust copying the script and playing around with, inserting echo bla etc will probably help you most to discover what triggers it#2021-06-0418:53dwhite44copying the original script and removing the execs does work#2021-06-0418:54borkdudeok, well, maybe keep doing that then :)#2021-06-0418:57dwhite44hmm. so interesting. even running the application with figwheel seems to be working using that. unless it’s just the brief temporary reprieve i sometimes get from making a change. guess i’ll know soon enough#2021-06-0419:50dwhite44this definitely seems to be at least a good solution for now. thanks everybody!#2021-06-0715:12Ian Fernandezthere is a way to pass the java classpath indo a deps.edn file?#2021-06-0715:13Ian FernandezI'm trying to pick the classpath that runs on a java -cp .... and trying to pass them to a deps.edn file#2021-06-0715:13Alex Miller (Clojure team)not into a deps.edn file, but you can at the Clojure CLI with clojure -Scp whatever#2021-06-0715:14Alex Miller (Clojure team)that will just override all classpath construction from the deps.edn file#2021-06-0715:14Ian Fernandezhum...#2021-06-0715:14Alex Miller (Clojure team)I'm not sure I understand what you're trying to do tbh#2021-06-0715:14Ian FernandezI can do this to make a pom.xml instead?#2021-06-0715:15Ian Fernandez(then it'll go out off the scope)#2021-06-0715:15Alex Miller (Clojure team)can you explain what "trying to pass them to a deps.edn file" means?#2021-06-0715:15Ian Fernandezif I can use
{:paths [,,,]}
with the files on classpath#2021-06-0715:16Ian Fernandezor something like this#2021-06-0715:17Alex Miller (Clojure team)I mean, the classpath is a string - you can split it on classpath splitter, and then copy that into the :paths#2021-06-0715:17borkdude@d.ian.b Do you mean, you want to add a directory with .jar files or .class files in it, to the classpath by adding them to deps.edn?#2021-06-0715:19Ian Fernandezyeah...#2021-06-0715:20borkdudeYou can do this using :local/root#2021-06-0715:17Alex Miller (Clojure team)I'm not sure why you want to do this#2021-06-0715:18borkdudeI'm a bit confused too#2021-06-0715:21Ian FernandezI'm trying to have a IntelliJ classpath to run a repl with tools.deps#2021-06-0715:22borkdudein that case it what Alex said: you can start it like clj -Scp <intellij-classpath>#2021-06-0715:22Ian Fernandezthe idea is that I don't have the IntelliJ classpath right now...#2021-06-0715:22Ian Fernandezright now the repl is executed by using:
java -cp ,,,, clojure.main -m user ,,,#2021-06-0715:23Ian FernandezI want to have a way to IntelliJ detect this classpath#2021-06-0715:23Alex Miller (Clojure team)you can get the classpath from a deps.edn file at the terminal with clj -Spath#2021-06-0715:26Ian Fernandezthe idea right now is that I don't have a deps.edn file#2021-06-0715:26Ian FernandezI want to build one#2021-06-0715:27borkdude@d.ian.b Are you using Cursive by any chance? I think it already has deps.edn support#2021-06-0717:45seancorfield@alexmiller I know :path and :extra-paths can use aliases-as-data. Can :jvm-opts? We’re finding that we have a common set of JVM options we want to apply in multiple contexts (aliases) and we’d rather not have to introduce a new alias just for those, but if we could add :jvm-opts :common-jvm-opts to each alias that would be nice.#2021-06-0717:45Alex Miller (Clojure team)Not currently#2021-06-0717:46seancorfieldThanks. Saves me grubbing around in the source. Do you think it would be worth adding?#2021-06-0718:37seancorfieldhttps://ask.clojure.org/index.php/10674/could-tools-alpha-support-opts-aliases-data-paths-extra-paths#2021-06-0718:12vlaaadwen tools.build#2021-06-0718:14vlaaadhehe joking. I've been on clojured and wanted to say that your presentation @alexmiller was fantastic, looking forward to tools.build!#2021-06-0718:15dpsuttonis that publicly available yet? my schedule didn't line up to catch it live unfortunately#2021-06-0718:27seancorfieldI attended the whole conference live from 1 am Pacific to 11 am Pacific — it made for a very “slow” Saturday, after the conference 🙂#2021-06-0718:21Alex Miller (Clojure team)Not available yet, hopefully soon#2021-06-0718:37seancorfieldhttps://ask.clojure.org/index.php/10674/could-tools-alpha-support-opts-aliases-data-paths-extra-paths#2021-06-0718:46borkdudeClojureD was great and it was very nice of you to come over there Alex! Enjoyed the presentation.#2021-06-0807:20pavlosmelissinosI have to update an old codebase to work with a new tool and just wanted to let you all know how happy the :exclusions key makes me 🙂#2021-06-0918:49dnolenis a version of tools-deps w/o JGit shipped yet?#2021-06-0919:04seancorfield@dnolen https://clojure.org/releases/tools#v1.10.3.814 — that happened back in March.#2021-06-0919:05seancorfield(but there were some bugs in that version, so you’ll need to use a most recent one)#2021-06-0919:10dnolenah k cool, great, just checking#2021-06-1020:17Alex Miller (Clojure team)New prerelease of Clojure CLI 1.10.3.875:
• New: support for specifying multiple functions with -X with -> semantics
• https://clojure.atlassian.net/browse/TDEPS-182 - Improve deprecation message to be more accurate
• https://clojure.atlassian.net/browse/TDEPS-183 - Fix -Sdescribe output to be valid edn on Windows
• https://clojure.atlassian.net/browse/TDEPS-179 - Fix incorrect classpath when :classpath-overrides removes path#2021-06-1103:03seancorfieldWe’ve simplified our CI build by using this to run multiple steps at once so it’s already a win for us — thank you!#2021-06-1020:21seancorfieldAwesome! Can you point me to an example of how that new -X stuff works? Is it literally just -X:myalias foo.bar/fn1 foo.bar/fn2 some.other/thing and each should take a hash map and return a hash map?#2021-06-1020:22Alex Miller (Clojure team)yes#2021-06-1020:22seancorfieldWhat happens if any of them return nil? Does exec just pass the original input hash map into the next function, or does it end up passing nil?#2021-06-1020:22Alex Miller (Clojure team)well importantly, the first function gets the kvs from the exec-args/ cli#2021-06-1020:22Alex Miller (Clojure team)it's like ->, so if one function returns nil, the next gets nil#2021-06-1020:24seancorfieldAh, so that also means that the functions in the 2nd pos on don’t need to take hash maps, right? It could be any data produced from the earlier fns?#2021-06-1020:25Alex Miller (Clojure team)I suppose :)#2021-06-1020:25Alex Miller (Clojure team)I will not claim that this is fully baked yet, so I reserve the right to change my answer before it becomes a stable release :)#2021-06-1020:25seancorfield(I’m already imagining how folks like @borkdude might think of abusing this 🙂 )#2021-06-1020:25seancorfieldevil chuckle#2021-06-1020:32dpsuttonthat seems really interesting#2021-06-1020:32dpsuttonwith low startup time that makes bash very close to a useable repl#2021-06-1020:33Alex Miller (Clojure team)repls are also very usable repls :)#2021-06-1020:33dpsuttonhaha very true 🙂#2021-06-1020:35vlaaad> -X:myalias foo.bar/fn1 foo.bar/fn2 some.other/thing
how does it know that it's (-> {} foo.bar/fn1 foo.bar/fn2 some.other/thing) and not (foo.bar/fn1 {'foo.bar/fn2 'some.other/thing}) ?#2021-06-1020:35borkdude@seancorfield could you help me understand how I could abuse this? ;)#2021-06-1020:35Alex Miller (Clojure team)symbols at the front are functions#2021-06-1020:36borkdudeand @dpsutton about startup time: how does this solve startup time?#2021-06-1020:36Alex Miller (Clojure team)fn* [kpath v]* map?#2021-06-1020:36dpsuttoni was thinking about sci/bb but at that point you could just pass in forms to be evaled#2021-06-1020:36vlaaadah, so clj-exec first consumes all symbol args as fn chain, and then rest args as map arg?#2021-06-1020:37Alex Miller (Clojure team)well we now have trailing map support ala 1.11 too#2021-06-1020:37vlaaad> fn* [kpath v]* map?
this is great explanation 😄#2021-06-1020:38Alex Miller (Clojure team)https://github.com/clojure/brew-install/blob/1.10.3/src/main/clojure/clojure/run/exec.clj#L100#2021-06-1020:38Alex Miller (Clojure team)to be more precise#2021-06-1020:38Alex Miller (Clojure team)parsing is done with that spec#2021-06-1020:48vlaaadJust tried it out on pwsh, it works#2021-06-1020:48Alex Miller (Clojure team)but of course :)#2021-06-1020:48vlaaadnot sure how I feel about the -X threading#2021-06-1020:48Alex Miller (Clojure team)thanks :)#2021-06-1020:50vlaaadI tried this:
clj -X clojure.core/pr-str clojure.core/read-string clojure.pprint/pprint :a 1
but when I was playing with it I felt the order is somewhat hard to follow, because arg-map is in the end, unlike in real ->#2021-06-1116:01rickmoynihan(defmacro <- [,,,],,,) :grinning_face_with_one_large_and_one_small_eye:#2021-06-1020:50Alex Miller (Clojure team)interested in feedback, so do more , and come back#2021-06-1020:52vlaaadintuitively I wanted to write this:
clj -X clojure.pprint/pprint clojure.core/read-string clojure.core/pr-str :a 1
because arg-map is closer to fn that will receive it. e.g.
(clojure.pprint/pprint (clojure.core/read-string (clojure.core/pr-str {:a 1})))#2021-06-1020:52Alex Miller (Clojure team)I don't know that this is something you're going to want to use all the time, but it is another dimension of composition available#2021-06-1020:52Alex Miller (Clojure team)the goal is not to replicate the repl. if you want that, use a repl#2021-06-1020:52vlaaadmm yeah...#2021-06-1021:00seancorfieldI guess my big test is going to be breaking depstar up into multiple -X-compatible functions so you can run parts of it or all of it in a pipeline 🙂#2021-06-1021:02Alex Miller (Clojure team)if you do do that, I would be very interested in hearing your questions/feedback. if I may be so bold, that might be a better thing to do in design space before actually cutting the code :)#2021-06-1021:06seancorfieldOh, definitely. I need to tease apart the various useful processes that depstar performs and figure out exactly what inputs and outputs each of them have.#2021-06-1021:06seancorfieldI have smaller things to test it on before that 🙂#2021-06-1021:09Alex Miller (Clojure team)the key thing is that inputs/outputs need to align, which harkens to namespaced keys ... but those fight with conciseness of what you want at the CLI#2021-06-1021:05vlaaadOkay, another argument why (fn1 (fn2 argmap)) might make sense:
PS C:\Users\Vlaaad\Projects\deps-test\lib> Get-Content .\deps.edn
{:aliases {:deploy {:ns-default deploy}}}
PS C:\Users\Vlaaad\Projects\deps-test\lib> Get-Content .\src\deploy.clj
(ns deploy)
(defn dev [args]
(merge
{:env :dev
:url "dev-cluster"}
args))
(defn prod [args]
(merge
{:env :prod
:url "prod-cluster"}
args))
(defn deploy [env]
(prn :deploy env))
Suppose I want to deploy my app. Currently I have to use this order: dev deploy
PS C:\Users\Vlaaad\Projects\deps-test\lib> clj -X:deploy dev deploy :a 1
:deploy {:env :dev, :url "dev-cluster", :a 1}
But this is unintuitive, as I would expect the at the command line first specify an "action" and then "env"
PS C:\Users\Vlaaad\Projects\deps-test\lib> clj -X:deploy deploy dev :a 1
:deploy {:a 1}
but with -> style the main action goes last instead of being first#2021-06-1021:08seancorfieldI don’t understand this at all. -> says “do things in this order” and that’s exactly what I would expect: setup the env, run the command.#2021-06-1021:10Alex Miller (Clojure team)agreed, the order is the most important thing here#2021-06-1021:11vlaaadso it's more like clj -X:build clean jar than clj -X:build deploy dev ?#2021-06-1021:12Alex Miller (Clojure team)yes#2021-06-1021:21seancorfieldclojure -X:depstar sync-pom compile jar for example 🙂#2021-06-1021:31vlaaadmm yeah, that makes a lot of sense#2021-06-1100:48phronmophobic> clojure -X:depstar sync-pom compile jar
If I wanted to add a prepare-assets step? How would that fit in? Is it like writing any other clojure function, or would it be like writing a depstar plugin? How likely is it that an idiomatic clojure function would also function as a cli target?
One of the reasons automating git is a pain is that there 1) a low level programmatic library and 2) a high level command line interface and not much in between. Maybe there's no way around it, but it seems like it's easy to fall into the trap where the cli is the primary interface and the programmatic interface becomes less idiomatic. It's a tough problem to crack since the clojure cli wants to tame the bashisms, clojurisms, and javaisms with just the right isms on top.#2021-06-1101:33seancorfieldI’ll try to choose the hash map contents/keys carefully but it would be just writing a function that took a hash map and return a hash map and then making sure it’s on the classpath. Some of it would depend on what inputs your prepare-assets function needed?#2021-06-1101:34seancorfieldI think having a task that computes the project basis (possible based on aliases provided as :exec-args) and then passes that through the pipeline would be helpful.#2021-06-1101:35seancorfieldI think we may all get a better sense of how this should work once we see tools.build — I suspect it will have a few “utility” tasks that standardize some of this setup. @alexmiller has indicate that tools.build will have a number of “standard” build steps built-in so it’ll be a case of other tools following that lead and building on the basic pipeline.#2021-06-1101:37seancorfielddepstar does a lot of “smart” stuff when building uberjar files (in terms of merging things) so once I have tools.build to play with, I’ll figure out how to make the “smart” part of depstar replace the (presumably, much simpler) uber/`jar` tasks.#2021-06-1102:21Alex Miller (Clojure team)There will be a basis building function. Well, there are multiple atm, I’m trying to sort all that out :)#2021-06-1022:15seancorfield@alexmiller What’s the thinking around -X functions that would require (shutdown-agents) calls at the end to avoid the 60 second “hang”? That’s been a long-standing issue with -main functions. I didn’t check the latest exec source but does it handle that? (since you clearly cannot chain exec functions that call shutdown-agents)#2021-06-1022:21Alex Miller (Clojure team)Haven’t thought about it#2021-06-1022:25seancorfieldI just looked at the exec.clj code and it does not call (*exit* 0) or (shutdown-agents) in the “happy” case…#2021-06-1022:34seancorfieldYeah, my first test of chaining tasks leads to a 60 second hang at the end because of that.#2021-06-1022:35seancorfield(I used to run the tasks individually and they had calls to (shutdown-agents) at the end — which I removed so I could chain them — and I can’t just call clojure.core/shutdown-agents via -X because it takes no arguments 😞#2021-06-1102:22Alex Miller (Clojure team)Yeah, we should probably do that for you#2021-06-1103:00seancorfieldThank you, yeah, that would be super-helpful!#2021-06-1103:03seancorfieldWe’ve simplified our CI build by using this to run multiple steps at once so it’s already a win for us — thank you!#2021-06-1323:50cflemingWhen invoking t.d.a via the CLI from Cursive, currently I have to search under the install dir for the uberjar since its location isn’t returned by -Sdescribe. Currently I look for a jar with “tools” in the name and a jar extension. I’ve just debugged a problem with a user where an older version of deps was being used in his case, presumably because his installer leaves older versions of the jar floating around (this is on Linux). Is it a reasonable assumption that the uberjar name will also contain the version? Everything I’ve seen so far seems to be named “clojure-tools-<version>.jar” so hopefully that’s reasonable.#2021-06-1407:09salami submitted a GitHub issue for this a while back and am glad to hear that it's being addressed: https://github.com/cursive-ide/cursive/issues/2105#2021-06-1408:34cflemingWow, you’re right, my apologies for not fixing that at the time and not making the connection to this issue! I have a fix for it which fixes the current problem, it’ll be in the next EAP.#2021-06-1400:51Alex Miller (Clojure team)right now, that's how the uberjar in the installer is named#2021-06-1400:52Alex Miller (Clojure team)not sure why you need the uberjar at all though#2021-06-1400:52Alex Miller (Clojure team)I think it may have been called other things in the past#2021-06-1402:15cflemingI could avoid the need for the uberjar using clj -A:deps -Spath along with -Sdescribe but that’s tricky for UI reasons in Cursive since describe returns immediately but that command takes some amount of time up to a lot. That needs running in the background with a progress meter, and also requires parsing output in order to provide some useful feedback to the user on what’s going on. Looking for the uberjar is just easier at the moment.#2021-06-1414:49pinkfrogIS add-lib deliberately not added in tool-deps?#2021-06-1414:50pinkfrogI am in search of the feature of dynamically adding a lib in the repl.#2021-06-1415:01andy.fingerhutIt is intentionally on a branch that is not the main/master branch, because the time/design/effort has not yet been expended to include it in the main/master branch.#2021-06-1415:02andy.fingerhutI have used it briefly before, and Sean Corfield uses it frequently, so you are free to use it if you wish.#2021-06-1415:02Alex Miller (Clojure team)We are still deciding on various questions about how to include it#2021-06-1521:44seancorfield@alexmiller Peter (Monks) raises an interesting point on ask about -X, given that — currently, at least — the exec stuff goes through this path:
exec "$JAVA_CMD" "${jvm_cache_opts[@]}" "${jvm_opts[@]}" "-Dclojure.basis=$basis_file" -classpath "$cp:$install_dir/libexec/exec.jar" clojure.main -m clojure.run.exec "${exec_args[@]}" "
I wonder whether it’s worth adding detection of --report as a CLI option and passing it into this line between clojure.main and -m?#2021-06-1521:48Alex Miller (Clojure team)it needs more thought. exec does some handling now and it will not necessarily always go through clojure.main#2021-06-1521:49Alex Miller (Clojure team)similarly, some of the choices in exec I like better than what's in main. the exec stuff may eventually make its way into clojure core itself too.#2021-06-1521:52Alex Miller (Clojure team)I have little interest in adding either more options to -X or more special cases :)#2021-06-1522:04seancorfieldThat’s fair. Just thought I’d ask. Personally, with the improvements to error reporting in 1.10, I hardly ever need to rummage around in the temp EDN file, and I’ve never needed to do that in our CI system (BitBucket Pipelines).#2021-06-1522:39Alex Miller (Clojure team)-X follows the lead of other Clojure CLI stuff and will print in the case of an ex-info and throw (resulting in printed stack) in other cases. All error messages in exec and tools.deps throw ex-info so this mostly only yields a stack in unexpected cases. so far, that has been mostly a good strategy.#2021-06-1522:55seancorfieldYeah, I’ve started to switch over to pretty much only throwing ex-info from my own code these days, figuring that having something upstream be able to print the ex-data along with the message will be more valuable than some “random” Java exception.#2021-06-1701:20seancorfieldI just want to go on record and say that my previous grumbling about needing to keep :test and :runner aliases separate to avoid accidentally triggering the :main-opts in :runner when trying to start a REPL with -A no longer applies if you switch over to -X for running tests (`:exec-fn cognitect.test-runner.api/test`). Thank you, Alex, for adding -X execution of tests to the runner!#2021-06-1701:21seancorfield(another reason for preferring -X over -M unless you have a very specific need for -main-style command-line args, IMO)#2021-06-1708:19borkdude@seancorfield The -A behavior should just not invoke the -main anymore, the behavior has been deprecated and will likely adopt to the REPL behavior in the future. Imo, it's got nothing to do with -X vs -M other than that the behavior for -A is currently inconsistent :)#2021-06-1716:21seancorfieldRight and the roots of my comment go back to the original -A behavior. I’m saying that just introducing -X alone is sufficient to negate my grumbling about splitting those aliases.#2021-06-1716:22seancorfieldI just didn’t realize that until recently (I’ve been on a hunt for :test:runner in my projects and merging them so -X:test is all I need).#2021-06-1708:53mkvlrran into this today, nuking ~/.gitlibs fixed it:
clojure -Srepro -Sforce -Sverbose -Stree ✘ 1
version = 1.10.3.855
install_dir = /usr/local/Cellar/clojure/1.10.3.855
config_dir = /Users/mk/.clojure
config_paths = /usr/local/Cellar/clojure/1.10.3.855/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/336449426.cp
Refreshing classpath
Error building classpath.
java.lang.NullPointerException
at clojure.tools.deps.alpha.util.dir$canonicalize.invokeStatic(dir.clj:30)
at clojure.tools.deps.alpha.util.dir$canonicalize.invoke(dir.clj:25)
at clojure.tools.deps.alpha.extensions.deps$eval1393$fn__1395.invoke(deps.clj:27)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__790$fn__792$fn__793.invoke(alpha.clj:403)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__505.invoke(concurrent.clj:34)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
#2021-06-1709:03mkvlrI guess this was the change that triggered it:
diff --git a/deps.edn b/deps.edn
index f44153b6572..602c70006b5 100644
--- a/deps.edn
+++ b/deps.edn
@@ -35,21 +35,17 @@
enlive/enlive {:mvn/version "1.1.6"}
io.pedestal/pedestal.route {:mvn/version "0.5.3"}
- io.pedestal/pedestal.jetty {:mvn/version "0.5.7"}
+ io.pedestal/pedestal.jetty {:git/url "" :sha "bd567123cf78d0d11b82e3bab94473c239b13382"}
image-resizer/image-resizer {:mvn/version "0.1.10"}
stylefruits/gniazdo {:mvn/version "1.0.1"}
- io.pedestal/pedestal.log {:git/url "" :sha "5ab1c9770472694f94d0835cda412ba753b28809"}
+ io.pedestal/pedestal.log {:mvn/version "0.5.9"}
io.pedestal/pedestal.service {:mvn/version "0.5.3"}
io.pedestal/pedestal.service-tools {:mvn/version "0.5.3"}
co.deps/ring-etag-middleware {:mvn/version "0.2.0"}
- ;; pinning jetty version to avoid incompatible changes with 9.4.36.v20210114 bundled with aws-api and other libs
- org.eclipse.jetty/jetty-proxy {:mvn/version "9.4.18.v20190429"}
- org.eclipse.jetty/jetty-http {:mvn/version "9.4.18.v20190429"}
- org.eclipse.jetty/jetty-util {:mvn/version "9.4.18.v20190429"}
- org.eclipse.jetty/jetty-client {:mvn/version "9.4.18.v20190429"}
+ org.eclipse.jetty/jetty-proxy {:mvn/version "9.4.35.v20201120"}
;; pedestal logging (from project template)
org.slf4j/jul-to-slf4j {:mvn/version "1.7.25"}#2021-06-1709:05mkvlroh, just saw GITLIBS_DEBUG, I’ll try that next time I run into this#2021-06-1709:13mkvlralso I was still on 10.0.3.814 when the error first occured:
clojure -Srepro -Sforce -Sverbose -Stree ✘ 1
version = 1.10.3.814
install_dir = /usr/local/Cellar/
#2021-06-1713:10souenzzoIs it a know issue?
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode
I'm having this error sometimes on CI (stacktrace on first comment)#2021-06-1713:10souenzzoError building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1882)
at java.base/java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2061)
at java.base/java.util.HashMap.putVal(HashMap.java:633)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:847)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:659)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:583)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:373)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:494)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.model.building.DefaultModelBuilder.importDependencyManagement(DefaultModelBuilder.java:1299)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:474)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:292)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$eval1076$fn__1078.invoke(maven.clj:107)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__790$fn__792$fn__793.invoke(alpha.clj:403)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__505.invoke(concurrent.clj:34)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
#2021-06-1713:27Alex Miller (Clojure team)not familiar, looks pretty weird#2021-06-1713:29Alex Miller (Clojure team)some googling seems to indicate this is likely a concurrency issue#2021-06-1713:30Alex Miller (Clojure team)could I trouble you to make an ask.clojure question for this?#2021-06-1713:49souenzzodunno exacly who to describe but tried to do my best
https://ask.clojure.org/index.php/10703/error-building-classpath#2021-06-1713:50souenzzoCollecting data to report the only "non-standard" thing that I found is the zulu VM.#2021-06-1714:13Alex Miller (Clojure team)yeah, I don't think the jvm matters#2021-06-1714:13Alex Miller (Clojure team)can you post the deps.edn, that was the more interesting part#2021-06-1718:29souenzzoCan't share, but it's a huge deps.edn, with a structure like seancorfield describes
projects/subproj1
projects/subproj2
projects/subproj3
projects/subproj-common
It uses CLJ_CONFIG="$(cat projects/subproj-common)", maven deps, git deps, local deps, datomic, etc...#2021-06-1719:37Alex Miller (Clojure team)I'm pretty sure it's related to parallel expansion of maven based projects#2021-06-1719:38Alex Miller (Clojure team)maven meaning git or local deps with pom.xml (not maven artifacts)#2021-06-1713:17v3gahmm I'm doing some experimenting with organizing here and I seem to have butchered my deps.edn. The repl starts but none of my dependencies are on the classpath. Can someone point me in the right direction?
https://gist.github.com/v3gal0g/ed79584f09f03ffc996b1ce389d9a3ef#2021-06-1713:23thheller@decim :aliases {:common {:extra-paths ["src/clj" "src/cljs] there is a " missing after src/cljs#2021-06-1713:50v3ga@thheller oops, it's actually correct there in my actual file. I made a edit to keep up with a change I made after the initial gist.#2021-06-1713:52v3gaironically i'm setting up shadow-cljs now and started making some adjustments to get things working... I'm sure this is just a cluster of newbie mistakes.#2021-06-1713:56practicalli-john@decim All your dependencies are within aliases. There is no top level :deps {,,,,}. So unless you include an alias when running a Clojure command then there wont be any dependencies on the class path.#2021-06-1713:58practicalli-johnFor example, to run the backend in development environment, I assume you would do something like clojure -M:common:backend:dev#2021-06-1714:00practicalli-johnPersonally I would move all the dependencies to a :deps {,,,} section before the aliases and only use aliases for optional things or to run supporting development tools like a test runner.#2021-06-1714:01practicalli-johnThis page and video might help we it comes to Clojure CLI projects https://practical.li/clojure/clojure-tools/projects/create.html#2021-06-1714:02v3ga@jr0cket OH!! That's how they work...#2021-06-1714:03practicalli-john🙂 There is also a reference page on defining aliases that might be of interest https://practical.li/clojure/reference/clojure-cli-tools/defining-aliases.html#2021-06-1714:04borkdude@jr0cket I think @thheller advices to keep backend and frontend deps separate. I was already doing this before I heard him say that#2021-06-1714:04borkdudeBut I guess one can also do that when there is a conflict#2021-06-1714:05practicalli-johnWhich is cool, so long as you include the alias names on the command line 🙂#2021-06-1714:05borkdudethe :common alias is probably not needed, you can just move that to the top level deps#2021-06-1714:05borkdudeas those are the common deps :)#2021-06-1714:06v3gaembarrassing. =P Thank you... yeah, I see whats going on now. I got a little organization happy without knowing what I was doing.#2021-06-1714:07practicalli-johnThats the best way to learn - as well ask asking for help here :)#2021-06-1714:15v3gaOk, appreciate it. It looks like i'm on the right path now.#2021-06-1821:13Cam SaulIf I'm using a :local/root dependency, is there any way to tell it to use a specific ~profile~ alias from its deps.edn?#2021-06-1821:17Alex Miller (Clojure team)not currently#2021-06-1821:18Alex Miller (Clojure team)if you want to vote, there is a request for that here https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects#2021-06-1821:19Alex Miller (Clojure team)but even better would be to describe your use case in a comment htere#2021-06-1821:24Cam SaulSure. Here's my specific use-case:
At Metabase we ship both an FOSS edition (under the AGPL) and an Enterprise Edition under a commercial license. We can't ship the Oracle JDBC driver in the FOSS edition because the license terms of the JDBC driver make it AGPL-incompatible. However, we can ship it with the EE version.
The Oracle driver is itself a subproject in the larger Metabase-core repo.
So I want to be able to have a top-level :ee aliases that merges in the :ee aliases from the :local/root stuff it pulls in. Example:
;; ./deps.edn
{:deps
{:local/root "oracle-driver"}
:aliases
{:ee
{:extra-deps {:local/root "oracle-driver", :alias :ee}}}}
;; oracle-driver/deps.edn
{:aliases
{:ee
{:extra-deps
{com.oracle.ojdbc/ojdbc8 {:mvn/version "19.3.0.0"}}}}}#2021-06-1821:24Alex Miller (Clojure team)please put it on ask, I'll lose this here#2021-06-1821:25Cam SaulNp#2021-06-1821:28seancorfield@camsaul You’ve introduced a new top-level key into your deps.edn file? I think that’s a bit of a “no-no” — you could put it under an alias though (aliases can be “just data”).#2021-06-1821:28Cam SaulSorry. Not :profiles. :aliases. I'll fix the example#2021-06-1821:29Cam SaulI'm translated it from a project.clj file for the example and got my wires crossed a bit#2021-06-1821:29seancorfieldAh, gotcha.#2021-06-2018:25thomIs there an equivalent of Leiningen's :global-vars that can be used in a deps.edn file? Is there some more tasteful way to ensure things like *warn-on-reflection* or *unchecked-math* are set everywhere?#2021-06-2111:43practicalli-johnIf these settings are for development only, then I would add this to a user namespace and include the file that contains that namespace on the classpath via an alias. This approach works for either Clojure CLI tools or Leiningen
https://practical.li/clojure/clojure-tools/projects/configure-repl-startup.html#2021-06-2018:27borkdude@thom704 there isn't but recently Alex said that "there should" in the #graalvm channel. Currently you can do this like:
clojure -M -e '(set! *warn-on-reflection* true)' -e "(require 'foo.main)" -e "(foo.main/-main)" or so
#2021-06-2107:07dominicmUnclear from https://github.com/clojure/tools.deps.alpha/wiki/Tools is there a tool for tracking the licenses of dependencies in a deps.edn?#2021-06-2119:55Cam SaulSorry if this is a another dumb question. How can I programmatically get all the paths to a transient dependency with tools.deps?
Example: org.apache.httpcomponents/httpclient is a transient dependency of both clj-http/clj-http and org.clojure/tools.deps.alpha. I tried resolve-deps , thinking maybe it would return both of those paths in :parents, but it doesn't:
(defn http-client-parents [deps]
(-> (t/resolve-deps
{:deps deps
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}}
nil)
(get-in ['org.apache.httpcomponents/httpclient :parents])))
(http-client-parents
'{clj-http/clj-http {:mvn/version "3.10.3"}})
;; ->
#{[clj-http/clj-http]
[clj-http/clj-http org.apache.httpcomponents/httpmime]
[clj-http/clj-http org.apache.httpcomponents/httpclient-cache]}
(http-client-parents
'{org.clojure/tools.deps.alpha {:mvn/version "0.11.931"}
clj-http/clj-http {:mvn/version "3.10.3"}})
;; ->
#{[org.clojure/tools.deps.alpha org.apache.maven.resolver/maven-resolver-transport-http]}
What's the right way to do this?#2021-06-2120:40Alex Miller (Clojure team)@camsaul I don't think the tools.deps api has a direct way to answer this question if I understand your question properly#2021-06-2120:41Alex Miller (Clojure team)I think you're asking, given a set of root deps, why is a particular lib included in the expanded dep set?#2021-06-2120:43Cam SaulSort of. If a transient dep is include in the expanded dep set I want to know all of the root deps that would have caused that dep to be included#2021-06-2120:43Alex Miller (Clojure team)the :parents keys are not part of the normative return of resolve-deps (that's an artifact of the expansion process)#2021-06-2120:44Alex Miller (Clojure team)but there are shortcuts in the expansion process that can cause the info in the return here to be incomplete#2021-06-2120:44Alex Miller (Clojure team)it is probably better to derive this information from the tree expansion (the code that backs -Stree)#2021-06-2120:46Alex Miller (Clojure team)if you call make-tree with the lib-map that will be an actual tree structure#2021-06-2120:47Alex Miller (Clojure team)you should be able to find the leaves you want and trace their path back up#2021-06-2120:47Cam SaulI actually did look at make-tree and -Stree but it does the same thing. The don't include httpclient as a dependency of clj-http/clj-http#2021-06-2120:47borkdude-Stree can also print EDN right? you can also parse that on the command line I guess#2021-06-2120:49Cam SaulHere's an example (gist because it's a lot of output): https://gist.github.com/camsaul/4e8c3977ce3f423e7bfd090d5c816da3
org.apache.httpcomponents/httpclient only appears once#2021-06-2120:49borkdudeclojure -X:deps tree :format :edn
#2021-06-2120:49Alex Miller (Clojure team)you can do that, he'd ask for programmatic#2021-06-2120:49Alex Miller (Clojure team)sorry, I said make-tree above but that's the old tree printer code which is kind of vestigial now#2021-06-2120:50Alex Miller (Clojure team)the newer stuff is under clojure.tools.deps.alpha.tree/trace->tree which importantly is fed by a trace log which comes from running resolve-deps with the trace flag set to true#2021-06-2120:56Alex Miller (Clojure team)(require '[clojure.tools.deps.alpha :as deps] '[clojure.tools.deps.alpha.tree :as tree])
(let [deps '{:deps {org.clojure/tools.deps.alpha {:mvn/version "0.11.931"}
clj-http/clj-http {:mvn/version "3.10.3"}}
:mvn/repos {"central" {:url ""}
"clojars" {:url ""}}}
libs (deps/resolve-deps deps {:trace true})
tree (trace->tree (-> libs meta :trace))]
(clojure.pprint/pprint tree))#2021-06-2120:56Alex Miller (Clojure team)some sketch code to get the tree (this will give you a ton of output)#2021-06-2120:58Alex Miller (Clojure team)the docstring for trace->tree gives you more info on the structure of the tree - basically it's a tree of map nodes keyed by lib, value is a map with a :children vector (which are more nodes)#2021-06-2120:59Cam Saulawesome, thanks! Looks like it shows up under both top-level libs there. I can use that to build what I need#2021-06-2121:00Alex Miller (Clojure team)importantly, also check the :include flag! this tree includes nodes that are NOT included in the final deps (:reason tells you why)#2021-06-2122:20Cam SaulI have one more question that I couldn't seem to find an answer for on http://ask.clojure.org. What do people with Java source files do to approximate Leiningen's :java-source-paths functionality? Of course, it's easy enough to do find java -name '*.java' | xargs javac -cp $(clojure -Spath) or whatever before running stuff, or even put that logic in an alias (e.g. clojure -X:compile-java ), but it's a nicer dev experience if I don't have to run two commands instead of one for everything I want to do.
I'm less worried about that tho, and more worried about the experience for contributors and other people using our project. I don't want to have to change our local dev server instructions from something relatively simple like lein run to something hairy like clojure -X:compile-java && clojure -X:dev-server .
If there were a way to add custom tools.deps "extensions"https://github.com/clojure/tools.deps.alpha/tree/master/src/main/clojure/clojure/tools/deps/alpha/extensions I think I could get this working pretty easily with something like
{:deps {my-project/java-source {:com.my-company.java-source/local "/path/to/java/src"}}
But I'm guessing that sort of extension support isn't something that's likely to make it into the CLI any time soon. Other than doing something like setting up CI to compile the Java sources and publish them somewhere (e.g. by uploading it to an S3 bucket structured as a Maven repo), is there a way I can avoid requiring people to run a separate compile command before they are able to do anything else with the project?#2021-06-2122:26Alex Miller (Clojure team)Coming very soon#2021-06-2122:27Cam SaulAwesome, that's great to hear!#2021-06-2512:45Jakub Holý (HolyJak)Hi! Does deps check for and download new snapshots automatically or do I / can I tell it to do so? Searching for "snapshot" at https://clojure.org/reference/deps_and_cli#_resolve_deps does not find much. 🙏#2021-06-2512:47Alex Miller (Clojure team)You can use -Sforce to force the cp to be recomputed #2021-06-2512:48Alex Miller (Clojure team)It will use the default snapshot update policy which is daily#2021-06-2512:49Alex Miller (Clojure team)If you want more new than that, you’ll need to remove portions of your local maven cache#2021-06-2512:50Alex Miller (Clojure team)I believe we have an existing ticket to allow setting the snapshot policy or more control over this#2021-06-2512:55delaguardois it possible to configure update policy using global maven settings.xml?#2021-06-2513:34Alex Miller (Clojure team)well tools.deps doesn't use that for this so won't matter#2021-06-2513:47dharriganDoes deps have support for s3p:// repos, aka private s3 repositories?#2021-06-2513:47dharriganA lein plugin, which our company uses is this project.clj to deps.clj#2021-06-2513:49rickmoynihanjust drop the p from the end of the protocol and it should work out of the box… iirc s3p is an old deprecated protocol#2021-06-2514:07Alex Miller (Clojure team)it's not deprecated, it's just specific to that leiningen plugin. that's not a "real" thing#2021-06-2513:51dharriganI will retry that, for it failed when I did it#2021-06-2513:51dharriganhmmm, strange#2021-06-2513:52dharriganI have this in my deps.edn...#2021-06-2513:52dharrigan:mvn/repos {"foo-releases" {:url ""}
"foo-snapshots" {:url ""}}}#2021-06-2513:52dharriganbut the resolution does this:#2021-06-2513:53dharriganDownloading: foo/components/0.2.0/components-0.2.0.pom from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.pom from foo-snapshots
Downloading: org/clojure/clojure/maven-metadata.xml from foo-snapshots
Downloading: org/clojure/clojure/maven-metadata.xml from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.jar from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.jar from foo-snapshots
Error building classpath. Could not find artifact foo:components:jar:0.2.0 in central ()
#2021-06-2513:54dharriganNot sure what is going on at all, why is it calling out to maven central (no artifact has been downloaded)#2021-06-2514:01rickmoynihanI’m pretty sure that’s just because it looks for the deps on all repos, which is why you’re seeing that#2021-06-2514:04dharriganOkay, a fallback. Fairy enough. However, it appears not to be able to download the artifact on s3.#2021-06-2514:08Alex Miller (Clojure team)correct on the error message - it cycles through all repos and will report only the last error#2021-06-2514:08dharriganAh, I think I've also discovered why it didn't download#2021-06-2514:08dharrigandouble checking#2021-06-2514:08Alex Miller (Clojure team)is that a private s3 repo or public?#2021-06-2514:08dharriganprivate#2021-06-2514:09Alex Miller (Clojure team)and you provided auth ?#2021-06-2514:09dharriganthat's what I'm double checking atm 🙂#2021-06-2514:09dharriganverifying my setup#2021-06-2514:09Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_maven_s3_repos#2021-06-2514:10Alex Miller (Clojure team)also important is that the AWS creds you use have a policy that allows s3 ops#2021-06-2514:10dharriganyup, all that works with lein, I'm doing a conversion to deps#2021-06-2514:11Alex Miller (Clojure team)but your log snippet above shows it successfully downloading stuff from the repo#2021-06-2514:11dharriganNo, it didn't#2021-06-2514:11dharriganit created the directories, but didn't download anything.#2021-06-2514:11dharriganso that message is misleading#2021-06-2514:12dharrigan(I wouldn't report it not downloading anything, without first checking if the directories actually contained stuff)#2021-06-2514:13Alex Miller (Clojure team)do you have metadata files in there or are they literally empty?#2021-06-2514:13dharriganliterally empty#2021-06-2514:13dharriganmetadata would have indicated that it yes, downloaded something (then my question would have been different, i.e., why metadata but no jar...)#2021-06-2514:14Alex Miller (Clojure team)can you try again with -Sforce ?#2021-06-2514:14dharriganThat's what I have been using#2021-06-2514:14Alex Miller (Clojure team)ok#2021-06-2514:14dharriganSo, I got it to download now, like this:#2021-06-2514:14Alex Miller (Clojure team)where are you putting AWS creds?#2021-06-2514:15dharriganin my .m2/settings.xml I had the the username/password set to my access key and secret access key (when I first attempted to verify that the conversion from project.clj to deps.clj was okay)#2021-06-2514:15dharrigan(following the instructions on the aws-vault exec foo -- clj -Sforce#2021-06-2514:16Alex Miller (Clojure team)what's aws-vault?#2021-06-2514:16dharriganthen that worked#2021-06-2514:16dharriganaws-vault list#2021-06-2514:18dharriganaws-vault list
Profile Credentials Sessions
======= =========== ========
default - -
dharrigan - -
foo foo sts.GetSessionToken:5h54m59s
foo:staging - -
#2021-06-2514:19Alex Miller (Clojure team)may be that whatever your ambient aws creds were were getting used over the user/pw creds#2021-06-2514:20dharriganI think it's the session credentials that should replace what was in the .m2/settings.xml, (I've removed the access key/secret key combo from the .m2/settings.xml)#2021-06-2514:22dharriganIf I do this:#2021-06-2514:22dharrigan❯ aws-vault exec foo -- env | rg AWS
_=/usr/bin/aws-vault
AWS_VAULT=foo
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ACCESS_KEY_ID=wibble
AWS_SECRET_ACCESS_KEY=wobble
AWS_SESSION_TOKEN=wibble-session
AWS_SECURITY_TOKEN=wobble-sesion
AWS_SESSION_EXPIRATION=2021-06-25T20:08:23Z
#2021-06-2514:22dharriganI can see that a lot of env variable are set, which I suppose the S3Transporter uses to successfully download the jar from.#2021-06-2514:25dharriganShould this be documented for those who come after me? A company that has a security policy like this, that requires that session tokens are gained and used, will affect anyone trying to download from s3 without this knowledge.#2021-06-2514:35Alex Miller (Clojure team)should what be documented?#2021-06-2514:36Alex Miller (Clojure team)s3 ambient creds are discussed in the docs#2021-06-2514:37dharriganThe instructions on the For authenticated repos, AWS credentials can be set in the ~/.m2/settings.xml on a per-server basis or will be loaded ambiently from the AWS credential chain (env vars, etc). is quite sufficient detail#2021-06-2514:39dharriganthey certainly hint at it, but don't expound upon it#2021-06-2514:56Alex Miller (Clojure team)I'm not sure what's special about the temporariness of the credentials#2021-06-2514:58Alex Miller (Clojure team)this is the relevant part:
AWS S3 credentials can be set in the environment using one of these mechanisms:
Set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Create a default profile in the AWS credentials file ~/.aws/credentials (older ~/.aws/config also supported).
Create a named profile in the AWS credentials file and set the environment variable AWS_PROFILE with its name.
Amazon ECS container and instance profile credentials should also work, but have not been tested.
For more information, most of the advice in this AWS document describes how credentials are located. Note however that the Java system properties options will NOT work with the command line tools (but would work if using the tools.deps.alpha library directly).#2021-06-2514:59Alex Miller (Clojure team)so, I think it does expound on it?#2021-06-2515:00dharriganIt's missing information on the session tokens, which are used in tandem with the access key and secret access key - but only under specific circumstances, i.e., when there is an AssumeRole going on.#2021-06-2515:01dharrigan(otherwise,the access key and secret access key, set in the .m2/settings.xml under username/password would have worked out-of-the-box)#2021-06-2515:02dharriganI'm highlighting a gap in the explanation that I came across. With the thought of being helpful to others in the future, who may encounter this too, I thought it would be beneficial to document it somewhere.#2021-06-2515:10Alex Miller (Clojure team)so you're talking about user/pw being insufficient when using session tokens#2021-06-2717:01dharriganHi, Sorry for the delay. Family and trying to reproduce the situation cleanly. On a clean OS, the only way I could get the artifacts to download from the private S3 repository was to ensure that there was no username/password set in the .m2/settings.xml and use aws-vault foo exec -- clj -Sforce to download the items. If I had the username/password set, it would not download (it would say it was downloading, create empty directories, but not actually download the items)#2021-06-2717:03dharriganSo, yes, a full set of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN etc., are required.#2021-06-2717:04dharriganI think this comes into play when a role doesn't have the ability to download, and has to temporarily assume the role (by using aws-vault for example, to ensure the correct ENV variables are set)#2021-06-2717:04dharrigan.#2021-06-2717:04dharriganI hope that helps.#2021-06-2515:32dharriganI think so yes, it wasn't sufficent for me when I had them in the .m2/settings.xml. However, let me validate that again by doing another test. I'll report back 🙂#2021-06-2606:23seancorfieldHow stable is this part of the tools.deps API:
user=> (require '[clojure.tools.deps.alpha.util.dir :refer [with-dir]] '[hf.depstar :refer [uberjar]] '[ :as io])
nil
user=> (with-dir (io/file (str (System/getProperty "user.dir") "/projects/api"))
(uberjar {:jar "../../../build/uberjars/api-2.0.0.jar" :main-class 'api.main :aot true}))
[main] INFO hf.depstar.uberjar - Compiling api.main ...
[main] INFO hf.depstar.uberjar - Building uber jar: ../../../build/uberjars/api-2.0.0.jar
nil
user=>
We're going to start replacing our build shell script stuff with a Clojure build.clj script or something similar and the shell script cd's into subproject folders to run commands and I figured out the above is equivalent... but it feels a bit like I'm relying on some internals there? I'm hoping with tools.build that there's some standard way to specify the "user directory" for running different tasks so a build script can navigate around a repository and run tasks in different places?#2021-06-2608:02dominicmI assume the aim is to avoid multiple jvms, so a faster build overall?#2021-06-2608:03seancorfieldYes, partly. And also just to get rid of bash scripts where possible 🙂#2021-06-2608:03seancorfieldGiven that tools.build is going to be taking us in this sort of direction, it seems reasonable to start planning.#2021-06-2608:04seancorfieldI just posted in #polylith a function I created that builds an uberjar for every projects entry in a Polylith workspace, which is where we're headed with this.#2021-06-2608:05seancorfield(but it's applicable anywhere that you have multiple subprojects that need a task applying to them in sequence)#2021-06-2608:07seancorfieldMy next task is to figure out whether classpath-isolated subproject test runners are achievable 🙂#2021-06-2608:08seancorfield(given that Polylith already has a version of this in its command-line tool, the answer is obviously "yes" but I just need to figure out how much work that is...)#2021-06-2608:29dominicmOnly polylith projects have that issue of needing classpath isolation because of how it swaps out namespaces right? Otherwise you could just load them all into a single JVM?#2021-06-2608:43dominicm(If clojure.test could be parameterized, you wouldn't even need to do that, you could just use a protocol!)#2021-06-2608:48dominicm@U04V70XH6 for cp isolation, you might be interested in https://github.com/SevereOverfl0w/clj-embed which was forked to fix a few bugs I found in upstream. It's behind the API times, but it's probably faster than shelling out.#2021-06-2616:54seancorfieldThanks @U09LZR36F and, no, re: Polylith swapping namespaces: this is more about wanting potential classpath isolation between subproject test runs. We have one particular subproject that requires a specific, old version of Jackson (before they changed the null merging behavior in 2.9.0) but the other subprojects can run with "any" version of Jackson. I haven't decided yet how I want to approach this part. Currently, we run a separate clojure process for each subproject when running tests but that's (obviously) slower than just loading everything and running all the tests in a single JVM...#2021-06-2616:56seancorfieldLooks like clj-embed does "more" in terms of isolation than I'd really need but I suspect it would get me closer. Thanks.#2021-06-2618:13athomasoriginalHi all. I’m just getting around to tackling the ~{:tag :a, :attrs {:href "https://clojure.org/releases/tools#v1.10.2.790"}, :content ("https://clojure.org/releases/tools#v1.10.2.790")}~ https://clojure.org/releases/tools#v1.10.2.790 warning now for my monorepo. I have read through Sean’s posts and conversations in this channel about monorepo setup with this new constraint.
I have outlined my https://gist.github.com/athomasoriginal/47913c6b946fc7416c3d839be101e600 and wanted to see if this aligns to the “recommended approach”?
> I use “recommend” lightly. My goal is to be sure the approach is not going against the grain too much or setting myself up for future pain.
Thanks!!#2021-06-2618:15seancorfieldSounds about right but I'll elaborate a bit on a couple of our monodeps setup...#2021-06-2618:17Alex Miller (Clojure team)I assume you mean :extra-paths in there, not :external-paths#2021-06-2618:17seancorfieldOur "workspace-level" deps.edn has an :everything alias which has :extra-deps listing each subproject as a :local/root dep and :extra-paths listing each subproject's test folder -- we use this alias for starting a global REPL to work with the entire codebase. I think that's an important affordance.#2021-06-2618:18seancorfieldThat top-level deps.edn file is where we declare any/all tooling -- but we don't have cljs subprojects so I don't know how that would change the equation (if at all).#2021-06-2618:22Alex Miller (Clojure team)why are app and app-server separate? app is what you build and app-server is what you deploy?#2021-06-2618:25Alex Miller (Clojure team)one thing coming is the ability for projects to publish that they must be "prepared" before use and for consumers of that project to ask to do that preparation#2021-06-2618:26Alex Miller (Clojure team)I'm trying to understand your bigger use case here and whether that could be used for advantage (having app-server consume app as a local dep, which has a prep step)#2021-06-2618:31athomasoriginal> I assume you mean :extra-paths in there, not :external-paths
Yes. My bad 😞#2021-06-2618:38athomasoriginal> why are app and app-server separate?
• app is the ClojureScript application (client)
• app-server is the web-server serving static assets and a REST API
My preference is separation because i’ve found it makes it easier to understand the backend/front end divide and helps avoid coupling tooling.
Before I go further though, does that clarify? I’m not sure I understand your comment: “app is what you build and app-server is what you deploy” :thinking_face:#2021-06-2618:40seancorfieldI suspect app-client and app-server might have been less confusing names 🙂#2021-06-2618:40athomasoriginalI think you are correct 😉#2021-06-2618:41Alex Miller (Clojure team)ah ok, that makes sense#2021-06-2618:42athomasoriginalI think I understand your build v. deploy question now though 🙂#2021-06-2618:44athomasoriginal> I’m trying to understand your bigger use case here and whether that could be used for advantage
Does the use case make more sense now? It might be outside the scope of the feature you’re considering though?#2021-06-2618:51Alex Miller (Clojure team)not sure, might still be a good match?#2021-06-2618:53Alex Miller (Clojure team)app would have some build process and publish in its deps.edn how to run it. use :path including the output of that process.
app-server would depend on app#2021-06-2618:53athomasoriginalYeah, that’s it.#2021-06-2618:54Alex Miller (Clojure team)there may be some tooling that will need to catch up to all that though#2021-06-2619:11athomasoriginal> That top-level `deps.edn` file is where we declare any/all tooling
@U04V70XH6 I am re-reading your post again, but in my scenario my driver for relying on paths external to the app-server project was only so I could have access to app-client s resources and target dirs because I feel it’s awkward to have those specified in the app-server project while your developing the app’s front end.
So I believe a difference is that I don’t actually need to work with clj code, I just did it to serve static assets in development.#2021-06-2709:32Shantanu KumarHi. I'm looking for a sample project/repo that maintains both project.clj and deps.edn - some way to specify the dependencies in just one place. Would appreciate any pointers.#2021-06-2709:43Shantanu KumarI've found https://github.com/RickMoynihan/lein-tools-deps#2021-06-2710:00borkdudeIt's fairly easy to keep a list of deps in an edn file and then generate the project.clj and deps.edn from a template, but this will limit you to non-git-deps#2021-06-2710:01borkdudeor just keep the deps.edn as the source of truth (since it's already just edn) and then generate the project.clj from that (using a template)#2021-06-2710:14mpenetI used to do that, it's brittle. Dependency resolution is different between the two, so you might end up in weird situations if you use stg like pedantic.#2021-06-2710:24borkdudePerhaps lein could one day use tools deps alpha as the resolution mechanism?#2021-06-2710:24borkdudeat least optionally#2021-06-2717:40seancorfield@U066J7E2U It's definitely an approach that has problems so I would ask why you want to do this? Rick's repo README does a great job of talking about the potential problems (due to the inherent mismatch between how lein and t.d.a handle dependency resolution) and the downsides of the trade off he makes. I created a similar tool for boot (prior to Rick's lein plugin, I believe) because we were using Boot but already had our dependencies external (in deps.edn -- albeit a different format at first). Here's mine https://github.com/seancorfield/boot-tools-deps -- again, the README talks about limitations and differences and I decided to archive the repo and discourage its use, because it was just too hard to make it work properly beyond the simplest cases! At work, we decided it was simpler to switch completely to the CLI and deps.edn and abandon trying to integrate the two approaches (we'd already switched from lein to boot years earlier for the flexibility Boot provided over Leiningen).#2021-06-2717:41seancorfieldMy advice would be to just switch from lein to t.d.a/CLI deps.edn and avoid trying to integrate them. Or stay with lein and ignore deps.edn for now.#2021-06-2718:51Shantanu KumarThanks, @U04V70XH6 I was trying to keep the benefits of both and the freedom (for developers) to choose either, but as you and others have mentioned, it's nondeterministic and error prone.#2021-06-2718:52borkdudeI do have a deps.edn and project.clj for a lot of open source projects, exactly for that reason and I never had any conflicts there, probably because libraries tend to have fewer deps#2021-06-2718:57seancorfield@U04V15CAJ I'm curious about what benefits you perceive for a library project to have both? (assuming the deps.edn/CLI side can do "everything" that the Leiningen side can)#2021-06-2718:59seancorfield(there are definite benefits -- for CLI users -- to have even a minimal deps.edn added to a Leiningen project, so that it can participate as a :git/url dep, but I don't see much benefit in having project.clj added to a deps.edn project if your dev/test/build setup is based on the CLI)#2021-06-2718:59borkdudeAs I just explained to @UKFSJSM38 :#2021-06-2718:59borkdudealso I want to be flexible for other maintainers#2021-06-2719:00seancorfield@U066J7E2U I think there's value in ~forcing~ "encouraging" Leiningen users to install the CLI and start learning deps.edn since it's the "official" tooling -- and we have tools.build on the horizon as well 🙂#2021-06-2719:06seancorfield@U04V15CAJ Fair enough. I'm looking forward to clojure -X:build test jar deploy as a CI process. We're not quite there yet, but I think that will beat what Leiningen offers. At work we just replaced our long build shell script with build.clj as a stepping stone toward tools.build, and the next version of depstar will offer the option of running pom, aot, and jar/`uberjar` as separate tasks, along with offering a :target-dir option.#2021-06-2719:07borkdudewhat's the syntax again for the -> semantics vs. providing key/vals?#2021-06-2719:13seancorfieldclojure -X:aliases func1 func2 func3 :some '"arg"' :and '[:more :stuff]' = (-> {:some "arg", :and [:more :stuff]} (func1) (func2) (func3)) -- you mean that?#2021-06-2719:14seancorfield(plus any :exec-args from the :aliases obvs)#2021-06-2719:14borkdudewhat if you want to pass func1 func2 as a key value pair?#2021-06-2719:14borkdudethe keys have to start with a colon?#2021-06-2719:15seancorfieldYou can always pass a hash map at the end: '{func1 func2}' -- just bear in mind those are then passed as symbols.#2021-06-2719:16seancorfieldclojure -X:aliases func1 '{func2 func3, :some "arg", :and [:more :stuff]}' -- like so (assuming you meant passing func2 func3 not func1 func2?)#2021-06-2719:19seancorfieldor clojure -X:aliases func1 :some '"arg"' :and '[:more :stuff]' '{func2 func3}' if you wanted to combine named args and a hash map. Just like Clojure 1.11 calls 🙂#2021-06-2719:21borkdudeah right, trailing maps#2021-06-2823:07metasoarousDoes anyone know if it's possible for deps.edn aliases to refer to or reuse each other? This would be pretty handy in certain scenarios, but doesn't seem to be possible. https://stackoverflow.com/questions/68170875/how-do-you-refer-to-or-reuse-one-alias-from-another-using-deps-edn#2021-06-2906:32practicalli-johnIt is possible to use many aliases with each other as has been stated in the stack exchange post, by chaining them together
I have found that explicitly including the aliases used to be much clearer as to the intent of the command and offers far greater flexibility.
Defining relationships between aliases seems like it could lead to greater complexity in the design and levels of abstraction behind what is otherwise can be a very clear chain of aliases.
Once you can have an alias depend on another, what is the precedence of any conflicting configs between aliases? Would we also need to have an excludes mechanism, as with dependencies?#2021-06-2919:41metasoarous> Once you can have an alias depend on another, what is the precedence of any conflicting configs between aliases? Would we also need to have an excludes mechanism, as with dependencies?
This is a great question/point! This is one clear argument for using a vector of aliases to be referred to instead of a set. Then "last wins", just as when you explicitly list several aliases in a cli invocation.#2021-06-2919:43metasoarousHowever, I don't know if much more is needed really. As long as there's no cycles (which should be an error), seems like you can simply resolve each alias by itself, and chain together the data from each such resolution at the next level up the tree using order for precedence, etc.#2021-06-2919:45metasoarousI think in many cases you are correct that explicitly listing out aliases may be advantageous. But I'd argue there are many others where it's pretty harmless, and saves folks a docs headache realizing that "oh, the alias I was trying to use doesn't work because I needed to include this other alias..."#2021-06-2921:06practicalli-johnI have not experienced any docs headache when creating over 30 aliases for https://github.com/practicalli/clojure-deps-edn
So I am afraid I am not able to relate to this point#2021-06-2823:18seancorfield@metasoarous It is not possible. But it is something people have requested quite often. You can vote for it here https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases#2021-06-2823:33metasoarousThanks for confirming, and pointing to the ticket!#2021-06-2823:26seancorfieldWe currently run our monorepo build script with clojure -M:build:everything:dev:test:runner build.clj ... and it sure would be nice to have :build be able to automatically "include" :everything:dev:test:runner!#2021-06-2900:51seancorfield@alexmiller Is it just an oversight that https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases has no associated Jira ticket for it?#2021-06-2900:53Alex Miller (Clojure team)probably#2021-06-2913:20il-tmfvHi!
I’m using 1.10.3.855 version of CLI and I’m getting strange (to me at least) results:
$ clojure -P -A:test:lint
Downloading: net/gcardone/junidecode/junidecode/0.4.1/junidecode-0.4.1.pom from central
Downloading: org/clojure/clojure/1.10.2/clojure-1.10.2.pom from central
Downloading: clj-kondo/clj-kondo/2021.03.22/clj-kondo-2021.03.22.pom from clojars
...
$ clojure -P -A:test
Downloading: com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.4.4/jackson-dataformat-cbor-2.4.4.pom from central
Downloading: com/fasterxml/jackson/core/jackson-core/2.4.4/jackson-core-2.4.4.pom from central
Downloading: com/fasterxml/jackson/core/jackson-core/2.10.0/jackson-core-2.10.0.pom from central
...
I’m trying to install all the deps with one command, but for some reason there are still “missing” deps when I run -P with just test alias (and extra download is happening on CI when test alias is run, clojure -P -A:test here is just to illustrate the problem).
Both test & lint aliases contain just :extra-deps key with one dep in each.
Do I miss something?#2021-06-2913:21borkdude@il.tmfv it could be that :test:lint results in a different resolution for jackson than only :test#2021-06-2913:21il-tmfvhow to check it?#2021-06-2913:21borkdudeyou could try -Stree for either#2021-06-2913:32il-tmfvYep, looks like it
I see that for test com.fasterxml.jackson.core/jackson-core is in different “place”
How to handle such situations? Run -P twice for each alias to have CI cache for all possible deps?#2021-06-2913:38Alex Miller (Clojure team)yes, you should run -P for whatever set of aliases you plan to run - the version choices are affected by the total set of libs you've selected#2021-06-2913:40il-tmfvgood, thank you guys 🤝#2021-06-2914:11dominicmAn aside - I really love that Clojure has a man page on my computer. Thank you for maintaining that 🙂#2021-06-2914:15Alex Miller (Clojure team);)#2021-06-2914:16dominicm(I hadn't seen the -P before, so I looked there)#2021-06-2914:16Alex Miller (Clojure team)whatever format that's called, it is the worst markup format#2021-06-2914:16Alex Miller (Clojure team)troff probably#2021-06-2914:17dominicmYeah, that's the one.#2021-06-2914:17dominicmhttps://sr.ht/~sircmpwn/scdoc/ You might be interested in trying this instead.#2021-06-2914:17dpsuttonof course the man page is by elana hashman. She is awesome#2021-06-2914:17dpsutton(just reading it now, didn't know it was there before)#2021-06-2914:18Alex Miller (Clojure team)she contributed it a few years ago and I've been keeping it up to date in the package#2021-06-2914:19dpsuttonits a nice touch. thanks!#2021-06-2914:31lreadYou probably are aware but https://docs.asciidoctor.org/asciidoctor/latest/manpage-backend/#2021-06-2914:32dominicmOh yeah it can. I forgot that clojure uses adoc, use the adoc man format thing if you can 😄 adoc is the best.#2021-06-3015:30gregHow to pass vector as one of args for -X ?
Example:
(ns xml)
(defn merge-files
"Merges given XML files. Options:
:files Vector of filenames, required
:output Filename for merge output, optional"
[{:keys [files output]}]
...)
Assuming there is no aliases specified in deps.edn, how to call merge-files? The attempt below seems to be invalid:
clj -X xml.merge-files :files '["file1.xml" "file2.xml"]' :output "file3.xml"
#2021-06-3015:37Alex Miller (Clojure team)why do you say it is invalid?#2021-06-3015:37Alex Miller (Clojure team)you also likely need to single quote the last arg#2021-06-3015:37Alex Miller (Clojure team)clj -X xml.merge-files :files '["file1.xml" "file2.xml"]' :output '"file3.xml"'
#2021-06-3015:38Alex Miller (Clojure team)alternately, you can now pass a trailing map in the latest version
clj -X xml.merge-files '{:files ["file1.xml" "file2.xml"] :output "file3.xml"}'#2021-06-3015:42gregthanks, works now, kind of
Actually previously I tried to pass the vector '[file1.xml file2.xml]'
I'm having now another issue: `Unqualified function can't be resolved: xml.merge-files'
The full call is like that
clojure -Sdeps '{:paths ["scripts"]}' -X xml.merge-files :files '["~/.m2/settings.xml" "/tmp/settings.xml"]' :output '"te.xml"'#2021-06-3015:47gregscripts is just a directory holding xml.clj file
xml.clj file defined internally (ns xml) and (defn merge-files)
Is there anything else I need to do?#2021-06-3015:47gregAlright, I should call clj -X xml/merge-files instead of clj -X xml.merge-files#2021-06-3015:56Alex Miller (Clojure team)yes, sorry!#2021-06-3015:58gregno no, my fault, thanks for help 🙂#2021-06-3022:08dpsuttoni've got a strange bug that i just cannot figure out. We build our release with a clojure cli based program. It has local-roots for a project that builds the actual uberjar, and it includes the paths :paths ["src" "resources"]. And for whatever reason, this resources path is not showing up on the classpath for my coworker when running this release script, but it is for everyone else#2021-06-3022:09dpsuttonI got him to run clojure -e "(run! println (clojure.string/split (System/getProperty \"java.class.path\") #\":\"))" and saved his output and my output and the only difference (after adjusting paths from his workspace to mine) is
diff ~/projects/work/diffs/my.classpath ~/projects/work/diffs/his.classpath
3d2
< /Users/coworker/workspace/metabase/bin/build-mb/resources
#2021-06-3022:10dpsuttonand we are both on Clojure CLI version 1.10.3.855#2021-06-3022:20Alex Miller (Clojure team)did you -Sforce on both?#2021-06-3022:22Alex Miller (Clojure team)does the directory exist on both?#2021-06-3022:22dpsuttonno. and i wish i would have told him to try that. are the cache's transitively used like that?#2021-06-3022:23dpsuttonyeah. dir exists, and the file it should find is there. it just seems to be a stale classpath. and i believe the resources addition was recent, so if there's a stale classpath around that would make sense. didn't think that it might use a cached one of a local root though.#2021-06-3022:24Alex Miller (Clojure team)changes in local dep deps.edn are not automatically picked up - you have to -Sforce to pick up those changes#2021-06-3022:24dpsuttonat both locations?#2021-06-3022:24dpsuttonor just the dep or just the top level?#2021-06-3022:24Alex Miller (Clojure team)top level is where it matters#2021-06-3022:24Alex Miller (Clojure team)since it's that classpath you care about here#2021-06-3022:25dpsuttonawesome. thanks. think he's in an interview but we'll try that. thank you very much as always#2021-06-3022:25Alex Miller (Clojure team)in the leaf local dep, any changes in deps.edn should make that project stale and recompute there in that project#2021-06-3022:26Alex Miller (Clojure team)but things that depend on it will not automatically detect that change#2021-06-3022:26dpsuttonoh i see. so the root's cache includes the old classpath and it doesn't know to invalidate#2021-06-3022:26Alex Miller (Clojure team)yep#2021-06-3022:27dpsuttongot a good model of it now. thanks#2021-06-3022:28Alex Miller (Clojure team)for the wrapper to know to check that file, it would have to understand the deps model, which would mean running Clojure code, which would negate the benefit of the cache :)#2021-06-3022:28dpsuttoni guess the cache doesn't include a hash of the deps.edn files. i've never looked inside of cpcache really.#2021-06-3022:29Alex Miller (Clojure team)I do think it could remember this info in the cache though and use that to determine the set of files to check for staleness#2021-06-3022:29Alex Miller (Clojure team)there is a ticket for this, just has never been something I've gotten around to#2021-06-3022:29dpsuttoni'll go look and vote for your next pass through. thanks#2021-06-3022:30dpsuttoni think this is it but it's pretty verbose: *loaded-libs* but the namespace didn't actually exist, when I removed it from *loaded-libs* it loaded fine. That to me sounds like a concurrent code loading issue, and some digging and monkey patching of require to dump stacktraces when called off the main thread and I have provisionally tracked it down to https://github.com/clojure/tools.deps.alpha/blob/master/src/main/java/clojure/tools/deps/alpha/util/S3TransporterFactory.java#L48 , I have seen the s3 transport do this in the middle of my test run: call back into clojure, which causes core.async and a ton of other deps to load, which races with code running the tests.#2021-06-3023:07hiredmanis there some way to call calc-basis synchronously?#2021-06-3023:11Alex Miller (Clojure team)you can pass -Sthreads 1#2021-06-3023:13Alex Miller (Clojure team)that's one source of multithreading in the download process, but doesn't actually get to that thing in S3TransporterFactory#2021-06-3023:14hiredmanI guess I could require clojure.tools.deps.alpha.util.s3-transporter in our build script, which would no-op the transport factory requiring it#2021-06-3023:14Alex Miller (Clojure team)I'm not sure exactly what your setup is but the S3 stuff is used to build a classpath via tdeps, so not sure why it's in process with your tests? (should be separate process)#2021-06-3023:15Alex Miller (Clojure team)unless you're using tdeps programmatically#2021-06-3023:15hiredmanour build script is#2021-06-3023:16Alex Miller (Clojure team)then yeah, I'd try to force the load earlier#2021-06-3023:17Alex Miller (Clojure team)I guess I could use the require lock in the background load#2021-06-3023:18hiredmandunno that would help, the test code being loaded is using require without the lock#2021-06-3023:19Alex Miller (Clojure team)true#2021-06-3023:20seancorfieldWe don't need the S3 stuff -- is there some env setting that prevents t.d.a. from even trying to load it? Or is the fact that it's part of that Java class's initializer making that impossible?#2021-06-3023:23Alex Miller (Clojure team)it's such a pain in the ass given how it's all initialized (see https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L166)#2021-06-3023:24Alex Miller (Clojure team)and there's a Named annotation on that class so ends in the META-INF in the jar etc#2021-06-3023:25Alex Miller (Clojure team)nothing like dependency injection frameworks to ruin everything good about dependency injection#2021-06-3023:25Alex Miller (Clojure team)it is intended to be tolerant of not finding the class (which is just a crutch for using it without compilation during dev)#2021-06-3023:25Alex Miller (Clojure team)so you could just delete the class out of the jar if you were feeling crazy :)#2021-06-3023:26Alex Miller (Clojure team)or use it as a git / local dep#2021-06-3023:27Alex Miller (Clojure team)but if you want something better, file a TDEPS ticket and I'll think about it better in a week or two#2021-06-3023:27hiredmanit defeated me trying to to even find how it was initialized because I was grepping for 's3' not 'S3' 🙂#2021-06-3023:28Alex Miller (Clojure team)if only I could just hand an instance to the thing that was using it rather than gen'ing a class and passing a class name#2021-06-3023:28seancorfieldYeah, when I use t.d.a from source -- via the add-libs branch -- I get the warning about S3 not being found but works 🙂 so maybe we'll just change out dependency from a maven version to a git sha and just live with the warning about S3 Transport not being loaded 🙂#2021-06-3023:30seancorfield@hiredman I sent you the :git/url coords, to save you looking it up.#2021-07-0104:40dpsuttonthat service locator's javadocs have the helpful
> Deprecated
> Use some out-of-the-box DI implementation instead.#2021-07-0104:43dpsuttoncan you set an instance with setServices(Class<T> type, T... services) with only a single instance in an array of Ts?#2021-07-0203:56onetomhttps://clojure.org/reference/deps_and_cli#_paths mentions that I can use aliases to assemble a value from aliases for the value for :paths.
is that possible to do something like this for :extra-deps and similar hash-maps, within aliases?#2021-07-0203:57onetommy motivation for doing this is to reduce some repetition in dependency definitions.#2021-07-0204:39seancorfieldNot at the moment. Only :paths can accept keywords and treat them as aliases I believe. Some tools support this, such as depstar and deps-deploy, for some of their exec args. I'd like to see this be more widespread.#2021-07-0210:45onetomthank you! i will have a look how you did it.#2021-07-0210:38dharriganIs it possible to mix, say a function execution that does something, i.e., -X:foo with a set of aliases -M:bar:baz, i.e., clj -X:foo -M:bar:baz#2021-07-0210:39dharriganso that foo would run first, then aliases next? Or perhaps I'm going about this the wrong way?#2021-07-0214:39Alex Miller (Clojure team)If I’m guessing what you want, no#2021-07-0214:39Alex Miller (Clojure team)Each clj call runs a single program#2021-07-0214:40Alex Miller (Clojure team)One new capability in latest release is that you can chain multiple functions together, but they run serially which I suspect is not what you want #2021-07-0214:41dharriganA bit of background, I'm porting a lein project.clj to deps. Part of our system uses a local docker stack, with random external ports. There's in-house lein plugin that introspects the docker stack running and spits out the ports for injection into the environment for development. I've ported that plugin to a little program that I can invoke using clj -X:foo. It's tedious to write out the ports each time (as the stack can bepulled up and down several times a day). So I wasn hoping to have something that ran a function (this one) then fired off the aliases that would then start up my repl etc...#2021-07-0214:41Alex Miller (Clojure team)You can use whatever you like to make two calls around clj, or you could write a program that did two things and call that#2021-07-0214:42Alex Miller (Clojure team)Chaining might work well for this#2021-07-0214:43dharriganserial execution is fine, but I wonder would it work with the repl programs, such as rebel-readline that uses a main method to start up#2021-07-0214:44dharriganso, it's like I want to run a function before then handing off to the -M aliases, perhaps that function, like in this case, is a pre-start bit of work#2021-07-0214:45borkdude@dharrigan Is there a need for the first script to run inside the Clojure JVM environment?#2021-07-0214:46dharriganno (in fact, all it does is shell out to docker to get a list of runnig services, then use a bit of clojure code to write them out neatly into edn format)#2021-07-0214:46Alex Miller (Clojure team)this won't work with clojure.main programs so you'd need a wrapper -X style entry point to get to the repl (which would maybe be something we could provide)#2021-07-0214:47dharriganInteresting!#2021-07-0214:48Alex Miller (Clojure team)the actual repl entry point function inside clojure.main is actually a vararg method so it would actually work as an entry point automatically IF you are using Clojure 1.11.0 alphas with trailing map support for kvarg functions#2021-07-0214:49Alex Miller (Clojure team)but you'd probably want a set of default supplied anyways, needing a wrapper#2021-07-0214:50dharrigangood point. I'll have a ponder over the weekend (I mean, I could always just do clj -X:foo && clj -M...... 🙂#2021-07-0214:51dharriganbut maybe babashka, with it's tasks may help out as well 🙂#2021-07-0214:51dharriganthank you @alexmiller and @borkdude!#2021-07-0214:51Alex Miller (Clojure team)yeah, there are many ways to skin this cat (which is why I don't see a need to add another one :)#2021-07-0214:52dharriganmeow!#2021-07-0219:02Chaseso tools.deps.alpha is for dynamically adding a library without restarting the repl right? How are folks using this universally in all their projects? I'm assuming putting it somewhere in their /clojure/deps.edn config? I'm thinking of putting it in my repl alias as I'm always using that to start a cider nrepl anyways#2021-07-0219:08Alex Miller (Clojure team)no, that feature is not available in the mainline code (it is available on an experimental branch and may be merged in some form at a future time)#2021-07-0219:13Chaseno worries, thanks#2021-07-0220:00seancorfield@chase-lambert I use that experimental branch all the time. My dot-clojure repo has that and a bunch of other aliases in it: see the usage notes here https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L202-L224#2021-07-0220:02seancorfieldI did a REPL-Driven Development talk about six months ago -- for both the London Clojurians and Provo Clojure -- and in that talk/demo I show how to use it with deps.edn to load new dependencies into the REPL while I'm working.#2021-07-0220:02ChaseDoes this mean if I add that in to my deps.edn I could do something like clj -M:add-libs:repl to run both aliases? I am still confused on how to call multiple aliases because I thought I read only the "last" alias allows for :main-opts or something#2021-07-0220:03ChaseI'll rewatch that talk, I think that is where I originally saw this possibility#2021-07-0220:03seancorfieldWith :main-opts only the last one wins, so it depends what is in your :repl alias?#2021-07-0220:05seancorfieldBut, yeah, in this case :add-libs only loads extra (git) deps, so it would combine with your :repl alias too.#2021-07-0220:06Chasethe repl main opts are basically nrepl middleware and "-i" to get a prompt#2021-07-0220:07Chaseit looks like your :add-libs alias needs some Dynamic Class Loader options going on or something though#2021-07-0220:10seancorfieldThat's really for Socket REPLs which don't currently have a DCL. Pretty sure your nREPL setup will have a DCL.#2021-07-0220:10Chasesounds good, I'll give this a shot, thank you#2021-07-0220:14seancorfieldFeel free to DM me with Qs if you have problems. There aren't many of using add-libs...#2021-07-0220:16Chasethanks! It seems to be working great in the repl. I was thinking it would add the dependency to my project's deps.edn for me though.#2021-07-0220:20seancorfieldNope, just in-memory. Which is why, in my demo, I do that weird dance with #_ and code actually in my deps.edn file 🙂#2021-07-0220:20seancorfieldSo I edit deps.edn to add the dep, then run add-libs on it in-situ to actually load the new dep into the running REPL.#2021-07-0220:21ChaseAwesome, I'll go check it out again soon. My lazy self also wants to find a way to auto load the (require '[clojure.tools.deps.alpha ...) at repl start too but I'll leave you fine folks alone on that (for now) haha#2021-07-0220:22seancorfieldPut it in user.clj#2021-07-0220:22seancorfieldClojure loads that when it starts up.#2021-07-0220:22ChaseI was thinking that's how it is done. So it just runs all the code found in that file automatically? It has to be on a certain path?#2021-07-0220:23seancorfieldYeah, so it's usually best to have a :dev alias that adds the path containing user.clj so it doesn't get into your production code. It just needs to be on the classpath.#2021-07-0220:24seancorfield(so you might have dev/user.clj and then :dev {:extra-paths ["dev"]})#2021-07-0220:25Chaseyep, just saw that here: https://github.com/prestancedesign/usermanager-reitit-example which is based on your own example repo. Thanks again#2021-07-0220:26ChaseI feel like the more I learn these tools.deps the more I'm going to keep wanting to run multiple aliases depending on what I'm doing but then this only one alias's main opts thing will be tripping me up constantly. That's not accurate though?#2021-07-0220:27seancorfieldThese days I run most stuff via -X, even if that means writing a little wrapper around someone else's actual -main function 🙂#2021-07-0220:28seancorfield:exec-args composes across multiple -X aliases.#2021-07-0220:34Chaseyeah it's time to sit down and really grok this official deps and cli reference docs I think#2021-07-0220:41seancorfieldIf you have the :new alias (from clj-new) and run clojure -X:new :template app :name chase/example you'll get a small, simple project that uses -X for most stuff.#2021-07-0220:42seancorfield(and soon these tools will be even easier to install and use -- see Alex's clojureD talk that was just released to YouTube today)#2021-07-0220:46ChaseIronically I decided not to use clj-new this time to force myself to learn these things myself. haha. Yeah I have all those clojureD talks lined up to watch, they all look interesting.#2021-07-0221:12seancorfieldIt was a great conference! I attended "live" and the virtual environment was really good.#2021-07-0221:13seancorfieldKind of brutal, time-wise, since I'm in California 🙂#2021-07-0221:14seancorfieldAt work, we're preparing for tools.build -- and we've been using deps.edn since 2018. Just recently we replaced our ad hoc bash-based build scripts with a short build.clj program (which my colleague is refactoring today), and we use -X invocation for a lot of tasks already.#2021-07-0221:16seancorfieldI've been updating depstar so it has multiple "tasks" that can be run individually via -X or together, but I'm waiting for tools.build to become available so I can finish that work off in a way that is compatible (with tools.build) so you should be able to swap out some of the built-in tasks for depstars version of them (sync pom.xml, compile Clojure, build a JAR or uberjar).#2021-07-0306:16Reily SiegelIs there any possibility/interest in integrating deps.edn with https://guix.gnu.org/? As a functional package manager, Guix seems to be relatively in-line with Clojure principles. There are a few advantages that I can think of to a guix integration. First, Guix allows an application/library to define dependencies on non-jvm components. Second, because packaging is a solved (ish) problem in Guix, many of the problems with Uberjars, such as diffuclty of embedding a JVM/JRE can be avoided. Finally, because Guix allows for more complex pacakge definitions than deps.edn, clojure code can (and currently is) automatically compiled by Guix upon installation. However, there is one major drawback to this integration. guix (at least in officially endorsed package channels) requires that all packages be built from source using a Guix build system. However, Clojurescript (which is depended on by many Clojure libraries via core.async) depends on many Google java libraries which are currently cumbersome to build from source in an automated fashion. Because of this, I have chosen to submit the clojure-tools package to an unofficial third-party package channel for now. I think, assuming we can overcome this build requirement (this is a one time cost), a Guix integration could be a major advantage for Clojure. Ideally, this would work in a similar manner to gitdeps, where a dep map could be specified like {my-guix-dep {:guix/package clojure-my-guix-dep}} .#2021-07-0312:35Alex Miller (Clojure team)I’m a little unclear if you are looking to make something available in guix or if you are looking to use guix as a source of dependencies, or both and what problem you’re trying to solve by doing so#2021-07-0312:39Alex Miller (Clojure team)If your are looking to establish interest in a feature or integration it’s best to ask at https://ask.clojure.org to gather votes. An important thing to keep in mind is how deps.edn dependencies are used, which is to gather paths to put on the classpath. If guix mostly holds non classpath things then it’s probably not useful to include#2021-07-0315:44Reily SiegelI'm mostly interested in using guix as a dependency source. Guix packages many java libraries (and some clojure libraries, despite the above issues with building CLJS) that use the class path. The idea behind guix integration is that a classpath library in guix could be packaged to have a dependency on a non-classpath package.#2021-07-0315:44Reily SiegelI will make a post about this on Ask Clojure as soon as I have a chance.#2021-07-0316:44hadilsHi! I am setting up a CI pipeline on buddy.works. This uses Docker containers. I am wondering if tools.deps depends on Maven to be installed. If not, how can I control where the jars are placed during installation?#2021-07-0317:11borkdude@hadilsabbagh18 you don't have to install mvn. I believe tools.deps supports :mvn/local-repo#2021-07-0317:12borkdudehttps://clojure.org/reference/deps_and_cli#_maven#2021-07-0317:15Alex Miller (Clojure team)Correct on both counts#2021-07-0318:28hadilsThank you very much @borkdude and @alexmiller!!#2021-07-0516:09borkdudecan we expect tools.deps CLI to flip the switch on -A to always launch a REPL rather than executing :main-opts anytime soon? I sometimes just want a REPL with the tests on the classpath, clj -A:test, and not execute the test runner#2021-07-0516:10borkdudeI know you can work around this by making moar aliases, but I'm just too lazy/tired to do that :-D#2021-07-0519:06practicalli-johnI did create some aliases for you if you want to use them 🙂
practicalli/clojure-deps-edn#2021-07-0516:11borkdudeI think the deprecation warning has been there long enough now, personally#2021-07-0516:46seancorfieldI don't know. I'm still running across users who don't even know about -X so they still expect -A to work as it always has (and they're sometimes very confused that -M has changed behavior). Like you, I feel -A has had that "legacy" behavior long enough at this point but I think we're more "leading edge" users than most Clojurians as far as the CLI goes?#2021-07-0516:47seancorfieldAs for the test runner, I've switched to -X for Cognitect's test-runner these days so -A:test just starts a REPL for me 🙂#2021-07-0517:16Alex Miller (Clojure team)I'm starting to think about thinking about removing the deprecation :)#2021-07-0517:23borkdudepreserve the old behavior? why?#2021-07-0517:29seancorfieldAnd then have -A and -M be identical? Oh, hey, I guess -R could be used/extended/brought back to start a Repl! 🙂 🙂#2021-07-0517:37borkdudeIn that case I'll wait a bit longer with upgrading my OSS CI configs ;)#2021-07-0517:48Alex Miller (Clojure team)sorry that was unclear, by "remove the deprecation" and I mean make it stop working#2021-07-0517:48Alex Miller (Clojure team)that is, no change in plans#2021-07-0517:56seancorfieldHahaha... OK, good. The sooner the better, IMO, but I understand that not all CLI users are there yet 🙂#2021-07-0516:37borkdudeWhere was that table again where you could see which version of the CLI packaged which tools jar?#2021-07-0516:38seancorfieldhttp://clojure.org/releases/tools#2021-07-0516:41borkdudeah, just found it in the zulip slack archive of the #clj-on-windows channel too :) thanks#2021-07-0518:58borkdudedeps.clj: A faithful port of the clojure bash script to Clojure
Used as native CLI, deps resolver in babashka and getting started REPL in Calva.
New release: v0.0.16
https://github.com/borkdude/deps.clj/releases/tag/v0.0.16#2021-07-0618:45seancorfieldWe've been working on replacing our build shell script with a Clojure build.clj script and we got part-way there and we're "happier". We've gone from several hundred lines of bash down to 100 and build.clj is about 250 lines of Clojure. For "build"-style stuff, it's all good, but we'd like to be able to integrate test setup and execution (DB migrations, running tests) into a single pipeline with build stuff (tagging, pom.xml sync'ing, AOT, JAR creation, deployment). That would mean running tests in the "context" of t.d.a which our build.clj relies on for the "build" stuff -- and t.d.a is a big ol' bag of dependencies that might interfere with running our code. So we find ourselves either running tests separately with just the "dev" dependencies -- and therefore requiring a "driver" script fronting the CLI (to do setup and run tests, and then to run our build.clj script for the rest of it) -- or running tests in the context of dev+t.d.a which doesn't feel ideal (since t.d.a doesn't end up in our artifacts). And that means we find ourselves wanting something like "Boot pods" to provide classpath isolation within a single JVM to run tests with a fixed set of dependencies...#2021-07-0618:46seancorfield(it's interesting to see that Polylith's poly tool has actually gone down this path for running tests so that it can isolate dependencies on a per "project" p.o.v)#2021-07-0618:47seancorfieldGiven the imminence of tools.build (he said, hopefully), what do people think about tooling that integrates test running with artifact building (to perhaps avoid spinning up multiple JVMs)? Is this a worthwhile goal or is it just not worth the hassle?#2021-07-0618:48seancorfield(Both Boot and Leiningen support this model of issuing a single command to run tests and build artifacts but I don't know how widely that capability is actually used in practice?)#2021-07-0618:53borkdude@seancorfield why do you need t.d.a on the classpath after it's done resolving deps / calculating a classpath?#2021-07-0618:55seancorfieldBecause build.clj uses t.d.a to calculate project basis information for various tasks.#2021-07-0618:56seancorfielde.g.,
(with-dir (io/file (str (System/getProperty "user.dir") "/projects/" p))
(let [{:keys [success] :as result}
(uberjar (-> (calc-project-basis) :aliases :uberjar :exec-args
;; cater for being _above_ the project dir:
(update :jar #(str "projects/" p "/" %))))]
(when-not success
(throw (ex-info (str "uberjar failed for " p) result)))))#2021-07-0618:56borkdudecan you compute the classpath and then set that manually using -Scp for the real tests?#2021-07-0618:56seancorfield(`with-dir` is from clojure.tools.deps.alpha.util.dir)#2021-07-0618:57seancorfieldThe point is to avoid spinning up additional processes/JVMs here.#2021-07-0618:57seancorfieldThat fragment above is run for each (sub) project that we want to build a JAR for.#2021-07-0618:58borkdudeBuilding a jar for a project doesn't actually need deps of that project on the classpath right?#2021-07-0618:58seancorfieldYou're missing the point.#2021-07-0618:59seancorfieldI already said that for build stuff, it all works as we want. Running tests is the problematic piece.#2021-07-0618:59borkdudegotcha#2021-07-0619:00seancorfieldI just used the above as an example of why/how we're using t.d.a in our build script -- as tools.build is going to do too, I'm sure.#2021-07-0619:00borkdudeSo you want different dependency sets for each test suite, within one JVM?#2021-07-0619:00seancorfieldIdeally, yes.#2021-07-0619:02seancorfieldWhich is what Boot could do, and what Polylith's test runner does -- and once we're 100% Polylith, we can just rely on that 🙂 but right now we're a mix of legacy subprojects and Polylith "bricks".#2021-07-0619:02borkdudeI think I would just run multiple JVMs as the other options sound a bit brittle, but this may just be my inexperience with the boot pod stuff.#2021-07-0619:03seancorfieldIt's fine if the expectation is "Run tests standalone, run build stuff together" and the ability to run tests in isolation is just a non-goal for tools.build but given Leiningen/Boot's ability to "run tests and build stuff" all-in-one it seems reasonable to ask 🙂#2021-07-0619:04seancorfieldIt's true that Boot's "pods" were indeed problematic but mostly around the supposed optimization of async refreshing and reusing of contexts.#2021-07-0619:07Alex Miller (Clojure team)I've built quite a bit of "basis-making" stuff in the imminent release. I have not done the work to potentially use it to feed something like test-runner but theoretically, it's a small gap#2021-07-0619:09seancorfieldThe "gap" seems to be around classpath-isolation, unless the tests are run in a separate process (which was what we were trying to avoid).#2021-07-0619:09Alex Miller (Clojure team)I'll take a look at that today - it has a couple possible interaction points of interest. tools.build has a facility for spinning processes and some stuff spinning from basis inside compile-clj but the useful bits are not abstracted out and that's been on my list to look at for a while#2021-07-0619:10Alex Miller (Clojure team)are you trying to avoid it as "having 2 things" or avoiding it because it's slower?#2021-07-0619:11seancorfieldFor now, it's not really a huge deal for us: we can run the tests in the same context as t.d.a (and we actually have one test that is connected to our build stuff that requires t.d.a) but t.d.a has such a huge spread of transitive dependencies. I think @hiredman figured out there are about two dozen deps that overlap with our code's deps that have different versions.#2021-07-0619:12seancorfieldMainly b/c we'd like to have a single clojure -X:some:aliases do a bunch of things rather than finding ourselves wanting a bash script to front that, just so it can determine that we really need to run clojure -X:some do a bunch and then clojure -X:some:aliases of things 🙂#2021-07-0619:13Alex Miller (Clojure team)believe me, I get it re deps :)#2021-07-0619:14Alex Miller (Clojure team)yeah, some of the new basis stuff would help you here. the cool thing about the basis is it's a Clojure map so perfectly amenable to being made in one jvm (and/or cached) and picked up for use in another#2021-07-0620:04hiredmanat one point while I was fighting with classloaders on friday I played a little with the basis file, and ultimately realized, for what I was trying to do it wasn't the right thing.#2021-07-0619:15seancorfieldOr even passed back and forth between two classpath-isolated "processes" in the same JVM 🙂#2021-07-0619:16Alex Miller (Clojure team)yes, although I'm not rushing to integrate that directly :)#2021-07-0619:18Alex Miller (Clojure team)I sometimes ponder whether you could have an architecture where you ran a "basis server" that just computed (and cached) classpaths, and another that spun empty jvms, then dynamically updated their classpaths loaded from the basis server#2021-07-0619:19Alex Miller (Clojure team)maybe that's reinventing nailgun, I dunno#2021-07-0619:23seancorfieldA more narrowly-focused problem to solve is just a test runner that is driven by deps.edn for a monorepo -- where different subprojects can have different deps. It's exactly what poly attempts to do so we may well just punt on this as something worth even solving since it will eventually "go away" for us. And it may well be that this is just a non-problem for folks who have a single testing context (i.e., not a monorepo).#2021-07-0619:25seancorfield(the poly test runner also has the benefit of incremental testing -- so it only runs tests that could be affected by files that have changed, and optionally for a specific "project" p.o.v)#2021-07-0619:28borkdudeI'm working (still exploring the possibilities) on an experimental native version of tools.deps.alpha. I don't expect Sean to adopt this (given its non-official status), but just for the sake of argument: it can calculate classpaths (or basis-es) very very fast, without spinning up a JVM. This would cut out one JVM instance at least, for the scripting part. But you would still have the "I want one JVM to run with different dep combinations/classloaders" problem, which is probably the biggest challenge anyway.#2021-07-0619:29borkdudeThere is a working version to download here: https://github.com/borkdude/tools-deps-native-experiment#2021-07-0619:32seancorfield"I don't expect Sean to adopt this (given its non-official status)" -- you know me well @borkdude 🙂#2021-07-0705:25plexusdoes tools.deps use maven repositories specified in the pom of dependencies?#2021-07-0705:26plexusWe have a library which wraps a Java library which is in its own maven repo (not on central or clojars). I thought it would be enough to point to that repo in our pom like this: https://github.com/lambdaisland/witchcraft/blob/main/pom.xml#L91-L95 but seems that isn't working#2021-07-0705:57hiredmanThere was something in the recent release notes about this#2021-07-0706:00hiredmanNot in the changelogs, I wonder where I read it, if I recall the gist of it is no, tools deps won't use repos specified in deps, and I believe the reasoning was to avoid potential supply chain attacks#2021-07-0706:06hiredmanThe changelog does link to this cve https://maven.apache.org/docs/3.8.1/release-notes.html#2021-07-0706:13seancorfield@plexus Yup, if you depend on something that is pulling its dependencies from "non-standard" repos, you need to list those explicitly in your own deps.edn for the reasons that @hiredman mentioned.#2021-07-0706:15plexusthanks, that's good to know#2021-07-0713:37dharriganI have in my deps.edn a set of custom mvn/repos for our company S3 bucket containing jars.#2021-07-0713:38dharrigan(confirming that having custom mvn/repos does work totally fine 🙂 )#2021-07-0718:39Bobbi TowersIs there a way to gain control over the way deps are downloaded? I'm living in the woods without high-speed internet and starting a new project often takes dozens of retries over several days. This happens because most downloads end up failing, even after seemingly gathering significant packages which are presumably "cleaned up" when it fails. Is there any way I could change this behavior, or work around it?#2021-07-0718:40Joshua SuskaloDependencies downloaded are cached in your local maven repository, and as long as the entire package successfully downloaded, it wouldn't be requested from the server again as long as you are requesting the same version.#2021-07-0718:41Bobbi TowersAh, so when it appears that one is "done", that really just means that it started#2021-07-0718:43Joshua SuskaloThe message for a dependency being downloaded occurs when the request is initiated to my understanding#2021-07-0718:44Alex Miller (Clojure team)there are two messages if you look closely - one for the metadata (.pom) used during dependency expansion, and one for the jar artifact itself#2021-07-0718:44Alex Miller (Clojure team)there is a control for the number of concurrent download threads and you might be better off with less concurrency, so you could try -Sthreads 1 to get that#2021-07-0718:45Bobbi TowersOh cool, thanks I'll try that :)#2021-07-0720:34hiredmanYou shouldn't be re-downloading things either, once downloaded they are cached in ~/.m2 , if you are using something like docker be sure to store that persistently some way#2021-07-0720:39Bobbi TowersWhat I really want is the ability to download them like torrents, or youtube-dl#2021-07-0720:43tvaughanIs it possible to download artifacts from maven repos using rsync?#2021-07-0720:57hiredmanfind a remote host with good bandwidth, run
clj -Sdeps '{:deps ...}' -Spath
then rsync over the ~/.m2 from there#2021-07-0721:09dominicmSome maven repos do have rsync open. clojars used to.#2021-07-0722:31Bobbi TowersOh I get it... that's a great idea! Thank you.#2021-07-0803:01hiredmanYou could even setup syncthing between your local m2 and remote m2, and wrap the above clj call in a single ssh command#2021-07-0808:04vlaaadWhat's the stance on renaming existing libraries to use reverse domain names (e.g. vlaaad/remote-repl -> io.github.vlaaad/remote-repl)? Should I do that? Should I also rename the root package?#2021-07-0817:17seancorfieldI renamed all my popular libs over a period of three or four months, double-publishing them under both the legacy group ID and com.github.seancorfield for a while (I am no longer doing that). antq seems to correctly detect this migration and will suggest the group change along with any version change when you run it against your deps.edn files.
I did this partly to raise visibility, partly because clj-new (and depstar) are trying to be "model citizens" and clj-new now generates new projects that follow this group/artifact pattern, and partly so that any new libs I release -- which have to go into a new VGN like com.github.seancorfield -- will be consistent will my existing libs. And that just happened with the latest version of what was expectations/clojure-test since the tooling involved in supporting cljs (new in the 2.0 version of the lib) filters out JARs whose names start with clojure so I had to change the artifact ID, which meant I could not publish it under the old non-VGN group ID!#2021-07-0808:07mpenetI personally do com.foo/lib-name and use foo.lib-name as root namespace in clj code, so foo.lib-name/do-something etc#2021-07-0808:08dominicm@vlaaad https://github.com/clojars/clojars-web/issues/801#2021-07-0808:33rickmoynihan@vlaaad: What is the point?
I thought the main purpose of the io.github.user/project convention was so git deps didn’t have to specify a :git/url and to make the artifact name for git deps less arbitrary.
Changing the artifact name for a maven dep seems like it’ll only cause pain (unless you also change the root package); but it feels like pointless churn to me.#2021-07-0808:35rickmoynihanIn the case of git deps, one thing I’m not clear on is whether io.github.user/project and user/project {:git/url ""} will canonicalise to the same thing? If so what about ssh urls?#2021-07-0808:37vlaaadYeah, I'm inclined to think changing the name will only cause confusion and errors...#2021-07-0809:04dominicmI would discourage you unless you're really sure. I predict a lot of pain in people's future if a lot of projects migrate without considering the transitive dependency clobbering problem.#2021-07-0809:16rickmoynihanYeah my point exactly#2021-07-0811:41Carlowhich branch should I use for add-lib @seancorfield? I understand the idea is a few years old now, is there some particular reason it has not been merged?#2021-07-0812:36Alex Miller (Clojure team)The most recent one is add-libs3. It hasn’t been merged because we have not decided on various aspects of the api and integration#2021-07-0817:10seancorfieldThe :git/url and :sha in my dot-clojure repo is pretty much always up-to-date for that, along with usage examples: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L202-L228#2021-07-0812:34Alex Miller (Clojure team)On the renaming question, I do not see any reason to rename any existing thing. Unless maybe it is a tool, and you want to make it easier to install, and no one else depends on it as a dep#2021-07-0814:18vlaaadNot like anyone cares, but I rewrote remote-repl's deploy script to a pure clojure (no more bash/powershell).
It's nice to iterate on build script in the REPL. The biggest part is pom, mostly because depstar doesn't put commit sha to pom's scm tag. In the end I decided to just create the whole pom from code — and look https://github.com/vlaaad/remote-repl/blob/master/build/build.clj#L40-L76 with Clojure!#2021-07-0817:18seancorfieldWe just switched out our build bash script at work for a build.clj "program" instead and we're liking that switch.#2021-07-0814:43Alex Miller (Clojure team)that will be even easier with tools.build :)#2021-07-0814:46vlaaadYay, that's what I hoped to hear :D#2021-07-0908:11rickmoynihanIncase you’ve not seen it, Alex talks about it in more detail here:
https://www.youtube.com/watch?v=BTAx-gFz6Ks&list=PLaSn8eiZ631nON7le-wdZxTR0c5bxqPYi&index=12#2021-07-0908:12rickmoynihanActually you obviously have seen it; as the naming convention thing was I think first mentioned here#2021-07-0910:46vlaaadI've seen it live :P#2021-07-0910:47vlaaadIt had that sweet copyright infringement at the beginning :D#2021-07-0814:48dharriganIs my understanding correct, that it will be a combination of deps.edn to define -X functions and tools.build to provide a toolchain?#2021-07-0815:23Alex Miller (Clojure team)well more you write a build program that uses tools.build, then use -X (or really -T but same idea) to invoke it#2021-07-0815:26dharriganperfecto!#2021-07-0815:26dharriganthank you 🙂#2021-07-0817:17seancorfieldI renamed all my popular libs over a period of three or four months, double-publishing them under both the legacy group ID and com.github.seancorfield for a while (I am no longer doing that). antq seems to correctly detect this migration and will suggest the group change along with any version change when you run it against your deps.edn files.
I did this partly to raise visibility, partly because clj-new (and depstar) are trying to be "model citizens" and clj-new now generates new projects that follow this group/artifact pattern, and partly so that any new libs I release -- which have to go into a new VGN like com.github.seancorfield -- will be consistent will my existing libs. And that just happened with the latest version of what was expectations/clojure-test since the tooling involved in supporting cljs (new in the 2.0 version of the lib) filters out JARs whose names start with clojure so I had to change the artifact ID, which meant I could not publish it under the old non-VGN group ID!#2021-07-0817:31Alex Miller (Clojure team)> filters out JARs whose names start with `clojure` #2021-07-0817:31Alex Miller (Clojure team)wat?#2021-07-0817:52seancorfieldI was pretty gobsmacked by that too... apparently it's something doo does? https://github.com/bensu/doo (I don't do cljs so this came from @kkinnear as part of three PRs to add standalone cljs support to the lib).#2021-07-0818:55Alex Miller (Clojure team)seems..... dubious#2021-07-0917:27Alex Miller (Clojure team)Happy to announce a pre-release of the Clojure CLI 1.10.3.905 and tools.build - read lots more here: https://clojure.org/news/2021/07/09/source-libs-builds
This is being provided for feedback and tire-kicking and there may be changes before it becomes a release.
Notably if you have been following the pre-releases, we've removed the multi-function execution support.#2021-07-0917:28Alex Miller (Clojure team)that's likely coming back, but might differ a bit#2021-07-0917:31dpsuttonthank you for releasing with the git history rather than copied to a new repo and everything "Initial Commit"#2021-07-0917:32Alex Miller (Clojure team)it's a tortured road ... :)#2021-07-0917:37dpsuttonbut its nice to see the thought in it. love reading how well-thought-out programs evolve to do things rather than just being presented wholly formed#2021-07-0918:01borkdude@alexmiller Congrats on the release. I think this implementation in process can be problematic when the buffer of either stdout or stderr is full and the process will wait for those streams to be consumed. Afaik this is why clojure.java.shell does that copying in futures while the process is running, rather than letting it complete beforehand.
https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/tasks/process.clj#L68-L70#2021-07-0918:03borkdudeAnd btw, the defaults are ProcessBuilder$Redirect/PIPE anyway, so no need to set those explicitly#2021-07-0918:08Alex Miller (Clojure team)thanks, need to set up a test for that#2021-07-0918:12borkdudeJust try to make the process print a couple of megabytes of output#2021-07-0918:12borkdudea few 100kbs should already be enough#2021-07-0918:13borkdudein combination with the :capture or :ignore option#2021-07-0919:09Alex Miller (Clojure team)logged as https://clojure.atlassian.net/browse/TBUILD-1 to track it#2021-07-0918:29seancorfield@alexmiller In the tools.build guide, it says:
Invocation that activates :dev environment will look like this:
clj -X:build jar :env :dev
The kv params are passed at the end and will be used to invoke the first function, clean, then the return of that will be passed to jar.
but clean is not being invoked here. I assume at some point it said clj -X:build clean jar :env :dev but the multi-function execution stuff got pulled?#2021-07-0918:31Alex Miller (Clojure team)Yeah, probably missed a couple places#2021-07-0918:30seancorfield(we're already using multi-function execution with -X at work so I'm not sure what we'll do regarding tools.build -- we will most likely hold off testing it until multi-function execution comes back in a later prerelease)#2021-07-0918:33Alex Miller (Clojure team)you're bleeding on the bleeding edge :)#2021-07-0918:34seancorfieldWell, I was so excited about the multi-function exec stuff that we adopted it immediately 🙂#2021-07-0918:36Alex Miller (Clojure team)it'll be back probably, just need to work out some of the potential ambiguities in arg parsing#2021-07-0918:41seancorfieldWhen I'm back at work (next Wednesday), I'll take a look at how/if it will affect us. I think, since we switched our build shell script to a build.clj Clojure script, we may have switched back from -X to -M until we knew what tools.build etc was going to look like in the wild, and I was only using multi-function -X from the CLI directly, rather than in our build system...#2021-07-0920:18dominicm@alexmiller If I'm reading the source correctly, the uber task clobbers duplicate files. That means data_readers.cljc won't work correctly, and also that the apache-2.0 licence isn't followed (they require preservation of the licence).
I think @seancorfield solved this by having merger functions which can be registered in some way for different matches of files.#2021-07-0920:20hiredmanhttps://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/tasks/uber.clj#L67-L73#2021-07-0920:21dominicmah, i did miss that then 😛#2021-07-0920:22dominicmThe LICENSE thing is still a problem though.#2021-07-0920:25seancorfieldYes, depstar does all sorts of stuff to ensure merging of various files properly (including the log4j2 plugin cache). I plan to offer near drop in replacements for jar/uber soon.#2021-07-0920:30dominicmI should probably get pack up to snuff with the new tools.build too.#2021-07-0920:32dominicmIt's been rotting while I wait for tools.build, so I could clarify the interfaces. But there's a parallelized jar generation tool in pack too which is probably worth pulling out independently.#2021-07-0920:34Alex Miller (Clojure team)yes, this is an area that I got to a point but would like to take further#2021-07-1018:15dominicmYou probably know this, but the maven shade plugin implements a lot of this already. Worth looking through for what's worth stealing at the least.#2021-07-1018:16Alex Miller (Clojure team)everything in tools.build is a clean room impl and I have largely avoided looking at other existing tools#2021-07-1018:16Alex Miller (Clojure team)I'd rather think more about what we should be doing#2021-07-0920:26dominicmI'm a little unsure about this :ensure key - it seems at risk of proliferation. e.g. what if I want to separate out my prod/dev builds into target/prod and target/dev so that I can build cleanly for production without any dev files leftover in there (short of cleaning, because then that breaks my background clojurescript build!).
Different libraries/companies are going to have different ideas of where built output should live. I understand the purpose is to ensure the classpath is pre-populated, but I'm not sure this is the best overall solution. Could the folder be a param somehow to the library being prepped?#2021-07-0920:39Alex Miller (Clojure team)can you talk more about the differences between dev and prod builds?#2021-07-0920:42seancorfieldHow would that be affected by a third-party library dependency that needs the CLI to run some task before the library could be used after checkout?#2021-07-0920:49Alex Miller (Clojure team)is that a question for me or dominic?#2021-07-0922:04seancorfield(for Dominic, sorry)#2021-07-1018:10dominicmReading the following messages around it being local to the dependency, my concerns are allayed. I was thinking of application-specific dev vs prod such as different dependency versions or similar.#2021-07-0920:40seancorfield@dominicm My reading is that it is local to that dep: for the CLI tools stuff to "prep" a checked out source lib before use the first time. So the :ensure directory is going to be within the .gitlibs checkout (only) and is going to be in the :paths vector for that dependency.#2021-07-0920:40Alex Miller (Clojure team)that's correct#2021-07-0920:45Alex Miller (Clojure team)git libs are per-SHA source checkouts and the assumption is that there is one way to prepare it to be on the classpath and if you do it once, you are done. (nothing is going to change in that sha)#2021-07-0920:48Alex Miller (Clojure team)moreover, all builds on the machine are going to share the sha output, so parameterization is problematic (without handling this more explicitly)#2021-07-0920:48hiredmanthat seems like it will run afoul of projects doing aot compilation with different versions of clojure?#2021-07-0920:49Alex Miller (Clojure team)aot compilation is still something best done when you compile your app#2021-07-0920:49hiredmanah, I see, prep is more for compiling java#2021-07-0920:49Alex Miller (Clojure team)prepping is intended for things like java compilation, resource copying/filtering, etc#2021-07-0920:50Alex Miller (Clojure team)the advice in https://clojure.org/guides/dev_startup_time doesn't change if you want to do dev time aot#2021-07-0922:51seancorfield@alexmiller Just to let you know: I switched depstar over to use t.d.a/`create-basis` and it does exactly what I need, so I can remove all my custom basis code now -- thanks!#2021-07-0922:52Alex Miller (Clojure team)Nice!#2021-07-0923:18seancorfielddepstar now has CLI tools support, using the latest prerelease:
clojure -Ttools install com.github.seancorfield/depstar '{:git/tag "v2.1.250"}' :as depstar
and then you can do
clojure -Tdepstar jar :jar MyLib.jar
#2021-07-0923:18seancorfield(it has :tools/usage {:ns-default hf.depstar} so that command invokes hf.depstar/jar)#2021-07-0923:53seancorfieldclj-new also has CLI tools support now:
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.314"}' :as new
# create a new app:
clojure -Tnew app :name myname/myapp
# create a new library:
clojure -Tnew lib :name myname/mylib
# create a new template:
clojure -Tnew template :name myname/mytemplate
# create a new project from a public template:
clojure -Tnew create :template electron-app :name myname/myelectron-app#2021-07-1016:40DerekWaiting to publish to clojars?#2021-07-1018:34seancorfield@UPEKQ7589 It doesn't need to be on Clojars for this: CLI tools takes it from GitHub.#2021-07-1018:42DerekThanks Sean. Very cool#2021-07-1022:28seancorfield(I put it up on Clojars anyway, just now, for folks who are using it via deps.edn)#2021-07-0923:58seancorfield@alexmiller One problem I ran into: if you use the new CLI tools stuff to install a tool that depends on an OLDER t.d.a, because the EDN for the tool has :git/sha in it but not :sha, when the tool is run, the older version of t.d.a trips over the lack of :sha. You can see this if you install clj-new 1.1.309 instead of 1.1.314 and then try that last line:
(! 1063)-> clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.309"}' :as new
Installed new
(! 1064)-> clojure -Tnew create :template electron-app :name myname/myelectron-app
Unqualified function can't be resolved: create
# ^ expected because 1.1.309 doesn't include :tools/usage
(! 1065)-> clojure -Tnew clj-new/create :template electron-app :name myname/myelectron-app
Failed with: Library com.github.seancorfield/clj-new has missing :sha in coordinate.
# ^ this is "unexpected" but happens because clj-new 1.1.309 depends on t.d.a 0.11.910 which doesn't recognize :git/sha
Execution error (ExceptionInfo) at clj-new.helpers/resolve-remote-template (helpers.clj:186).
Could not load artifact for template: electron-app
Tried coordinates:
[electron-app/boot-template "RELEASE"]
[electron-app/lein-template "RELEASE"]
For more detail, enable verbose logging with :verbose 1, 2, or 3
Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-7876746592438264246.edn#2021-07-1000:03seancorfield(this may be unique to clj-new because of the way it tries to dynamically resolve/load dependencies to find the specified template? the error EDN file does not mention the missing :sha failure at all)#2021-07-1000:11Alex Miller (Clojure team)yeah, that's just the nature of it. you can still use :sha though#2021-07-1000:43seancorfieldI think you're missing the point: :git/sha is what is installed by the CLI, not clj-new.#2021-07-1000:44seancorfieldI think this will cause problems for any t.d.a-based tool that is installed via the new CLI but relies on an older t.d.a (that does not recognize :git/sha) to create an environment for executing code?#2021-07-1000:46seancorfieldThe only difference between clj-new 1.1.314 which works and 1.1.309 which doesn't, is the version of t.d.a that they depend on.#2021-07-1000:46seancorfield(and the new release now has :tools/usage in deps.edn but that's not relevant for this)#2021-07-1001:00Alex Miller (Clojure team)doesn’t work at install time or at tool execution time?#2021-07-1004:33seancorfieldI showed exactly what doesn't work in the main channel... Let me know if I need to explain it better but it should be an exact repro.#2021-07-1013:09respatializedI have a kind of open ended question about deps.edn and library/program design. Say I want to offer users of my library / program two different usage patterns that have different dependencies: the use case I'm thinking of here would be one version of a program that runs purely in memory and one that uses a database for persistence. For those users who want to run it only in memory, is there a way for them to pull in my library without pulling in the DB deps as well?#2021-07-1013:33borkdude@afoltzm usually you make these dependencies optional, by just not making them dependencies of your lib#2021-07-1013:33borkdudeif you want to provide the storage-related functionality, you tell users to include extra deps#2021-07-1018:08practicalli-johnI guess I missed this, it seems the : is now not used between the flags , or is the : between the flag and first alias name optional (maybe it always was and I didnt notice it)?
clojure -X:project/new
clojure -Xproject/new
That's probably a clue to get the latest version and do some UX testing 🙂#2021-07-1018:10practicalli-johnAesthetically I like the initial colon, especially when using multiple aliases clojure -M:fig:cider-cljs, feels more consistent. I appreciate I seem to be weird in that I like typing... :rolling_on_the_floor_laughing:#2021-07-1018:12dominicm@jr0cket iirc, officially only -X:fig:cider-cljs is supported, the other happens to work and the code was intentionally written to work with that kind of input.#2021-07-1018:13Alex Miller (Clojure team)aliases should always use :#2021-07-1018:14Alex Miller (Clojure team)tool names are symbols and should not use a colon (with -T)#2021-07-1018:14Alex Miller (Clojure team)in some cases, aliases without colons are tolerated (but don't do that)#2021-07-1018:18practicalli-johnAh, so T is in the same style of flags as J and S it would seem. I should definitely get the latest and have a look at clojure --help . Thanks all.#2021-07-1018:20Alex Miller (Clojure team)when you find it is lacking, please let me know :)#2021-07-1018:21Alex Miller (Clojure team)I do have some todos in the syntax documentation area for next week#2021-07-1018:39practicalli-johnIn the clj -h I think this could potentially be misleading, especially those new to Clojure CLI and the concept of aliases
exec-opts:
-Aaliases Use concatenated aliases to modify classpath
-X[aliases] Use concatenated aliases to modify classpath or supply exec fn/args
-M[aliases] Use concatenated aliases to modify classpath or supply main opts
-P Prepare deps - download libs, cache classpath, but don't exec
It assumes it is known that an alias is :new and not just the name new . It is stated that aliases are keywords earlier in the texts, but that can easily be missed
Perhaps adding a specific example as with -X:deps section
Programs provided by :deps alias:
-X:deps mvn-install Install a maven jar to the local repository cache
-X:deps git-resolve-tags Resolve git coord tags to shas and update deps.edn
I have always found a few specific examples help clarify the general approach#2021-07-1018:47Alex Miller (Clojure team)The help here is something I have struggled to make simultaneously small and useful :)#2021-07-1018:31borkdude@jr0cket Did you know clojure has a man page as well? :-) You might in fact be the person who mentioned it here, don't remember.#2021-07-1018:32practicalli-johnI did know, but wasnt the one who mentioned it. I will read it (when I've installed the new version)#2021-07-1018:39practicalli-johnIn the clj -h I think this could potentially be misleading, especially those new to Clojure CLI and the concept of aliases
exec-opts:
-Aaliases Use concatenated aliases to modify classpath
-X[aliases] Use concatenated aliases to modify classpath or supply exec fn/args
-M[aliases] Use concatenated aliases to modify classpath or supply main opts
-P Prepare deps - download libs, cache classpath, but don't exec
It assumes it is known that an alias is :new and not just the name new . It is stated that aliases are keywords earlier in the texts, but that can easily be missed
Perhaps adding a specific example as with -X:deps section
Programs provided by :deps alias:
-X:deps mvn-install Install a maven jar to the local repository cache
-X:deps git-resolve-tags Resolve git coord tags to shas and update deps.edn
I have always found a few specific examples help clarify the general approach#2021-07-1018:42practicalli-johnThe man clojure page looks very nice and much clearer I believe, e.g
clj-opts
-Jopt Pass opt through in java_opts, ex: -J-Xmx512m
-A:alias...
Concatenated aliases for REPL execution, ex: -A:dev:mem
-X:alias... fn KPATH V
Exec alias to invoke a function that takes a map, with keypath/value overrides. Function must either be supplied or be in :exec-fn argmap for alias
-M:alias...
Exec clojure.main, either from opts in alias or command line
#2021-07-1018:59EddieI am learning about how aliases can be used as replacement for the maven “provided” dependency scope and I am having trouble understanding how to accomplish a very specific kind of compilation task.
My AOT compiled application is deployed into an environment that supplies some pre-compiled deps. The application references classes in these pre-compiled deps and thus they must be on the compile-time classpath. However, I do no not want these deps (or their transitive deps) in the final uberjar.
I understand how aliases allow for these deps to only be provided during the “build” task, but I have not been able to figure out how to declare these deps as being compile-time only. My current understanding is that this is https://ask.clojure.org/index.php/9110/scope-in-deps-edn-should-be-added-and-not-deleted-from-pom-xml?show=9117#c9117 not supported by tools.deps because aliases can be used to same the effect. However aliases don’t have any control over if a dep is included in an uberjar and it would be the responsibility of the assembly/build tool (for example https://github.com/seancorfield/depstar) to handle the exclusion of certain deps in the uberjar. Is this understanding correct?#2021-07-1019:00EddieIn case it is helpful, my particular environment is a https://spark.apache.org/ that provides Spark as a pre-compiled dep at runtime.#2021-07-1019:13Joshua SuskaloI believe you are right with the build tool handling the exclusion of certain deps. It turns out that depstar has a way to specify specific namespaces/classes not be included in the final uberjar. You may look at that to solve this particular issue.#2021-07-1019:36EddieThanks! I played around with the :exclude option on depstar but (as far as I could tell) you would have to carefully identify all the unnecessary transitive dependancies and curate a set of regex that exclude them by name. I asked over in #depstar to see if I was missing something.#2021-07-1021:05seancorfieldSpark is a use case that depstar is specifically designed to work with. That is why the compile alias stuff was added @U7XR2PZFW#2021-07-1021:06seancorfieldI'm not at a computer to give you more details. You need to read the depstar docs closely about aliases.#2021-07-1021:59seancorfieldThis is the relevant issue @U7XR2PZFW https://github.com/seancorfield/depstar/issues/75#2021-07-1022:01seancorfield(so this was Storm, but the same thing applies for Spark, and the approach is to separate out "provided" deps into an alias so it doesn't become part of the JAR but specify that alias to the "compile" phase of depstar so it is available for AOT compilation)#2021-07-1101:36EddieThank you @U04V70XH6! I was indeed missing the :compile-aliases option in depstar.
I’m still mentally decoupling the notion of “scopes” used in maven into the independent parts that we prefer in Clojure. Based on everyone’s responses it seems like my mental model is correct: It is not the role of deps.edn to declare the separation between compile-time and runtime classpaths, but rather this is handled by an assembly/build tool (like depstar or tools.build). Furthermore these tools leverage aliases (which is a construct provided natively by deps.edn) to compose a classpath.
This is clear to me now, and I like it a lot! Thanks all. Super helpful, as always. 🙂#2021-07-1019:14Alex Miller (Clojure team)I can't speak to depstar, but the new https://clojure.org/guides/tools_build library allows you to construct a "basis" (a classpath execution context) in your build with whatever modifications you need. Additionally, you can construct a different basis for different parts of your build. So for example you could define an alias with the compile-time library:
(def compile-basis (b/create-basis {:aliases [:compile-libs]}))
;; use that compile-basis with b/compile-clj
(def runtime-basis (b/create-basis {})) ;; default basis, matches project with no aliases
;; use runtime-basis to b/write-pom (declares your dependencies in the pom from the basis for publishing)
;; use runtime-basis with b/uber to build the uber jar#2021-07-1019:37EddieThanks @U064X3EF3! I will spend some time learning tools.build. It sounds like it has the level of control that I require.#2021-07-1022:01seancorfielddepstar has options specifically for this use case (well, originally added to support Storm). I explained in the other thread.#2021-07-1019:15Alex Miller (Clojure team)the key here being that it's a program and you can do whatever you want :)#2021-07-1021:15cflemingWith the new tools.build stuff, has there been any improvement in constructing classpaths with deps that do not include Clojure, to allow it to be used for non-Clojure projects (or non-Clojure modules in mixed-language projects)? Last time I looked at this that was impossible since the system deps.edn always pulls it in and there was no way to override that.#2021-07-1021:27borkdude@cfleming I've been doing this in bb for a while using classpath overrides#2021-07-1021:28cflemingDo you mean overriding the Clojure jar with e.g. an empty jar or something?#2021-07-1021:28borkdude:classpath-overrides {org.clojure/clojure ""
org.clojure/spec.alpha ""
org.clojure/core.specs.alpha ""}
#2021-07-1021:28borkdude(sorry for indentation, copied out of context)#2021-07-1021:29borkdudeI use this for babashka dependencies which do not need clojure on the classpath#2021-07-1021:29cflemingNo worries - that’s interesting, I’ll try that, thanks. I’m surprised that works, I must admit. I’d be interested in Alex’s view on whether that’s defined behaviour or a happy accident 🙂#2021-07-1021:29borkdudeI learned this from Alex :)#2021-07-1021:33cfleming@borkdude I guess that means you always build them using an alias like :no-clojure or something similar, is that right?#2021-07-1021:34borkdude@cfleming This is what's used in bb:
https://github.com/babashka/babashka/blob/b6e65c08a8fef8d04d9ad46685a99b4902e52e89/src/babashka/impl/deps.clj#L54-L76#2021-07-1021:35cflemingThanks, yeah, I was just looking at your repo. I see you also have :classpath-overrides {org.clojure/clojure nil ... } in one place, does that work similarly?#2021-07-1021:36borkdudethat's similar yes.#2021-07-1021:36borkdudeThe call to deps/-main is calling into deps.clj which is my re-implementation of the Clojure CLI in Clojure itself#2021-07-1021:38borkdudeso I'm basically calling as if I'm writing -Sdeps ... on the command line#2021-07-1021:39cflemingInteresting, thanks. I’ll play around with that and see if I can make it work for building Cursive.#2021-07-1116:43seancorfield@alexmiller Q about basis calculation: I'm not sure if this is a bug, but it caught me out (and was raised as a bug against depstar). If you call create-basis with :aliases, the :deps entry in the basis you get back does not reflect any :extra-deps in those aliases -- but it does reflect :replace-deps from those aliases. I was using that basis as the argument to t.d.a's sync-pom which only uses :deps and :paths when constructing the pom.xml file, so when depstar tried to create/sync a pom file, it looked as if :aliases were being ignored. My "fix" for this in depstar is to do the following:
(let [{:keys [resolve-args] :as basis}
(t/create-basis (cond-> {:aliases aliases}
repro (assoc :user nil)))]
(update basis :deps merge (:extra-deps resolve-args)))
so that when I pass that to sync-pom, the :deps entry includes the :extra-deps from the :aliases and they appear in the <dependencies> section, which is what users expect. It seems that if you do clojure -A:some:aliases -Spom, the behavior is correct: any :extra-deps from those :aliases do end up in the pom.xml file (presumably because they are added to :deps by the time sync-pom is called?) so this feels like a bug in t.d.a to me.#2021-07-1121:32seancorfieldIn the tools.build guide, it seems to indicate that the :build alias would contain :deps {io.github.clojure/tools.build {:tag "v0.1.2" :sha "81f05b7"} -- is that something specific to -T? I would have expected :replace-deps {io.github.clojure/tools.build {:git/tag "v0.1.2" :git/sha "81f05b7"} for a tool (`:replace-deps` instead of :deps and :git/ qualifiers on the tag and SHA).#2021-07-1121:36Alex Miller (Clojure team)the :deps is coming from the merged deps.edn so is part of the input to the basis calculation#2021-07-1121:38Alex Miller (Clojure team)for the deps to use when writing a pom, you should really use the root deps from :libs (the output of the basis). I'm not sure what write-pom in tools.build uses, might be wrong there#2021-07-1121:39Alex Miller (Clojure team):deps and :replace-deps have always been synonyms in aliases#2021-07-1121:40Alex Miller (Clojure team)we had basically switched to making :replace-deps the primary but when using it with -T, :deps makes more sense to me - this is something I'm still thinking about#2021-07-1121:46Alex Miller (Clojure team)yeah, write-pom is still using :deps. that's the bug imo. logged as https://clojure.atlassian.net/browse/TBUILD-3#2021-07-1122:54seancorfieldI tripped over this in tools.deps.alpha BTW, not tools.build, so I think this affects both libraries?#2021-07-1213:10Alex Miller (Clojure team)yes. I expect ultimately that the deps version is going to go away and build will be the way to do this#2021-07-1216:23Alex Miller (Clojure team)fixed TBUILD-3, and also updated similar code for clj -X:deps mvn-pom , clj -Spom was already doing this#2021-07-1218:20seancorfieldInteresting. So the expectation is that tools like depstar and clj-new that currently depend on t.d.a and calculate a basis would instead depend on tools.build in the future?#2021-07-1218:41Alex Miller (Clojure team)yes, I'm expecting the pom writing to eventually live only in tools.build and to phase out the -Spom and -X:deps mvn-pom options#2021-07-1218:48seancorfieldHmm, I would want more control over where the pom.xml file gets written before I could switch over, I think. Doesn't write-pom put the file into Maven-style directory structure? Can it update-in-place, like -Spom currently does?#2021-07-1218:53Alex Miller (Clojure team)currently, no, but I expect it's config to expand#2021-07-1218:53Alex Miller (Clojure team)these changes are not imminent#2021-07-1218:54Alex Miller (Clojure team)are you using -Spom or -X:deps mvn-pom right now?#2021-07-1218:54seancorfieldNo, because I use t.d.a as a library (in depstar) to update a pom.xml in place or write it to a specific target directory.#2021-07-1218:55Alex Miller (Clojure team)ok, thx#2021-07-1218:55seancorfieldclojure.tools.deps.alpha.gen.pom/sync-pom specifically.#2021-07-1218:56seancorfieldhttps://github.com/seancorfield/depstar/blob/develop/src/hf/depstar/pom.clj#L46#2021-07-1122:20seancorfieldYay me! I found a bug 🙂 Thanks for the other answers. And the :tag vs :git/tag Q observation @alexmiller?#2021-07-1213:09Alex Miller (Clojure team)I haven't had a chance to look into it yet#2021-07-1122:22seancorfield(I have hf.depstar.api/jar and hf.depstar.api/uber as drop-in replacements for the tools.build.api versions working locally -- ~still trying to decide whether I want to also expose the pom and aot stuff from ~ -- the API for those two functions is too different from how depstar currently works so I've cleaned up hf.depstar so that it can offer an alternative "build" approach via -X/`-T` or in a build.clj script... but now I need to write a lot more documentation!)#2021-07-1218:20seancorfieldInteresting. So the expectation is that tools like depstar and clj-new that currently depend on t.d.a and calculate a basis would instead depend on tools.build in the future?#2021-07-1201:51lgesslerI'm having an issue getting depstar to work--I have tools.namespace in extra-deps but when i clojure -X:depstar it can't find tools.namespace. Details in thread#2021-07-1201:52lgesslermy :depstar alias in deps.edn:
{:exec-fn hf.depstar/uberjar
:exec-args {:aot true
:jar "target/server.jar"}
:extra-deps {org.clojure/tools.namespace {:mvn/version "1.0.0"}
thheller/shadow-cljs {:mvn/version "2.15.0"}
com.github.seancorfield/depstar {:mvn/version "2.1.253"}}}#2021-07-1201:52lgesslerthe error:
$ clojure -X:depstar
Exception in thread "main" Syntax error compiling at (user.clj:1:1).
at ...
Caused by: java.io.FileNotFoundException: Could not locate clojure/tools/namespace/repl__init.class, clojure/tools/namespace/repl.clj or clojure/tools/namespace/repl.cljc on classpath.#2021-07-1201:53lgesslerMoving tools.namespace and shadow-cljs to my main :deps key solves the issue, but that's not ideal of course. Think I'm doing something silly here, would appreciate any insights#2021-07-1202:30hiredmanThe depstar alias is for the dependencies of depstar#2021-07-1202:31hiredmanAdding dependencies there doesn't make them available to your code#2021-07-1202:31hiredmanWhich is why adding them to your main deps key makes it work#2021-07-1202:32hiredmanNot sure why that is of course not ideal, if you depend on it, you need to specify it in your deps#2021-07-1202:47lgesslerthat's right--I think I have other problems I need to figure out, actually... I don't think that I depend on shadow-cljs or tools.namespace anywhere in my code. My fried brain thought I depstar needed me to specify them but that can't be right obviously, it'd be pulling in whatever it needs on its own#2021-07-1202:57lgesslertotal PEBKAC--was including my src/dev by mistake, which did use the dependencies i seemed to need to include 🙂 thanks @U0NCTKEV8#2021-07-1205:48seancorfieldFor future reference, there's a #depstar channel so we can keep #tools-deps for general t.d.a / deps.edn questions.#2021-07-1211:54delaguardoI have some problems with the pre-release version of clojure cli.
here is my ~/.clojure/deps.edn
❯ cat ~/.clojure/deps.edn
{:aliases
{:socket-repl {:exec-fn clojure.core.server/start-server
:exec-args {:name "server"
:port 5555
:accept clojure.core.server/repl
:server-daemon false}}}}
and when I try to start socket-repl in my project using clj -A:test -X:socket-repl :port 5557 it exits quietly with 0 status code#2021-07-1212:37borkdude@delaguardo Have you also tried -X:test:socket-repl ?#2021-07-1212:38delaguardoyes, same result (#2021-07-1212:43borkdudeperhaps it starts a server and then quits immediately?#2021-07-1212:45delaguardomost likely this is what is happening#2021-07-1212:45borkdudehmm, on my machine it works, so it does indeed seem to be happening with the newest one only?#2021-07-1213:02delaguardoyes, :version "1.10.3.905" only that version behaves like this#2021-07-1213:02delaguardobtw, I’m using mac os#2021-07-1213:12Alex Miller (Clojure team)seems like a bug#2021-07-1213:13Alex Miller (Clojure team)can you file on https://ask.clojure.org ?#2021-07-1213:14Alex Miller (Clojure team)I assume this is due to changes in -X exiting where we now (System/exit 0)#2021-07-1213:26delaguardosure, will do in few minutes#2021-07-1213:26Alex Miller (Clojure team)actually, I made an update for this for the next release so don't need to file#2021-07-1213:26delaguardook)#2021-07-1213:27Alex Miller (Clojure team)thx for the report#2021-07-1215:10borkdudeA thought: threading functions with -X on the command line works best when functions:
- Return the input map (let's call it context) with stuff added and don't return a completely new context
- Use namespaced keywords for options to avoid conflicts between function inputs and outputs.#2021-07-1215:11borkdudeConsidering this, I think you'd have to design most functions from scratch to work well with this pattern (i.e. you don't get a nice cmd line UX "for free" most of the time). But the namespaced keywords could make stuff boilerplatey as well.#2021-07-1215:28Alex Miller (Clojure team)yes, that is the tension#2021-07-1215:28Alex Miller (Clojure team)the key is that you very quickly are programming, so just put it in a program or use the repl#2021-07-1215:29Alex Miller (Clojure team)the chaining/threading is useful in a relatively narrow set of cases#2021-07-1215:58Joshua SuskaloIt could also be implemented fairly easily here with an extra alias
clj -X:thread :aliases "[:alias-1 :alias-2]" :arg-map "{:some-arg 1 :other-arg 2}"
You could even put this in your user configuration so that you don't need to re-implement this alias for every project you'd want to do it in.#2021-07-1215:59Joshua SuskaloOf course if you're using this as a part of the build process you probably would want to include it in every project.#2021-07-1216:01Joshua SuskaloThe only significant challenge here would be computing the correct set of dependencies based on what aliases are being loaded for this, which I imagine is part of why it doesn't seem like a great idea to add it to the Clojure CLI core.#2021-07-1216:01Joshua SuskaloJust write it as a program like Alex says.#2021-07-1221:09didibusIn the guide I see:
{:paths ["src"]
:aliases
{:build {:deps {io.github.clojure/tools.build {:tag "TAG" :sha "SHA"}}
:ns-default build}}}
Which can then be used with -T option. So any alias can also be a tool then, its just that the tool tool will not install things as an alias, but has its own separate global file to track what is installed?#2021-07-1221:14seancorfield-T:alias vs -Ttool-name#2021-07-1221:15seancorfieldThey are different. -T:alias is very similar to -X:alias except that it replaces the execution context implicitly (the docs explain this more accurately).#2021-07-1221:15seancorfield-Ttool-name is how you run a tool that has been -Ttools install'd and is not in your deps.edn file.#2021-07-1221:15DerekIs there a ~/.clojure/tools.edn file (or something analogous) that tracks installed tools? I enjoy being able to take my ~/.clojure/deps.edn file from machine to machine#2021-07-1221:15didibusHum, kind of confusing, so the colon makes a big difference. It seems that for -X and -A the colon was optional and with or without did the same thing#2021-07-1221:16seancorfieldWith :replace-paths and :replace-deps, we've been able to run things as "tools" for some time -- it's how clj-new and depstar have both expected to be used, for example.#2021-07-1221:16didibusI feel it be nice if alias just had precedence on installed tools, and you could use either or#2021-07-1221:17seancorfield@dpassen1 The tools are installed into ~/.clojure under a tools folder.#2021-07-1221:18borkdude> Is there a ~/.clojure/tools.edn file (or something analogous) that tracks installed tools? I enjoy being able to take my ~/.clojure/deps.edn file from machine to machine
This was also something I expressed when I first heard about the tools idea, but I think the answer to this is to just make an alias in your deps.edn as always#2021-07-1221:18seancorfield(! 1015)-> ls -l ~/.clojure/tools/
total 24
-rw-r--r-- 1 sean staff 184 Jul 11 09:51 depstar.edn
-rw-r--r-- 1 sean staff 184 Jul 9 16:55 new.edn
-rw-r--r-- 1 sean staff 126 Jul 9 11:09 tools.edn
#2021-07-1221:18didibusOn a team, ideally no one has to install tools, and just git clone should have it setup so all needed tools are already there, so it be nice if I could have the tools all setup as an alias and ran with -T#2021-07-1221:18seancorfield(I haven't tried it, but I think you can just copy that folder from machine to machine)#2021-07-1221:19borkdude@didibus just make an alias in your project's deps.edn as always?#2021-07-1221:20didibusSure, but I think I should call out then that the new tools feature is not useful to my team#2021-07-1221:20didibusAnd that as it is, it seems we'll just continue to use aliases#2021-07-1221:22didibusBut, I don't want to speak too soon, I've yet to play with it fully. But I was really expecting it to be more in-line with the declarative style of deps.edn to define apps#2021-07-1221:22seancorfieldI suspect it would be useful to individuals on your team who want to install and manage their own additional tooling 🙂#2021-07-1221:22didibusYa, for that it would, but they might also find it better to add an alias so they can sync their tools install between desktop and laptop, and whenever they need to change laptop or rebuild their desktop instance#2021-07-1221:23didibusI also personally would have liked a way to mess with tools at a project level#2021-07-1221:24seancorfieldI have ~/.clojure/ under version control and that's how I keep my laptop and desktop in sync.#2021-07-1221:24didibusThe pain-point with alias was having to find the correct definition for it. Tools solve that, but it be nice to have something similar for alias. Like if I define an exported alias in my deps.edn for my lib#2021-07-1221:25didibusThe same way you can define :tools/usage#2021-07-1221:26seancorfieldI just tested on my laptop: I git pull'd the updated ~/.clojure repo from GH and ran clojure -Ttools list and the same tools/versions are installed there as on my desktop (without actually needing to install anything).#2021-07-1221:27didibusOk, that's good, so it seems its just that tools don't pollute your deps.edn, but are also declarative.#2021-07-1221:27didibusNow I'm just concerned with reproducibility#2021-07-1221:27didibusBut I guess those tools I'll just use aliases for#2021-07-1221:29seancorfieldWell, the ~/.clojure/tools/<toolname>.edn file includes the git tag/sha so you can guarantee the version -- is that what you're concerned about?#2021-07-1221:29seancorfieldhttps://github.com/seancorfield/dot-clojure/blob/develop/tools/new.edn for example#2021-07-1221:30didibusI'm concerned with like, pulling an older commit of some project/application, and having even the tools be the exact same version, so I can get a complete repro of the setup#2021-07-1221:30seancorfieldOh, well, for project-specific tooling you should still use aliases in the project's deps.edn definitely.#2021-07-1221:31seancorfieldThat's why we vendor the whole Clojure CLI into our repo at work so we can reliably have the exact same version of the CLI on every machine.#2021-07-1221:31seancorfield<repo>/build/clojure/bin/clojure is what we use in all of our scripts.#2021-07-1221:33didibusYa, that's a reasonable practice. Though I wish there was first class support for this. Like I wish in my deps.edn I could declare a dependency on the Clojure CLI, tools.deps and tools.build versions to use.
I know it sounds like a weird bootsrap issue, but basically you use the globally installed clojure cli and tools.deps to pull down the defined version of clojure cli and tools.deps in the current project.#2021-07-1221:34didibusAnd same for all tools and aliases and everything#2021-07-1221:36borkdudethere is a JVM version of the clojure CLI#2021-07-1221:36borkdudedeps.clj, non-official, by yours truly#2021-07-1221:37didibusHaha, of course, why am I not surprised!#2021-07-1221:37didibusHum... but that might actually be useful. Maybe I could switch away from my custom launcher to using that.#2021-07-1221:38borkdudeclojure -Sdeps '{:deps {borkdude/deps.clj {:mvn/version "0.0.16"}}}' -M -m borkdude.deps -M -e '(+ 1 2 3)'
#2021-07-1221:39didibusThough... couldn't the normal CLI also be bundled as a git lib or even in a Maven repo? As far as I know you are not restricted to java in them#2021-07-1221:40didibusIt just needs to pull down the shell script#2021-07-1221:40borkdudeI guess it could, but running a shell script from a jar is a bit cumbersome#2021-07-1221:41borkdudehowever, gitlibs might be more useful there#2021-07-1221:41didibusTrue, but with git libs, that shouldn't be a problem since it pull down the folders and files#2021-07-1221:44borkdudePerhaps the new prep phase can install the script in a bin dir ;)#2021-07-1221:45seancorfieldYeah, that is definitely a new piece of OSS project's that we'll have to pay careful attention to -- since it could do "anything".#2021-07-1221:19didibusYa, to me it seems tools -install could have just added a tool alias in your ~/.clojure/deps.edn along with the new -T option#2021-07-1221:19didibusAnd that way there could have also have a project tool install option, to do the same in the project deps.edn#2021-07-1221:45didibusOk, so not trying to critique, I'm trying to wrap my head around on tools and all that's all. But, since they cannot include the project deps and path, it seems kind of limited what can actually be a tool no?
Like you can't make clj-kondo a tool, or nrepl a tool, etc.#2021-07-1221:45Alex Miller (Clojure team)you can do those things#2021-07-1221:46seancorfield@didibus How do you think depstar knows how to build JARs?#2021-07-1221:46Alex Miller (Clojure team)those are all tools that require some "basis" (execution context) to run in#2021-07-1221:46Alex Miller (Clojure team)functions are provided to compute a basis#2021-07-1221:47borkdudedo all tools have to depend on tools.build/tools.deps.alpha to be able to do that?#2021-07-1221:47Alex Miller (Clojure team)yes, unless you are picking up the basis from an .edn file or something#2021-07-1221:47Alex Miller (Clojure team)and that is something I'm thinking about#2021-07-1221:48borkdudeI guess one could combine a lib + tools.build to make an installable tool as a new project#2021-07-1221:49borkdudeso the lib doesn't have to depend on tools.*#2021-07-1221:49Alex Miller (Clojure team)what is the reason to not depend on tools.*?#2021-07-1221:50borkdudeless deps, more good. graalvm compatibility, etc.#2021-07-1221:50seancorfieldI would say "most people" don't care about that though... 🙂#2021-07-1221:50Alex Miller (Clojure team)less deps is theoretically better, but I'm not here an actual problem (yet) there#2021-07-1221:51Alex Miller (Clojure team)and if you're building a tool to run via the CLI, not sure why graal matters for that use case#2021-07-1221:51seancorfieldIt helped depstar tremendously, in terms of ease of adding features, once I stopped worrying about "no deps" and just embraced t.d.a 🙂#2021-07-1221:52didibusI don't know, seems it could cause incompatibilities both ways. Like if the tool depends on the installed tools.deps, if the installed one is too old or too new might break the tool.
On the other hand, if the installed one supports new features to define paths or deps the tool might not pick up on those if it uses an old tools.deps#2021-07-1221:52Alex Miller (Clojure team)an actual problem is a deps conflict between what a tool needs and what tools.* needs#2021-07-1221:52borkdudewell, e.g. clj-kondo can run both as a JVM tool and a graal-compiled tool. I'm extremely cautious with adding stuff to it which makes the binary bigger or causes incompatibilities for example#2021-07-1221:52Alex Miller (Clojure team)I have seen one or two cases related to maven libs there#2021-07-1221:53borkdudebut I would probably make a "wrapper" for the clojure CLI tool case if that were a problem#2021-07-1221:53seancorfield(and I did run into that -- as I reported yesterday(?) -- where tools installs things and assumes :git/sha will be recognized, but I installed a version of clj-new that depended on an older t.d.a and when it tried to resolve deps, the tool runner failed because there was no :sha in the new.edn, only :git/sha)#2021-07-1221:54seancorfieldI have since updated #2021-07-1221:55borkdudealso loading less code could make things faster. loading tools.build/deps.alpha when it's not necessary for all lib uses cases seems a bit wasteful#2021-07-1221:55didibusDo the tools deps merge with the project deps? Like could you override the tools deps?#2021-07-1221:56seancorfieldThat question doesn't even make sense to me... they are completely separate...#2021-07-1221:56seancorfieldTools are run in an isolated context, insofar as they ignore any project deps.edn file.#2021-07-1221:57didibusYa, maybe I just find that weird haha#2021-07-1221:57seancorfieldWhy? They're tools -- they're not part of your project.#2021-07-1221:57didibusBecause with aliases, you have full control, if the alias needs tools.deps.alpha, I can pin wtv version I want for the alias in my project#2021-07-1221:58didibusSo if I use a newer tools.deps.alpha with git/sha support, I can try using it with the alias as well, and chances are it works#2021-07-1221:59seancorfieldYou can still use whatever you want with aliases. That's completely separate from tools.#2021-07-1221:59didibusI guess conceptually, I find it weird that a tool is not somehow related to a project. Like clearly I could have one project that has a deps.edn from 3 years ago, and another one using a new deps.edn with all the new features. So maybe I need the version of the tool compatible with both depending on the project#2021-07-1222:00seancorfieldLike I said earlier: if you want a "tool" tied to a project, use an alias in the project's deps.edn file.#2021-07-1222:01seancorfieldYou can still run it "like a tool" with -T:the-alias.#2021-07-1222:01Alex Miller (Clojure team)tools are a way to give a user-specific name to a lib/coord#2021-07-1222:02Alex Miller (Clojure team)with support for install at the CLI (no file twiddling) and invocation by name#2021-07-1222:02didibusAh, ya so that would solve it no? So I could have installed depstar as an alias and used override-deps in the alias to depend on the newer tools.deps.alpha and ran it with -T:depstar. Assuming you had not updated depstar to the latest, this would have been one workaround correct?#2021-07-1222:04seancorfieldYes, rather than install depstar as a "tool" and using -Tdepstar.#2021-07-1222:07didibusOk, well at least there is that. So I guess one is convenient, the other solves edge cases. Which is good to have both#2021-07-1222:03seancorfieldI'm already loving that I have -Tnew app and -Tnew lib and -Tdepstar jar and -Tdepstar uberjar available for any project now, without needing aliases in my ~/.clojure/deps.edn file 🙂#2021-07-1222:04didibusOk, but what are you loving about it over aliases? Just that you don't need to manually edit your ~/.clojure/deps.edn and find the correct alias from some readme file? Or is it that you like to keep the deps.edn uncluttered?#2021-07-1222:05seancorfieldIt will allow me, over time, to remove a lot of stuff from my deps.edn file -- and the tools install stuff makes it easier to find the right versions of tools and upgrade/downgrade them: I can do it from the CLI instead of editing my deps.edn file.#2021-07-1222:05didibusOk, I guess I just don't see how that could not all have been done for aliases as well, just providing a cli for it#2021-07-1222:06didibusWhich would have naturally extended to project specific uses as well#2021-07-1222:07seancorfieldA lot of people don't want a CLI tool editing their files.#2021-07-1222:07borkdudeI guess it comes down to the "brew install" vs "nixos" way of doing things. Personally I'm leaning towards the deps.edn approach to keep things reproducible but it's nice to have options, I guess.#2021-07-1222:08seancorfieldThe nice thing is that we have both approaches available now: locally-installed tools (with a manifest that can be sync'd across machines) and alias-bound tools, either in the project deps or your own personal deps.#2021-07-1222:09didibusYa, I prefer the deps.edn approach as well "nixos" style. But I guess I'll deal with it 😛
I think it would have been fine to have like a convention of a aliases folder (or even the tools folder), but where you can just have more files that define more aliases.#2021-07-1222:09didibusYa, but we don't have the thing I wanted the most, to have a way to install aliases from the CLI 😞#2021-07-1222:10seancorfieldAs I just said "A lot of people don't want a CLI tool editing their files." 🙂#2021-07-1222:10borkdude@didibus That's just a script away#2021-07-1222:10didibusNo its not, it requires full buy-in from the library authors#2021-07-1222:10borkduderoll your own#2021-07-1222:10didibusYou need to declare your alias in your lib in a way a cli could figure out what to install#2021-07-1222:11seancorfieldThat's why the tools approach pays attention to :tools/usage in your library's deps.edn file 🙂#2021-07-1222:12borkdudeI don't have a lot of churn in my deps.edn anyway, I edit it probably maybe once a month or less#2021-07-1222:12didibusRight, but so that's for tools, would have been nice for aliases as well#2021-07-1222:13borkdudeI don't see why I would want to install depstar as a system-wide tool. I want a reproducible version of depstar in my project in CI.#2021-07-1222:13borkdudeFor any tool really#2021-07-1222:14seancorfieldWe've just cut our project aliases way, way down (now that we have a build.clj script instead of trying to use bash and the CLI and aliases instead of actual functions).#2021-07-1222:15didibusThat's unrelated no? Its just all the functionality of these disparate aliases are all merged into tools.build#2021-07-1222:15seancorfield@U04V15CAJ With depstar you may be right. clj-new makes more sense as a global tool.#2021-07-1222:16seancorfield@didibus No, that change was unrelated to tools.build.#2021-07-1222:16borkdudeI usually write projects with my own bare hands without templates, but yes, that's a good example :)#2021-07-1222:16seancorfieldOur build.clj existed before tools.build was released 🙂#2021-07-1222:17didibusOk, but still confused, what does that relate to tools or aliases?#2021-07-1222:19didibusYour build.clj is a new tool?#2021-07-1222:19seancorfieldSigh. I'm commenting on how few aliases we have in our project deps.edn and that they don't change very much -- unless we do wholesale project-level changes. And we made some changes earlier this year (which I blogged about) specifically to avoid having developer tooling in the project, i.e., so developers could use any tooling they wanted.#2021-07-1222:20seancorfieldNo, build.clj is "just" a Clojure script. It replaced our build bash script (and a few auxiliary bash scripts).#2021-07-1222:21seancorfieldI guess you've missed folks here talking about switching over to using Clojure programs to do "build stuff" with instead of various non-Clojure scripts...#2021-07-1222:22didibusOh I see, all my Clojure build stuff are already Clojure programs 😛#2021-07-1222:55Jason@seancorfield when can we expect the "monorepo part v" post? 🙂#2021-07-1223:14seancorfieldSoon @jasonhlogic soon! We're working on finalizing our build.clj stuff -- now that we can align it with tools.build now that's been released! -- and then I can write the next article, about how we switched to using Clojure for our test/build pipeline and how we adopted tools.build 🙂#2021-07-1223:16seancorfield(I'll also have more to say about Polylith, since we're now up to 16 projects, 3 bases, and 23 components!)#2021-07-1223:32Jasonshould shadow-cljs be a tool under the new regime?#2021-07-1223:59seancorfieldI thought Shadow-cljs was primarily installed via npm or something?#2021-07-1300:02Jasonit has a clojure part too. See:
https://shadow-cljs.github.io/docs/UsersGuide.html#_high_level_overview#2021-07-1300:03seancorfieldRight, but hardly anyone uses that directly I thought?#2021-07-1300:04Jasonmy deps.edn has
:main-opts ["-m" "shadow.cljs.devtools.cli" "watch" "ui"]
where ui is a subproject in the monorepo#2021-07-1300:04seancorfieldOK, so it's a -m invocation if you do that -- which is not how -X/`-T` work.#2021-07-1300:05seancorfieldTools invocation assumes the "exec fn" model, where you invoke a specific function, passing in a hash map.#2021-07-1300:06Jasoncould that not be a wrapper around shadow-cljs?#2021-07-1300:07seancorfieldSomeone -- theller or someone else -- could write a -X-compatible entrypoint/wrapper for it I guess.#2021-07-1300:17Jasonprior to tools.build, i was trying to do too much in deps.edn. task-like aliases, logical deps sets, version defaulting for subprojects. a lot of aliases use shadow-cljs but only as an entrypoint to start a process. that feels more like a tool to me?#2021-07-1300:25seancorfieldI don't do any cljs so I'm not familiar with it (I dabbled with figwheel-main a bit last year).#2021-07-1300:26seancorfieldWhen I saw that shadow-cljs was so node-centric, I went looking elsewhere. I don't do JS/node and don't have any of that tooling installed (and don't want to).#2021-07-1300:26seancorfieldBut you would need a -X-compatible entry point in a GitHub project to use it as a tool (or in a Clojars artifact).#2021-07-1300:30Jasoni understand your npm concerns but it allows js lib access just as maven coordinates allows java lib access. a useful hack.#2021-07-1300:31Jasoni'll see if theller is interested#2021-07-1303:08didibusWould be cool to make shadow-cljs a tool I think. But since it requires npm anyways, and npm already supports local installs, it might still be best to keep it using npm#2021-07-1305:29thhellershadow-cljs does not make sense as a tool. it is not standalone and needs to be part of the project classpath#2021-07-1305:31thhellershadow-cljs is two parts right now, unfortunately I named them the same name. the shadow-cljs CLI tools published as an npm package, which handles only the CLI. and then the real thing shadow-cljs the CLJ library, where all compilation and stuff happens#2021-07-1305:31thhellerthe npm package has always been optional, I wrote about it here https://code.thheller.com/blog/shadow-cljs/2017/11/18/the-many-ways-to-use-shadow-cljs.html#2021-07-1305:33thhellerso a "tool" could technically replace the npm part but that doesn't make too much sense IMHO. maybe there could be a namespace for an :exec-fn-like API though#2021-07-1305:36thhellerwell maybe it could be a tool. by the nature of the CLJS compiler it needs the project classpath, which I guess it could construct via deps.edn itself and manage its own secondary JVM much like the npm shadow-cljs tool does now#2021-07-1305:36didibusWhat I gathered is that tools are expected to depend on tools.deps.alpha and use its api to build the classpath for the folder it is run in#2021-07-1305:37didibusSo a tool which needs a project classpath would still get one by using tools.deps.alpha as a library for getting it from the deps.edn file#2021-07-1305:40thhellerwell or it could just run a secondary clojure -M process similar to what the current shadow-cljs CLI npm thing does#2021-07-1305:43thhellerbut since you'll still need npm anyways for actual JS packages I'll probably keep the CLI package too. Starts fast and does everything you'd want. For clj purists a :exec-fn namespace seems useful, more so than a tool#2021-07-1305:43didibusYa I guess it could do that too.#2021-07-1305:45thhellertools.build like stuff is already fully supported and has been from the beginning so nothing to do there. meaning everything you can run from the command line you can also run from the CLJ REPL as regular functions#2021-07-1305:45didibusYa, it does seem to possibly fit better as an alias with -X#2021-07-1305:46didibusNot sure that's even really useful, but if people are currently invoking it with clojure cli using -m as an alias, I guess it would probably be slightly better for them#2021-07-1305:49thhellermost people using it directly today (without the npm CLI) use it in embedded mode though. so starting it as part of their user/start or so, together with their CLJ server#2021-07-1305:50thhellersome have integrated it with libs like integrant, component or mount#2021-07-1305:51didibusWhat you said about tools.build I hadn't thought, ya I guess someone can just create themselves build tasks in build.clj that just call into shadow-cljs clojure API and then they can run shadow commands using the clojure cli with -T:build alongside their other commands they'd want#2021-07-1305:52thhelleryeah many people do this today via either clj or https://shadow-cljs.github.io/docs/UsersGuide.html#clj-run#2021-07-1305:52thhelleror lein of course. tools.build users will likely use the direct CLJ api a lot more than the CLI#2021-07-1305:53didibusYa most likely#2021-07-1305:55didibusHum, actually it seems now that I think of it, that all "build tool" or "build" related tooling will be better offered as a clojure lib to be used within tools.build. While pure "tools" would be the kind that don't care about the project you are in and are more like generic Clojure tools you'd want installed globally, like if I wrote a grep tool in Clojure and you want to "install" it so you can use it.#2021-07-1305:57thhelleryeah, thats actually how shadow-build (predecessor to shadow-cljs) started. only functions, no command-line API. just got too tedious to create the same few functions over and over again in each project, so a proper CLI and thus shadow-cljs was born 😉#2021-07-1306:00didibusInteresting, I wonder if a similar thing will happen with tools.build, like if people will feel copy/pasting all the basic tasks will start to be tedious.#2021-07-1306:01didibusThough it has the advantage that each function becomes a "command" on the cli, even though a bit tedious: clj -T:build command#2021-07-1306:08thhelleryeah, that is my concern with tools.build. just the same functions in every project, likely generated by some template. we will see I guess. doesn't have to be a bad thing, makes things easiert to customize (if you know what you are doing)#2021-07-1317:17Jason@U05224H0W just for clarity, would you be against adding the introspection keys to shadow-cljs's deps.edn? https://clojure.org/reference/deps_and_cli#_best_practices_for_tool_authors#2021-07-1317:22thhellernot at all but as I described you really cannot run it as a tool currently. shadow-cljs needs full access to the project classpath which tools don't get (AFAIK)#2021-07-1317:22thhellerso adding those things really doesn't do anything#2021-07-1317:24thhelleralso shadow-cljs uses lein and the deps.edn is not really up to date currently#2021-07-1317:24seancorfieldTools can use tools.deps.alpha or, now, tools.build to calculate the project basis and that includes a project classpath, which you can then load dynamically, if you need to. But I agree that if you have a "tool" that really needs to execute in the context of a project's classpath -- and test runners are a good example -- then making them a "tool" makes little sense.#2021-07-1317:25Jasoni concede the point 🙂#2021-07-1317:28Jasonbut i do have a few apps which use the documented shadow-cljs deps setup, so it would be nice if that was up to date (which is a different discussion)#2021-07-1317:29seancorfield(and probably belongs in #shadow-cljs at this point 🙂 )#2021-07-1317:29thhellerwhats wrong with :main-opts? that doesn't go away 😉#2021-07-1317:31seancorfield:main-opts is great for some types of invocation (where argument processing needs to be *nix-compliant in some way or arguments can be kind of ad hoc -- and all derived from strings) but exec fn/args is nicer from a programmatic, composability p.o.v. and all exec fns take standardized arg formats (unrolled EDN hash maps, basically).#2021-07-1317:33seancorfieldFrom a tool author's p.o.v., exec fn/args means that you don't have to write any argument parsing, and your entry points can (now) all have their help/documentation rendered from the command-line (via the new -X:deps help/doc stuff -- although I haven't quite gotten my head around how exactly it works).#2021-07-1317:36thhellershadow-cljs watch app is probably the most common command. kinda gets annoying with -X IMHO. so :main-opts looks better#2021-07-1317:37thhellerbut I'm open to adding stuff for CLJ purists so whatever makes sense I'll add#2021-07-1317:39seancorfieldAlso, from a user's p.o.v., exec args are great because they combine across multiple aliases -- :main-opts is always "last one wins" and does not combine across aliases.#2021-07-1317:39seancorfieldSo they both have their useful spheres.#2021-07-1317:40seancorfield(and, yes, simple commands like watch app do become more verbose with -X which is why some commands are simply better off with -M / -m)#2021-07-1321:40didibusI feel the word "tool" will confuse countless people going forward, that a "build tool" is not a Clojure "tool", but just an "alias".#2021-07-1223:42Jasoni've never been happy with how it fit into the semantics of my deps.edn aliases#2021-07-1314:13dharriganI'm curious, when executing a function using the -X switch, the JVM never exits, I have to put an explicit (System/exit 0) or similar. That's a bit surprising. I would have thought that as soon as the function has exited, the JVM would shut down.#2021-07-1314:14dpsuttondoes it stop after about 30 or 60 seconds?#2021-07-1314:14dharriganYes#2021-07-1314:15dharrigandaemon?#2021-07-1314:15dpsuttonyeah its the agent threadpool i believe. you can call the really cool-named function (shutdown-agents).#2021-07-1314:15dharrigantrying#2021-07-1314:16borkdudeyeah, the thing is that when you thread multiple functions and the non-last function calls that, it might screw things up :)#2021-07-1314:17dharriganyup#2021-07-1314:17borkdudebut perhaps the "wrapper" of this thread could call it as the last op#2021-07-1314:17dharriganthat works#2021-07-1314:18dharriganI wonder if that could be added to the docs, i.e., -X:deps help/doc thing, and I can get it to give me help for a whole :ns but when I try to get help for a single function, I just get
(! 1036)-> clojure -X:deps:uberjar help/doc :fn hf.depstar/pom
-------------------------
(! 1037)-> clojure -X:deps:uberjar help/doc :fn pom :ns hf.depstar
-------------------------
There are no examples in the docs so I'm not sure what is supposed to work...#2021-07-1317:44seancorfieldThe following works:
(! 1038)-> clojure -X:deps:uberjar help/doc :ns hf.depstar
Entry point for clojure -X options.
-------------------------
hf.depstar/aot
([options])
-X entry point for AOT compilation.
... lots of help for every function ...
#2021-07-1317:46dharrigan@seancorfield only fixed on a pre-release#2021-07-1317:56Alex Miller (Clojure team)Either qualified :fn or :ns is supposed to work#2021-07-1317:57Alex Miller (Clojure team)Please file a jira if you have a repro that doesn’t work#2021-07-1318:00seancorfieldI'll stick it up on ask for others to vote for it, and you can create a Jira in whichever is the appropriate project...#2021-07-1318:06seancorfieldhttps://ask.clojure.org/index.php/10778/x-deps-help-doc-does-not-work-for-a-function#2021-07-1318:06seancorfield(I initially thought it might just be a problem with stuff brought in via an alias but apparently not)#2021-07-1405:53seancorfieldclojure -Ttools install com.github.seancorfield/depstar '{:git/tag "v2.1.267"}' :as depstar
Major documentation overhaul, aligns with tools.build.#2021-07-1410:42dominicmFor -X is there yet a strategy for substitution (i.e. where I can't manually do edn escaping) so that I can do something like clj -X:foo :bar-key "$(wl-paste)" ? I'm only ever expecting strings when doing this.#2021-07-1411:54borkdude@dominicm
$ clj -X clojure.core/prn :foo \"$(echo "dude")\"
{:foo "dude"}#2021-07-1411:55dominicm@borkdude that isn't general purpose though, what if my command outputs a "? e.g. clj -X clojure.core/prn :foo \"$(echo '"dude')\"#2021-07-1411:56borkdudeyou want an escaped Java string?#2021-07-1412:00borkdude@dominicm
$ escaped=$(echo '"dude' | bb -e '(prn (slurp *in*))')
$ clj -X clojure.core/prn :foo $escaped
{:foo "\"dude\n"}
#2021-07-1418:39pbrownI came across a new warning after updating TDA recently:
WARNING: Use of :paths external to the project has been deprecated, please remove: ...
What's the rationale behind disallowing such paths? Some of my builds include paths that symlink into /tmp, which look 'internal' e.g. :extra-paths ["target/cljs-resources"] but become 'external' when canonicalized in clojure.tools.deps.alpha/validate-paths.
(I use /tmp to include build inputs from previous steps, and because I can mount it as tmpfs for speed and to avoid wearing out my SSD.)#2021-07-1418:44seancorfield@alexmiller ☝️:skin-tone-2:#2021-07-1419:24Alex Miller (Clojure team)> What's the rationale behind disallowing such paths?
Projects that reach outside their project do not play well when moved to other people's machines, particularly as gitlibs#2021-07-1419:25Alex Miller (Clojure team)I don't understand "to avoid wearing out my SSD" but I do understand "for speed" :)#2021-07-1419:27Alex Miller (Clojure team)this may be a place where some careful path handling could keep the path to the in-project link. if you like, file a question on https://ask.clojure.org and I can add it to the queue of things to look at#2021-07-1419:38pbrownAh makes sense, thanks! Sounds good, I'll file it there.#2021-07-1419:53dominicm@borkdude right, but ideally without needing an extra tool in there. Not everyone has bb installed (yet)#2021-07-1419:55hansbugge$ echo -n '"bla' | clj -M -e '(prn (slurp *in*))'
"\"bla"
🙂#2021-07-1419:55borkdudebb runs normal Clojure. You can use normal Clojure is bb is not there, so there's always an alternative#2021-07-1419:57dominicmYeah, true. Not particularly fast though. 2.46s overhead cost for it. And not great to type at the terminal either.#2021-07-1419:57Alex Miller (Clojure team)can you use something like this?
clj -X:foo :bar-key '"'$(wl-paste)'"'
#2021-07-1419:58dominicm@alexmiller that's going to fail if there's a space in the paste 🙂#2021-07-1419:58Alex Miller (Clojure team)I don't think it will#2021-07-1419:59Alex Miller (Clojure team)% echo $FOO
a b c
% clj -X clojure.core/prn :a '"'$FOO'"'
{:a "a b c"}#2021-07-1420:00dominicmI get a different result:
$ VAR='foo bar baz'
$ clj -X clojure.core/prn :bar-key '"'$VAR'"'
Unreadable arg: "\"foo"
#2021-07-1420:00dominicmIs this perhaps a bashism?#2021-07-1420:01Alex Miller (Clojure team)you could surround $VAR with "" there again if you need to, not sure why mine is different#2021-07-1420:01Alex Miller (Clojure team)maybe bash version - Mac uses old bash#2021-07-1420:01dominicmHmm, no. Shellcheck says that's still not right, even with bash.#2021-07-1420:02dominicmFor me, it works on zsh but not with bash or dash.#2021-07-1420:02borkdudeMaybe just don't bother with -X and use -M in this case?#2021-07-1420:05dominicmThe world seems to be moving to -X though.#2021-07-1420:06Alex Miller (Clojure team)clj -X clojure.core/prn :bar-key '"'"$VAR"'"' seems like it works on both zsh and bash#2021-07-1420:08dominicm@alexmiller That doesn't work for anything containing a quote.
❯ VAR='b"'
❯ clj -X clojure.core/prn :bar-key '"'"$VAR"'"'
{:bar-key "b"}
#2021-07-1420:08dominicmShould be {:bar-key "b\""}#2021-07-1420:11Alex Miller (Clojure team)have you heard the good word about putting your data in files? :)#2021-07-1420:11borkdudeMoving to -X causes shell-related pains like this and the solution is different for each shell/OS, which is a pity for a platform like the JVM which is supposed to be write-once, run anywhere, but yes, it seems the world is moving towards that.#2021-07-1420:12dominicmI'm thinking about things like -X some.fn/bar '[:db :password]' "$(pass show -p password-for-db)" or other things where users want to use the clojure stuff as a swiss army knife of dynamic parts.#2021-07-1420:13Alex Miller (Clojure team)env vars is probably a better choice for that particular case (where you don't want your passwords in your command lines, visible in process commands)#2021-07-1420:16dominicmWell there're other things too, like script clj-new (which I've done for testing my templates before). You get into the game of escaping your temporary names. Rewriting clj-new to take env vars doesn't make sense.
I'm looking for parameterization a la carte, rather than hard-coded into functions.#2021-07-1420:17Alex Miller (Clojure team)well you're making these calls from some shell, so there's no way to avoid the eccentricities of that API#2021-07-1420:18Alex Miller (Clojure team)if you want more control, write a clojure program into a file and run it#2021-07-1420:18Alex Miller (Clojure team)or pipe it over stdin with clojure.main :)#2021-07-1420:25dominicmClojure is worse than a shell here. Process substitution is much harder than $() for example. Shells are optimized for combining things in ways like this on the fly.#2021-07-1420:26dominicmMaybe we need a type system. clj -X clojure.core/prn ^String :foo-bar "$(echo 'b"zt')" and then -X can just treat it as a literal string. Of course, that breaks if someone accidentally echoes something like ^String as their key. So we'll need an escaping system, just in case.#2021-07-1420:32dominicmIn case not obvious, I'm trolling.#2021-07-1420:26Joshua SuskaloSounds like the job of a tiny library for use with #babashka#2021-07-1420:29borkdudeI think most of the pain with shells + -X comes from having to quote strings. If that was not needed, it would probably a bit smoother.#2021-07-1420:39Alex Miller (Clojure team)Agreed, still an open topic #2021-07-1420:30dominicmSomething that would work without sub-shells or crazy escaping, although is eh compared to normal shell usage, would be FOO="$(echo 'b"zt')" clojure -X clojure.core/prn :foo-bar '#env FOO' where env is doing (System/getenv).#2021-07-1620:55dominicmhttps://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core_print.clj#L200-L221
This looks reasonably easy to port to a C/zig program that would be portable & tiny enough to include in a clojure distribution for making this escaping reasonably easy.
It's like how xargs ports stdin to args, this ports shell args to clojure args so cargs? 😄#2021-07-1708:10dominicm#!/bin/sh
printf '"'
sed 's/\\/\\\\/g' |
sed "s/\r/\\\\r/g" |
sed 's/$/\\n/g' | tr -d '\n' |
sed "s/\t/\\\\t/g" |
sed 's/"/\\"/g' |
sed "s/\f/\\\\f/g" |
# \b, but sed uses \b for word boundary
sed 's/\x08/\\b/g'
printf '"'
That seems to work as an edn escaper. I could see this going into the clj tool so you can do printf '"' | clj arg#2021-07-1708:14dominicmCompared with the native zig version it's much slower, but also portable so probably worth it 🙂
Overall, both are fast.#2021-07-1904:22didibusOr a GraalVM version like https://github.com/borkdude/deps.clj which can properly work on Windows as well (which I wish the official cli followed this model instead of using bash)#2021-07-1420:44souenzzo@dominicm why not -X app.start-script/prod where (defn prod [] (sh/sh "pass" "show" "-p" ....)) ?#2021-07-1509:15dominicmThe use of pass is local to myself, not to the whole team.#2021-07-1509:15dominicmWhen I'm working from the shell I'll often use parameter subsitution in various ways, change it, etc.#2021-07-1420:47seancorfieldThat would have to be (defn prod [_] ...) to be invoked with -X but at this point I agree with Alex: write a (Clojure) script for something that is too complex to do easily from the command-line.#2021-07-1423:24Cam Saulsorry if this is a dumb question, but how do I clojure -X:deps prep a library that's only included under :extra-deps for a specific alias? At first I assumed either clojure -X:my-alias:deps prep or clojure -X:deps:my-alias prep would work, but they both fail with the libs must be prepared before use message. I tried clojure -X:deps prep :basis '{:aliases #{:my-alias}}' after reading the https://github.com/clojure/tools.deps.alpha/blob/655ea0a037f041e365cc56db0dc8181268c29998/src/main/clojure/clojure/tools/cli/api.clj#L68 but that doesn't work either, since {:aliases #{:my-alias}} isn't actually a basis (although it could be used to create one, e.g. with tools.build create-basis) -- I'm not sure how to get an actual basis from the CLI. I tried clojure -X:deps basis '{:alias #{:my-alias}}' thinking maybe that would return a basis that I could then pass to -X:deps prep (e.g. something like clojure -X:deps prep :basis "$(clojure -X:deps basis '{:alias #{:my-alias}}')") but basis doesn't write any results to the CLI.
How do I prep libs in an alias?#2021-07-1423:52Alex Miller (Clojure team)At this moment, you can't but come back next week :)#2021-07-1620:55dominicmhttps://github.com/clojure/clojure/blob/b1b88dd25373a86e41310a525a21b497799dbbf2/src/clj/clojure/core_print.clj#L200-L221
This looks reasonably easy to port to a C/zig program that would be portable & tiny enough to include in a clojure distribution for making this escaping reasonably easy.
It's like how xargs ports stdin to args, this ports shell args to clojure args so cargs? 😄#2021-07-1500:11Cam Saulone more (probably dumb) question: is there a way to deps prep a top-level deps.edn project? My main use case in compiling a Java file in a project similar to what Leiningen :java-source-paths does -- my workaround was to add a separate java/deps.edn (with the :deps/prep-lib key) and then have the top-level deps.edn include ./java as a :local/root dependency. Not such a huge deal, but since compiling the Java source file requires some dependencies already specified in the top-level deps.edn I ended up having to have a few duplicate deps between deps.edn and java/deps.edn#2021-07-1500:14seancorfield@camsaul Sounds like you should be using tools.build at this point? It has a javac task.#2021-07-1500:19Cam SaulI'm using tools.build javac as part of the :deps/prep-lib step. The problem is I want that prep step to run (or at least fail with the error saying it needs to be ran) before doing anything -- running tests, starting a REPL, etc.
The prep step works fine if I create an extra deps.edn file in my Java source dir and refer to it as a :local/root dep, I'm mainly asking if there's a way to do it without creating the extra deps file to avoid having to specify duplicate deps in both the top-level deps.edn and the extra one I created in my ./java source dir#2021-07-1500:22seancorfieldAh, gotcha.#2021-07-1500:23seancorfieldWait, the prep lib stuff is only going to run once -- and then the dependency is "prepped" -- so it won't run when you change the Java code or anything like that.#2021-07-1500:33Cam Saulyup, but that's fine with me for the time being. At Metabase we have exactly one Java source file (we use Liquibase for app DB migrations and the file is a custom class subclassing a generic class that forces it use the utf8mb4 collation when creating new tables MySQL/MariaDB, so it basically has to be there and be compiled before doing anything else). I wrote that file maybe 4 years ago and haven't touched it since.
My main motivation is so that anyone who clones the git repo and tries to run it locally doesn't need to jump thru too many hoops to do it. clojure -X:deps prep && clojure -X:run (or whatever the new multiple-function equivalent of that is) is fine for now -- and I have that more or less working -- but having to specify the Liquibase dependency twice in two separate deps.edn files makes me a little itchy#2021-07-1513:17Joshua SuskaloYour prep-lib step is just running another alias, so to prep your root lib you just execute that alias from the command line#2021-07-1513:17Joshua Suskalo@camsaul#2021-07-1513:18Joshua SuskaloAnd if you want to ensure that alias is run only once, then make it take a :force key you can provide from the command line, and if it isn't provided, then it'll create a lockfile and ensure it won't run again.#2021-07-1514:55Alex Miller (Clojure team)New prerelease of the Clojure CLI is now available: 1.10.3.912
• Fix help/doc and help/dir to better load namespaces
• TDEPS-187 - Make help/doc and help/dir work with tools via something like:
◦ clj -A:deps -Ttool help/dir
• Fix -X/-T to use (agent-shutdown) instead of (System/exit 0) on success to avoid forced exit for server processes#2021-07-1515:50seancorfield@alexmiller Should I expect this to give help on depstar? clojure -A:deps -Tdepstar help/doc -- it doesn't, it gives help on clojure.tools.cli.api instead.#2021-07-1515:52seancorfieldThese work
505 clojure -A:deps -Tdepstar help/doc :ns hf.depstar
506 clojure -A:deps -Tdepstar help/doc :fn hf.depstar/pom
but given that depstar has :ns-default I would have expected the above to work.#2021-07-1515:52seancorfieldAlso, :deps/`-T` is not very friendly with tools that do their own logging due to the slf4j no-op lib:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/sean/.m2/repository/org/slf4j/slf4j-nop/1.7.25/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/sean/.m2/repository/org/slf4j/slf4j-simple/1.7.30/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
#2021-07-1516:00Alex Miller (Clojure team)Can you log these somewhere?#2021-07-1516:02Alex Miller (Clojure team)Just in general, if you have a bug please report it in something I can track rather than here. I am just beyond my ability to track all the things#2021-07-1516:16seancorfieldI wanted to check whether you expected it to work, i.e., that it is a bug, first... I'll put it on ask now.#2021-07-1516:23seancorfieldI added a comment to https://ask.clojure.org/index.php/10782/help-doc-would-be-useful-for-ttools since I don't know whether you'll want to reopen the ticket or create a new one.#2021-07-1515:11Joshua SuskaloWould this cause issues with server processes which use agents?#2021-07-1515:19Alex Miller (Clojure team)that's not common (and they would already be in trouble with the prior impl which hard exited). they would need to block the main thread from exiting.#2021-07-1515:19Joshua SuskaloFair#2021-07-1515:19Alex Miller (Clojure team)the thing we're trying to avoid above is an early exit when a server has non-daemon threads still active (socket repl server being the obvious case)#2021-07-1515:21Joshua SuskaloAaah, makes sense.#2021-07-1515:57seancorfield@alexmiller While I remember: I have a use case for wanting just the various deps.edn read and merged -- reading :alias data, without actually resolving deps, so t/find-edn-maps and then t/merge-edns without any additional steps. This cropped up with depstar which I had switched to t/create-basis for that and it caused problems for someone in CI who was passing a custom basis into the tool.#2021-07-1515:58Alex Miller (Clojure team)I’m not understanding what you’re asking for#2021-07-1516:16seancorfieldt/create-basis causes deps to be downloaded, yes?#2021-07-1516:18seancorfieldt/find-edn-maps & t/merge-edns don't. And that's important for the user that reported the bug. This is in the context of you saying t.d.a's create-basis might go away in favor of tools.build's version, and I just wanted to point out that EDN manipulation is a likely use case for tooling, without full calculation of a basis (and downloading stuff).#2021-07-1516:40Alex Miller (Clojure team)t.d.a's create-basis is not going away#2021-07-1516:40Alex Miller (Clojure team)if you want to use find-edn-maps and merge-edns , then use them, they are part of the api#2021-07-1516:41Alex Miller (Clojure team)the only thing I said might go away is -Spom / -X:deps mvn-pom (I think)#2021-07-1517:17seancorfieldPerfect! Thank you! Just wanted to make sure.#2021-07-1616:12seancorfieldWe are officially using tools.build at work for part of our build.clj script (although not for anything very substantial: just process, instead of some ProcessBuilder/interop stuff, and some file copying/deletion).#2021-07-1616:14seancorfieldPart of me really wants to have build tasks that are like "run the tests and if successful build the JAR" but you can't "run the tests" if your build script is invoked via :replace-deps (or just :deps). Am I barking up the wrong tree with that or are folks resigned to having "run the tests" always be a separate clojure invocation to "build the JAR"?#2021-07-1616:18Alex Miller (Clojure team)why can't you run the tests in your build script? something to do with bring in the test runner dep?#2021-07-1616:21seancorfieldIf your build script is invoked with just "." for paths and just c.t.b for :deps, how do you get the test runner on the classpath to invoke it and how does the test runner find all your source/test code to actually run it?#2021-07-1616:22seancorfield(currently, we're invoking the build script with a bunch of extra paths/deps and just running all the tests with c.t.b and t.d.a on the classpath as well which is what caused the S3 code loading race conditions that @hiredman was complaining about the other week)#2021-07-1616:23seancorfieldunless you use c.t.b process to shell out and invoke clojure -X:test... I guess which is... ugh!#2021-07-1616:47Alex Miller (Clojure team)that's one option (personally I prefer to have it as a separate thing)#2021-07-1616:47Alex Miller (Clojure team)if you have it as a separate thing, you can use a combo of create-basis, java-command, and process to execute it#2021-07-1623:10seancorfieldFWIW, I just started to look into this and whilst it might work fine for a -main with regular string command-line arguments, it doesn't look so easy to replicate a -X style invocation, which is what we're using. We have a custom wrapper for test-runner's API that does some per-run setup and then invokes the runner, both of which happen via the -X entry points right now.#2021-07-1623:14seancorfieldIf I'm following the CLI logic correctly, it looks like that routes through clojure.run.exec which I believe is only in the exec.jar and can't be added to a basis easily?#2021-07-1623:24seancorfieldI thought maybe I could depend on brew-install as a git dep but the deps.edn there isn't a "real" one (`:paths ["x"]` etc) so that's not possible.#2021-07-1700:16seancorfieldOK, I've worked around that by temporarily copying clojure.run.exec into our repo so it can be on the classpath and I have a working version of running tests in a subprocess this way. I noticed that clojure.run.exec assumes the clojure.basis system property exists, so I had to propagate that into :java-opts so it could read the basis...#2021-07-1700:24seancorfieldFYI
(let [test-dirs (map #(pr-str (str % "/test")) projects)
aliases [:everything :dev :test :runner]
basis (b/create-basis {:aliases aliases})
cmd-args
(b/java-command {:basis basis :main 'clojure.main
:java-opts
;; propagate the basis from the CLI:
[(str "-Dclojure.basis=" (System/getProperty "clojure.basis"))
;; ideally we should pull all the :jvm-opts from the :runner
;; alias into this setup...
"-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
"-Dlog4j2.configurationFile=log4j2-silent.properties"]
:main-args
["-m" "clojure.run.exec"
;; need :exec-args from this alias:
"--aliases" ":runner"
"ws.build.runner/test"
":dirs" (str "[" (str/join " " test-dirs) "]")]})
{:keys [exit]}
(b/process cmd-args)]
(when-not (zero? exit)
(System/exit 1)))#2021-07-1701:46Alex Miller (Clojure team)Yeah, you should rely on nothing about clojure.exec.run’s api - it changes frequently in breaking ways#2021-07-1701:47Alex Miller (Clojure team)But I think having support to invoke a -X entry point as part of tools.build is really interesting#2021-07-1705:16seancorfieldSome sort of tool.build.api/exec perhaps? I'll try converting a few more of our build tasks over and see what abstraction I can pull out of it...#2021-07-1705:17seancorfieldI think with a decent wrapper, this can be quite clean -- but it needs some sort of standard -X style adapter for java-command?#2021-07-1915:23Alex Miller (Clojure team)added https://clojure.atlassian.net/browse/TBUILD-6 to track this#2021-07-1917:32seancorfieldThanks for that @alexmiller -- do you want me to share with you the code I created for that?#2021-07-1917:43Alex Miller (Clojure team)if you like#2021-07-1616:48Joshua SuskaloThis is the sort of thing that I feel like just, the task runner, is good for.#2021-07-1616:52seancorfield@alexmiller That sounds very low-level and a lot of work when the Cognitect test-runner has a nice -X function so it could be run in-process, if only there was a nice way to have basis-isolated contexts in c.t.b 🙂#2021-07-1617:12Alex Miller (Clojure team)well, not planning to make that#2021-07-1617:12Alex Miller (Clojure team)or at least not soon#2021-07-1617:13Alex Miller (Clojure team)basically re-making boot pods at that point#2021-07-1617:13seancorfieldYeah, they were great but they were also problematic 😞#2021-07-1617:13Alex Miller (Clojure team)which is probably cool, but farther down the list#2021-07-1617:14seancorfieldI'd be interested in seeing an example in the tools.build docs of using "a combo of create-basis, java-command, and process to execute it", preferably for test-runner...#2021-07-1617:14Alex Miller (Clojure team)but if someone else wanted to make it, I think the provided basis stuff probably does all the hard work, just need to combine it with classloader isolation stuff#2021-07-1617:14Alex Miller (Clojure team)@seancorfield can you file a ticket for that at clojure-site ?#2021-07-1617:15seancorfieldYeah, which @hiredman has done at work but it's kinda gnarly (classloader isolation). And it's something that Polylith has also had to do. I think everyone reinvents that particular wheel right now.#2021-07-1617:15seancorfieldSure, I'll go open an issue on the GH repo for that...#2021-07-1617:18seancorfieldhttps://github.com/clojure/clojure-site/issues/538#2021-07-1617:19Alex Miller (Clojure team)thx#2021-07-1619:31Cam SaulIs there any way to customize the JAR manifest when using tools.build uber? https://docs.liquibase.com/tools-integrations/extensions/extension-upgrade-guides/lb-2.0-upgrade-guide.html. Also having it not include Multi-Release: true automatically (even if some of our libs are multi-release) would be nice -- log4j is several orders of magnitude slower in a Multi-Release JAR. Maybe a way to override the default mainfest that tools.build generates with my own custom one?#2021-07-1619:45Alex Miller (Clojure team)currently, no - could you ask it at https://ask.clojure.org and I'll turn it into a ticket from there#2021-07-1619:46Cam Saulsure, np. Thanks!#2021-07-1620:11seancorfield@camsaul depstar lets you customize the manifest and has a drop-in replacement for tools.build.api/uber in hf.depstar.api#2021-07-1623:02Cam Saul@seancorfield thanks, I'll take a look at that#2021-07-1623:10seancorfield(I added some notes to your ask post about it)#2021-07-1620:13seancorfieldI'm surprised about log4j and the MR stuff but I will point out that if you're combining libs that use log4j2 (not 1.x) you'll also run into the problem that tools.build's uber task does not merge plugin cache files properly (which depstar does).#2021-07-1620:37borkdude> This is the sort of thing that I feel like `just`, the task runner, is good for.
FWIW, this is also one of the main use cases of the bb task runner. https://book.babashka.org/#tasks#2021-07-1621:09borkdudeOr find a better solution to pass literal strings to -X. I think that would make things between shells and clojure smoother in general (https://clojurians.slack.com/archives/C6QH853H8/p1626295192392500?thread_ts=1626294554.390300&cid=C6QH853H8)#2021-07-1621:43borkdude$ echo '(ns foo) (defn -main [& {:strs [-a -b --long-opt]}] (prn {:a -a :b -b :long-opt --long-opt}))' > src/foo.clj
$ clojure -M -m foo -a foo -b bar --long-opt baz
{:a "foo", :b "bar", :long-opt "baz"}
;)#2021-07-1701:19Cam SaulI'm not sure if this is a known issue, but tools.build doesn't work with Potekmin defprotocol+ (which skips redefining the underlying var/interface class if it already exists) -- at least not in cases where another namespace that require the one defining the defprotocol+/etc. come before the one defining it alphabetically.
For example, suppose I have a namespace stuff :
(ns stuff
(:require stuff.protocols))
and a namespace stuff.protocols:
(ns stuff.protocols
(:require [potemkin :as p]))
(p/defprotocol+ MyProtocol ...)
tools.build/uber emits a compile.clj file that's more or less
(binding [*compile-path* "/tmp/compile-clj9219850936397513089/compile-clj"] ; number is random for each run
(compile 'stuff)
(compile 'stuff.protocols))
and then runs it. (compile 'stuff) of course loads stuff.protocols as a side effect, but this should mean stuff.protocols (and thus MyProtocol) should also ends up getting compiled as a side effects (if I understand correctly? Please correct me if I don't). It seems like what is happening is that the subsequent call to (compile 'stuff.protocols) doesn't emit the init code for MyProtocol (since it already exists) and ends up stomping over the previously-compiled version of stuff.protocols. (This seems to be what's happening because the I have two different stuff/protocols$loading__1234__auto____<number>.class files in the end, but from decompiling protocols__init.class it seems like only one is actually getting used.)
Now what's interesting is that https://github.com/technomancy/leiningen/blob/4d8ee78018158c05d69b250e7851f9d7c3a44fac/src/leiningen/compile.clj#L154-L163, but this code works with lein uberjar. AFAIK, the only difference is Leiningen does not put the working-compile-dir (e.g. /tmp/compile-clj9219850936397513089/compile-clj) on the classpath for the compilation process. This means the classes underlying MyProtocol won't already exist no matter how many times stuff.protocols gets compiled. The resulting uberjar when compiled with Leiningen only has a single stuff/protocols$loading_1234_auto__<number>.class which supports this theory.
I have been able to get tools.build to work with either one of a couple hacks. The first was to tweak clojure.tools.build.tasks.compile-clj/write-compile-script! to include the lines
(require 'potemkin.macros)
(intern 'potemkin.macros 'equivalent? (constantly false))
in the compile.clj script it generates (this effectively disable Potemkin's skip-class-redefinition logic).The second was to tweak compile-clj to remove working-compile-dir from the classpath. What's interesting tho is that even with the latter hack, tools.build still ends up generating significantly more files (it still has the duplicate _loading__number_auto__ classes, for example). With Metabase, this ends up being a significant different in the number of files. Here are the number of classes in the entire uberjar and for compiling a single namespace with both tools.deps and Leiningen:
# with tools.build
$ jar -tf target/uberjar/metabase.jar | wc -l
102171
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
51
# with Leiningen
$ jar -tf target/uberjar/metabase.jar | wc -l
94706
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
33
I haven't worked out why there's still duplicate classes with tools.build yet. I wrote some bespoke compilation code that just finds all my namespaces with tools.namespace.find and compiles them, and the numbers I get are similar to Leiningen's. I'd love to hear any ideas as to why the difference persists.
Does *compile-path* actually need to be on the classpath? The documentation for it says
> This directory must be in the classpath for 'compile' to work.
But clearly it works even if it's not...
I'd love to hear any suggestions people have about how to deal with this. I have a feeling since Potemkin is a bit of a controversial library a fix specially for it might not be something I should hold my breath for. I think another way to fix this issue without removing the *compile-path* from the classpath would be to compile things in a topological dependency order (https://github.com/cloverage/cloverage/pull/303) but it seems a little complicated for tools.build... happy to submit a PR/patch if it's welcome tho#2021-07-1701:44Alex Miller (Clojure team)You can specify the namespaces to compile in order with compile-clj - did you try that?#2021-07-1701:44Alex Miller (Clojure team)It’s not expected that the default will work for every library, particularly for issues like this#2021-07-1701:56Cam Saulwe have a little over 400 namespaces for Metabase, so it's a bit too much to do by hand. Let me try programmatically determining the appropriate compile order for them and seeing if tools.build works without the hacks if I do that#2021-07-1715:02vemvhonestly Potemkin has a quite extensive history of being problematic for various of its functionality#2021-07-1702:14Cam SaulOk, by passing in the namespaces in topological order (I basically just copied the Cloverage code I wrote last year to do it programmatically) it does actually work without the hacks, and doesn't create any duplicate files. The bad news is my build.clj file is ~200 LoC now (just to replicate lein uberjar -- nothing else like install or jar). Not so bad for me since I've been writing Clojure around 9 years now but I feel like potentially writing that much code to deploy a project has the potential to trip up newcomers to the language#2021-07-1702:28Alex Miller (Clojure team)I don't think newcomers have 400 namespaces :)#2021-07-1702:29Alex Miller (Clojure team)but would very much like to make this work oob#2021-07-1702:15dpsuttonWe probably have tools namespace on the class path. Can’t that topo sort them?#2021-07-1702:19Cam Saulyeah that's what I ended up doing.
tools.namespace topo sort needs a dependencies graph, and to build a dependencies graph you have to find all your namespaces (using tools.namespace.find) and then read the ns forms in each of them (using ns.file and ns.parse).
It ended up being ~50 LoC to do all that#2021-07-1702:20dpsuttonAh bummer. Although I was thinking that actually might come in handy so you can ensure all the ones that depend on Potemkin can go first#2021-07-1702:26Alex Miller (Clojure team)if you want to contribute that code, that would be cool to have built in#2021-07-1702:27Alex Miller (Clojure team)assuming that works :)#2021-07-1702:33Cam SaulSure, I'd be happy to get it up to library quality and contribute it. What's the process for tools.build? Open an issue in JIRA and attach a patch? Or open a PR in the GitHub repo?#2021-07-1704:11Alex Miller (Clojure team)Jira and patch in https://clojure.atlassian.net/browse/TBUILD#2021-07-1704:11Alex Miller (Clojure team)Don’t stress about the api, I’ll adjust as needed#2021-07-1914:33Alex Miller (Clojure team)I added a jira for this at https://clojure.atlassian.net/browse/TBUILD-7#2021-07-1702:39hiredmanJust generate a dummy file that requires all your namespaces and only compile that file#2021-07-1702:41hiredmanimo anything that calls compile more than once is going to break things#2021-07-1702:41hiredmanIf you do the dummy file you don't need to sort anything#2021-07-1703:06hiredmansomething like https://github.com/hiredman/tools.build/commit/3709c1b9748519a373908528406fccbb86b678fb#2021-07-1703:06Cam Saulthanks! I'll have to try that#2021-07-1703:07hiredmanI haven't run it, so it may need some debugging#2021-07-1703:08hiredmanhrmm I see a bug already, the namespace needs ~'#2021-07-1714:04FiVoHey, I am using the prerelease cli 1.10.3.905 and I am having a dependency that points to a local jar via :local/root . I am getting the error
Error building classpath. Manifest type :jar not loaded when finding deps for ... in coordinate ...
When using the cli version 1.10.3.855 everything runs fine. Just wanted to report.#2021-07-1721:34Alex Miller (Clojure team)thanks, fixed for next release#2021-07-1719:44EddieJust wanted to +1 the conversation about compilation order. I think I hit a similar issue as @camsaul just yesterday. I have bit more detail https://stackoverflow.com/a/68413578/4297413.#2021-07-1721:40Drew VerleeShould -x:deps find-versions be working for Clojure CLI version 1.10.3.855? it says function not found: clojure.tools.cli.api/find-versions, asking because the example is in the docs already 🙂#2021-07-1721:43Alex Miller (Clojure team)no, only working in prerelease#2021-07-1721:44Alex Miller (Clojure team)docs should probably guard that, but I'm hoping it's not more than a week or so till things come back together#2021-07-1722:18rafaeldelboniIs there a way in deps.edn to set that the type of my dependency is pom only? Like this one https://mvnrepository.com/artifact/io.zonky.test.postgres/embedded-postgres-binaries-bom/13.2.0#2021-07-1722:20Alex Miller (Clojure team)no, that's not supported in deps.edn#2021-07-1809:46henrik42Is it planed? We're using it a lot with our maven build to encapsulate/export version infos, plugins etc.#2021-07-1812:20Alex Miller (Clojure team)You can file a request on http://ask.clojure.org to gather interest…#2021-07-1905:40didibusI get Improperly specified VM option 'TieredStopAtLevel=1 -J-XX:SharedArchiveFile=clojure.jsa' when I call clojure like this: clojure -J-XX:TieredStopAtLevel=1 -J-XX:SharedArchiveFile=clojure.jsa -A:repl it seems maybe I don't understand how to pass more than one JVM option?#2021-07-1905:49Alex Miller (Clojure team)that's how I would expect it to work#2021-07-1905:51Alex Miller (Clojure team)if you throw bash -x at the beginning of that, you'll see the actual command being run#2021-07-1905:51Alex Miller (Clojure team)I can't repro what you're reporting#2021-07-1905:51didibusOh, ok it might have something to do with that I'm calling it inside a bash script with command, if I just type it at the terminal it works, so probably on my end#2021-07-1914:02Alex Miller (Clojure team)New prerelease of Clojure CLI ~1.10.3.916~ 1.10.3.920 with some bug fixes:
• TDEPS-187 - fix #2 for using help/doc and help/dir with tools (but also a lot of internal cleanup)
• Fix regression with using local jar deps
• Make slurp-deps more tolerant of missing deps files#2021-07-1915:32seancorfieldI'll get that installed as soon as I've had my coffee and let you know how it goes. Thank you!#2021-07-1915:33seancorfield(and I'll update clj-new to use the newer t.d.a and verify that I can remove the workaround I added)#2021-07-1915:39Alex Miller (Clojure team)actually, it's buggy, new version coming :)#2021-07-1915:40Alex Miller (Clojure team)1.10.3.920#2021-07-1917:09seancorfield(verified that with t.d.a 0.12.1003, I can remove the workaround from clj-new -- thank you!)#2021-07-1919:06seancorfieldI ran into this error on one of our old servers (which we'll be replacing soon):
Cloning:
Checking out: at 688245ea6cc8b3b173b79b861938444fcd9e3e1e
Error building classpath. Unable to checkout 688245ea6cc8b3b173b79b861938444fcd9e3e1e
git: 'worktree' is not a git command. See 'git --help'.
It has git 1.7.1 -- I gather that's not a supported version of git at this point?#2021-07-1919:49ghadioh the ~humanity~ security#2021-07-1919:53seancorfieldWe haven't used git on that server for a long time -- I'd forgotten it was actually installed (since git is not installed on any of our production servers) -- but I accidentally introduced a git dep into something our deployment script was doing (now removed).#2021-07-1919:55seancorfieldSo I guess, in a way, this was a good error to trip over so that I didn't accidentally put something on production that still relied on a git dep...#2021-07-1919:45Alex Miller (Clojure team)right#2021-07-1919:45Alex Miller (Clojure team)this is mentioned in the deps reference - we expect Git ≥ 2.5#2021-07-1919:46Alex Miller (Clojure team)(that came out about 5+ years ago)#2021-07-1919:54seancorfieldI'll be glad when our migration to new servers is completed at work 🙂#2021-07-2003:03idasmy apologies, i asked this question a while back in this channel, but i've forgotten the response many kind folks gave me earlier.
i'd like to a make mixed JVM language project using both clojure and java, where clojure code is calling java code. what is the best way to structure this kind of a project using clj-tools/tools-deps? is there an existing example project to look at?#2021-07-2003:21Alex Miller (Clojure team)you can use the brand new tools.build to help you with such a thing#2021-07-2003:22Alex Miller (Clojure team)I don't have a good example for you (yet) but that would be a great one to add to the guide at https://clojure.org/guides/tools_build#2021-07-2003:22Alex Miller (Clojure team)but really it's making a build script that uses tools.build javac task to compile java into a classes dir, which is on the :paths#2021-07-2004:02idasyeah, this kind of project scenario definitely looks like a great use case for tools.build. a while back when i asked this question i was referred to github projects that used deps.edn -X aliases#2021-07-2004:58didibusI'm not sure how much Java you intend to have, if its majority Clojure, tools.build should work great with its javac task, and you can also leverage java-command and process for slightly more complicated tasks. If you intend to have a good chunk of Java like 50% java, or you have an existing Java code base and want to start adding some Clojure to it, you might be better served by a Java build tool like Gradle that uses a Clojure extension such as Clojurephant or nebula-clojure-plugin. These will make it easier to use Eclipse or IntelliJ for the Java development, run Java code coverage, unit tests, checkstyle, Lombok, java doc, and all that, if that's needed for your use case.#2021-07-2014:25idasmostly clojure. just a few java classes for some more performant sections. mainly because my clojure understanding isn't that great yet#2021-07-2018:52didibusYa, then tools.build will be good enough, and better for the Clojure#2021-07-2004:59seancorfieldJust remember that we're a Clojure-focused community here wanting to use some Java with Clojure 🙂#2021-07-2005:21seancorfieldAt work, we've already adopted tools.build and we're leveraging java-command and process to run various things that are outside the scope of the built-in tasks, such as database migrations and other environment setup. We're also running our tests via those two functions -- so we can run clojure -T:build run-tests and that creates a basis for running our test suite and then creates a java-command to invoke Cognitect's test-runner with the appropriate arguments, and then process to actually run it. So far it's all working pretty nicely (but see TBUILD-6 for -X style invocation which was something we needed to implement ourselves, so far).#2021-07-2009:57FiVo@idas I made a simple example project testing exactly that the other day: https://github.com/FiV0/tools.build-testing#2021-07-2014:26idasNice! Thanks for sharing this!#2021-07-2011:46EdHi. If I was to be trying to generate an uberjar with tools.build, is there a way to generate a manifest file so we can java -jar ... to start it?#2021-07-2011:54Ednevermind ... I'm just being dumb ... found it 😉#2021-07-2015:07borkdudehttp://blog.fogus.me/2021/07/20/clojure-builds-as-an-amalgamation-of-orthogonal-parts/ (thanks @fogus for the write-up)#2021-07-2015:16dominicmThat's giving me a certificate error 😧#2021-07-2015:17dominicm> Web sites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for http://blog.fogus.me. The certificate is only valid for the following names: http://s444.sureserver.com, http://www.s444.sureserver.com#2021-07-2015:20fogus (Clojure Team)adding an s to the protocol should fix it.#2021-07-2015:22dominicmI am using https 🙂 That's what gave me the error.#2021-07-2015:22borkdude@fogus does https://blog.fogus.me/2021/07/20/clojure-builds-as-an-amalgamation-of-orthogonal-parts/ work for you? For me it doesn't#2021-07-2015:22dominicmWho uses http:// these days? 😄#2021-07-2015:23fogus (Clojure Team)Weird. Looks like I have to check my cert.#2021-07-2015:26fogus (Clojure Team)OK, should be fixed now..... 🤷#2021-07-2015:26dominicmYep!#2021-07-2015:27seancorfield@fogus You say "One interesting aspect of the jar target is that it uses a source pom as the base for the new pom." -- it might make it clearer to new tools.build users if you additionally clarify that "the new pom" is written to the :class-dir folder. Folks who've used clojure -Spom will be used to the CLI updating the pom.xml file in place.#2021-07-2015:27fogus (Clojure Team)Thanks @U04V70XH6 I'll add that#2021-07-2015:28seancorfield(and :class-dir is not as intuitive as :target-dir, I suspect, for folks used to Leiningen terminology since Leiningen writes pom.xml directly into target as I recall?)#2021-07-2015:29seancorfieldI have to keep reminding myself that :class-dir is "inside the target folder"! 🙂#2021-07-2015:31seancorfieldHopefully, I'll get around to writing up our tools.build usage at work soon. Need to figure out how one piece is going to be tackled, which I've discussed with @alexmiller (TBUILD-6). I'm kinda holding off until I have a better sense of whether that's going to get addressed sooner, or later 🙂#2021-07-2015:31fogus (Clojure Team)I'm not keen on class-dir or target-dir --- I wonder if there's an even better name for that place?#2021-07-2015:32seancorfieldNaming is hard 😕#2021-07-2015:32fogus (Clojure Team)absolutely.#2021-07-2015:32borkdude:out-dir ?#2021-07-2015:34seancorfieldI mean, in tools.build it is more a "classes" folder than just a "target" folder. In depstar, I used :target-dir but that's for the top-level folder -- it always puts classes in that folder if you compile .clj files, but it puts the JAR and the pom.xml directly into that top-level folder, like Leiningen does (and it doesn't copy intermediate files there -- to improvement performance: they are copied directly from their original locations into the JAR file -- but I can see the additional flexibility that copying files brings).#2021-07-2015:34dominicmcljs already uses out-dir#2021-07-2015:34dominicmMight be confusing#2021-07-2015:35borkdudeor it might be consistent with other clojure tooling ;)#2021-07-2015:37dominicmExcept you'd likely have the cljs out-dir pointing inside the clj out-dir#2021-07-2015:41fogus (Clojure Team)directories are hard too!#2021-07-2019:45didibusInteresting, so tools.build intends to limit itself only to packaging tasks?
And it would still be expected that other development and release tasks would use other tools? And be orchestrated through aliases again? Such as running tests, linting, doc generation, formatting, starting repl, etc. ?#2021-07-2019:47seancorfield@U0K064KQV I'm not sure where you get that impression from the above thread?#2021-07-2019:48Alex Miller (Clojure team)Many of those already have good answers available now, so there is no need to recreate them in tools.build#2021-07-2019:48seancorfieldAt work, we're using tools.build to run database migrations, run tests, build JARs, push deployments...#2021-07-2019:48Alex Miller (Clojure team)But if you want a build that does other stuff, it’s your program, so go for it!#2021-07-2019:49didibusWell it talks about an amalgamation of orthogonal tools, and explains that tests for example are not meant to be a part of tools.build, so some other tool must be used and configured as an alias#2021-07-2019:49seancorfieldYour build script can invoke a test runner.#2021-07-2019:51seancorfieldOur CI at work is basically just clojure -T:build run-all-tests and clojure -T:build build-and-deploy-jars.#2021-07-2019:52seancorfield(and run-all-tests is an exec function that runs our database setup, migrations, search engine setup, content publishing, and then our complete test suite 🙂 )#2021-07-2019:53didibusI know but that's not how Fogus set it up in his article. That's why I was curious. My thoughts was that you'd orchestrate all tasks with tools.build, and just that maybe it doesn't "yet" have a built-in task for everything, but could eventually or those could be added by other libs which follow the same pattern and can be used inside build.clj
So that later the CLI would have a way to chain tasks and compose them, and I was expecting the current arg on the build steps would probably eventually be a way to pass context around between steps.
But in the article I understood that testing maybe isn't meant to be a part of that, and so other aliases are expected to be used for those. Maybe I understood wrong, but that's two very different philosophies though.#2021-07-2019:53seancorfield"builds are programs" 🙂#2021-07-2019:57seancorfieldGiven that your tests generally need to be run in the context of your test runner but not in the context of your build script (something that has caused us a bit of grief until we took the subprocess approach), you could just use
clojure -X:test && clojure -T:build build-and-deploy
and that's nice and simple and composable.#2021-07-2019:59seancorfield(our monorepo setup is sufficiently complex that clojure -X:test is not enough to run our tests -- because the test runner needs to be told about all the separate subprojects that contain tests -- which is why we're using exec functions in our build.clj script to figure that out and create the right context for executing the test runner!)#2021-07-2019:59seancorfieldI will be blogging about this in due course...#2021-07-2020:10didibusI was mostly interested in understanding the vision as things evolve. At first I thought that the vision is that all tasks would be managed by tools.build. And so in the future, you wouldn't create a test runner that is a CLI, or a lein-plugin, etc. It would just be a Clojure lib which you'd call inside build.clj
Thus all tooling would just become Clojure libs, and build orchestration would become a Clojure program in the form of build.clj
And the CLI would play nicely with build.clj, such as being able to call each of the "tasks" defined in it individually, or in the future compose an order of them together.
As for the "context" each build tool must run under, like what classpath, what root folder, what resources, what user, JVM args, properties, etc. Well I thought that was maybe still an open question. But that either you'd fork a new process, or possibly you could have sub-builds running with a different context. Something like maybe I can have a test-build.clj where tasks in there run under the test alias.#2021-07-2020:12didibusBut now having read the article, I see maybe the vision is different, and actually build.clj is really meant for things that can happen under the build context, like packaging a jar. But everything else is still meant to be used as aliases, so other build tools would be -X or -M invokable, and shell would be used to compose and orchestrate them.#2021-07-2020:33Alex Miller (Clojure team)I would go more with the latter vision :)#2021-07-2020:35Alex Miller (Clojure team)tools.build is a TOOL to BUILD artifacts (it's right there in the name!) :)#2021-07-2020:36Alex Miller (Clojure team)that said, your build program (build.clj or whatever) is your program - it can use stuff from tools.build to make artifacts, or do other things using functionality from other libraries#2021-07-2020:38Alex Miller (Clojure team)and then deps.edn aliases are a way to set up other tools a level above that - at that level you're choosing a classpath, and building artifacts and running tests and generating docs may all need different classpaths#2021-07-2020:39Alex Miller (Clojure team)we're not actively running at trying to provide functionality to run multiple programs - you have many OS-specific tools designed to do that#2021-07-2020:52seancorfieldFor "most" projects -- where a simple clojure -X:test can run all the project's tests -- I would expect folks to use tools.build for "just" the build-related stuff (packaging, versioning, etc) since they already have a solid command for running tests and, with something like deps-deploy, they have a solid command for installing to the local ~/.m2 or up to Clojars. For more complicated projects, tools.build provides a bunch of useful stuff that helps orchestrate other tasks -- the create-basis/`java-command`/`process` stuff -- and in those cases it is a nice "leg up" to writing more complex "build" scripts. So I'm pleased with what it provides.#2021-07-2020:55Alex Miller (Clojure team)you can use clj -X:deps mvn-install for installing locally. we may end up making something for deployment - I think there is more that can be done there.#2021-07-2020:55seancorfieldFor "just" building JAR/uberjar files, I'll probably stick to depstar, even for small projects -- especially with nasty stuff like the log4j2 plugins cache file merging: that alone prevents us using tools.build.api/uber.#2021-07-2021:00seancorfieldI will probably add a tools.build-based JAR building script to library projects created by clj-new though (the log42j stuff does not affect libraries, and I want to encourage folks to use tools.build).#2021-07-2021:01seancorfield(once the new CLI changes get to a stable release so that I don't introduce a dependency on a prerelease CLI if you create a new project with clj-new!)#2021-07-2021:02seancorfieldFor the app template in clj-new, I could also add a tools.build-based script but have it use hf.depstar.api/uber for that step 🙂#2021-07-2101:19didibusI see, but I guess there is tools.build the library, and there is also changes to the CLI related to tools.
So tools.build is just a lib with whatever useful convenient functions that may help you with writing a build program and doing common things.
I guess more interesting is the convention of having a file called build.cljwhich it establishes, and to have a -T invokable alias called build in your deps.edn.
While just a convention, this is starting to look like a build framework, as it will become the expected frame for how a project sets up its build and how someone would expect to be able to run the project build tasks by using clojure -T:build <task>
Similarly, I'm guessing from this convention, if you were to poke around some projects and wanted to understand what are all the build steps available for it, one would think to look at the build.clj
It seems the -T option of the CLI and build.clj being established as the ns-default are really the important bits here. As well as what's coming to chain -T commands from the CLI
> We expect to be able to do these both together on the command line but that is a work in progress
>
As well as the convention to parameterize using a key/value map, and have the "tasks" functions return that map for whenever support for command chaining is added.
Personally when I see all this, I would find it weird to expect other build related tools, like tests and linting for example, to be setup as.`-X`. If so, now you have to learn a whole other tool and it's commands and all that, and won't be able to use whatever the command chaining when it lands for -T.
So I feel it's really asking for people to release their additional tools in the same convention and style as tools.build , and for projects to adopt the build.clj as the place where tasks are defined.
For example, I'd expect someone could say release a nRepl task where you'd do:
(ns build
(:require [clojure.tools.build :as b]
[nrepl.nrepl.build :as nb]))
(def lib 'my/lib1)
(def version (format "1.2.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(defn nrepl [params]
(nb/start-nrepl
{:basis basis
:src-dirs ["src"]})
params)
And similar for testing, linting, and all others.
At least I feel this is the convention that's being established. If that happened, the use of -X or -M would eventually get smaller and smaller.#2021-07-2101:31Alex Miller (Clojure team)you often don't need that level of customization and control though. for example I have a couple of aliases in the tools.build deps.edn for testing and doc gen https://github.com/clojure/tools.build/blob/master/deps.edn#2021-07-2101:32Alex Miller (Clojure team)tools.build isn't (currently) released as an artifact and doesn't need a build and doesn't have a build.clj#2021-07-2101:34seancorfield@U0K064KQV I would hope that with the improved support for source libs in the latest CLI/`tools.deps.alpha` we will see more libraries switch to that mode and perhaps stop publishing JARs at all.#2021-07-2101:34Alex Miller (Clojure team)if I could loosely cut up the world (ignoring client-side cljs stuff atm):
• tools - usually released as git deps, usually don't need a build
• libs - usually deployed as jars to clojars or released as source deps to git. may have a "simple" build.clj that makes a source jar
• apps - not deployed, usually have a build.clj that makes an uberjar#2021-07-2101:35seancorfieldEven though both clj-new and depstar are currently published to Clojars, there's really no reason for me to do so -- both libraries could be installed as local "tools" (`clojure -Ttools install`) and clj-new arguably should be; and depstar is likely to have a career as a library to supplement tools.build, at least for a while, and that can be added as a source dependency.#2021-07-2101:37seancorfield"other build related tools, like tests and linting" -- those aren't "build related" though.#2021-07-2101:38Alex Miller (Clojure team)note that the new usage stuff for tools is taken from the deps.edn from source-based deps so you can only use that feature with source-based git deps right now (I've thought of some options for artifacts but nothing implemented atm)#2021-07-2101:39seancorfieldLeiningen blurred the lines because it runs tests, runs a REPL, builds JARs, writes poms, creates new projects, etc, etc, etc -- and with plugins it runs linters and all sorts of other tools. It's "easy" but not "simple" in the Clojure sense. I don't think we should expect to have "One Tool to rule them all".#2021-07-2101:43seancorfieldMost of my projects today have these three commands:
clojure -X:test
clojure -T:jar
clojure -T:deploy
and I can use the former to run tests using any version of Clojure I want to check (either through aliases in the project or my user deps.edn file). A lot of projects don't need any more than that -- and if we move more to source libs we'll see less need for the latter two commands.#2021-07-2101:43Alex Miller (Clojure team)leiningen was just drafting maven in that respect#2021-07-2101:47didibusI think all these names are kind of irrelevant. People have tasks they need to run, those are needed during development, debugging, CI, deployments, release and all that. Generally languages have a task framework so all these tasks play nice with each other and can also become part of the ubiquitous language between developers. Because these tasks often also required specific dependencies to run, such task framework is often coupled with a dependency manager.
In my opinion this task framework is a good thing. Now a specific framework like lein maybe is too hard to extend or branch out when you need to do complicated things. But the idea of having at least a common convention you can be familiar with is pretty good.#2021-07-2101:48didibusI'm okay with -X, but it makes tools.build the odd one out. Like why isn't it also just an alias to be called with -X then?#2021-07-2101:49Alex Miller (Clojure team)it can be#2021-07-2101:50Alex Miller (Clojure team)but the main difference is that -X uses your project classpath + mods. -T does not start from your project classpath.#2021-07-2101:51Alex Miller (Clojure team)testing is typically your project classpath (-X) + testing framework#2021-07-2101:51Alex Miller (Clojure team)building is typically NOT your project classpath (-T) + build framework#2021-07-2101:52Alex Miller (Clojure team)the other subtle difference is that -T uses :paths ["."] by default so you can reach things like a build.clj at the root without needing to add that to your alias#2021-07-2101:53seancorfieldI just edited my "three commands" above since :jar and :deploy could both be invoked with -T now -- that just wasn't the case before.#2021-07-2101:53seancorfieldNow running tests is the odd one out @U0K064KQV 🙂 🙂#2021-07-2101:54Alex Miller (Clojure team)I am still working out the best way to talk about this stuff. there is a lot of intention in it but I have been living in it too long to have the beginner mind. so I appreciate these questions didibus!#2021-07-2101:56didibusSo the "framework" here is really with -X and -T. Like if I want to create a task that others would use I should have it support proper invokation on either -T or -X.
But if I need as a user to do anything more complex, which is where lein failed, I would need to write my own program, in build.clj
Now in my own program, I would need to probably leverage other things to make it easier for me, like tools.build.
So I guess anything setup for -T or -X can be used as a library as well, since it's just a clojure function that takes a map.
But -X and -T currently don't compose at the command line. And -X has the issue when used as a lib that it might require a very different classpath and root folder and all that#2021-07-2101:59seancorfield@U0K064KQV It's early days with this stuff. I expect we'll get multi-function invocation back for both -X and -T (right @alexmiller?) so it really will come down to knowing whether a "tool" (in the broader sense) needs to run in your project classpath (`-X`) or not (`-T`) and some tools won't care (`:jar` and :deploy examples above could be either).#2021-07-2102:01didibusSo personally I find the -T and -X confusing. I feel it would have been easier if the alias could say it's a tool or not. So as a user you just copy/paste and -X like always. It seems a burden to need to remember which invokation is the right one for the alias. I don't know maybe there's a reason for it needing to be user provided every time?#2021-07-2102:02seancorfieldWhat would happen if you used -X:foo:bar and :foo declared it wanted the project classpath and :bar declared it didn't? You still can't compose those two types of tools.#2021-07-2102:03didibusBut my biggest maybe uncertainty I have right now, and hopefully no hard feelings haha I am just trying to provide some feedback I can be absolutely wrong too.
It's around being able to compose and reuse tasks. It's great whenever the -X and -T command composition is going to come, but there also needs a way to compose those in a Clojure program no? Specifically the build.clj ? Is that the issue your referring to Sean?#2021-07-2102:04seancorfieldIf you don't mind using :replace-paths ["."] :replace-deps {...} then you can use -X for tools that don't need your project classpath.#2021-07-2102:04didibusWell, that problem you described is the same-ish as being able to compose -X in a program no? That they might need different classpath and all?#2021-07-2102:04Alex Miller (Clojure team)that's out of scope imo#2021-07-2102:05Alex Miller (Clojure team)maybe future in-scope, but not right now. at the CLI you are always running one program in one classpath.#2021-07-2102:06didibusYou mean one function? Or like you'll be able to run multiple functions just not in different classpaths?#2021-07-2102:06Alex Miller (Clojure team)yes#2021-07-2102:06Alex Miller (Clojure team)like multiple functions in a build program#2021-07-2102:06seancorfieldMultiple functions in a single program in a single classpath.#2021-07-2102:07Alex Miller (Clojure team)but I think the use cases for that are limited - that's more the exception than the norm#2021-07-2102:07Alex Miller (Clojure team)> I feel it would have been easier if the alias could say it's a tool or not.
is a good question#2021-07-2102:08didibusOk ya. Maybe this is where I latch on to tools.build Because it lets you use deps.edn to form a basis to run another task in a sub-process. If that other task is a -X test.runner for example, well then build.clj looks like a great way to compose those things. And then I could do clojure -T:build copy-config compile test#2021-07-2102:08didibusEven though each of those might need different classpaths#2021-07-2102:08seancorfieldWell, you can't easily run a -X function in a subprocess right now.#2021-07-2102:08Alex Miller (Clojure team)I don't think you're going to be happy forking processes to do those things#2021-07-2102:08seancorfieldThat's what TBUILD-6 is about 🙂#2021-07-2102:10didibusI mean, my only alternative right now is shell and that just forks processes. If I do: clojure -X:copy-config && clojure -T:build compile && clojure -X:test#2021-07-2102:11seancorfieldtools.build makes it possible to run "arbitrary" java commands as subprocesses but it's more work that just clojure.java.shell/sh or even ProcessBuilder interop -- but in exchange you get conveniences for creating the classpath (via basis creation) etc.#2021-07-2102:11didibusAnd as a user, eventually I'd expect that I wouldn't need to fork the process myself, but people would have their -X function takes an optional basis that then forks it and executes it within the context of the basis for me.#2021-07-2102:12seancorfield@U0K064KQV As I said yesterday, maybe instead of contemplating this in the abstract, you should use tools.build and build some of this stuff yourself and see how well it does or doesn't work for you.#2021-07-2102:12didibusIf they are an -X that needs to run in specific deps and all that#2021-07-2102:14seancorfieldFor what it's worth depstar already forks a process to run AOT compilation (so that it can guarantee it happens in the correct classpath/environment). But I don't think it's realistic to expect test runners to do that when it's just extra complexity and overhead: they could simply run in your project's classpath without any of that.#2021-07-2102:15seancorfieldLeiningen and Boot essentially did force everything to work "the same way" so you have a barrier to entry immediately for folks looking to write simple tools that can be used from the CLI.#2021-07-2102:15didibusTest runners need to run in the project test classpath though, which will often add mocking libs and other test only deps#2021-07-2102:16seancorfieldWell, project classpath + test runner + test deps. That's still different to "no project classpath and having to create the basis and spawn a new process" etc.#2021-07-2102:18didibusYa, but you can say tell users:
Add the test runner to the build alias deps
Add a :test alias to deps.edn and add your test deps in there.
Copy this function inside your build.clj
Add this require to build.clj
Run: clojure -T:build test
Done.#2021-07-2102:19seancorfieldOr:
Add a :test alias to deps.edn and add your test deps and the test runner in there.
Run: clojure -X:test
Done.#2021-07-2102:19seancorfieldWhat if they don't have build.clj? A lot of projects won't need it, as Alex said above.#2021-07-2102:21seancorfieldIf you have a pure Clojure source library and you expect folks to use it via git then you really only need :test alias and run it via -X.#2021-07-2102:22didibusYes that's an alternative, but then I need to know if it's -X:test or -T:test or -Ttest or -T:build test cause now I'm dealing with different tasks using all four.
And if I'm a power user, and I need to do more things before my tests can run, say generate some client, grab some creds, and I want to put those step in my build.clj and then start the test runner all with -T:build test ?#2021-07-2102:23seancorfieldThis is something I have to consider for clj-new -- how simple should the generated projects be? Currently, they have a deps.edn with aliases for :test, :jar, :deploy and can be run via -X. But should those (simple) projects all have build.clj by default? Probably not, so maybe I need new template names, or I handle this via additional command-line options or something else...#2021-07-2102:24seancorfieldYou're boiling the ocean: you're trying to solve every use case and trying to make them all the same. I don't think that's a very helpful or productive approach.#2021-07-2102:25seancorfield-Ttest would be a locally-installed tool (in ~/.clojure/tools) and is different from -T:test -- and you would know because you'd installed that tool yourself (`clojure -Ttools install some/coord '{...}' :as test`)#2021-07-2102:28didibusThat one I'm not super concerned, though I think a convenience if like -T:test if no such alias is in the project or global deps.edn to check if a global tool of that name exists would be nice too.
But ya, I'm more thinking in terms of the convention#2021-07-2102:28didibusLike it seems it be more straightforward to me to just establish build.clj as the build script with all entry points into all tasks. And for the Clojure CLI to become the task runner for them.#2021-07-2102:32Alex Miller (Clojure team)that is the not the model#2021-07-2102:32didibusOtherwise people will still have to lean on shell scripts, makefile, lein, npm scripts, Babashka tasks runner, etc.#2021-07-2102:33Alex Miller (Clojure team)that's what those things are good at#2021-07-2102:33Alex Miller (Clojure team)the original and still intent of the CLI is to run programs in a classpath#2021-07-2102:33Alex Miller (Clojure team)a build that makes an artifact is one example of a program#2021-07-2102:33didibusHum... Okay maybe I just don't see what problem tools.build is solving then?#2021-07-2102:34Alex Miller (Clojure team)it's helpers to write programs that make artifacts#2021-07-2102:35didibusOk, so it's just another depstar and company? But tasks are still meant to be standard deps.edn and be clojure programs either invokable as -X, -T or -M and orchestrated with some other tool like Makefile or a shell ?#2021-07-2102:35Alex Miller (Clojure team)yes#2021-07-2102:36didibusAh! 🤯, ok, well maybe it was just me being very confused, but I'd suspect maybe some others are similarly confused.#2021-07-2102:37Alex Miller (Clojure team)I started working on tools.build in Feb 2020 so depstar was not quite the same level of popularity then#2021-07-2102:38didibusFair fair. This really likes clarifies things. I totally projected my problem and latched on tools.build thinking it solved it, which was that -X and -M alias are hard to make work together in a coherent way and orchestrate#2021-07-2102:39Alex Miller (Clojure team)@U04V70XH6 on the clj-new question above, I think the intent of what you're building matters - tool/lib/app#2021-07-2102:40Alex Miller (Clojure team)for a tool, I'd say no build.clj.
for a lib, I think I'd lean towards no but maybe having a basic build.clj that made a source jar would be useful.
for an app, definitely.#2021-07-2102:41seancorfieldAgreed. But tools and libs don't "need" JAR/deploy -- unless they do 🙂 and an app can be a simple think or might be complex enough to warrant a build.clj file. Right now, clj-new's lib (and template) templates have a :jar alias that uses depstar to build a source JAR.#2021-07-2102:41Alex Miller (Clojure team)well, I'd stick with having a build for that then#2021-07-2102:42seancorfieldOnce the CLI gets to a stable release, I'll def. revisit the whole clj-new template thing.#2021-07-2102:42Alex Miller (Clojure team)I mean, I think that's the most common case right now - building a lib with a source artifact deployed to clojars#2021-07-2102:43Alex Miller (Clojure team)I'm in another window trying to debug the Windows updates so that we can move towards a stable release :)#2021-07-2102:44didibusAs an aside, and maybe a selfish request, but I would like it if all libs still defaulted to being able to publish a Jar to either Clojars or Maven Central.
It makes companies like mine which have policies that deps must only come from internal repository mirrors and where GitHub and git is not something they support being mirrored from, mean that I'd have to manually import all source only libs and do a jar myself to put in our mirrored Maven repo. (I had to do that with the Cognitect test runner)#2021-07-2102:44seancorfieldJust so everyone reading this (long!) thread is on the same page regarding clj-new and lib projects right now:
(! 588)-> clojure -Tnew create :template lib :name alexmiller/example
Generating a project called example based on the 'lib' template.
The lib template is intended for library projects, not applications.
(! 589)-> cat example/deps.edn
{:paths ["src" "resources"]
:deps {org.clojure/clojure {:mvn/version "1.10.3"}}
:aliases
{:test
{:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "1.1.0"}
io.github.cognitect-labs/test-runner
{:git/url ""
:sha "62ef1de18e076903374306060ac0e8a752e57c86"}}
:exec-fn cognitect.test-runner.api/test}
:jar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.267"}}
:exec-fn hf.depstar/jar
:exec-args {:jar "example.jar" :sync-pom true}}
:install {:replace-deps {slipset/deps-deploy {:mvn/version "0.1.5"}}
:exec-fn deps-deploy.deps-deploy/deploy
:exec-args {:installer :local :artifact "example.jar"}}
:deploy {:replace-deps {slipset/deps-deploy {:mvn/version "0.1.5"}}
:exec-fn deps-deploy.deps-deploy/deploy
:exec-args {:installer :remote :artifact "example.jar"}}}}
(! 590)-> ls example/
CHANGELOG.md LICENSE README.md deps.edn doc pom.xml resources src test
(edited to show a library project!)#2021-07-2102:48didibusI'm not sure how many companies are in this boat, but all the ones that use services like artifactory or AWS code artifact might have it#2021-07-2102:48Alex Miller (Clojure team)the cool thing about templates is you can have more than one :)#2021-07-2103:54seancorfieldI realized I missed a bit of the back and forth up above around depstar and just wanted to provide a bit of historical context: I took over depstar in October 2018 and by February 2020 it was already doing AOT, Multi-Release stuff, etc but it was all still trying to do it in the project classpath (which was getting painful because it meant depstar could not have any dependencies except Clojure itself). It didn't become a "tool" -- running separately from the project classpath -- until December 2020, at which time it was leveraging t.d.a etc. So I think that really highlights how 2020 has been a critical year for a lot of the work around deps.edn and CLI in terms of massive changes/evolution in tooling.#2021-07-2015:35dharriganI have a little question#2021-07-2015:37dharriganIt's too early, given the new tools build feature, and I think a consensus has to arise (maybe?), but would the thoughts be around including the build clj programs as part of your project, or perhaps bundling them up in a jar and depending up that jar (i.e., if one has say 10 or so Clojure projects, it would feel perhaps a bit burdensome to include the build scripts for every single project, but perhaps to depend upon a jar that includes those scripts might be the way?)#2021-07-2015:38dharriganWhat's the thoughts around this?#2021-07-2015:40dominicm@dharrigan I think a common build script for common uses cases makes loads of sense#2021-07-2015:40dominicmnotleiningen.build.clj 😛#2021-07-2015:44seancorfieldSince your build script can be an arbitrary namespace full of functions, it could also be a library.#2021-07-2015:46seancorfieldBut the "variables" between projects are going to be the group/artifact/version/etc which would need to be per-project -- so you could have most of your programmatic build in a library and just the variables part in your local build.clj which the library then requires to get at those...#2021-07-2015:47seancorfieldAnd you could treat your "build" library as a "tool" and clojure -Ttools install it so it's available "everywhere".#2021-07-2015:47dpsuttoninteresting, but they probably need to be kept from any jars. Since they are all single namespace any other build.clj files in a jar could interfere with finding your own build.clj right? And this would rely on your source directories position in the path to always resolve your own#2021-07-2015:48seancorfield@dpsutton build.clj isn't in src so it doesn't end up in the JAR.#2021-07-2015:48dpsutton> but would the thoughts be around including the build clj programs as part of your project, or perhaps bundling them up in a jar and depending up that jar
was responding to a suggestion for them to end up in jars#2021-07-2015:48seancorfieldI took that to mean "build script as library" rather than the "project JAR"#2021-07-2015:49dpsuttonah#2021-07-2015:49Edyeah ... I think he's talking about sharing common build tasks between multiple repos#2021-07-2015:50dharriganmy intention (and probably I'm misunderstanding), is that my project deps.edn would have a com.foo/build dep and in that build.jar would be the scripts that are common across all projects.#2021-07-2015:50dharrigandoes it work that way?#2021-07-2015:51EdI think you add a :deps keyword to your tooling alias#2021-07-2015:51Edand that defines the dependencies that the tool uses when it runs ... independent of the dependencies of your project itself#2021-07-2015:51Edright?#2021-07-2015:51seancorfieldYes, the build is "just" a dependency.#2021-07-2015:53dharriganRight, so something like this?#2021-07-2015:53dharrigan{:paths ["src"]
:aliases
{:build {:deps {com.foo/build {:mvn/version "0.1.0"}}
:ns-default build}}}#2021-07-2015:53seancorfieldWhen you run clojure -T:some-alias it does :replace-paths ["."] I think which makes it easy to pick up a script in the root of the project, but our :build alias at work also includes a dependency on depstar as a library and on a small artifact uploader "library" locally.#2021-07-2015:54seancorfieldOur :build alias:
:build
{:deps {org.clojure/clojure {:mvn/version "1.11.0-alpha1"}
io.github.clojure/tools.build {:git/tag "v0.1.3" :git/sha "688245e"
:exclusions [org.slf4j/slf4j-nop]}
;; force an updated version of t.d.a:
org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"
:exclusions [org.slf4j/slf4j-nop]}
;; add depstar for building uberjars:
com.github.seancorfield/depstar {:git/tag "v2.1.267" :git/sha "1a45f79"
:exclusions [org.slf4j/slf4j-nop]}
;; and local build tools:
worldsingles/build {:local/root "build"}
poly/base-artifact-uploader-cli {:local/root "bases/artifact-uploader-cli"}
poly/artifact-uploader {:local/root "components/artifact-uploader"}}
:ns-default build}#2021-07-2015:55seancorfield(we have some code in our build.clj and its deps that rely on the 1.11 hash map arg processing, hence needing to declare Clojure in that alias)#2021-07-2015:55dharriganis that on your parent directory, so the various sub projects would see the :local/root "build" directory?#2021-07-2015:56seancorfieldWe execute all CLI commands from the project root with its deps.edn file. Each subproject has its own deps.edn file, including the build subproject.#2021-07-2015:57dharriganI see.#2021-07-2015:57seancorfieldDocs on using depstar with tools.build for those curious: https://cljdoc.org/d/com.github.seancorfield/depstar/2.1.267/doc/tools-build-usage#2021-07-2015:57Eddo you start a different repl for developing build scripts from developing the application?#2021-07-2015:58Edor do you start a jvm with them all on the classpath?#2021-07-2015:59seancorfield@l0st3d Our dev REPL gets started with -M:everything:dev:test:runner plus whatever local tooling a developer wants. Our build REPL can be started with -M:build -i build.clj -r#2021-07-2015:59seancorfield(`:dev` and :test are our Polylith aliases for the portion of our monorepo that we've migrated, :everything and :runner are the "equivalent" aliases for the portion that has not yet been migrated)#2021-07-2016:00seancorfieldSee various posts on http://corfield.org for our monorepo setup with CLI/`deps.edn` and our migration to Polylith.#2021-07-2016:01Edcool ... so if you need to debug the build script, your current workflow is to start a new repl from your "all the modules in the system" repl?#2021-07-2016:01seancorfieldMy personal REPL for work is started like this:
SOCKET_REPL_PORT=5000 clojure -Sforce -M:rebel:reflect:jedi-time:reveal:j14:classes:everything:dev:test:runner:dev/repl
👀#2021-07-2016:01Edah ... so no ... you've got them all on there#2021-07-2016:03seancorfieldOn a branch we have the :build alias above and a fully-migrated setup, where we run tests in the subprocess via java-command and process in our build script. On our main branch we still have :build with :extra-deps etc... and that's where I still need :build in my REPL startup command.#2021-07-2016:04seancorfield(hopefully that edit makes it less confusing)#2021-07-2016:06Ed😉 ... I'm easily confused#2021-07-2016:06Edthanks for sharing ....#2021-07-2016:06EdI'm just interested in the potential workflows with these tools#2021-07-2016:07seancorfieldOur dev setup is pretty fluid -- we've been changing our build/CI pipeline a lot in the last month or so, trying to align with tools.build.#2021-07-2016:08seancorfieldIn CI we have (on that fully-migrated build branch):
# ensure we have all the deps for our cache
- ../build/clojure/bin/clojure -P -A:build:everything:dev:test:runner
- FORCE_POLLING_ONLY=true
../build/clojure/bin/clojure -T:build all-tests-ci
- ../build/clojure/bin/clojure -T:build tag-build-and-upload-all
(we have the Clojure CLI vendored into our repo so we can rely on a known version on all servers)#2021-07-2016:11Edwhere I'm currently working have some python based tools that are used to build docker images and wrap terraform / aws calls, etc ... and there's talk of migrating away from python to something more clojurey since it's mostly building clojure apps ... babashka tasks is on the list ... and I'm just looking around at this to see what is offered 😉#2021-07-2016:15seancorfieldWe previously had a mix of bash scripts and some Clojure stuff. Now it's all Clojure (although we still have a legacy app that relies on ant for build/deploy 🙀).#2021-07-2016:15Edoh the horror#2021-07-2016:17borkdude@l0st3d doing a talk on babashka tasks in one hour and thirteen minutes at London Clojurians #2021-07-2016:17Edyes .. I know ... I'm afraid I can't make it 😞#2021-07-2016:18borkdudeIt will be recorded #2021-07-2016:20Edyeah ... I figured it would ... I've been ~stalking you on the youtubes~ watching your videos ... it's all much appreciated 😉#2021-07-2017:34seancorfield@alexmiller The 1.10.3.920 prerelease seems to be broken for -T?
(! 1203)-> /usr/local/Cellar/clojure\@1.10.3.912/1.10.3.912/bin/clojure -Sforce -T:build uberjars :projects '[api auth]'
Running: git describe --tags
Running: depstar on api
2021-07-20 10:32:33,489 INFO [hf.depstar.aot] - {} - Compiling api.main ...
2021-07-20 10:32:56,259 INFO [hf.depstar.uberjar] - {} - Building uber jar: projects/api/../../../build/uberjars/api-1.0.0.jar
Running: depstar on auth
2021-07-20 10:33:06,394 INFO [hf.depstar.aot] - {} - Compiling auth.main ...
2021-07-20 10:33:25,174 INFO [hf.depstar.uberjar] - {} - Building uber jar: projects/auth/../../../build/uberjars/auth-1.0.0.jar
(! 1204)-> !!:gs/912/920/
/usr/local/Cellar/clojure\@1.10.3.920/1.10.3.920/bin/clojure -Sforce -T:build uberjars :projects '[api auth]'
Unqualified function can't be resolved: uberjars#2021-07-2017:38Alex Miller (Clojure team)I don’t see that in local things I’ve done? Can you do something I can repro?#2021-07-2017:39seancorfieldIt looks like the fix for the -X regression removed "${exec_args[@]}" and the --aliases ... piece is missing now? Based on https://github.com/clojure/brew-install/commit/8b556085a5e23cbce0a5cbc55766f4f1018d7b10#2021-07-2017:39Alex Miller (Clojure team)No longer needed#2021-07-2017:39Alex Miller (Clojure team)Lots of changes in the handoff of this data #2021-07-2017:40Alex Miller (Clojure team)You may need to -Sforce possibly to remake basis#2021-07-2017:42seancorfieldOh... this is because I stashed a version of clojure.run.exec to implement our exec-command stuff!#2021-07-2017:42Alex Miller (Clojure team)I can't tell what you're doing here, but if your exec copy is involved, that is broken#2021-07-2017:42seancorfieldSorry...#2021-07-2017:42Alex Miller (Clojure team)yeah, I mentioend that to you yesterday :)#2021-07-2017:43seancorfieldRight. I didn't make the connection between updating the CLI and updating 😕#2021-07-2017:49seancorfieldAll fixed. Although now I can't easily tell the difference between a -X/`-T` invocation of build.clj and a -M invocation and my exec-command code doesn't work any more...#2021-07-2017:50Alex Miller (Clojure team)for what reason do you need to tell the difference?#2021-07-2017:51Alex Miller (Clojure team)-T can be seen as a pre-stage for -X that gathers the tool's lib/coord and its usage info from the deps.edn and then executes as a -X program#2021-07-2017:52Alex Miller (Clojure team)there is new stuff in the basis (:execute-args) that may be able to tell you something useful, but not sure what you need#2021-07-2017:53seancorfieldOur build script had a mode where you could invoke it with -M and "regular" arguments and it would run multiple functions with interspersed arguments as part of a transition from our old shell script.#2021-07-2017:55Alex Miller (Clojure team)not sure if you're writing more but I don't understand what the problem is from that#2021-07-2017:56seancorfieldAs a shortcut, the script didn't need a -main, it just read *command-line-args* at the end of the script 🙂 Now it needs a -`main` for that.#2021-07-2017:58seancorfieldI'm not sure what to do about invoking the subprocess via -X now since the updated exec.clj reads the basis from the system property location and that's where it tries to get :execute-args from -- so my trick of passing that system property through and being able to specify the additional aliases for exec-args no longer works.#2021-07-2018:00seancorfield(this is all on a branch so it's not a blocker -- but it does mean that we can't use that branch unless we roll back to CLI 1.10.3.912 and the previous exec.clj code)#2021-07-2018:00seancorfieldI hope TBUILD-6 is going to be implemented "soon" so we can move forward with running -X style subprocesses from the build script... 🙂#2021-07-2018:01Alex Miller (Clojure team)well I think where things are now is substantially better in a lot of ways so not planning on going back (change let me delete a lot of fiddly code in multiple places)#2021-07-2018:01Alex Miller (Clojure team)I'm not likely to work on TBUILD-6 soon#2021-07-2018:02Alex Miller (Clojure team)(partly because I have 3 vacations in the next 2.5 weeks) and partly b/c I think other stuff is higher priority#2021-07-2018:03Alex Miller (Clojure team)one possible way to go with it is to move exec into something consumable and the recent api changes make me feel better about that option, but that's a bunch of stuff to do#2021-07-2018:04seancorfieldYeah, definitely an improvement for the system as it stands. I may take the old exec.clj and put it under a different name and use it "as-is" which will allow us to continue working on exec-command without breaking what the CLI is doing with updates to exec.clj.#2021-07-2018:25seancorfieldOK, I copied the old exec.clj to ws.clojure.run.exec for now so we can continue with our exec-command experiment, and now the following three types of invocation all work:
# legacy command-line invocation:
clojure -M:build -m build test lowlevel test worldsingles-web
# interactive "build REPL":
clj -M:build -i build.clj -r
# new "tools" invocation:
clojure -T:build run-tests :projects '[lowlevel worldsingles-web]'
with the latter running tests in a subprocess via our exec-command stuff and that older clojure.run.exec code.#2021-07-2018:26seancorfield(to make the legacy invocation work, I had to add :paths ["."] to the :build alias because -T does that but -M does not)#2021-07-2018:30seancorfieldAnd just in case it sounds like I'm grumbling, I'll just reiterate that I really appreciate all the work you're doing on the CLI and the build tool chain -- thank you, @alexmiller -- and enjoy your vacations and I'll try to stop pestering you for a few weeks 🙂#2021-07-2018:35Alex Miller (Clojure team)io.github.clojure/tools.build {:git/tag "v0.1.4" :git/sha "169fef9"} is now available
• jar - add support for custom :manifest attributes
• uber - add support for custom :manifest attributes
• create-basis - make more tolerant of missing deps.edn file
• update tools.deps.alpha dependency to latest#2021-07-2023:03seancorfieldRelated to the new ability to reference a project via :git/tag and :git/sha -- since t.d.a checks that the tag and the short SHA match, that means that any docs for the project that give examples of installation have to be updated after the "release" (after the tag has been applied)... which means that any service, such as cljdoc, which fetches doc source from GH based on a tag will pull back the outdated docs with an old SHA. Any thoughts on how this could be addressed?#2021-07-2023:05seancorfield(currently, cljdoc requires a JAR release to clojars but some of us have been discussing the possibility of cljdoc importing direct from GH so this scenario becomes more likely)#2021-07-2023:28Alex Miller (Clojure team)for tools.build and test-runner, I tag, then update the changelog and readme with tag+sha#2021-07-2023:29Alex Miller (Clojure team)I don't think there's any way around that with the sha (although you could update the docs with the tag before you tag#2021-07-2023:30Alex Miller (Clojure team)(also note that in the special case of clj -Ttools install - the sha is not required and will be inferred from the tag)#2021-07-2023:35seancorfieldYeah, it only affects projects that expect to be used as source libs that have the short SHA in their docs -- and their docs are generated by an external system based on a tag.#2021-07-2023:35Alex Miller (Clojure team)one helpful tool might be to look up the sha (although then maybe you're just opening up to all the problems we're trying to avoid)#2021-07-2023:36Alex Miller (Clojure team)git ls-remote --short v0.1.4 for example#2021-07-2023:37seancorfieldI'll see what the cljdoc folks suggest. This came up for depstar since I document how to use it with tools.build and I wanted to use a source dep but then realized it would always have to be out of date...#2021-07-2023:37Alex Miller (Clojure team)it is definitely annoying#2021-07-2023:41seancorfieldHah, and I just shot myself in the foot by updating depstar's deps.edn file and referring to test-runner via the new coords and my CI failed because it's not on a recent enough CLI version 😐 Easy to get used to these new affordances!#2021-07-2023:57Alex Miller (Clojure team)I have done that over and over in the last few months#2021-07-2103:40markaddlemanIs there a magic incantation to get clj -X... 's :exec-fn to return a status code to the shell? I want to use clj -X... to execute my regression suite before a deploy but I need the bash script to check that the tests pass#2021-07-2103:43didibusIt returns whatever the program you call with -X returns I believe. So it depends on what it is you are calling with it#2021-07-2103:43markaddlemanThat's what I thought but it's not working. I must be doing something wrong. Thanks for the confirmation#2021-07-2103:43didibusAre you doing (System/exit 1) ?#2021-07-2103:44didibusI might be wrong, I really just assumed so, maybe there's something weird with -X wrapping things#2021-07-2103:45markaddlemanI'm not doing (System/exit ...) The function is just returning an integer#2021-07-2103:46didibusIf you return an integer, I think that just goes to output. I think you need System/exit to set the process status code#2021-07-2103:46markaddlemanThat makes sense. Thx#2021-07-2103:50didibusActually just returning an integer just has it go in the void lol. You need to print to have it go to the output. And (System/exit 0) or (System/exit 1) to set the status#2021-07-2103:51didibusI just tried#2021-07-2104:02seancorfieldThe idea is that a "failed task" should generally throw an exception (an ex-info for the best handling) and the CLI exec stuff will handle it.#2021-07-2104:03seancorfieldRight now the return value is not used. It was when you could chain -X functions (and I'm sure it will be again in the future -- like the -> macro).#2021-07-2104:04seancorfieldAll my -X functions return a hash map -- usually the hash map they were passed in, unless they have something to add it to. That makes them chainable/threadable. And they throw an exception if they "fail".#2021-07-2113:21markaddlemanThrowing an exception did the trick. Thanks!#2021-07-2104:30Alex Miller (Clojure team)-X/-T should either return 0 for success or 1 if an exception is thrown out of the function#2021-07-2104:30Alex Miller (Clojure team)so depends on the test runner function to some degree#2021-07-2104:35seancorfieldRight, but what was missing (in that thread) was understanding that what the invoked function returns does not affect the status -- I commented that you either throw an exception for failure or you just return data for success.#2021-07-2104:43didibusWould you say an explicit call to System/exit is bad practice inside a function meant to be called with -X or -T ?#2021-07-2104:43seancorfieldYes.#2021-07-2104:44seancorfieldSuch functions should either return data or throw exceptions. In a world where functions can be chained (threaded), those functions should probably return the data they were passed as an argument, possibly with additional keys (or perhaps they should remove some of the keys they were given).#2021-07-2104:46seancorfield(I think it's a bit odd to see some of the build functions just returning nil right now but maybe when we get threading back those functions will either get updated to return their input data, or functions that return nil will cause the threading code to pass the original input data along to the next function?)#2021-07-2105:11Alex Miller (Clojure team)a build is a thing you write for your project. if the build functions aren't parameterized, there's no reason to do so#2021-07-2105:34seancorfieldSure, the internal functions, but the entry points, which all get passed arguments... I guess it'll depend on how much folks would use threading/chaining -- assuming we get that feature back (I see :fns in the latest clojure.run.exec so it looks like we'll get it back?).#2021-07-2106:02Alex Miller (Clojure team)the entry points in most of the build scripts I've written are not parameterized#2021-07-2106:12seancorfieldRight, but they still have to accept a single argument? Because that's the hash map of stuff coming from the command line...#2021-07-2105:00didibusMakes sense, System/exit would kill the chain even on success otherwise#2021-07-2105:58didibusIt looks like -T fails when there is no build.clj file and using ns-defaults, but it works if using fully qualified on the command line:
{:paths ["src"]
:aliases
{:foo {:ns-default foo}}}
(ns foo)
(defn bar
[params]
(println "Foo Bar")
params)
> clojure -T:foo bar
Execution error (FileNotFoundException) at clojure.run.exec/requiring-resolve' (exec.clj:34).
Could not locate build__init.class, build.clj or build.cljc on classpath.
clojure -T:foo foo/bar
Foo Bar
#2021-07-2106:04seancorfieldThat makes sense to me. In the first case you are asking it to resolve build/bar and it can't because no build ns file exists.#2021-07-2106:05seancorfieldIn the second case you are asking it to resolve foo/bar and it loads a foo ns file and resolves bar within it.#2021-07-2106:06seancorfieldIt's not that "there is no build.clj file" so much as you are asking for a ns that doesn't exist -- it's just the same as doing clojure -M -m no-such and getting
(! 592)-> clojure -M -m no-such
Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate no_such__init.class, no_such.clj or no_such.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.
#2021-07-2106:07Alex Miller (Clojure team)where is build even coming from?#2021-07-2119:08didibusThat's what I'm wondering too.
The only thing is I did have it as ns-default build at first and then changed it, could it be some kind of caching issue?#2021-07-2119:09Alex Miller (Clojure team)try -Sforce ?#2021-07-2121:09didibus-Sforce did the trick#2021-07-2106:08tianshuHi, I want to use tools.build with this
io.github.clojure/tools.build {:git/tag "v0.1.4" :git/sha "169fef9"}
But tools.deps says Error building classpath. Library io.github.clojure/tools.build has missing :sha in coordinate.
My CLI version is Clojure CLI version 1.10.3.855 . Is there something I missed?#2021-07-2106:09Alex Miller (Clojure team)that's the stable release - the new git dep coords are only in the latest prerelease#2021-07-2106:09Alex Miller (Clojure team)1.10.3.920 is latest#2021-07-2106:11tianshuGot it, thanks!#2021-07-2106:28tianshuWhere can I find the latest CLI source code, is it part of Clojure project?#2021-07-2106:42tianshuI found this. linux-install-1.10.3.920.sh#2021-07-2111:48Alex Miller (Clojure team)That’s the installer#2021-07-2106:08seancorfieldOh, I thought he had :ns-default build but maybe not?#2021-07-2106:08seancorfield@doglooksgood You need the prerelease version for that format of coordinate#2021-07-2106:09tianshuA prerelease version of tools.deps?#2021-07-2106:09dpsuttonwhat's the best way to get a repl up with build.clj on the classpath? is it just clj -A:build and then eval the ns form? Load-file? Or is there a way to essentially get "." on the classpath with -T but not invoke a build function. Pretty simple problem to solve just making sure i didn't overlook something obvious#2021-07-2106:11seancorfield@doglooksgood The CLI. You need 1.10.3.912 at least.#2021-07-2106:11tianshuOkay, thanks!#2021-07-2106:11seancorfield@dpsutton what we do at work is clj -M:build -i build.clj -r#2021-07-2106:12seancorfieldThen build is already loaded and you can just call (build/some-fn {})#2021-07-2117:20Alex Miller (Clojure team)Clojure CLI prerelease 1.10.3.929 available
• TDEPS-189 - Port -T changes to Windows scripts
• Script cleanups in bash#2021-07-2117:20Alex Miller (Clojure team)Nothing big there, but if Windows users want to kick the tires, go for it#2021-07-2117:25seancorfieldIs there anything non-Windows users might need to test in this update?#2021-07-2117:39Alex Miller (Clojure team)nothing specific, just cleaned up some dead code mostly in the bash#2021-07-2119:15seancorfield@alexmiller With tools.build.api/copy-dir, the :replace hash map would be stuff like {"{{version}}" "1.2.3", "{{username}}" "seanc"} -- a hash map with strings for keys and values?#2021-07-2119:22Alex Miller (Clojure team)yes#2021-07-2119:30seancorfieldI realized that with a small wrapper around tools.build I can create a really nice, stripped down version of clj-new 🙂#2021-07-2119:30seancorfield(and that answer just confirms what I was thinking about that)#2021-07-2121:35didibusIt seems that copy-file will overwrite a directory as a file, which I mean maybe is a cool feature, but I feel its unintuitive behavior, I've never used a linux command that behaved like this:
e.g.:
;; output was a directory, but it will now become the tempo.clj file after calling this copy-file
clojure -T:build-api copy-file :src '"./src/tempo.clj"' :target '"./output"'
#2021-07-2121:40Alex Miller (Clojure team)You expected it to be in target?#2021-07-2121:42didibusYa, my expectation was if you don't specify the filename in :target it uses the same filename as the source file. But I'd be okay with just failing saying output is a folder. Overwriting the folder to be the file seems dangerous#2021-07-2121:45didibuscp in linux does the former#2021-07-2121:46didibusAt least it seems copy-file fails if the directory is not empty#2021-07-2121:49didibusAlso weird is if do:
;; src is a directory with files in them, output is an empty directory
clojure -T:build-api copy-file :src '"./src"' :target '"./output"'
Then copy-file succeeds, but nothing seemingly happens. Like output is still an empty directory, and src is stil a directory with files in it.#2021-07-2122:46Alex Miller (Clojure team)as the docstring says, "Copy one file from source to target, creating target dirs if needed." perhaps you are looking for copy-dir?#2021-07-2200:42didibusNone of the behavior I describe do this though?#2021-07-2200:43didibusThe latter one doesn't copy or create anything, the former overwrites an existing folder#2021-07-2200:49didibusThe latter I tested just to see what happens in that edge case. I think I'd expect a failure in that case, not a silent do nothing but succeed.
Though well at least it's not destructive. But the former feel wrong, to overwrite a directory with a file? At least the doc-string I think should be explicit of it. Cause it's a very easy fat-finger incident I think#2021-07-2200:57didibusAnyway, just reporting on the behavior in case you weren't aware of those edge cases. If its by-design that's fine too.#2021-07-2216:45seancorfield@U0K064KQV While I too am a bit surprised that copy-file overwrites an (empty) directory, I'm also sympathetic to the fact that it does it because a) it's called copy file, b) it has :src and :target args not :target-dir. I think my preference would to add an :overwrite arg, that defaults to true and could be set to false so it would throw an exception if :target exists in any form.#2021-07-2216:46seancorfieldThe default behavior would still be to silently overwrite :target, regardless of what it is, but if you want to be paranoid, you could use :overwrite false, and you could decide whether to delete the :target and re-copy or whatever.#2021-07-2216:48seancorfieldOn the same subject, I'd also like to see :recurse false and :throw-on-missing true as options on delete -- but all of these are just "nice to have" knobs and dials as far as I'm concerned: the core functionality does what is needed for build programs -- and you have copy-file is just this Java call: https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/util/file.clj#L59 -- so this is the semantics of Java's standard library itself.#2021-07-2222:07seancorfieldAnd these are the only options it provides: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/StandardCopyOption.html @U0K064KQV#2021-07-2222:09seancorfieldBottom line: t.b.api/copy-file does exactly what Java's Files/copy command does, with the sole addition of calling (.mkdirs target-file) to auto-create any missing directories in the :target path.#2021-07-2400:46didibusI was thinking might be different because it exposed a:
> sun.nio.fs.UnixCopyFile/copy exception
And the name of that made me think it could be OS specific.
Seems from the javadoc that file attributes are OS specific and undefined how they will copy, but the rest should be same.#2021-07-2400:56didibusMeh, ok, I guess this is just how nio copy works with REPLACE_EXISTING option, java is dumb 😛#2021-07-2401:34seancorfield"java is dumb" good, "clojure is dumb" bad :rolling_on_the_floor_laughing:#2021-07-2209:44borkdudeI have looked into making bb compatible with some of the namespaces of tools.build, most of them work, except those which depend on tools.deps.alpha (since it contains many dependencies that can't run from source with bb, like the mvn and s3 stuff).
What I noticed is that some namespaces use tools.deps.alpha but only one function from it: join-classpath which is a pretty basic function which doesn't need any mvn/s3 stuff.
$ rg "deps/"
clojure/clojure/tools/build/tasks/compile_clj.clj
50: deps/join-classpath)
clojure/clojure/tools/build/tasks/create_basis.clj
17: "Wrapper for deps/create-basis, but ensure relative paths are resolved
30: (deps/create-basis params))))
clojure/clojure/tools/build/tasks/process.clj
93: deps/join-classpath)]
(https://github.com/clojure/tools.deps.alpha/blob/dec88af58b9e6f6e2f1949c94fbd1d93eecc5959/src/main/clojure/clojure/tools/deps/alpha.clj#L611)
In fact, it seems only the create-basis namespace really needs tools.deps.alpha and the rest could just use join-classpath function that could be "inlined" into tools.build itself.
This might also benefit load time if you're not going through create-basis in a build step (perhaps you already have the basis in a file somewhere for example), avoiding to load tools.deps.alpha in that case.
Just wanted to drop this thought here in case it makes sense for other use cases as well.
Another approach is to "podify" the entire tools.build library which should be possible now that there is a native version of tools.deps.alpha possible (https://github.com/babashka/babashka/issues/951).#2021-07-2216:41seancorfield@borkdude create-basis itself is two useful parts: the find/read/merge EDN file part (I've had variants of that to access alias data in several projects), and the dependency resolution (download/clone/etc). Only the second part needs t.d.a So it would be nice to have a small lib with just the first part -- that many tools could use, if they don't need the whole dep/res stuff.#2021-07-2216:42seancorfield(I'd also like to see the EDN arg parsing and require/resolve portion of clojure.run.exec as a separate lib too -- but that's a separate discussion 🙂 )#2021-07-2216:50Jakub Holý (HolyJak)Is there a way to force a particular minimal version of clojure CLI? I have few times run into problems where my projects failed on other people's computers because they had a too old version of the CLI 🙏
So something like :clijure-cli/min-version 1.10.3.855 in deps.edn that would trigger a warning when run with an older version of the CLI would help.#2021-07-2216:54seancorfieldI run into that a lot too -- and have shot myself in the foot a couple of times by using a new CLI/t.d.a feature in a script and then having it fail on a server that hasn't been updated. I think it's important to recognize that we now have three "cogs" in that machine that tooling might be version-sensitive about: the CLI itself, the version of t.d.a being used, and the version of Clojure itself. Right now, the latter is controllable by specifying a minimum org.clojure/clojure version in your project but the other two are not (and I've been bitten by both but mostly by the CLI version).#2021-07-2216:56seancorfieldNow, the problem is that the CLI itself isn't checking deps.edn and if the cache files are not stale will not even run the step to build the basis etc (which is what reads/analyzes deps.edn).#2021-07-2220:40jjttjj~Hi are there any known issues with using ~
Cloning:
~printed and then it hangs forever. I can clone the project from the command line successfully just using ~ wait I think I might see what the issue is now#2021-07-2221:27jjttjjFor what it's worth I had to change GIT_SSH env var to "C:\Windows\System32\OpenSSH\ssh.exe" (it was plink before). This didn't immediately fix things but after bashing on the keyboard a bunch (not ultimately changing anything else, as far as I'm aware), things just randomly started working#2021-07-2220:44borkdude@jjttjj you can try to debug using GITLIBS_DEBUG set to true#2021-07-2221:24jjttjjThank! That's definitely useful#2021-07-2319:17jjttjjIs there a proper way to use the project root path in :paths? :paths ["."] used to work but now seems deprecated. This is useful for using scripts from gist#2021-07-2319:19borkdude@jjttjj does :paths [""] work?#2021-07-2319:20jjttjjNope, same deprecated message#2021-07-2319:21seancorfield:paths ["."] works just fine. In fact that's specifically what -T does behind the scenes.#2021-07-2319:21jjttjjWARNING: Use of :paths external to the project has been deprecated, please remove: C:\my\project\.clojure\local
#2021-07-2319:21seancorfield@jjttjj You need to provide a bit more context and perhaps a repro example.#2021-07-2319:22jjttjjAh, sorry, disregard! I realize what this is. Coming from a setting in my root deps.edn not this one.#2021-07-2319:22borkdude{:paths [""]} also worked for me#2021-07-2319:22jjttjjSorry about that, thanks for the help#2021-07-2319:22seancorfieldHmm, never thought to try that... interesting...#2021-07-2319:23borkdudeit's probably better to use the dot though#2021-07-2417:35souenzzoHey
I'm having a wired exception. It is probably something with localstate, like removing .m2 .gitlibs should fix it
But it's a crypt exception, I think that we can try to understand and improve this error message. At least, know which file cause this exception
https://gist.github.com/souenzzo/7ab1a37ad172efef71a6765b13413704#2021-07-2417:50seancorfield@souenzzo My first thought was that it might be a version conflict on org.clojure/data.xml in your environment, since it says that a Var in that lib isn't defined -- but when I looked at blame on clojure.data.xml that element-nss has been in there for six years at this point so that theory doesn't seem terribly likely.#2021-07-2417:52seancorfieldDid you succeed in making the error go away? Can you reproduce it?#2021-07-2417:58seancorfieldCould there be an 0.0.8 version of data.xml in your transitive dependencies? That didn't have those Vars (but it didn't have some of the same namespaces at all).#2021-07-2419:39souenzzo(some #{"/home/souenzzo/.m2/repository/org/clojure/data.xml/0.2.0-alpha6/data.xml-0.2.0-alpha6.jar"}
(string/split (System/getProperty "java.class.path")
#":"))
=> "/home/souenzzo/.m2/repository/org/clojure/data.xml/0.2.0-alpha6/data.xml-0.2.0-alpha6.jar"
#2021-07-2419:45souenzzoIt's something with the fact of tools.build being in my classpath#2021-07-2420:18seancorfieldYeah, having tools.build be "in addition" instead of "replacing" can lead to some problematic situations, but mostly around tools.deps.alpha in our experience.#2021-07-2420:19seancorfieldThat's why we run tests in a subprocess and avoid having our code on the classpath when using tools.build.#2021-07-2420:20seancorfieldI'd have to see more of your deps.edn file and how you're invoking the build script to be able to provide any more specific feedback, I suspect.#2021-07-2420:41seancorfield@jgomo3 Saw your post on ClojureVerse -- if you hop into #tools-deps here, you can get more "real-time" assistance on tools.build etc.#2021-07-2423:36greg@seancorfield I've seen today one of your talks from the Jan this year about REPL development. It included demoing of Reveal and add-libs. The idea of adding libs in a runtime is so nice, but I've got a question. How do you deal with adding newer version of libs? Or loading a fork?
Today I had this issue that I loaded a lib from github, then I forked it, modified it, updated deps with my fork, called add-libs on updated deps, but at the end, I still had the old version at hand.
Also, when I checked (:libs (clojure.tools.deps.alpha.repl/current-basis)) it still contained the info about the old dep. At the end I had to restart REPL as usual, but it would be nice to keep it on if possible. How do you tackle this kind of issue?#2021-07-2423:51seancorfieldYou can't add a newer version of a library you have already loaded, without starting a new isolated Clojure instance (which you can do without restarting the JVM but it's non-trivial). If I'm iterating on a library locally for which I have the source, I just use "load file" in my editor and eval every form I change -- that will override any previously loaded (older) version, just like evaluating any other source code.#2021-07-2423:54seancorfieldI don't find I need to update libraries all that often while I'm actually developing.#2021-07-2423:54gregOk, that is actually not the bad idea. The IDE can fetch new deps, and then you can reload necessary files.#2021-07-2423:56seancorfieldIt feels like it's taken me a long time to settle on a nice workflow in the REPL. I'm very productive with it now and my REPLs all tend to be long-lived at this point (often weeks, sometimes months).#2021-07-2521:55stuartrexkingHow do I add an additional alias to clj -X:deps tree. The alias dev as extra-deps, but they don’t show up in the tree when I clj -X:deps:dev tree#2021-07-2521:57seancorfieldUse -Stree instead of -X:deps tree#2021-07-2521:58seancorfieldclojure -Stree -A:dev for example @stuartrexking#2021-07-2522:05borkdudeIt means -A:foo:bar:baz#2021-07-2522:06borkdudeif you do -X:foo:bar:baz for example, the deps from aliases foo and bar are included on the classpath. also those of baz, but you will only eXecute a function from the baz alias.#2021-07-2522:06borkdudeThe last one wins.#2021-07-2522:09seancorfieldAlthough :exec-fn is "last one wins", :exec-args are merged across all the aliases you specify. That can be a very useful trick to use aliases to modify the behavior of an exec function.#2021-07-2522:11seancorfieldSo if each of :foo, :bar, and :baz have both :exec-fn and :exec-args, then the function that gets invoked will be :baz's :exec-fn but with a merged hash map of the :exec-args from all three aliases.#2021-07-2522:13seancorfieldNo. For a brief while you could specify multiple functions tho' but that went away.#2021-07-2522:13borkdudewas that only in a pre-release?#2021-07-2522:13borkdudethe threading thing?#2021-07-2522:13seancorfieldclojure -X:foo:bar:baz my.first/func another/func third/func -- was that only in a prerelease? Dunno. I'm always using one of the latest prereleases so I can't remember.#2021-07-2522:14seancorfieldI think it'll come back at some point but there were ambiguities in terms of argument parsing...#2021-07-2522:15borkdudeIn general when you write :foo:bar:baz it means you invoke :baz (exec, main, tool) but with additional deps from aliases foo and bar#2021-07-2522:17seancorfieldThe -X behavior with multiple aliases is not documented. The behavior for :main-opts (and thus -A/`-M`) is documented, as it is for :paths and several other things.#2021-07-2522:18seancorfieldSearch this page https://clojure.org/reference/deps_and_cli for multiple to see the other documented behavior.#2021-07-2522:20seancorfieldThere's definitely a lot of "assumed knowledge" in the deps.edn/CLI docs, unfortunately. I think partly because it has changed a lot over time and partly because a lot of the early users have always been in this channel providing feedback, even before features got implemented, so some of those discussions just never made it into the docs...#2021-07-2522:26seancorfieldSeveral of the (older) sections on that reference page have bulleted lists with the keys they relate to and a bullet that says "If multiple alias maps with these keys are activated..." but the -X stuff is introduced in a different way near the top of the page and then the argument parsing is described in detail at the bottom of the page, but there's no simple bulleted list of its keys and behavior.#2021-07-2522:27seancorfield(and now that I read it, I see concatentating is misspelled five time on that page)#2021-07-2522:27seancorfieldI've never noticed that before and I've read that page so many times...#2021-07-2522:28borkdudeperhaps it's a new Clojure word?#2021-07-2522:29seancorfieldI think adding a Function execution section to the overall Execution section, between Main execution and Prepare for execution would be a good start. It could list the four keys that affect function execution (`:exec-fn`, :exec-args, :ns-default, and :ns-aliases) and the "If multiple alias maps with these keys are activated..." bullet.#2021-07-2522:31seancorfieldhttps://clojure.org/guides/deps_and_cli#override_deps shows an example#2021-07-2522:32seancorfieldI think that's the only example in the docs tho'...#2021-07-2522:32seancorfieldEasy to miss, two thirds down that page.#2021-07-2522:34seancorfieldProbably, since that would be the first "mention" of multiple aliases assuming someone reads the entire guide first then then entire reference manual...#2021-07-2522:35seancorfieldHow I start my REPL:
SOCKET_REPL_PORT=5000 clojure -Sforce -M:rebel:reflect:jedi-time:reveal:j14:classes:everything:dev:test:build:runner:dev/repl
👀#2021-07-2522:36seancorfieldNo, aliases can't be amalgams of other aliases (a feature lots of people have requested).#2021-07-2522:36seancorfieldMaybe I should write a blog post about my REPL? :thinking_face:#2021-07-2522:37seancorfield:everything:dev:test:build:runner all comes from our work project deps.edn and the other aliases from my dot-clojure deps.edn (in the repo on GitHub). Just FYI.#2021-07-2522:38seancorfieldYeah, I've been putting off some chores I should go and do (it's just too hot here).#2021-07-2607:44flowthingThere seems to be a weird issue related to -X and futures with Clojure CLI 1.10.3.929:
$ clojure --version
Clojure CLI version 1.10.3.929
$ clojure -Srepro -X clojure.core.server/start-server :name '"server"' :port 5555 :accept clojure.core.server/repl :server-daemon false
$ clojure -Srepro -J-Dclojure.server.repl="{:port 6666 :accept clojure.core.server/repl}"
$ nc localhost 5555
user=> (def x (future 1))
Execution error (RejectedExecutionException) at java.util.concurrent.ThreadPoolExecutor$AbortPolicy/rejectedExecution (ThreadPoolExecutor.java:2057).
Task
That is, future throws when connecting to a socket REPL server started via -X, but not when connecting to a socket REPL server started via the clojure.server.repl system property.#2021-07-2607:47flowthingI cannot reproduce the issue with 1.10.3.875.#2021-07-2607:48flowthingThe -X invocation doesn't work at all on 1.10.3.882, so I can't (easily) narrow it down any further.#2021-07-2614:25seancorfieldI can repro but can't explain the difference @U4ZDX466T You should put this up on http://ask.clojure.org so it doesn't get lost as Slack drops history.#2021-07-2614:25flowthingSure, will do later today.#2021-07-2609:15gregWhat's your take on putting aliases within project deps.edn vs home's .clojure/deps.edn? I understand that some of the tools vary among Clojure devs (REBL/reveal, srepl/nrepl, IDEs/editors), so make sense to keep them locally specific, but lots of stuff could be shared, plus it is nice to have a sensible defaults for any newcomers. What your thoughts on this one? What kind of aliases do you shares within your projects' deps.edn?#2021-07-2614:23seancorfield@grzegorz.rynkowski I think the project deps.edn should contain only aliases relevant to the project itself. If you want a "shared .clojure/deps.edn file", create a separate repo for that (and, now, any standard "tools" you want them to have installed, if you're starting them on the latest prerelease), and have them set up their environment based on that.#2021-07-2614:28borkdudeI don't know the right answer to "I want to check my emacs settings into your open source project so it does the right thing when I open CIDER", but I generally tell people to just document this for whoever wants to use it, instead of committing it in the project#2021-07-2614:29borkdudeOn the the other hand, I sometimes see some IDEA settings in open source Clojure or Java projects 🤷#2021-07-2614:30borkdudeyeah#2021-07-2614:30borkdudewhat if the next person wants a slightly different .idea or .gitignore?#2021-07-2614:31borkdudeBut when it comes to .lsp or .clj-kondo config, I do think it belongs into the project, since it's about the code itself and improves the tooling related to the code.#2021-07-2614:31borkdudeI think there might be a grey area there#2021-07-2614:35borkdude@deleted-user as an aside: I avoid the usage of the magic "jack-in" stuff#2021-07-2614:35borkdudeI always start a REPL in a terminal and then connect to them manually#2021-07-2614:35borkdudewhy? because I don't want to have my dev process as a child of whatever editor. when I close my editor or open a too big of an XML file, I have to restart my dev process. too brittle.#2021-07-2614:36borkdudealso I don't want to rely on whatever deps gets injected by the tooling.#2021-07-2614:45seancorfieldMy team start their REPL and then connect their editor to it. It's definitely our preferred approach. Partly because REPLs tend to outlive the editor and partly because you know exactly how the REPL is getting started instead of some "jack-in magic".#2021-07-2614:45borkdudeThat's what I said ;)#2021-07-2614:45seancorfield(and this really isn't about #tools-deps stuff -- greg's original Q was)#2021-07-2614:49borkdudeI think I still don't understand lein profiles#2021-07-2614:49ghadiPretty sure no one does#2021-07-2614:49borkdudealthough I am using them, they are a bit magic when it comes to whatever profile is active when#2021-07-2614:50borkdudein deps.edn this is fully explicit and I like that#2021-07-2614:53borkdudeA jar doesn't have system properties set. You have to set those when you invoke the jar, I think?#2021-07-2614:53borkdudeunless you use some properties file and a tool that reads it perhaps?#2021-07-2614:53borkdudeyou can use environment variables for this#2021-07-2614:55borkdudeJava properties also work#2021-07-2614:55borkdudebut it depends when you capture and process those#2021-07-2614:55dpsuttonfor those dev tooling, i'd probably just check if those namespaces were available, and if so, use the middleware with requiring resolve. Then jars wouldn't have those dev deps and you don't even need to configure anything#2021-07-2614:57borkdude@deleted-user you want specific ring middleware for development, is this the problem?#2021-07-2614:58seancorfieldI would put the dev-specific stuff in a :dev alias and use that when developing (and it would be omitted when building the JAR).#2021-07-2614:58borkdudeI don't see how this related to lein or deps.edn specifically, but there are ways to get this done through a dev profile/alias#2021-07-2614:59seancorfieldJack-in can be told to use aliases.#2021-07-2615:00seancorfieldBut my advice would be to stop using "magic".#2021-07-2615:00borkdudeI don't recommend the jack-in stuff but I think you can configure the tooling to use a specific alias ;)#2021-07-2615:00seancorfieldIt's much harder to optionally exclude stuff than to optionally include other stuff.#2021-07-2615:01borkdudethe CIDER docs will probably have some entry for this#2021-07-2615:01borkdudecalva also has something for this, I bet#2021-07-2615:03dpsuttonyou can ask in #clojure or somewhere. we can help you through things. But seems like this is getting towards generic project management rather than specific tools-deps questions#2021-07-2615:03borkdudehttps://github.com/clojure-emacs/cider/issues/2396#2021-07-2615:03borkdude( I googled for: CIDER dev alias clojure )#2021-07-2615:04borkdudeok#2021-07-2615:06pezYou can also ask in #calva , w/o the jack-in hate added to the answers, even. 😀#2021-07-2615:06seancorfieldRe: JAR building -- if you can get to the place where you use -M:dev for dev/test and -M:prod for production, you can use the :prod alias when building the JAR.#2021-07-2615:09borkdude@deleted-user if you need to persist settings in a jar, the only way is to flush them to a file in the resources, e.g. under META-INF/my-tool/settings.edn#2021-07-2615:09borkdudeor to assemble some clojure file using templating or so#2021-07-2615:11borkdudewhen you (def foo (System/getProperty "build.time-property")) this won't be captured and written into the jar somehow#2021-07-2615:11borkdudenext time you run the jar, the property is gone#2021-07-2615:14seancorfield(for what it's worth, we run our code in dev/test, the same way it would run in production -- without any special middleware etc -- partly to avoid these problems and also, mostly, to avoid "works-in-dev, break-in-production" issues)#2021-07-2622:07dpsuttonwas gonna ask this on http://ask.clojure.org but i don't see a category for tools.build in the dropdown and that's giving me pause if that's a good place for prerelease version. But i'm observing the following error
> /var/folders/h/3fpgc8pj4fb2w3mfqfl59fr0000gn/T/uber10889152638484484/license/LICENSE (Not a directory)
and the stacktrace
build=> (b/uber {:class-dir class-dir
:uber-file "uberjar.jar"
:basis basis})
FileNotFoundException /var/folders/_h/3fpgc8pj4fb2_w3mfqfl59fr0000gn/T/uber5850854519152411962/license/LICENSE (Not a directory)
java.io.FileOutputStream.open0 (FileOutputStream.java:-2)
java.io.FileOutputStream.open (FileOutputStream.java:298)
java.io.FileOutputStream.<init> (FileOutputStream.java:237)
java.io.FileOutputStream.<init> (FileOutputStream.java:187)
clojure.tools.build.tasks.uber/explode (uber.clj:66)
clojure.tools.build.tasks.uber/explode (uber.clj:53)
clojure.tools.build.tasks.uber/uber/fn--16677 (NO_SOURCE_FILE:636)
clojure.core/run!/fn--8813 (core.clj:7717)
clojure.core.protocols/naive-seq-reduce (protocols.clj:62)
clojure.core.protocols/interface-or-naive-reduce (protocols.clj:72)
clojure.core.protocols/fn--8181 (protocols.clj:169)
clojure.core.protocols/fn--8181 (protocols.clj:124)#2021-07-2622:14seancorfieldOut of curiosity, if you add depstar as a library, and use hf.depstar.api/uber instead of t.b.api/uber, does it work? (It should be API-compatible)#2021-07-2622:15seancorfieldPer https://cljdoc.org/d/com.github.seancorfield/depstar/2.1.267/doc/tools-build-usage if you aren't sure what I mean...#2021-07-2622:21dpsuttoni'll have to try unfortunately. but i'm quite interested in that#2021-07-2622:22dpsuttoni suspect this is happening when another license/LICENSE exists already#2021-07-2622:22seancorfieldCould be. I think there's already a bug open against t.b.a/uber for problems with multiple instances of a file.#2021-07-2622:23seancorfieldIt doesn't concatenate license or notice files right now (it should). It doesn't handle log4j2 plugin cache files either. But I think that's all on the roadmap.#2021-07-2622:23dpsuttonoff to jira#2021-07-2622:23seancorfieldThe end goal is, in theory, that depstar can be retired in favor of t.b.a 🙂#2021-07-2719:33dpsuttondepstar worked a treat. i think the filesystem as an intermediate can be tricky when going from case sensitive jars, possibly case sensitive FS, back to case insensitive jars#2021-07-2720:10seancorfield@U11BV7MTK If you're on Windows, try building an uberjar with tools.build for a project that depends on uncomplicate/neanderthal 🙂#2021-07-2720:11seancorfieldIt contains an aux.clj file, which is a reserved name on Windows -- so you can't explode the neanderthal library to a directory and then copy it all into a JAR.#2021-07-2720:11seancorfieldhttps://github.com/seancorfield/depstar/issues/8 -- that was a big part of why I stopped doing that step.#2021-07-2720:14seancorfield(I haven't checked whether Neanderthal has renamed that namespace - it might have)#2021-07-2720:17dpsuttonhaha that is not fun at all 🙂. i don't have access to a windows box unfortunately though#2021-07-2720:26seancorfieldI do but it isn't set up for Clojure on the Windows side -- only on the WSL2 side, otherwise, I'd verify that the above issue exists with tools.build and log a bug if it still happens...#2021-07-2822:42blueberry@U04V70XH6 I renamed aux.clj to auxil.clj long time ago (due to this unfortunate Windows behavior).#2021-07-2822:48seancorfield@U086AG324 Good to know! Thank you! It's weird that after all these decades, Windows still treats several filenames as reserved...#2021-07-2823:06dpsuttoni think that's really cool of them to take backwards compatibility so seriously. gotta respect that com binaries from the 80s still run on windows 10#2021-07-2823:22seancorfieldlooks for an emoji for stagecoach or something similarly old... 🙂#2021-07-2622:08dpsuttonit seems this is from the jar .m2/repository/xml-apis/xml-apis-ext/1.3.04/xml-apis-ext-1.3.04.jar which has an entry for license/LICENSE.#2021-07-2622:12seancorfieldI'd say put it on ask with the tools.deps category...#2021-07-2622:12dpsuttoni'm not seeing that here#2021-07-2622:12dpsuttonor is that a tag?#2021-07-2622:12seancorfieldCategory Contrib#2021-07-2622:13seancorfieldAnd I see tools.build under Contrib so you should be able to tag it that way.#2021-07-2622:13dpsuttonyou are 100% correct. thanks sean#2021-07-2715:51dpsuttonI've narrowed the problem down. One jar has an entry license/LICENSE and another jar has a LICENSE. When tools.build attempts to copy the contents of the jars it blows up. osx is case insensitive so there cannot be a file LICENSE and a folder license. But removing this issue just leaves how to resolve when a jar has a file named X and another jar has a folder named X.#2021-07-2715:55dominicmOh, because tools.deps just makes a folder and converts that to a jar. @seancorfield I'm not sure how depstar works, and your code is probably more straightforward than mine (mine got a bit fancy), but in pack.alpha jars are created using outputstreams and there's no intermediate filesystem required. That would help with having consistent behaviour for case sensitivity that matches the underlying format.#2021-07-2715:56dominicmPack got fancy because it supports generating jars within jars (or jars within zips for lambdas) and it will parallelize that as much as possible.#2021-07-2715:57dpsuttoni think depstar just consumes directly from the jar as a source without the intermediate local filesystem copy#2021-07-2715:57dpsutton(read before typing. sorry)#2021-07-2716:28seancorfieldRight, depstar copies everything directly from wherever it is (directory or JAR) into the target JAR (as a ZipFileSystem, as I recall). I don't know what depstar would do if it finds license (file) and license/LICENSE (folder containing file), but I think the ZipFileSystem is case-sensitive even on macOS, in which case a LICENSE file won't conflict with a license/LICENSE directory?#2021-07-2716:31didibusOn compile task what exactly does :filter-nses options do? Will it make it so only classes with the prefix are included in the class-dir? Or will it make it so only namespaces with the prefix are compiled which means the resulting class-dir will also include all transient classes which might not start with the prefix?#2021-07-2716:37seancorfieldI dunno. I'd have to look at the source code 🙂#2021-07-2716:37seancorfieldOr maybe try it out on a project to see what it does... 🙂#2021-07-2716:37Alex Miller (Clojure team)The former#2021-07-2716:38seancorfieldDoes it actually filter on the namespace or the class name?#2021-07-2716:40Alex Miller (Clojure team)Namespace prefix is what you specify#2021-07-2720:28jjttjjAny tips for what I should look for in my clj -Stree output when I have some dependency issue?
When I add tech.ml.dataset to my project, when I'm also using aleph as a webserver and make a web request, I get the error
16:16:39.778 [manifold-pool-2-1] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
There is no immediate connection in the deps tree between tech.ml.dataset and aleph or manifold#2021-07-2720:29seancorfield@jjttjj What makes you thing it's a dependency problem?#2021-07-2720:29seancorfieldIs it purely the addition of that dependency with no code changes at all?#2021-07-2720:30jjttjjexactly#2021-07-2720:30jjttjjIf I remove the dependency it works again#2021-07-2720:30seancorfieldOK, and what's in your deps.edn file for the dependencies?#2021-07-2720:31jjttjjIt's a lot of stuff#2021-07-2720:31jjttjjWondering if there's a good way to narrow it down a bit#2021-07-2720:31seancorfieldCan you just share the tech.ml.dataset dep and the aleph and manifold deps? Then I can try to repro#2021-07-2720:32jjttjjSure one sec I'll throw that together#2021-07-2720:35hiredmaneasier just to share the rest of the stacktrace#2021-07-2721:08jjttjjOh yeah, here it is:
17:06:57.893 [manifold-pool-2-3] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
at clojure.lang.RT.contains(RT.java:853)
at clojure.core$contains_QMARK_.invokeStatic(core.clj:1494)
at clojure.core$contains_QMARK_.invoke(core.clj:1486)
at ring.middleware.flash$flash_response.invokeStatic(flash.clj:21)
at ring.middleware.flash$flash_response.invoke(flash.clj:14)
at ring.middleware.flash$wrap_flash$fn__100872.invoke(flash.clj:39)
at ring.middleware.session$wrap_session$fn__101193.invoke(session.clj:108)
at ring.middleware.keyword_params$wrap_keyword_params$fn__101239.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__101297.invoke(nested_params.clj:89)
at ring.middleware.multipart_params$wrap_multipart_params$fn__101595.invoke(multipart_params.clj:171)
at ring.middleware.params$wrap_params$fn__101619.invoke(params.clj:67)
at ring.middleware.cookies$wrap_cookies$fn__101072.invoke(cookies.clj:214)
at ring.middleware.absolute_redirects$wrap_absolute_redirects$fn__101792.invoke(absolute_redirects.clj:47)
at ring.middleware.content_type$wrap_content_type$fn__101740.invoke(content_type.clj:34)
at ring.middleware.default_charset$wrap_default_charset$fn__101764.invoke(default_charset.clj:31)
at ring.middleware.not_modified$wrap_not_modified$fn__101721.invoke(not_modified.clj:61)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:384)
at aleph.http.server$handle_request$fn__55977$f__48214__auto____55978.invoke(server.clj:170)
at clojure.lang.AFn.run(AFn.java:22)
at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:62)
at manifold.executor$thread_factory$reify__48096$f__48097.invoke(executor.clj:47)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:832)
Having trouble getting the minimal repro to fail this way at the moment#2021-07-2721:12hiredmanthe flash and session stuff is all tied to browser state#2021-07-2721:13hiredmanso my guess is the http://tech.ml depedency has nothing to do with it, what triggers it is if your browser has a session cookie set and tries to send it or something#2021-07-2721:25hiredmanhttps://github.com/clj-commons/aleph/issues/254#2021-07-2721:25hiredmanlooks like ring-session has(had?) incompatibilities with aleph in the past#2021-07-2723:31jjttjjI hope this isn't too off topic, but here's a minimal example of the error.
It does have to do with the ring session stuff it seems, I'm just confused by why the dataset repo would cause this
https://github.com/jjttjj/aleph-dataset-bug-repro1
Here's a diff of the -Stree with and without the dataset dep
https://www.diffchecker.com/JdvUqlE7
But excluding the ones that the dataset diff pulls in didn't change anything#2021-07-2723:37seancorfieldIt brings in org.ow2.asm/asm 9.0 instead of 5.2 -- the latter is a dependency of core.async#2021-07-2723:38jjttjjTrue but adding these exclusions didn't fix the particular error https://github.com/jjttjj/aleph-dataset-bug-repro1/blob/3481f4b6ba288bcf9e1c84f72c23fe9d58cbc6cf/deps.edn#L17#2021-07-2720:36hiredmanit will show you exactly what is happening#2021-07-2720:37seancorfieldSure, but that won't answer the original Q which was what to look for in clj -Stree.#2021-07-2720:37hiredmanit likely isn't anything to do with deps like that#2021-07-2720:38seancorfieldIf adding a dep without changing any code causes this... and removing the dep again solves the problem... without changing any code...#2021-07-2720:39hiredmanwell, the rest of the stacktrace will tell us#2021-07-2720:39hiredmanI sort of suspect code is changing#2021-07-2720:39seancorfieldIs static init in a class/jar invoked if the new dep is never referenced in code? I thought it wasn't until the class was referenced...#2021-07-2720:40hiredmanaleph can be a little cagey about when it is handling you a fully realized value, and when you are getting a something deferred, which can trip people up when they treat like "just ring"#2021-07-2720:40seancorfieldtech.ml.dataset sure pulls in a lot of transitive deps: https://github.com/techascent/tech.ml.dataset/blob/master/project.clj#2021-07-2720:40hiredmanwhich that error looks like a text book case of#2021-07-2720:41hiredmanthe rest of the stracktrace would tell us right away#2021-07-2720:42hiredmanhttps://github.com/techascent/tech.ml.dataset/blob/master/project.clj#L104 looks like a huge problem#2021-07-2720:45didibusYa, I think we need a way to discourage that, and when its needed, to encourage people to add a step in their Jar where they only include class files that are required, so it doesn’t pull in all the dependencies as well and the classes that might be better of as source.#2021-07-2720:46didibusBy “a way”. I’m thinking more through education#2021-07-2720:46hiredmanyou are talking about filtering the output of compile?#2021-07-2720:46didibusYes, like go compile -> class-dir -> filter -> jar#2021-07-2720:47hiredmanthat is also a bad idea, compile generates the classes it needs and there is no guarantee the same classes will get generated again later#2021-07-2720:47seancorfieldThat's for :uberjar though, not lein jar right?#2021-07-2720:47didibusWell, really it goes:
src ------------------------------> jar
compile -> class-dir -> filter -> jar#2021-07-2720:47hiredmanlein had(has?) a feature to do that#2021-07-2720:47hiredmanit is gross#2021-07-2720:48didibusHum, no it doesn’t exactly, but I think it had something called excision which you could use for it#2021-07-2720:48didibusWhat’s gross about it?#2021-07-2720:48hiredmando you want me to go find the commit for it?#2021-07-2720:49hiredmanwhen the compiler generates classes those classes have dependencies on each other based on the names of the classes#2021-07-2720:49hiredmana second run of the compiler in some other time and place will likely not generate the same names for classes#2021-07-2720:50hiredmanso when you delete or filter out classes, you run the risk of next time your code is loaded, the compiler won't generate the same class names, which will result in classes your compiled code needs being missing#2021-07-2720:51didibusThat’s true, its a prefix filter, and it relies on the fact that class names are always prefixed the same even if the part after changes#2021-07-2720:52hiredmanhttps://github.com/technomancy/leiningen/commit/c4271c07c697c948716c2dd1af86f77cef3f9a4c is the lein feature that was default to on at one point being default to off because it broke things#2021-07-2720:55didibusHum, I mean that might just be because it failed at doing so.
But I’m talking about something more explicit, well you need to specify what to include, which is normally very little. So you’re left with mostly source files, and a few select class files.#2021-07-2720:57didibusOk ya, so I looked up how I do it, I use namespace-munge to find the class name from a namespace.#2021-07-2720:59didibusAnd I have another logic for gen-class that takes the gen-class :name
And then in the build you need to explicitly list the namespaces and gen-class names you want to include their compiled classes in the Jar.#2021-07-2721:01hiredmanhttps://groups.google.com/g/clojure-dev/c/r3A1JOIiwVU#2021-07-2721:02didibusAnd so what I do is that I include everything that in: <namespace-munge>*.class from the compiled dir into the Jar, for each explicitly listed namespaces.
And for gen-class, you can just take the full :name from the gen-class and add .class after, since its guaranteed to have that name always#2021-07-2721:03hiredmanyou will notice that you completely left out the case that the mailing list thread calls out (protocols)#2021-07-2721:05didibusHum, wouldn’t the protocol also be prefixed with the namespace-munge ?#2021-07-2721:06hiredmancompilation is transitive#2021-07-2721:07hiredmanso if I am namespace N and require and use a protocol from namespace A, compiling N will generate a classfile for A's protocol#2021-07-2721:08jjttjjOh yeah, here it is:
17:06:57.893 [manifold-pool-2-3] ERROR aleph.http.server - error in HTTP handler
java.lang.IllegalArgumentException: contains? not supported on type: manifold.deferred.Deferred
at clojure.lang.RT.contains(RT.java:853)
at clojure.core$contains_QMARK_.invokeStatic(core.clj:1494)
at clojure.core$contains_QMARK_.invoke(core.clj:1486)
at ring.middleware.flash$flash_response.invokeStatic(flash.clj:21)
at ring.middleware.flash$flash_response.invoke(flash.clj:14)
at ring.middleware.flash$wrap_flash$fn__100872.invoke(flash.clj:39)
at ring.middleware.session$wrap_session$fn__101193.invoke(session.clj:108)
at ring.middleware.keyword_params$wrap_keyword_params$fn__101239.invoke(keyword_params.clj:53)
at ring.middleware.nested_params$wrap_nested_params$fn__101297.invoke(nested_params.clj:89)
at ring.middleware.multipart_params$wrap_multipart_params$fn__101595.invoke(multipart_params.clj:171)
at ring.middleware.params$wrap_params$fn__101619.invoke(params.clj:67)
at ring.middleware.cookies$wrap_cookies$fn__101072.invoke(cookies.clj:214)
at ring.middleware.absolute_redirects$wrap_absolute_redirects$fn__101792.invoke(absolute_redirects.clj:47)
at ring.middleware.content_type$wrap_content_type$fn__101740.invoke(content_type.clj:34)
at ring.middleware.default_charset$wrap_default_charset$fn__101764.invoke(default_charset.clj:31)
at ring.middleware.not_modified$wrap_not_modified$fn__101721.invoke(not_modified.clj:61)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at ring.middleware.x_headers$wrap_x_header$fn__100835.invoke(x_headers.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.AFunction$1.doInvoke(AFunction.java:31)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:384)
at aleph.http.server$handle_request$fn__55977$f__48214__auto____55978.invoke(server.clj:170)
at clojure.lang.AFn.run(AFn.java:22)
at io.aleph.dirigiste.Executor$Worker$1.run(Executor.java:62)
at manifold.executor$thread_factory$reify__48096$f__48097.invoke(executor.clj:47)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:832)
Having trouble getting the minimal repro to fail this way at the moment#2021-07-2721:08hiredmanlike, the classfile format is not crazy, just parse the classfiles, figure out which ones reference each other, instead of reinventing this completely broken worse is better 80% solution stuff over and over#2021-07-2721:09hiredmanyeah, that has nothing to do with the ml library#2021-07-2721:09hiredmanthat is the flash middleware not being able to handle what aleph is passing it#2021-07-2721:11didibusWell, the point is to not include transitive .class files, and so only include what you need to be locked down and made accessible from Java. But I had not looked into protocols I don’t think. So you can’t use a compiled protocols from some non compiled namespace?#2021-07-2721:11hiredmanit is complicated#2021-07-2721:12hiredmanaot compiled code can get loaded from a different classloader, so you end up with visibility issues#2021-07-2721:14hiredmanso if you have a defrecord that inline satisfies some protocol, it implements the interface behind that protocol under the hood#2021-07-2721:14didibusHum, I’ve only ever used it for gen-class and gen-interface when I write a Java lib in Clojure. So I guess I haven’t encountered the weirdness around protocols yet.#2021-07-2721:15hiredmanso when that defrecord's class file is loaded the jvm needs to be able to find that interface class file in the classloader defrecord is loaded from or a parent#2021-07-2721:16hiredmanbut in the aot case the defrecord maybe loaded in the app or system (or whatever its callled) classloader, where the defprotocol when it is run defined the class in the compilers dynamic classloader which is a child of the app or system#2021-07-2721:17didibusYa I see, so the compiled interface for the protocol is loaded in the parent classloader, and than the dynamicclassloader when it compiles the inline implementation for it that will fail to find the class.#2021-07-2721:17hiredmanno, the reverse#2021-07-2721:18didibusOh, reverse, oh so if you don’t AOT the defprotocol, but AOT your call site? Wait so is a AOT protocol works with a non-aot namespace that extends it?#2021-07-2721:19hiredmansame outcome, but in this case the defrecord's class (created via aot) would be loaded in the parent, but the defprotocol would create the interface it needs in the child (the aot'ed interface having been deleted by filtering)#2021-07-2721:20didibusI see, so it fails both ways.#2021-07-2721:21hiredmanif in your project you have a namespace N, and you use some library A, and that library defines a protocol P, and in N I created a record R that inline satisfies P, and then aot N, and prefix filter out all the classfiles that don't start with N, the interface for P gets deleted#2021-07-2721:22didibusOk ya I get it.#2021-07-2721:23didibusWouldn’t gen-class have this issue as well? Like if the gen-class is compiled, but its implementing namespace is not. Then wouldn’t the gen-class be loaded in the parent classloader and when trying to import the corresponding implementing clojure namespace it would fail to find it? Since that one is compiled at runtime by the child class loader?#2021-07-2721:25hiredmanI forget but I believe gen-class refers to clojure functions via vars through the namespace, so no name based classfile links#2021-07-2721:26didibusHum, ya might be that’s why it works#2021-07-2721:27didibusIt uses loadWithClass#2021-07-2721:27hiredmanit has been approximately a billion years since I used gen-class for anything other than just getting a main class to launch, and maybe half a billion since I've done that#2021-07-2721:29didibusIt seems the most reliable way is still to just have Java shims#2021-07-2721:30didibusBut even for protocols, then if you want a lib that works with Java and Clojure, you’re looking at not making it a protocol, but instead making it an Interface in Java.#2021-07-2721:30hiredmanor just don't aot libraries#2021-07-2721:31didibusYou need too for them to work as a Java lib though, that’s my use case#2021-07-2721:31didibusAnd not force people into using a Clojure Java API interface, but make them believe they are just dealing with a real java lib#2021-07-2721:35hiredmanI have been known to just use clojure.asm.* to generate the exact bytecode I want, if I had to deliver a java api like that I would be very tempted (likely foolishly) to do that.#2021-07-2721:46didibusI remember seeing a lib that was basically S-expr to jvm bytecode for Clojure… can’t remember, but that could be a simpler way of doing the same#2021-07-2721:48didibusOh, JiSE, hum but it does not support interfaces 😞
I guess though, even with clojure.asm.* I’m not sure how I would generate something that is both a valid Protocol when consumed from Clojure, and a valid Interface when consumed from Java#2021-07-2721:50hiredmanI've always thought it would be interesting to attach a protocol definition to an existing interface instead of generating a new one#2021-07-2721:52didibusWhat do you mean by attach?#2021-07-2721:54hiredmana protocol is sort of a pair of dynamically extendable state and a java interface, satisfying inline in a deftype or defrecord uses the interface, extending to an existing type uses the dynamically extendable state#2021-07-2721:54hiredmanso when you defprotocol you get both those things, because it creates some mutable state and defines the interface#2021-07-2721:54hiredmanbut I dunno, it seems like you could just pass the interface in#2021-07-2721:59didibusLike if you already have an interface… Ya, so just go like this interface is also now this protocol which just adds that state machinery to it.
That would kind of allow all interfaces in Clojure core to become protocols as well without breakage in a way.#2021-07-2721:59didibusAnd make all Java interfaces a protocol too. It would be neat.#2021-07-2722:35seancorfield(ns my.entry.point
(:gen-class))
(defn -main [& args] ((requiring-resolve 'my.impl/main) args))
🙂 Problem solved!#2021-07-2722:36seancorfieldAOT will only compile my.entry.point and it only depends on clojure.core stuff which is already compiled. No transitive compilation issues.#2021-07-2722:39dpsuttonquick check: what's the difference between specifying an alias with :deps vs :extra-deps? Does :deps clobber the top level and extra-deps is additive?#2021-07-2722:55seancorfield:deps and :replace-deps are "the same" I think, but I've noticed that when I use -A or -M it seems that those dependencies still combine with other aliases which sort of surprised me so I'm not sure whether something else changed in the latest t.d.a around that...#2021-07-2722:59seancorfieldLooking at the t.d.a code, it seems like :replace-deps generally merge but if you use a "tool" context, you only get :replace-deps -- and it looks like :deps is treated as a legacy alias for :replace-deps (and in one case it seems like t.d.a will merge them if both are present so together they replace )#2021-07-2723:00seancorfield(defn tool
"Transform project edn for tool by applying tool args (keys = :paths, :deps) and
returning an updated project edn."
[project-edn tool-args]
(let [{:keys [replace-deps replace-paths deps paths]} tool-args]
(cond-> project-edn
(or deps replace-deps) (merge {:deps (merge deps replace-deps)})
(or paths replace-paths) (merge {:paths (vec (concat paths replace-paths))}))))
So the output is just what was in the tool-args and that will override the top-level :deps (and :paths) from the project deps.edn file.#2021-07-2723:31jjttjjI hope this isn't too off topic, but here's a minimal example of the error.
It does have to do with the ring session stuff it seems, I'm just confused by why the dataset repo would cause this
https://github.com/jjttjj/aleph-dataset-bug-repro1
Here's a diff of the -Stree with and without the dataset dep
https://www.diffchecker.com/JdvUqlE7
But excluding the ones that the dataset diff pulls in didn't change anything#2021-07-2813:30Alex Miller (Clojure team)hey all, I've been out for the last week or so and I'm declaring Slack bankruptcy so if you posted something above, I didn't read it. as always, if you have a bug or feature request, please ask it at https://ask.clojure.org so I do not lose track of things. I get to everything there (eventually).#2021-07-2816:04seancorfieldI think the only thing that hasn't yet made it into ask is a discussion about "multiple aliases" in the docs and that was partly because we haven't formulated what the problem really is that needs addressing...#2021-07-2816:05seancorfieldAnd that maybe belongs as one or more issues on the clojure-site repo anyway...#2021-07-2820:28Derek1.10.3.933 was released? cool 😎#2021-07-2820:42Alex Miller (Clojure team)still working on release notes, patience :)#2021-07-2820:51DerekI saw the homebrew tap change#2021-07-2820:51Derekjust excited#2021-07-2821:01Alex Miller (Clojure team)Clojure CLI https://clojure.org/releases/tools#v1.10.3.933 is now available. Changes since last stable release:
• deps.edn
◦ https://clojure.org/reference/deps_and_cli#_git
• If a git library name follows the repo convention names, the :git/url can now be inferred (`:git/url` can also be specified explicitly and takes precedence)
• :git/tag and prefix :git/sha can now be specified instead of the full sha. Both must point to the same commit.
◦ :sha has been renamed to :git/sha but the original is still supported for backwards compatibility
◦ :deps/prep-lib - a new top-level key can be used to say how a source lib should be prepared before being added to the classpath. This key’s value is a map with :alias, :fn, and :ensures. See https://clojure.org/reference/deps_and_cli#prep for more info.
◦ :tools/usage - a new top-level key can be used to provide the :ns-default and :ns-aliases context for a tool
• https://clojure.org/reference/deps_and_cli#tool_install - git-based programs that can be installed with a local name. Tools can provide their own usage context in deps.edn.
◦ Added new auto-installed tool named tools with https://clojure.github.io/tools.tools install, list, remove. See https://clojure.org/reference/deps_and_cli#tool_install#.
◦ Install a tool with clojure -Ttools install <lib> <coord> :as <toolname>
◦ Run a tool with clojure -T<toolname> fn (also takes -X style args)
• https://clojure.org/reference/deps_and_cli
◦ New -T option is like -X (executes a function) but does not use the project classpath, instead uses tool classpath (and adds :paths ["."] by default). -T:aliases is otherwise same as -X. -Ttoolname resolves named tool by name and uses that tool lib.
◦ https://clojure.atlassian.net/browse/TDEPS-198 - -X and -T will not wait to exit if futures/agents have been used
◦ https://clojure.atlassian.net/browse/TDEPS-182 - Improve deprecation messages to be more accurate
◦ https://clojure.atlassian.net/browse/TDEPS-183 - Fix -Sdescribe output to be valid EDN on Windows
◦ https://clojure.atlassian.net/browse/TDEPS-179 - Fix incorrect classpath when :classpath-overrides removes path
◦ Delay computation of local-repo path (don’t compute at load time)
◦ Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.12.1003
• New -X:deps programs:
◦ find-versions - to find versions of Maven or git libs or tools
◦ prep - use to https://clojure.org/reference/deps_and_cli#prep source libs
◦ help/dir - to list available functions in a tool namespace
◦ help/doc - to list docs for a tool namespace or function
#2021-07-2821:02dpsuttondoes this mean 933 is a stable release?#2021-07-2821:04seancorfieldYes, per https://clojure.org/releases/tools#2021-07-2821:04Alex Miller (Clojure team)yes#2021-07-2821:05borkdudecongrats#2021-07-2821:06seancorfieldThanks in particular for TDEPS-198!#2021-07-2821:06vemvCongratulations!
Unrelated question, I have in mind a program that would take a deps.edn file as input, and would compute a derived deps.edn as the output. We can consider it a pure defn written in vanilla clojure.
(under Lein this would be a "plugin")
Now let's say I want people to use my program idiomatically, having a good integration with other tools, IDEs etc. Is there a "tools.deps way" to do it?#2021-07-2821:06seancorfield@vemv Sounds like a classic use of the new "tools" functionality!#2021-07-2821:07seancorfieldWrite it as a defn that accepts a hash map of options. Put it up on GH and tag it. clojure -Ttools install libname '{:git/tag "v1.2.3"}' :as thing and then clojure -Tthing#2021-07-2821:08seancorfield(assuming you want it as a public tool for others to use?)#2021-07-2821:09seancorfieldOtherwise, if it's just local, put it in an ns that's accessible via an alias and clojure -T:myalias (with :myalias specifying :exec-fn and :exec-args etc and the path to make the ns available).#2021-07-2821:10seancorfieldAs for "IDEs", that's going to depend on the IDE. But "tools" can be invoked from the REPL easily enough if you go with the alias route.#2021-07-2821:10vemvThanks so far!
Yes I suspected the new tools might be related however I never got the gist of them. I just see that one can add/remove them but what do they do?#2021-07-2821:11seancorfieldThey are "installed" via an EDN file under ~/.clojure but don't affect your deps.edn.#2021-07-2821:11seancorfieldOtherwise they're just like any code that you would otherwise use via -X in earlier releases of the CLI.#2021-07-2821:12seancorfield(with the key difference that -T doesn't include the project classpath when invoking the function -- just the tool itself)#2021-07-2821:13seancorfieldI have both clj-new and depstar installed as "tools" locally so they don't need to be in my (user or project) deps.edn files and I can just say clojure -Tnew app :name myname/myapp to create a new application project, for example.#2021-07-2821:15seancorfieldAnd you can now easily find tagged versions of tools (and libraries) without needing to trawl Clojars or the source repo:
(! 700)-> clojure -X:deps find-versions :lib com.github.seancorfield/clj-new
Cloning:
{:git/tag "v0.1.0"}
{:git/tag "v0.2.0"}
...
{:git/tag "v1.1.321"}
{:git/tag "v1.1.324"}#2021-07-2821:18vemvIs there a particularly handy doc on tools ? I've tried on two different days to get them but I still don't
for example clojure -Tnew app :name myname/myapp appears to run an arbitrary clojure program
but what I want to do is to read the current project's deps.edn file (after applying any user-provided aliases, which I cannot know in advance), and compute in return another deps.edn value for tools.deps to use
i.e. I want to affect how tools.deps itself works for a given project.#2021-07-2821:18borkdudewouldn't it be "easier" if the output sorted the newest version first?#2021-07-2821:49seancorfield@U04V15CAJ It would likely scroll off the screen -- better to have the newest last so by the time the command has finished outputting all versions, the most recent ones are still visible.#2021-07-2821:52borkdudegood point#2021-07-2821:52borkdudeor perhaps it could just show a max number of version#2021-07-2821:52borkdudeI don't see why I would want to have all of the version back to the beginning, usually#2021-07-2821:59seancorfieldYeah, it does seem a bit redundant for projects with long histories...#2021-07-2821:19borkdude@vemv you want to transform the deps.edn in the local dir?#2021-07-2821:19borkdudei.e. rewrite it?#2021-07-2821:19borkdudewhat does a user provided alias have to do with the problem: the deps.edn doesn't change because of an alias?#2021-07-2821:20borkdudeor do you want to calculate a "basis" and then make tools deps use that?#2021-07-2821:20vemv> @vemv you want to transform the deps.edn in the local dir?
yes, but only at runtime
read the computed deps.edn after aliases -> change it -> return it to tools.deps for it to continue whatever it was doing#2021-07-2821:20borkdudeah so you won't actually change a file, I get it#2021-07-2821:21vemv> what does a user provided alias have to do with the problem: the deps.edn doesn't change because of an alias?
the computed value changes, I cannot just slurp deps.edn
(I think you got that now. Posting for completeness)#2021-07-2821:22borkdudewhat you maybe want is invoke a java process with a basis#2021-07-2821:22borkdudehttps://github.com/clojure/tools.build/blob/fc2e1f82988d2200ee66f21b1e757385ef443e9e/src/main/clojure/clojure/tools/build/tasks/process.clj#L80#2021-07-2821:23borkdudeonce you get the basis from the tools.build create-basis function you can create a java process with that#2021-07-2821:24borkdudehttps://github.com/clojure/tools.build/blob/fc2e1f82988d2200ee66f21b1e757385ef443e9e/src/main/clojure/clojure/tools/build/tasks/create_basis.clj#L16#2021-07-2821:25vemvSounds good. Would the final solution be idiomatic?
An important requirement is that this should be concise and standard for users to do / IDEs to integrate
Just like when one tells people "use Lein plugin foo", which is a well-understood thing to follow#2021-07-2821:25Alex Miller (Clojure team)it sounds like you want to dynamically modify the deps that are being computed and used by the CLI, and that is not a thing#2021-07-2821:26borkdudeisn't -Sdeps something you could use for that as well?#2021-07-2821:27Alex Miller (Clojure team)there are a variety of ways you can dynamically compute/modify a classpath, but that's not available in a pluggable way in the CLI#2021-07-2821:27borkdude(note: -Sdeps is merged, not a replacement for the local deps.edn)#2021-07-2821:29borkdudeperhaps describing a concrete use case would help find a better fitting solution#2021-07-2821:29vemvYes I checked out clojure --help and it seems to have the necessary building blocks. Can be done
At the same time I was afraid of it being bit of a hassle for others to integrate. Hence the "easy as a plugin" remark#2021-07-2821:30borkdudenote that dealing with the CLI can be prone to OS/shell-specific issues#2021-07-2821:30vemvExactly#2021-07-2821:30borkdudeBut invoking a java process using a basis is not.#2021-07-2821:31vemv[...] > but that's not available in a pluggable way in the CLI
has it been considered? should I ask on ask.clojure? Hope the use case is not a no-go#2021-07-2821:51vemvcc/ @U064X3EF3#2021-07-2822:05Alex Miller (Clojure team)not something I'm very interested in#2021-07-2822:05Alex Miller (Clojure team)we have all kinds of affordances provided to modify the deps in common ways#2021-07-2822:06Alex Miller (Clojure team)if it becomes dynamic requiring code to run, all tools and external users have to understand how to do that#2021-07-2822:06Alex Miller (Clojure team)this is imo at odds with the notion of deps as data#2021-07-2822:06Alex Miller (Clojure team)I haven't heard what you actually need to do - start there#2021-07-2822:26vemvcheers. I want to make https://github.com/clojure-emacs/enrich-classpath easy/standard to use for deps.edn users
It currently is both a vanilla clojure lib and a Lein plugin. It takes dependencies as data and returns a superset of those dependencies as data.
For Lein users, I can instruct them to install a plugin. Not sure of what I should tell deps.edn users to do
The functionality should work in arbitrary projects, and in face of user-provided aliases chosen at runtime. It should play fine and composably with any other core deps.edn functionality.#2021-07-2823:06Alex Miller (Clojure team)thanks, that's useful#2021-07-2823:10Alex Miller (Clojure team)so I think there are a few parts to this...
1. given deps/aliases etc, determine the full set of libs
2. for each lib, ensure that the javadoc or source jar is downloaded for the lib to the local repo
3. create a classpath that contains the original classpath + source/javadoc jars#2021-07-2823:29vemvthanks! The devil in the details resides in how to impl 3 . Would understand if it needs some hammock time.
For example clojure --help gives me some building blocks, but the overall solution would seem excessively bespoke.
Ideally people could just use a defcustom https://github.com/clojure-emacs/cider/blob/5aaaa3a01310f5f9118465f5371a75dfea35b856/cider.el#L160 (or whatever customization convenience the IDE provides) specifying a standard tools.deps flag with the would-be "plugin" id. So that it doesn't feel like an invasive or non-standard solution#2021-07-2823:38Alex Miller (Clojure team)sorry had to step away#2021-07-2823:41Alex Miller (Clojure team)1. is well covered by tools.deps create basis stuff
2. is actually not directly supported by tools.deps at all (although you can target the lower level utils and probably get some use out of it for that)
3. creating the classpath is not hard once you have the set of things. there is not really any way to get in the loop right now to replace the classpath used by the CLI and I remain dubious that that's a good thing to do. The CLI does have -Scp which lets you just supply a classpath and replace everything. So you could create the classpath, then shell out to clojure -Scp ... or whatever, but that's a lot grosser than what you're looking for#2021-07-2823:41Alex Miller (Clojure team)I don't think this is an area we're likely to change soon#2021-07-2823:46vemvThanks, yes it's useful to get a feel of how likely are things to change
I think it's doable to craft my own script that does the right thing, then users would be customize this other defcustom https://github.com/clojure-emacs/cider/blob/5aaaa3a01310f5f9118465f5371a75dfea35b856/cider.el#L146
Then again a bespoke script doesn't necessarily compose well with other stuff (what if someone else also wanted to promote a similar script for a different feature?)
But it can be OK to just release an "MVP", see if people like the ugly-ish script and then there could be a popular demand for something more standard#2021-07-2823:48vemvFor further context, the underlying feature has been desired for many years, I still see people asking about it on #cider every other week#2021-07-2900:01Alex Miller (Clojure team)Is there some reason all these files need to be on the classpath?#2021-07-2900:01Alex Miller (Clojure team)seems like for javadoc especially there are likely to be conflicting resources#2021-07-2900:01Alex Miller (Clojure team)seems like mucking with the classpath is just asking for trouble. surely IntelliJ and Eclipse and other Java IDEs don't do that#2021-07-2900:07vemv> Is there some reason all these files need to be on the classpath?
yes, without them on the classpath CIDER can't possibly access the source artifacts, necessary for its Java jump-to-source integration to work.
Similarly, other features work by javadoc parsing#2021-07-2900:08vemvI've used this plugin for the entirety of 2021 without a problem, would be open to a specific insight though#2021-07-2900:12vemv> surely IntelliJ and Eclipse and other Java IDEs don't do that
it's a good observation. CIDER is fundamentally runtime-powered. Java IDEs can afford a more static approach#2021-07-2900:24Alex Miller (Clojure team)I still don't see why "being on the classpath" is necessary for anything you just listed. it's just opening files and doing file stuff.#2021-07-2900:25Alex Miller (Clojure team)another idea - if you either exploded or uberjarred all the files, you could just add one mega thing to the classpath#2021-07-2900:28vemv> I still don't see why "being on the classpath" is necessary for anything you just listed. it's just opening files and doing file stuff.
I agree, it's something I've considered but relying on files to be randomly opened breaks (reinvents?) the Resource abstraction.
Arbitrarily many other tools can be based on Resources, i.e. they rely that if there's some interesting source or javadoc to parse, then it's on the classpath. I can't coordinate with everyone (including future tools!)#2021-07-2900:29vemv> another idea - if you either exploded or uberjarred all the files, you could just add one mega thing to the classpath
maybe. what I'm doing right now is intentfully placing all sources/javadocs at the tail of the classpath. It's one the few things that made the plugin reliable#2021-07-2821:32borkdudeyou want to make a launcher it seems that injects some deps or whatever? you still haven't described it fully#2021-07-2821:33vemvwe're clear on the requirements now, let's keep it concise#2021-07-2821:34borkdudeactually the requirements still aren't that clear to me, but I'll go do something else ;)#2021-07-2821:35vemv> ways you can dynamically compute/modify a classpath [in a pluggable way]
these are them cheers#2021-07-2821:42seancorfieldI thought you were asking about a standalone process to produce a modified deps.edn file -- your requirements were certainly not clear to me 🙂#2021-07-2821:58vemvThat's slightly besides the point. A. Miller made a precise summary of what was being asked, so asking for a further use case seemed redundant from then on#2021-07-2901:38Oliver GeorgeI'm curious to try Clojure CLI https://clojure.org/releases/tools#v1.10.3.933 - is there usually a delay before brew upgrade clojure/tools/clojure will find it? brew tells me "Warning: clojure/tools/clojure 1.10.3.822 already installed"#2021-07-2901:40Oliver GeorgeI suspect my brew-fu is at fault. Perahps there's some tap specific update needed so brew knows about new releases.#2021-07-2901:58seancorfieldMaybe you just need to run brew update first @olivergeorge?#2021-07-2901:59Oliver GeorgeTa. I got there in the end. brew update didn't seem to help but doing a version specific brew install worked (at least clashed with what I had but after fetching the new version).#2021-07-2902:00seancorfieldYeah, I always do the explicit @version install (and then the brew link dance)...#2021-07-2902:08Alex Miller (Clojure team)brew upgrade clojure/tools/clojure should work - it's updated there#2021-07-2902:08Alex Miller (Clojure team)I'm still waiting on homebrew to merge to homebrew core#2021-07-2902:14seancorfield(! 707)-> ls /usr/local/Cellar/clojure
clojure/
🙂#2021-07-2906:32flowthing> Thanks in particular for TDEPS-198!
Seconded, many thanks!#2021-07-2909:23dominicmI'm struggling to use -X:deps help/doc . I think it's my misunderstanding. It's unable to work because my tools are in a namespace that requires namespaces in a library, but -X:deps replaces the deps instead, so I get "Could not locate clojure/data/csv__init.class, clojure/data/csv.clj or clojure/data/csv.cljc on classpath.".
Shouldn't -X:deps run as well as my project dependencies to make this work?#2021-07-2913:18Alex Miller (Clojure team)can you put your full command line?#2021-07-2917:10seancorfield@dominicm Yeah, I've only been able to get help/doc to work in very limited situations because :deps replaces your project dependencies.#2021-07-2917:11seancorfieldI'll write up a repro on ask...#2021-07-2917:12Alex Miller (Clojure team)help/doc is really intended for tool use cases more than project use cases, but happy to consider things more fully#2021-07-2920:33dominicmNothing I saw solved for the project case. Here's my use-case: I have some functions in my project which I routinely want to run for doing things like installing generated data to the db, etc. These functions could be run from the REPL, but they're also good for the CLI too (with a little prn sprinkled in).
I want the full set of project dependencies in there, not just a small subset I can specify using :extra-deps. https://ask.clojure.org/index.php/10888/help-doc-doesnt-work-with-project-dependencies and TDEPS-201 captures my wants well, and mentions solutions I think would work for me.#2021-07-2920:36seancorfieldYeah, the alternate root deps.edn alias version of :deps that does :extra-deps instead of :replace-deps seems like a good compromise. I have occasionally found myself wanting to just add t.d.a to something I'm doing so it would help with that too.#2021-07-3016:17cldwalkerCongrats on the official release! The project use case would be nice to have#2021-07-2917:14seancorfieldhttps://ask.clojure.org/index.php/10888/help-doc-doesnt-work-with-project-dependencies#2021-07-2917:16seancorfield@alexmiller I had it working with a tool before but now I can't figure out what I had to do for that... can you remind me?#2021-07-2917:19seancorfieldAh, I remembered!
clojure -A:deps -Tnew help/dir
That lists functions in the default namespace of the tool.#2021-07-2917:20Alex Miller (Clojure team)yeah, you'll either want that or clojure -X:deps:whatever help/dir#2021-07-2917:24seancorfieldOK, so this works:
(! 766)-> cat deps.edn
{:aliases {:csv {:extra-deps {org.clojure/data.csv {:mvn/version "RELEASE"}}}}}
(! 767)-> clojure -X:deps:csv help/doc :ns clojure.data.csv
Reading and writing comma separated values.
...
but without an alias, none of the project's basic :deps would work and that's "expected" because this is intended for "tools", not projects?#2021-07-2917:25seancorfieldI almost feel like there should be some "magic" alias that means "use the project :deps" 🙂#2021-07-2917:28seancorfieldI will raise an issue against clojure-site to add examples to the reference docs to show the two usages for tools...#2021-07-2917:32seancorfieldhttps://github.com/clojure/clojure-site/issues/548 for reference.#2021-07-2917:42Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_help_functions - a start#2021-07-2917:44seancorfieldI suspect folks will wonder what -Tnew is so maybe my-tool instead?#2021-07-2917:45seancorfield(although you could use -Tdeps-graph since that is actually used as an example on that page)#2021-07-2917:46seancorfieldThis stuff is hard -- you never know what assumptions folks bring with them to a page of docs like that.#2021-07-2917:46Alex Miller (Clojure team)I switched it to tools, which is built in#2021-07-2917:46Alex Miller (Clojure team)also added a couple more notes and link to the api docs#2021-07-2919:20mkvlrCongrats about the release, really excited about the new things. Ran into this after upgrading to Clojure CLI version 1.10.3.933 (artifact changed, we encountered this with a private dep):
clj -Srepro -Sdeps '{:deps {org.clojure/data.csv {:git/url "" :sha "aa9b3bdd3a1d3f6a7fe12eaab76b45ef3f197ad5" :tag "main"}}}' ✘ 130
Error building classpath. Library org.clojure/data.csv has invalid tag: main
Fixed by removing the :tag which previous versions would just ignore. Not sure if it should be considered a breaking change.#2021-07-2919:52Alex Miller (Clojure team)yes, I'd consider that a bug and it kind of ties in with some issues we have with git-resolve-tags#2021-07-2919:52Alex Miller (Clojure team)have you been using git-resolve-tags btw?#2021-07-2920:08mkvlrnot me personally (I’m used to manually looking up the sha on github and pasting it) but I think my colleague who added the dep is using it#2021-07-2920:09Alex Miller (Clojure team)the new :git/tag expects only a valid tag (not a branch name like main)#2021-07-2920:09Alex Miller (Clojure team)the old :tag with git-resolve-tags would happily resolve that#2021-07-2920:10Alex Miller (Clojure team)so there are important differences here. I just tend to assume no one is using that stuff but clearly somebody is#2021-07-2920:38seancorfieldI have an example in my user deps.edn of using gitlibs/resolve for turning a branch name into a SHA: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L177-L180 -- presumably @mkvlr could use that instead of git-resolve-tags?#2021-07-2920:10mkvlrah, and there’s no way to specify a branch and use tools.deps to update the sha from it anymore?#2021-07-2920:10Alex Miller (Clojure team)at the moment, not really#2021-07-2920:11mkvlrok, not a big deal imo#2021-07-2920:11mkvlrI guess being able to set a minimum CLI version would be nice, so we can upgrade to the newer :git/tag keys#2021-07-2920:13mkvlror is there currently anything I’m missing except asking everybody on the team to upgrade when relying on newer tools.deps features?#2021-07-2920:14mkvlr(though of course the problem will only go away once everybody is on a version that supports requiring a minimum version, so not immediately helpful)#2021-07-2920:21mkvlranother thought: could :git/ref be a thing to support in the future for mutable things like branches together with git-resolve-refs ? Also not sure if it would be nice to support :git/tag without needing a :sha. I feel like deleting or overwriting a released tag can be similar to force pushing a :sha out of the repo which can be gone once GC’d. But also see the upside of being explicit…#2021-07-2920:23Joshua SuskaloThese sorts of things I think goes against the idea of deps as data, since there's no reproducible way to run your project if it has a mutable dependency. Personally I'd favor having a -T tool which would update your dependencies in your deps edn based on a key like :git/ref , and then it just updates the sha in the dep with rewrite-clj#2021-07-2920:23Joshua SuskaloAlthough it'd probably be like :tool-name.git/ref for the actual key#2021-07-2920:28mkvlr@suskeyhose to be clear I was talking of a ref as a way to update a sha, so it would still be immutable. And tags I consider it to be best practice for them to not change (but can be forced to be overwritten) so I’m on the fence on that one.#2021-07-2920:28Joshua SuskaloAh, I see#2021-07-2920:30Joshua SuskaloThen we were suggesting basically the same thing, but as a tool vs as a core part of tools deps I suspect#2021-07-2920:31Joshua SuskaloWell, not quite if your suggestion was to validate that the ref does match that sha#2021-07-2920:33dominicmNothing I saw solved for the project case. Here's my use-case: I have some functions in my project which I routinely want to run for doing things like installing generated data to the db, etc. These functions could be run from the REPL, but they're also good for the CLI too (with a little prn sprinkled in).
I want the full set of project dependencies in there, not just a small subset I can specify using :extra-deps. https://ask.clojure.org/index.php/10888/help-doc-doesnt-work-with-project-dependencies and TDEPS-201 captures my wants well, and mentions solutions I think would work for me.#2021-07-2920:33mkvlrwe both want to update the sha, which git-resolve-tags used to do that (for both tags and branches, now only for tags). Think we all agree a tool is the way forward#2021-07-2920:38seancorfieldI have an example in my user deps.edn of using gitlibs/resolve for turning a branch name into a SHA: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L177-L180 -- presumably @mkvlr could use that instead of git-resolve-tags?#2021-07-2920:45mkvlryeah, though I’m not sure I’m ever going to change my habit of just copy & pasting the sha from GitHub, comes with the upside of seeing the commit etc 🙈#2021-07-2921:36borkdudeNow that that deps.edn has some new rules, the linting for it in clj-kondo needs to be updated. Please test your deps.edns and report any false positives, please!
$ clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:git/url "" :sha "05cc5aca58d89215a295799df8e27fa1a9874885"}}}' -M -m clj-kondo.main --lint deps.edn
Checking out: at 05cc5aca58d89215a295799df8e27fa1a9874885
linting took 52ms, errors: 0, warnings: 0
You can report false positives in this issue: https://github.com/clj-kondo/clj-kondo/issues/1318
🙏#2021-07-2921:45borkdude(especially when you're using the new features)#2021-07-3001:24Cam SaulI haven't really dug in to it much but was wondering whether other people run into this with the CLI. It occasionally gets into a weird state and barfs on launch when a deps.edn file (usually one from a :local/root dependency) was updated. Nuking all the .cpcache directories consistently fixes it.
I've had maybe 4 people on my team run into this now (including just now with the latest 933 release)... just wondering if this is a known issue or maybe I've set things up the wrong way somehow #2021-07-3002:14seancorfieldYes. Transitive deps in local deps don't get recalculated. Use -Sforce#2021-07-3002:14hiredmanI would expect that if you are both using local/root and a high churn of deps for those local/root projects#2021-07-3002:15seancorfieldIt's a long standing known issue#2021-07-3017:56cap10morganDoes the new prep feature need tools.build? What defines the :fn in the referenced :alias?#2021-07-3017:59cap10morganor is that just clojure.core/compile and you're telling it to invoke that as the main under the referenced alias?#2021-07-3017:59cap10morgan(in the example in the docs)#2021-07-3018:11Joshua SuskaloI believe the prep feature just runs an alias the same way -X does. It does not require tools.build.#2021-07-3018:15cap10morganHmm, that leaves me somewhat confused what the :fn attribute is referencing then#2021-07-3018:17Joshua SuskaloSome aliases specify a default ns, or otherwise don't provide a function and expect it to be provided on the command line. Since there's no user-controlled command line to prep a lib, that fn argument is what you want to pass on the command line.#2021-07-3018:20cap10morganoh ok I think that makes sense. will probably be clearer once I try it. 🙂 thanks!#2021-07-3018:21Joshua Suskalono problem!#2021-07-3019:52mpenetAre there workarounds to tdeps-174 now ?#2021-07-3019:54mpenetI remember talks about allowing more control aver the "chain" of deps files or add something like aero tags to deps.edn reader but glancing at the changelogs I didn't see anything related#2021-07-3020:05seancorfield@mpenet I've blogged about how we have restructured our monorepo/deps files (four parts in the series so far over several months) but where we've settled on now -- based on guidance from Alex and a bunch of experience with different approaches -- is to have a project deps.edn with dev/test aliases configured that treat the subprojects as :local/root libraries with their own deps.edn files, then we have a "project" subproject for each artifact we build which has a standalone deps.edn file that treats the parts it needs as :local/root libraries. That allows us to control a unified dev REPL/test environment but also have specific "build" environments for each "project" (artifact). We've also started to adopt Polylith which works much the same way but has a custom tool (that reads/merges deps.edn files) so it can perform incremental testing and do so in the context of each "project" as well as in the "dev" context.#2021-07-3020:05seancorfieldTL;DR: we no longer care about TDEPS-174 🙂#2021-07-3021:13mpenetI d prefer to avoid using a custom tool to merge/read edn files, so it seems 174 is still relevant. I ll read the latest post, I might have missed it#2021-07-3021:17seancorfieldWell, Polylith's tool poly does a bunch of reading and merging, and it also does classpath isolation stuff to run tests in-process. In our own code at work, we're using tools.build and create-basis, java-command, and process to construct contexts to run tests (as subprocesses, rather than in-process).#2021-07-3021:19mpenetI know about Polylith, but this is not something I could use in that context. I am also not sold on the approach. But maybe that changed recently too. I ll look again#2021-07-3021:22mpenetFrankly if I had to impose some external tool/script to handle it I d rather have a simple "launcher" that allows more deps edn to the chain at "call" time. That seems less intrusive, but that's also not something I want to do.#2021-07-3021:24mpenetI am curious to know if 174 & co is still considered as something to be improved or not#2021-07-3021:25mpenet(by tools.deps)#2021-07-3021:36seancorfieldI get the impression that this is a problem the core team don't feel is worth solving in "core" since t.d.a exists and folks who need more deps.edn files can "easily" write their own in a few lines.#2021-07-3021:38seancorfield(we use t.d.a in our build script to do stuff that's outside the CLI scope but pretty simple to write ourselves)#2021-07-3023:44dpsuttoni've seen this error show up in CI infrequently. anyone recognize it or know what i can do to mitigate it or lower the frequency?
Downloading: org/clojure/data.json/2.0.2/data.json-2.0.2.pom from central
Downloading: enlive/enlive/1.1.6/enlive-1.1.6.pom from clojars
Downloading: org/ccil/cowan/tagsoup/tagsoup/1.2.1/tagsoup-1.2.1.pom from central
Downloading: org/jsoup/jsoup/1.7.2/jsoup-1.7.2.pom from central
Error building classpath. class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1882)
at java.base/java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2061)
at java.base/java.util.HashMap.putVal(HashMap.java:633)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:847)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependency(DefaultModelValidator.java:659)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveDependencies(DefaultModelValidator.java:583)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:373)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:494)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:440)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:430)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:292)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$eval1076$fn__1078.invoke(maven.clj:107)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__790$fn__792$fn__793.invoke(alpha.clj:403)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__505.invoke(concurrent.clj:34)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Exited with code exit status 1
#2021-07-3023:46hiredmanwhat version of tools deps? try adding -Sthreads 1 to your clj call#2021-07-3023:47hiredmanthat error is due to unsafe concurrent modification of a hashmap, internally in maven, I believe it has been fixed (I haven't seen it in a while)#2021-07-3023:48Alex Miller (Clojure team)That’s a known issue, hoping to work on it in a couple weeks#2021-07-3023:48dpsuttonok. thank you much. i'll just kick the box when i see it. thanks#2021-07-3023:57seancorfieldI'm a bit surprised we've never seen that at work -- we don't use the -Sthreads option. Quite a few do seem to hit it. I wonder why it seems to affect some people more than others?#2021-07-3100:00hiredmanI've seen it, but not in some time. My most recent command usages all include -Sthreads because I added it back when I was fighting with classloaders and the thread the s3 downloaded launches#2021-07-3100:09seancorfieldI'll rephrase that then: I'm a bit surprised I've never seen that at work -- I don't use the -Sthreads option. 🙂#2021-07-3101:05hiredmanI wonder if it is more likely if your m2 isn't completely populated#2021-07-3101:26ghadiRace condition on unsynchronized HashSet?#2021-07-3101:48hiredmanthat is my guess based on similar stacktraces#2021-07-3106:24dharriganOh, I thought I was the only one. I get it from time to time - I just rerun clj and it usually works the next time.#2021-08-0200:31seancorfieldJust in case anyone wants to know how to run tests using build.clj (instead of clojure -M:test or clojure -X:test as a separate step), I have added an example to next.jdbc's repo, since it uses a barebones invocation of the Cognitect test-runner: https://github.com/seancorfield/next-jdbc so you do clojure -T:build run-tests#2021-08-0200:32seancorfield(this is only really useful if you want a build command that runs your tests as part of a bunch of other stuff -- otherwise you're just spinning up a subprocess for no good reason)#2021-08-0200:33seancorfieldNote: this is not a -X-style invocation of test-runner (even though clojure -X:test is how I normally run tests in that project)!#2021-08-0202:03markaddlemanI'm getting this error when my deps.edn includes `
techascent/tech.ml {:mvn/version "6.001"}
I've tried deleting my entire .m2/repository directory but the error reoccurs.
Any idea what's going on?#2021-08-0202:09seancorfield@markaddleman try adding -Sthreads 1 to your CLI invocation.#2021-08-0202:12seancorfieldFWIW, clojure -Sdeps '{:deps {techascent/tech.ml {:mvn/version "6.001"}}}' works fine for me so there's no inherent problem with that lib.#2021-08-0202:13markaddlemanThanks. I'm getting a new error now:
Error building classpath. Could not transfer artifact args4j:args4j:jar:2.0.26 from/to central (): status code: 416, reason phrase: Range Not Satisfiable (416)
I think I've made progress#2021-08-0202:13markaddlemanI'm also on very spotty wi-fi so I#2021-08-0202:13markaddlemanSo I'll try again when I get home#2021-08-0202:19seancorfieldAh, spotty wifi. Yeah, definitely -Sthreads 1. And you may need to delete your ~/.m2/repository and start fresh to avoid corrupted files.#2021-08-0202:20markaddlemanYes, thanks#2021-08-0202:38markaddleman-Sthreads 1 did the trick. Thanks!#2021-08-0214:36Jeff Evansis it possible to specify BOM dependencies via deps.edn ? ex: clojure -A:alias1:alias2 -M foo ? Or do you just clojure -M:alias1:alias2:foo? Is there a difference, or one to be preferred over another?#2021-08-0219:53dpsutton(foo here is an alias that specifies :main-opts#2021-08-0219:53seancorfieldI would use the second format.#2021-08-0219:54dpsuttondo you have a reason for that? just preference, unambiguous merging of main opts, etc?#2021-08-0219:55seancorfieldI pretty much only use -A now if I'm starting a REPL or have a very specific use case where -A is the right approach, e.g., clojure -A:deps -Tnew help/doc#2021-08-0219:56dpsuttonI see. This is for a coworker that needs an nrepl main so it kinda feels the same as that to me: clojure -A:dev -M:nrepl kinda feels the same#2021-08-0219:56seancorfieldI've always used just one flag and a concatenated list of aliases...#2021-08-0219:57seancorfieldI think it's just personal preference. I find a single flag option "clearer" in that you immediately see "I run a -main" or "I run exec function" (or "I start a REPL") -- -M or -X (or -A).#2021-08-0219:58dpsuttonyeah. i guess i like the other version for precisely that reason clojure -A:classpath-additions -M entrypoint#2021-08-0219:58dpsuttonok. well as long as its not wrong 🙂#2021-08-0220:00seancorfieldThe docs seem to encourage my style:
clojure [clj-opt*] -X[aliases] [a/fn] [kpath v]* kv-map?
clojure [clj-opt*] -M[aliases] [main-opts]
#2021-08-0220:01dpsuttonthat's a good point. I guess i was thinking of when the changeover happens such that -A will no longer allow main options to "win".#2021-08-0220:02dpsuttonbut i think you've convinced me now that it is wrong, but perhaps happens to work#2021-08-0220:03seancorfieldAlso, where it describes how various options (in deps.edn) are provided, it says "should be provided as an alias in one of the deps sources and provided to the Clojure tools using -A (for REPL invocation), -X (for function execution), or -M (for clojure.main execution)." which reads to me as "you use either -A or -X or -M" (even though you can provide -A and either -X or -M).#2021-08-0221:10dominicm@dpsutton RE tooling, I'm not sure what cider's opts are atm, but I have been thinking about switching my setup over to taking a list of aliases which are passed to -A as you can't combine -M or -X with the nrepl start code anyway. The downside to that is how main-opts concatenate, so I can have an alias containing :main-opts ["-i", "init.clj"] to inject some startup code, but I suppose -M:user:provided:aliases is just as valid so it's probably OK.#2021-08-0221:51dpsuttoncider used to use just trailing main args with -M -m nrepl/cmdline host port but if there were any profiles in the provided aliases those would actually dominate. so now it uses inline -Sdeps {:cider/nrepl {:main-args ..}} and then adds the profile cider/nrepl as the last profile so it wins under any strategy#2021-08-0310:41borkdudeclj-kondo was updated to support the newest linting tools in deps.edn in clojure CLI 1.10.3.933. it's released in 2021.08.03.#2021-08-0315:11tianshuThe lastest tools-deps seems have different way on calculate paths?#2021-08-0315:55seancorfield@doglooksgood Different to... some other tool? Do you mean classpath or some other paths?#2021-08-0323:23tianshuMy project has a setup to start shadow-cljs with a few aliases. like [:dev :test :cljs :cljs-test]. In :test alias I have a :extra-paths which add "test/clj" . After upgraded to latest version of tools-deps, test/clj is not collected in CLASSPATH, but it should be. If I move the :extra-paths from :test to :dev the CLASSPATH is correct. Since there are few tools cider, shadow and tools-deps. I'm not sure where the problem comes from.#2021-08-0323:26dpsuttonwhen CIDER starts up at the top of the repl it should print the startup command. You can copy that and then continue your investigations from a repl to do exactly what CIDER is doing for you, and then eliminate CIDER's middleware and other things to see if CIDER is the culprit. If not, you have removed one extra player and can continue investigating#2021-08-0323:36tianshu@dpsutton good idea, I'll have a try#2021-08-0323:43dpsuttonyou can also work from the other side. just start up your project with the aliases you want and assert that your classpath has whatever entry you want. if it doesn't work with all tooling removed it just won't work at all#2021-08-0520:35Cam SaulI was trying to rework a build script as a -T alias, and I've just noticed that it's missing from ~a lot of the~ the man page.
• The man page for clojure has been updated to mention that -T exists in the synopsis but doesn't actually describe what -T does outside of that
• ~`-T` is also not explained anywhere in the Deps and CLI Guide (AFAIK)~
~The only place I found that ~#2021-08-0520:38Joshua SuskaloThe deps and cli reference has been updated with references to -T#2021-08-0520:39Cam SaulI don't see -T mentioned once in https://clojure.org/guides/deps_and_cli#2021-08-0520:40Joshua SuskaloYeah, sorry, I was mistaken, I sometimes confuse the reference for the guide.#2021-08-0520:41Joshua SuskaloThe guide has been updated for this release because it does contain information about preparing libs (https://clojure.org/guides/deps_and_cli#prep_libs), but you're right, no references to -T#2021-08-0521:17seancorfieldIt's discussed in the reference section https://clojure.org/reference/deps_and_cli#2021-08-0521:18seancorfieldNot sure whether it belongs in the guide section, except maybe as a note to read the relevant parts of the reference?#2021-08-0521:18seancorfieldThere's a separate guide for tools.build stuff: https://clojure.org/guides/tools_build#2021-08-0521:19seancorfield(and I'm happy to answer Qs in-depth since we've switched our entire setup at work to use clojure -T:build ... for everything!)#2021-08-0604:07seancorfieldI'm looking for feedback on clj-new 2.0. Feel free to jump into #clj-new or just reply here in a thread. Thank you!#2021-08-0612:23tvaughanA setup for developing in a container #2021-08-0617:15seancorfield@U0P7ZBZCK Can you elaborate on that? I don't understand why developing in a container is different locally, from the p.o.v. of clj-new?#2021-08-0617:37tvaughanSorry, I only read as far as "your wish-list" on first glance. I think what I'm after is simply a template for developing in a container. So yeah not really a helpful contribution after reading the rest of the post#2021-08-0617:52seancorfieldI meant "I don't know what a template for developing in a container would look like" -- I don't understand why it would be different to the default app or lib template.#2021-08-0617:52seancorfieldI'm happy to add/modify the built-in templates to make "developing in a container" easier -- but I don't know what that entails.#2021-08-0618:02tvaughanAh ok. For example, I structure my projects where the top-level directory is "what runs on my laptop," e.g. what's required to create and run the container and then "what runs inside the container" is in a sub-directory, like https://gitlab.com/tvaughan/kibit-runner. For larger projects involving multiple containers, as well as limiting the host mounted volume to each project's sub-directory instead of the top-level directory (see the top-level Makefile) this layout has been really helpful#2021-08-0618:44seancorfieldSee https://github.com/seancorfield/deps-new/issues/4 -- that will be possible out-of-the-box via :target-dir (which exists) and whatever modification of :overwrite I end up with.#2021-08-0618:57tvaughanAwesome @seancorfield#2021-08-0608:47gmercerIs there a seperate tools.build channel ? Should there be ?
Just wanting some help on correctly setting the Main-Class in the jar MANIFEST.MF#2021-08-0617:43seancorfield@gmercer Currently this channel serves tools.build as well as t.d.a -- I'll update the topic to make that clearer.#2021-08-0623:27Cam SaulWith the Clojure CLI is there an easy way to make it use a different version of the tools.deps.alpha or tools.build libs (for local development)? Or would I have to build and install a new version of the CLI?
I want to point the CLI to a fork to t.d.a on my local machine... I have a patch that adds support for an optional :aliases argument to clojure.tools.cli.api/prep (i.e., lets you do clojure -X:deps prep :aliases '[:my-additional-alias-to-prep]' ). I have it working locally with a -T command I wrote to wrap the call to c.t.cli.api/prep but wanted to actually confirm it works with -X:deps so I don't waste Alex's time with untested patches#2021-08-0623:30seancorfield@camsaul I think you would need to modify clojure, around line 274:
tools_cp="$install_dir/libexec/clojure-tools-$version.jar"
to add the path to your local t.d.a ahead of that.#2021-08-0623:31seancorfieldtools_cp="/path/to/your/t.d.a:$install_dir/libexec/clojure-tools-$version.jar"
#2021-08-0623:33seancorfieldOh, wait, no. It's easier than that. You just want -X:some-alias (instead of -X:deps) where :some-alias loads your t.d.a instead.#2021-08-0623:34seancorfieldI think you could even use :deps in a user or project deps.edn which would "win" over the root deps.edn alias, which is:
:deps {:replace-deps {org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"}
org.slf4j/slf4j-nop {:mvn/version "1.7.25"}}
:ns-default clojure.tools.cli.api
:ns-aliases {help }}#2021-08-0623:34seancorfieldI used to have :deps in my user deps.edn pointing to the add-lib3 branch on GitHub (but renamed it because it shadowed the root :deps alias!).#2021-08-0623:36Cam Sauloh yeah, adding a :deps ~profile~ alias and replacing t.d.a. with a :local/root coordinate did the trick. Thanks @seancorfield#2021-08-0623:36seancorfields/profile/alias/ 🙂#2021-08-0623:38Cam Saulhaha I'm still getting used to that#2021-08-0623:38seancorfieldIs there an http://ask.clojure.org question or JIRA ticket relating to this?#2021-08-0623:39seancorfield(I don't see anything on ask but may not have searched for the right terms)#2021-08-0623:42Cam SaulI don't think there's an http://ask.clojure.org question or JIRA ticket, but I asked Alex about it 3 or 4 weeks ago in this Slack channel and he said it's something he was planning on adding in the near future. Thought I could save him a bit of trouble since I have it working locally. I'll create an Ask question and JIRA issue for it in a second#2021-08-0623:56seancorfield(mostly so I can go vote for it 🙂 and also look at the patch out of curiosity)#2021-08-0700:18Cam Saulhttps://ask.clojure.org/index.php/10915/support-preparing-deps-extra-deps-in-aliases and https://clojure.atlassian.net/browse/CLJ-2652. Patch is attached to the JIRA issue... actually a really simple change#2021-08-0700:33seancorfieldOh, that's really nice and elegant!#2021-08-1000:09Alex Miller (Clojure team)there is a new Clojure CLI prerelease out that you could try if you like:
• TDEPS-199 Use default http-client in S3 transporter
• Cache S3 transporter for a repo
• Fixed session cache to work properly across threads / binding stacks for better perf
• Replace specific maven version range requests with non-range request to reduce repo metadata lookups#2021-08-1000:11Alex Miller (Clojure team)Mainly, it should be faster to make classpaths. It may also improve some of the Maven concurrency issues as a side effect. I wasn’t working on that explicitly, but may be improved. So if you are using -Sthreads 1 on a ci server to avoid that, I’d love to hear if it helps#2021-08-1002:14Alex Miller (Clojure team)I guess I forgot the version 1.10.3.939#2021-08-1020:14Alex Miller (Clojure team)new Clojure CLI prerelease with some additional performance enhancements: 1.10.3.943
• Load and cache Maven settings once for perf
• Cache version range resolution results for perf#2021-08-1020:19Alex Miller (Clojure team)in the original case I've been working on the total download time has gone from ~58 sec to 7 sec - this is just one use case and is pretty heavily impacted by what transitive deps you have, whether version ranges are in use, whether you're using an s3 repo, etc. and of course, if you have a cached classpath, nothing changing here is being run, so caveat emptor#2021-08-1020:22seancorfieldSounds like it might benefit CI setups quite a bit @alexmiller?#2021-08-1020:32Alex Miller (Clojure team)I think so, yes#2021-08-1020:33Alex Miller (Clojure team)would appreciate anyone kicking tires (and turning off -Sthreads 1 if you're using that - I believe I have probably mitigated a lot of the maven concurrency issues we were seeing so this should not be needed and will of course be faster). have not worked on the git download concurrency issues, so you may still need it for that.#2021-08-1020:34Alex Miller (Clojure team)if you do find a reproducible (-ish) case for maven concurrency issues, would love to have a deps.edn that demonstrates that#2021-08-1020:56seancorfieldI opened a ticket at work to vendor in the .943 prerelease. Should get to that tomorrow and run some local tests then see how it does on our CI setup.#2021-08-1116:22seancorfield@alexmiller So far, so good. Definitely seems faster to download everything.#2021-08-1021:05deactivateduserSorry if this has been answered (my search-fu is weak), but if I want to use t.d.a. programmatically (as a library), specifically with add-libs support, is the add-lib3 branch on GitHub still my best source? I notice it’s 5 months / 113 commits behind master, hence the question.#2021-08-1021:25Alex Miller (Clojure team)yeah, it hasn't been merged recently. not sure it will work with Clojure CLI as is#2021-08-1021:27Alex Miller (Clojure team)I just merged master into it in case that helps you#2021-08-1021:27deactivateduserThanks! I’ll give it a spin and see what happens.#2021-08-1021:29Alex Miller (Clojure team)note also that master is actually ahead of the stable release, so you're effectively getting the changes mentioned above too#2021-08-1021:30Alex Miller (Clojure team)just out of curiosity, what do you use it for?#2021-08-1021:40deactivateduserI’m messing about with a little “jig” project to make it easier to find, introspect, and prototype with 3rd party Clojure or Java libraries from the REPL, without having to mess with deps (either deps.edn or -Sdeps) and restart the REPL.
Obviously it will have the same limitations as add-libs itself, but that’s still better than my current workflow (look for a library in my browser, express a dep to it, (re)start my REPL to pick it up, start exploring the library’s API, realise it doesn’t meet my needs, pick an alternative, …repeat…).#2021-08-1021:40deactivateduserIt’s open source (ofc), and happy to share a link once I finish battling with git identities on this new(ish) laptop…#2021-08-1021:41deactivateduserBe warned that it is pretty hacky - it’s intended to be a cheap & cheerful tool to support my actual development, not something I’m super focused on in and of itself.#2021-08-1105:01deactivateduserJust reporting back in to say that as you predicted the add-libs3 branch doesn’t work.
When invoked with clj -Sdeps 'org.clojure/tools.deps.alpha {:git/url "" :sha "02809dbd41d4fe6e269ee67316d054a04bc70678"}' -r, an IllegalArgumentException is thrown and the REPL fails to start.
When invoked with that coordinate in a deps.edn file and started with clj, the REPL starts, and the clojure.tools.deps.alpha.repl namespace can be required. But calling add-libs always returns nil, albeit with a tremendous amount of Apache HttpComponents logging output if the dependency doesn’t yet exist in ~/.m2 (and the JAR file and associated metadata files appear to be downloaded correctly - they just don’t get added to the active classloader).#2021-08-1120:32Jacob EmckenI'm was looking for a way to do lein install but with cli tools. I found clojure -X:deps mvn-install but that requires me to reference a jar file. So I guess my question is... how to I build the jar file then? 😅#2021-08-1120:33Alex Miller (Clojure team)tools.build of course :)#2021-08-1120:33Alex Miller (Clojure team)https://clojure.org/guides/tools_build#2021-08-1120:47Jacob EmckenI cloned the Clojure library https://github.com/nextjournal/beholder tbh I don't understand how I am supposed to build it#2021-08-1120:48seancorfieldYou generally don't need to install libraries locally to use them. Just declare them as dependencies in your project.#2021-08-1120:48Jacob EmckenI know that but I changed something in it that I want to try out#2021-08-1120:49seancorfieldOK, so use :local/root as a dependency on it.#2021-08-1120:49Jacob Emckenin a lein project 😉#2021-08-1120:49seancorfieldHaha... OK, well, there's your problem 🙂#2021-08-1120:50seancorfieldOK then probably the easiest way to make a JAR from beholder is to use depstar, since beholder doesn't have a build script or any aliases for that.#2021-08-1120:51Jacob Emckento figure out what that is#2021-08-1120:58seancorfieldHere's the steps you need:
(! 1026)-> git clone
#2021-08-1120:58seancorfield(edited to show actual commands and results)#2021-08-1121:03seancorfieldThen you can depend on it (via lein or here in a CLI REPL):
(! 1033)-> clj -Sdeps '{:deps {beholder/beholder {:mvn/version "0.1.0"}}}'
Clojure 1.10.3
user=> (require 'nextjournal.beholder)
nil
user=> (nextjournal.beholder/watch prn ".")
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
#object[io.methvin.watcher.DirectoryWatcher 0x65bb9029 "#2021-08-1121:04dominicmbeholder officially uses https://github.com/applied-science/deps-library for it's jars/release process. So that's probably the supported path.#2021-08-1121:04dominicmThat includes the install step too#2021-08-1121:05seancorfieldThe beholder project only seemed to have a :release alias so I wasn't sure what would be needed without also tracking down that library and learning about it 🙂#2021-08-1121:05seancorfield@jacob429 Hope the above gets you going...#2021-08-1121:05dominicmI hadn't heard of it before, but searched for "release.edn" on github and it came up 🙂#2021-08-1121:07dominicmOh, so it's actually as simple as clj -M:release install#2021-08-1121:08seancorfield(pity there's no docs in beholder or its deps.edn file to make that clear since they're using a deps/release process that is not common/mainstream 🙂 )#2021-08-1121:09dominicmIs there a standard release process yet? (depstar doesn't count)#2021-08-1121:10seancorfieldI think depstar is the most widely used JAR builder for deps.edn-based projects currently? But I hope it will be able to go away once tools.build's uber task does more merging stuff.#2021-08-1121:10winsomeI'd like to have deps.edn be my source of truth and use it to generate my pom.xml - is there a way to specify what should go in the <scm> tag? Right now cljdocs integration is broken because I don't have a source link in there.#2021-08-1121:10winsomeOh, I'm using depstar, btw, with :sync-pom true#2021-08-1121:11seancorfield@winsome that's why clj-new generates a fully-fledged pom.xml into new projects.#2021-08-1121:11winsomeCan I use clj-new on an existing project, or is that just like lein new?#2021-08-1121:12seancorfieldIt's just like lein new. But you could use it to create a new project somewhere, with the same qualified project name that your current project has, and then copy the generated pom.xml file into your project.#2021-08-1121:12seancorfieldI'm working on deps-new which will be able to overlay new files into existing projects (and is much, much simpler to write new templates for).#2021-08-1121:13winsomeThat sounds good, thanks. And deps-new is a great idea!#2021-08-1121:13seancorfieldAdding a full-featured pom.xml file is a good new feature for deps-new -- I'll create an issue.#2021-08-1121:14winsomeyeah, it's something I never think of until it's release time, which can be quite distant from creation time (never in most cases :p)#2021-08-1121:14seancorfieldhttps://github.com/seancorfield/deps-new/issues/10#2021-08-1121:25Jacob Emcken@dominicm Thanks clojure -M:release install seemed to work#2021-08-1122:45salamis there a way to configure the clj/`clojure` commands globally so that we can apply a particular command line argument to all clj/`clojure` calls? for example, i would like to set --report stderr globally so that i don't have to change each and every existing call to these commands.#2021-08-1122:45Alex Miller (Clojure team)not currently#2021-08-1122:45salamok. thanks, Alex.#2021-08-1122:46Alex Miller (Clojure team)in general, I think -J-Dclojure.main.report=stderr is better for the Clojure CLI as it composes better with main args and -X etc#2021-08-1122:47Alex Miller (Clojure team)you could make an alias with :jvm-opts ["-Dclojure.main.report=stderr"] that could be combined with any other alias#2021-08-1122:48Alex Miller (Clojure team)this question is for default jvm opts https://ask.clojure.org/index.php/7834/support-default-jvm-opts if you want to vote for it (which would let you do the thing above more broadly)#2021-08-1122:49Alex Miller (Clojure team)but I think adding such a thing needs much more consideration in how it combines#2021-08-1122:53salamunderstood. i'll go with the alias approach for now as we already have :dev alias that's referenced at every call site to clojure. thank you.#2021-08-1208:10raymcdermottquestion on procurers: is there a DB procurement option eg from code stored in Datomic or similar?#2021-08-1208:13raymcdermottIf not, how much of the infrastructure is reusable once the code has been procured?#2021-08-1212:30Alex Miller (Clojure team)Nothing like that now. The procurer impl needs to be able to find and download a lib and determine it’s deps.#2021-08-1212:30Alex Miller (Clojure team)Depending how you stored stuff, could reuse lots of existing pieces#2021-08-1212:47raymcdermottMy thinking was having a project that looked something like {:project "epic" :code {:ns "my.foo" :code "(def bar 42)"}} which could be stored directly in a DB that kept history. Then you pick the latest view of it, serialize the namespaces to files (urgh but OK if we have to) and add the namespaces to the CLASSPATH. Which feels close to what you do with git deps. I would rather avoid the files but not sure if this is possible. Is it?#2021-08-1212:49raymcdermottare you open to a PR on this kind of thing?#2021-08-1212:51Alex Miller (Clojure team)It’s all ultimately file based (you’re always making a classpath), so can’t avoid files. (I have made similar db-based dynamic classloaders before, but that’s attacking a different level)#2021-08-1212:52Alex Miller (Clojure team)Not interested in adding that to tools.deps but it’s all open multimethods so no need to be in the project#2021-08-1212:55Alex Miller (Clojure team)The one big caveat is that the Clojure CLI currently has no hook to load such a thing (not a problem when using tools.deps directly as a lib). I’ve been pushing that off as a problem to solve when it needed solving#2021-08-1212:56Alex Miller (Clojure team)All of the extension points are defined in clojure.tools.deps.alpha.extensions#2021-08-1212:59raymcdermottok, thanks. Understood#2021-08-1217:06seancorfield@alexmiller With t.d.a, if it encounters the same dependency in multiple places in the dependency tree, both with git SHAs, can it compare them in some circumstances but not others, or does it always throw Unable to compare versions for... regardless of whether one SHA is an ancestor of the other?#2021-08-1217:10Alex Miller (Clojure team)it can compare them in any case where there is an ancestor relationship and otherwise fails#2021-08-1217:11Alex Miller (Clojure team)in the latter case, there is no way to determine which is preferred (so you'll need to resolve it explicitly at the top level)#2021-08-1217:13Alex Miller (Clojure team)the more descendant one is preferred (as it is presumably the newer one)#2021-08-1217:17seancorfieldThanks.#2021-08-1302:28seancorfieldFor some tools.build fun, I switched HoneySQL over to using that for all its CI needs: https://github.com/seancorfield/honeysql/blob/develop/build.clj -- CircleCI and GH Actions run this now. Multi-version Clojure testing, ClojureScript testing, Eastwood, Readme testing, JAR-building. Only the Clojars deploy is manual (and currently has to use -X so :pom-file can be provided -- the -M version of deps-deploy doesn't support that option).#2021-08-1302:46seancorfield(TIL: git rev-list --count HEAD produces 1 when run in a GH Actions script???)#2021-08-1302:48dpsuttoni can recreate that with git clone --depth 1 <repo>#2021-08-1302:48dpsuttonwhich is probably analogous to what gh is doing#2021-08-1303:03seancorfieldAh, thanks @dpsutton -- yeah, I think you're right that it just clones the most recent commit and nothing else.#2021-08-1312:48lread@seancorfield You can ask GitHub Actions to clone it all via fetch-depth: 0, https://github.com/clj-commons/rewrite-clj/blob/bd3905d0dcc40a499dc080e1e0b8f665f9b39a16/.github/workflows/release.yml#L15-L18.#2021-08-1317:12dorabIs there a way to prevent build/wite-file from doing a pr-str on the :content? In that way, it is different than spit.#2021-08-1317:13Alex Miller (Clojure team)sure, pass it the string you want to write#2021-08-1318:04dorabTaking the discussion to tools-build#2021-08-1317:14Alex Miller (Clojure team)btw, I've created #tools-build if anyone wants to have future discussion there#2021-08-1318:41seancorfield@alexmiller If t.d.a encounters two git deps with the same URL (and same SHA) but two different :deps/root values, could that cause the "cannot compare versions" error?#2021-08-1318:42seancorfieldThis is the error (reformatted a bit):
Unable to compare versions for repro-lib/root:
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"} and
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"}
{:lib repro-lib/root, :coord-x {:local/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"}, :coord-y {:local/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"}}
Unable to compare versions for repro-lib/root:
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/red/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"} and
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/blue/a252586fe4831fe8a719571793cfe81f1f4f7e88/Root"}
#2021-08-1318:43seancorfieldThis is the issue where this has come up https://github.com/polyfy/polylith/issues/108 and the repro case is explained/linked in that.#2021-08-1318:46seancorfieldI initially wondered whether it might be a case-sensitivity issue -- the repo has Root, Blue, Red but the :local/root and :git/url deps both have blue and red instead. But then I noticed it is comparing across two different URLs (repro-lib/red and repro-lib/blue) with the same SHA and both have Root? (This might also be an issue with Polylith not respecting :exclusions though).#2021-08-1319:30Alex Miller (Clojure team)whether it's a case problem will likely depend on your os - if you're using a case-sensitive file system, then probably an issue#2021-08-1319:31Alex Miller (Clojure team)those errors above (while in gitlibs dir) are local deps, which can never be compared, as they are just seen as arbitrary files with no known relationship#2021-08-1319:32Alex Miller (Clojure team)so prior questions about gitlibs are irrelevant as those are not at play here#2021-08-1319:33seancorfieldWell, other than these being git deps -- and having :local/root deps inside them?#2021-08-1319:34Alex Miller (Clojure team)but ... they're NOT git deps#2021-08-1319:34Alex Miller (Clojure team):local/root == local dep#2021-08-1319:37seancorfieldFrom the project's deps.edn: https://github.com/imrekoszo/MinimalBugRepro/blob/no-common-transitive-deps/deps.edn#L4-L9#2021-08-1319:37seancorfieldNote that they are both git deps to the same repo but different :deps/root values.#2021-08-1319:37Alex Miller (Clojure team)there are like 3 repro projects, sorry#2021-08-1319:39Alex Miller (Clojure team)although my comment on your errors above stands - those are local deps#2021-08-1319:40seancorfieldThose git deps point to subprojects that have deps like https://github.com/imrekoszo/ReproLibrary/blob/master/Red/deps.edn#2021-08-1319:40seancorfieldand that root project has no deps#2021-08-1319:41Alex Miller (Clojure team)if you have 2 git deps, that both point to local deps within their git repo, then the idea that you're "in" a git dep is lost and there is no way to compare those two deps#2021-08-1319:42Alex Miller (Clojure team)as in the error above, you just have two pointers to the file system#2021-08-1319:42Alex Miller (Clojure team)we've talked about this general problem before here wrt monorepos and there is a ticket for it#2021-08-1319:43seancorfieldOK, so this is related to :deps/root? I'm just trying to understand exactly what combo causes this. It looks like have two "identical" git deps with different :deps/root values is what is causing the transitive deps to end up being incomparable?#2021-08-1319:44Alex Miller (Clojure team)I think you have something like:
<main>
git dep A (+ sub path)
local dep to ../foo
git dep B (+ sub path)
local dep to ../foo#2021-08-1319:45seancorfieldI double-checked and was on the wrong branch above (Imre has simplified the repro quite a bit now), and get this error:
Unable to compare versions for repro-lib/green:
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/red/4237c6492172dc6a4102ccfc34df6316ccd1734b/green", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/red/4237c6492172dc6a4102ccfc34df6316ccd1734b/green"} and
{:local/root "/Users/sean/.gitlibs/libs/repro-lib/blue/4237c6492172dc6a4102ccfc34df6316ccd1734b/green", :deps/manifest :deps, :deps/root "/Users/sean/.gitlibs/libs/repro-lib/blue/4237c6492172dc6a4102ccfc34df6316ccd1734b/green"}
so they are ultimately two paths to the "same" lib, but via different :dep/root -- yes, exactly the above!#2021-08-1319:46seancorfieldSo the paths to green here are via red in one path and blue in the other -- hence uncomparable b/c they're no longer just "git deps".#2021-08-1319:46Alex Miller (Clojure team)comparisons are made based on two local coords there which (from tools.deps pov) point to arbitrary places in the filesystem, which cannot be compared#2021-08-1319:48Alex Miller (Clojure team)I think this is the same general problem as https://clojure.atlassian.net/browse/TDEPS-132#2021-08-1319:49seancorfieldYup, looks like it. Thank you for sticking with me on that -- we got there in the end 🙂#2021-08-1319:49Alex Miller (Clojure team)I think the "A while ago we discussed the option of a “nearest git repository” on slack. I wanted to record that. But it’s also lacking a problem statement. The idea was a new coordinate type which would search for its nearest repo. In the case of working within the repo, it would search upwards and find the .git folder. In the case of a git/url, it would do the same. This would solve the transitive dependency problem." idea in the comments is interesting but I have not spent a lot of time thinking about it.#2021-08-1319:49Alex Miller (Clojure team)you really want a local dep that retains the git context of its dependee#2021-08-1319:51Alex Miller (Clojure team)and then comparisons would basically need to use git coord comparison rather than local coord comparison#2021-08-1319:51seancorfieldSo git dep A + sub path X and git dep A + sub path Y both become (different) local deps, in essence?#2021-08-1320:01Alex Miller (Clojure team)depends what you mean by that notation :)#2021-08-1320:02Alex Miller (Clojure team)in any case, I'm not going to work on this today#2021-08-1322:17seancorfield@alexmiller I'm looking at how t.d.a handles :exclusions. It's only documented for :mvn/version coords but it looks like it actually works for all types of coords?#2021-08-1322:17Alex Miller (Clojure team)yes#2021-08-1322:18seancorfieldThanks. Oh, and the problem above ultimately traced back to Polylith's test tool not respecting :deps/root -- @imre helped me track that down.#2021-08-1615:19tengstrandIs this something that needs to be fixed @seancorfield?#2021-08-1617:43seancorfield@U1G0HH87L I submitted a PR for the fix.#2021-08-1618:33tengstrandYour changes looks good! My approval of the PR didn’t trigger the CI build, so I’m looking into that.#2021-08-1618:36tengstrandIt’s merged now!#2021-08-1618:41seancorfieldThank you! Woohoo! I'm a contributor now! 🙂#2021-08-1618:42seancorfieldI would have included tests if I could have figured out easily where to put them and how to trigger that code path 😐#2021-08-1410:20henryw374I want to change the default timezone in my jvm. In the past, with lein, I think I just set the user.timezone property.
with clj, if I haven't set it myself, I find that property is already set, in my case to Europe/London. If I do try setting it with :jvm-opts ["-Duser.timezone=\"America/New_York\""]
I find that prop is set to "GMT+01:00" , via (System/getProperty "user.timezone") .
If I run a java main, ie no clj, the prop is not set by default. Java 11.
anyone got any pointers? Thanks#2021-08-1419:05seancorfield:jvm-opts has to be under an alias -- not at the top level. You can use -J-Duser.timezone=... as a command-line argument to clj` tho'...#2021-08-1419:06seancorfield(! 1427)-> clj
Clojure 1.10.3
user=> (System/getProperty "user.timezone")
"America/Los_Angeles"
user=> ^D
(! 1428)-> clj -J-Duser.timezone=America/New_York
Clojure 1.10.3
user=> (System/getProperty "user.timezone")
"America/New_York"
user=>
#2021-08-1419:06seancorfield@U051B9FU1 ☝️:skin-tone-2:#2021-08-1419:58henryw374Thanks I'll have a go#2021-08-1506:09henryw374all good now. you got me on the right path. thanks#2021-08-1411:59Alex Miller (Clojure team)clj just calls java, there is no special handling here#2021-08-1416:13henryw374Hmm. Ok thanks#2021-08-1506:08henryw374ah my fault, I had escaped quotes around the zone name which are not needed. so `
:jvm-opts ["-Duser.timezone=America/New_York"]
`
works.#2021-08-1714:48imretried a brew upgrade today:
% brew upgrade clojure/tools/clojure
==> Upgrading 1 outdated package:
clojure/tools/clojure 1.10.3.933 -> 1.10.3.943
==> Upgrading clojure/tools/clojure
1.10.3.933 -> 1.10.3.943
==> Downloading
Already downloaded: /Users/ikoszo/Library/Caches/Homebrew/downloads/5c6b0abbbfd924d4e8b2057a3a9f89f28760bf1ef52d1467abee7d379fa0737f--clojure-tools-1.10.3.943.tar.gz
Error: An exception occurred within a child process:
NoMethodError: undefined method `path' for nil:NilClass
Did you mean? paths
#2021-08-1714:49imredid I mess something up here?#2021-08-1715:01Alex Miller (Clojure team)I think someone else reported this somewhere - what mac os are you on?#2021-08-1715:02Alex Miller (Clojure team)I think it's something in brew itself#2021-08-1715:04Alex Miller (Clojure team)yeah, this same error was reported last week on macOS 12.0 beta#2021-08-1715:10imremacos 11.5.2#2021-08-1715:11imrelet me try a debug upgrade#2021-08-1715:12Alex Miller (Clojure team)can you run brew doctor ?#2021-08-1715:12imresure thing#2021-08-1715:13Alex Miller (Clojure team)seems like often this error occurs if xcode is out of date#2021-08-1715:13Alex Miller (Clojure team)according to the googles#2021-08-1715:14imredoctor does report my xcode is out of date#2021-08-1715:14imreI'll try to update that#2021-08-1715:16Alex Miller (Clojure team)xcode-select --install
#2021-08-1716:07imrecheers, that indeed fixed it!#2021-08-1716:08imreinteresting tho, the xcode update had to be forced, it did not show up in any lists#2021-08-1717:29cap10morganIs there a way to specify a subdirectory in a git library in my deps.edn? I.e. the pom.xml and src dir are not in the root of the git repo.#2021-08-1717:34rwstauner:deps/root ?#2021-08-1717:35rwstaunerOptional key :deps/root
Specifies the relative path within the root to search for the manifest file
#2021-08-1717:35rwstaunerhttps://clojure.org/reference/deps_and_cli#_dependencies#2021-08-1717:36cap10morganBet that's it! Thanks @clj149 (and for the reference link)!#2021-08-1720:27Alex Miller (Clojure team)New prerelease of Clojure CLI version 1.10.3.949 is now available. The only change in this is an update to all deps (aws api, maven resolver, maven core). I did find one pretty glaring breaking regression in the latest maven core which I suspect will be rolled back in the next release, but it's patched around in tools.deps. One other notable change is that the latest maven core fixes a lot of concurrency issues that were introduced in the prior version (which does coincide with our increase in seeing those issues with the Clojure CLI). So, hopefully, this is better behaved in that respect. Anyways, if anyone wants to try it out, would be grateful for any reports.#2021-08-1720:39cap10morganI started looking into what it would take to add lein project.clj manifest support to tools.deps.alpha and wanted to see what might or might not be acceptable w/r/t adding additional deps:
1. OK to pull in leiningen-core and use its project.clj reading code?
2. better to reimplement just enough project.clj reading in tools.deps.alpha to get what we need?
3. don't work on this right now because reasons? :)#2021-08-1720:41Alex Miller (Clojure team)1. no#2021-08-1720:42Alex Miller (Clojure team)2. unsure what the effort/reach of doing this without 1 yields#2021-08-1720:43cap10morganyeah. it looks like there is potentially a decently-small subset of the project.clj reading code we could copy from leiningen-core b/c we don't need / want e.g. profiles support, etc.#2021-08-1720:45Alex Miller (Clojure team)the impl should look like clojure.tools.deps.alpha.extensions.deps - that's basically the same set of extension methods that need to be implemented (plus adding the manifest-type detector at the top of clojure.tools.deps.alpha.extensions)#2021-08-1720:45cap10morganyep, makes sense#2021-08-1720:45souenzzo@cap10morgan should be possible to develop somthing like that:
deps.edn
{:aliases {:lein {:deps {...lein-deps-connect ...} :fn lein-deps-connect}}}
Then you run clj -X:lein uberjar and it will generate an uberjar following project.clj spec#2021-08-1720:46cap10morganthat's not quite what I'm trying to do#2021-08-1720:47cap10morganI'm trying to add support to tools-deps so that you can specify a lein-only dependency (most often as a git lib) and it will read from the project.clj (vs. needing to add a pom.xml to the repo)#2021-08-1720:51souenzzoyeah it will be harder.#2021-08-1720:57cap10morganyep 🙂#2021-08-1720:48cap10morgan@alexmiller what is the top level repo I need to build a clojure command w/ my modified tools.deps.alpha?#2021-08-1721:01cap10morganhttps://github.com/clojure/brew-install is what I want I think#2021-08-1721:14Alex Miller (Clojure team)the easiest way is to actually NOT do that#2021-08-1721:15Alex Miller (Clojure team)instead, modify your which clojure in the line that starts tools_cp=" to include at the beginning /path/to/your/tools.deps.alpha/src/main/clojure:#2021-08-1721:16Alex Miller (Clojure team)then your own tools.deps source is hooked into your existing clj (make sure to use -Sforce to ensure you're triggering classpath gen)#2021-08-1721:23cap10morganah, beautiful. thanks!#2021-08-1723:45cap10morganGot a first basic cut at project.clj manifest support working: https://clojure.atlassian.net/browse/TDEPS-204#2021-08-1803:17cflemingFor those having problems with Cursive and git deps with recent versions of Deps, the new EAP fixes it: https://twitter.com/CursiveIDE/status/1427831439660179465#2021-08-1813:23imreworks on my machine 🙂#2021-08-1820:00dominicmhttps://github.com/clojure/tools.deps.graph I'm surprised there're no instructions for use with -T !#2021-08-1820:01dominicm(ah, there is on http://clojure.org though)#2021-08-1820:08dominicmI give up, what am I doing wrong?
~
❯ clj -Ttools
Error building classpath. Bad coordinate for library , expected map: nil
#2021-08-1820:12seancorfieldclojure -Ttools <command> <args>#2021-08-1820:12seancorfielde.g., clojure -Ttools install lib/name '{coords}' :as alias#2021-08-1820:12seancorfieldand then clojure -T<alias> <args>#2021-08-1820:13seancorfield@dominicm Like on the README for clj-new:
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.331"}' :as clj-new
# create a new app:
clojure -Tclj-new app :name myname/myapp
# create a new library:
clojure -Tclj-new lib :name myname/mylib
# create a new template:
clojure -Tclj-new template :name myname/mytemplate
# create a new project from a public template:
clojure -Tclj-new create :template electron-app :name myname/myelectron-app#2021-08-1907:55dominicm@seancorfield I get the same error, even for clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.1.331"}' :as clj-new
I'm on "1.10.3.933" (via -Sdescribe)#2021-08-1908:04dominicmLooks like the void linux build is broken, it's missing tools.edn#2021-08-1914:06Alex Miller (Clojure team)not sure who/where that's maintained but that is a relatively new part#2021-08-1914:06Alex Miller (Clojure team)you can manually install it though by just creating the file at ~/.clojure/tools/tools.edn#2021-08-1914:08Alex Miller (Clojure team)content is https://github.com/clojure/brew-install/blob/1.10.3/src/main/resources/clojure/install/tools.edn#2021-08-1916:25seancorfield@U09LZR36F How did you install the 1.10.3.933 CLI?#2021-08-1920:09dominicm@U064X3EF3 I maintain it! 🙂 It's fixed now.#2021-08-1920:10dominicmhttps://github.com/void-linux/void-packages/pull/32579 void is now on 934 + with tools.edn present#2021-08-1920:10dominicmIt's not too difficult to maintain for anyone tbh. Anyone can send the prs if they're so inclined.#2021-08-1909:13rickmoynihanI’ve just been playing around with some of the new -Ttools support…
One thing that took me a while to figure out is how to get the help / introspection on tools working .
Is there anyway that this invocation could be made easier somehow in the future, perhaps with some “easy” -H switch in the shell script that expands to the right invocation?
i.e. going from these:
$ clojure -A:deps -Tnew help/doc
$ clojure -A:deps -Tnew help/dir
To something like this:
$clojure -Hnew doc
$clojure -Hnew doc
Or am I missing an easier way to do this sort of thing?#2021-08-1916:27seancorfieldThe clj-new README provides that command:
> As of Clojure CLI 1.10.3.933, you can get help about the functions available in clj-new:
clojure -A:deps -Tclj-new help/doc
#2021-08-1916:27seancorfieldI should add it to the deps-new help tho'...#2021-08-1916:31seancorfield(updated readme for deps-new)#2021-08-1912:53Alex Miller (Clojure team)Not an easier way for tools at the moment but this is certainly not a finished area#2021-08-1913:36simongrayI have a pretty standard src folder structure, but I have a certain subdirectory that I would like to be excluded from my root :paths (e.g. in this case src/dk/wordnet/prototypes) and added to the :extra-paths of an alias instead. Can this be accomplished without me having to change the layout of the source directory?#2021-08-1913:39Alex Miller (Clojure team)No#2021-08-1914:00simongrayWell that sucks#2021-08-1914:00simongrayOk, I guess I will make a separate src dir for that code then#2021-08-1914:01simongrayNot too bad, just wanted to know if there was a less intrusive way to make the change#2021-08-1916:27seancorfieldThe clj-new README provides that command:
> As of Clojure CLI 1.10.3.933, you can get help about the functions available in clj-new:
clojure -A:deps -Tclj-new help/doc
#2021-08-2005:32seancorfieldDoes "tool" installation support projects in a subfolder of the repo? I'm thinking of Polylith's tool which would sort of be clojure -Ttools install com.github.polyfy/polylith '{:git/tag "v0.2.0-alpha11" :git/sha "f7bae45" :deps/root "projects/poly"}' :as poly (although it does not yet have a -X style entry point).#2021-08-2005:33seancorfieldI'm wondering which deps.edn the CLI would use and what :tools/usage would look like?#2021-08-2012:57Alex Miller (Clojure team)The tool definition is just a tool name that resolves to a lib/coord so I think that should work#2021-08-2012:58Alex Miller (Clojure team)The usage is just ns-default/ns-aliases so that does not seem affected#2021-08-2012:59Alex Miller (Clojure team)I would expect it to use the one at projects/poly#2021-08-2012:59Alex Miller (Clojure team)I haven’t tried any of this though :)#2021-08-2016:31seancorfield@alexmiller I went ahead and tried it, and it works "as expected". I plan to add a "tools" API to Polylith this weekend so it can leverage the new CLI features.#2021-08-2309:38Ben SlessIs there a way to work nicely with a tool which does not take a map as its argument?#2021-08-2309:40delaguardoyou can create a proxy function which will accept a map as an argument an call the tool you want to use#2021-08-2309:41Ben SlessSo if I'm getting it from github I have to either provide a compatibility layer on my own or MR the author?#2021-08-2309:43delaguardoif this “tool” is hosted on github I think it wasn’t designed to work as a tool for clojure cli, am I right?#2021-08-2310:25Ben Slessyes#2021-08-2310:31delaguardothen I think the best option is to file an issue about additional clojure cli support and in the mean time add a wrapper into your project#2021-08-2317:25seancorfield@UK0810AQ2 The -X/`-T` stuff is still pretty new so a lot of library/tool maintainers haven't caught up with it yet. For -T, a project needs :tools/usage in its deps.edn file, as well as a compatible entry point. Most project maintainers seem amenable to adding this I think.#2021-08-2317:25seancorfieldPolylith just accepted a PR to make it installable via -Ttools install and Cognitect's test-runner added a -X-compatible API namespace a while back after I raised that issue.#2021-08-2317:26seancorfieldIn both cases, I started out by writing my own exec fn wrapper that called into the project's code -- until the issue was resolved directly in the project.#2021-08-2418:57Jeff Evansanyone know what might be causing this kind of error, when invoking clojure -X:deps prep
Cloning:
Checking out: at 1e7c019730dc6f9e38793170c8801c5950516b60
Downloading: org/clojure/tools.namespace/1.0.0/tools.namespace-1.0.0.pom from central
Downloading: org/clojure/tools.namespace/1.0.0/tools.namespace-1.0.0.jar from central
Execution error (IOException) at java.lang.ProcessImpl/forkAndExec (ProcessImpl.java:-2).
error=2, No such file or directory
more context in thread#2021-08-2418:58Jeff Evansthis is running inside a Docker image which itself is running via a GitHub action, so there is some degree of virtualized craziness going on. Clojure itself is being installed, via:
# install clojure version needed for Metabase
curl -O
chmod +x linux-install-1.10.3.933.sh
./linux-install-1.10.3.933.sh
#2021-08-2418:59Jeff EvansI also couldn’t quite figure out how to tell it to spit out the command it’s trying to fork and exec#2021-08-2419:06Alex Miller (Clojure team)are you using compile-clj?#2021-08-2419:07Alex Miller (Clojure team)if so, that's the most likely culprit as it forks a jvm to run the compile, and my best guess is that the directory it's trying to do that in doesn't exist#2021-08-2419:07Jeff EvansI’m not sure… I never explicitly call such a command from this script#2021-08-2419:07seancorfield@U0183EZCD0D Do you know what "prep" your dependencies have declared? That would at least tell you what it is trying to run.#2021-08-2419:08seancorfield@U064X3EF3 Is there some hidden "verbose" or debug flag that could be turned on here to help?#2021-08-2419:10Jeff EvansI’m afraid I still haven’t grokked deps enough to be able to understand that question 😞#2021-08-2419:18Jeff Evansactually, I figured it out, that java is not on $PATH apparently
{:type java.io.IOException,
:message
"Cannot run program \"java\" (in directory \"/app/source/java\"): error=2, No such file or directory",
:at [java.lang.ProcessBuilder start "ProcessBuilder.java" 1128]}#2021-08-2419:22seancorfieldFor the future, you'd be looking for :deps/prep-lib in deps.edn in one or more of the dependencies, which tells -X:deps prep what code needs to be run.#2021-08-2419:23Alex Miller (Clojure team)sorry I was in a meeting#2021-08-2419:24Alex Miller (Clojure team)I have not (yet) added any debugging here, but there is a lot of stuff that gets spit into the temp directory where this is happening and on error, it is left so you can examine the entrails#2021-08-2419:25Jeff Evansthis one?
Full report at:
/tmp/clojure-5839141627426774183.edn
#2021-08-2419:26Alex Miller (Clojure team)that's a separate thing, but also useful#2021-08-2419:26Alex Miller (Clojure team)that's the error reporting output from clojure.main#2021-08-2419:26Alex Miller (Clojure team)user=> (import [java.nio.file Files] [java.nio.file.attribute FileAttribute])
user=> (.toString (.toFile (Files/createTempDirectory "compile-clj" (into-array FileAttribute []))))
"/var/folders/_4/rhwlxzf92fj4727zsjk8xjtw0000gp/T/compile-clj17809906038309626783"
#2021-08-2419:27Alex Miller (Clojure team)should show you approximately the directory where it's happening (the number at the end will be different, but just look for the newest directory and that will have some stuff in it#2021-08-2419:28Alex Miller (Clojure team)there should be a compile.clj script there, and a compile.cp classpath file#2021-08-2419:28Alex Miller (Clojure team)although I don't think either would help you with the problem above#2021-08-2419:29Jeff Evansright, yeah, this ended up just being something silly. I appreciate the pointers, though, which I will consult next time. thanks to both of you!#2021-08-2500:56kennyPrefixing this with: I don't believe this to be any issue with tools.deps.
Has anyone seen "Premature end of Content-Length delimited message body" before? Maven issue with this particular artifact? At the moment, I can consistently reproduce this.
clj -Srepro -Sdeps '{:deps {com.ibm.icu/icu4j {:mvn/version "68.2"}}}'
Downloading: com/ibm/icu/icu4j/68.2/icu4j-68.2.jar from central
Error building classpath. Could not transfer artifact com.ibm.icu:icu4j:jar:68.2 from/to central (): Premature end of Content-Length delimited message body (expected: 11,963,048; received: 569,344)#2021-08-2501:03Alex Miller (Clojure team)If you delete that artifact from your ~/.m2 and re-get the dep, is it repeatable?#2021-08-2501:05Alex Miller (Clojure team)That sounds like a server side maven central thing but not sure#2021-08-2501:07kennyYes#2021-08-2517:42Joshua SuskaloDoes the :fn key need to be present in the :deps/prep-lib map if the alias that will be run already has an :exec-fn specified?#2021-08-2517:55Joshua SuskaloAh, I see, so prep-lib doesn't support an alias which has a function and args specified. That's unfortunate.#2021-08-2518:01Joshua SuskaloI guess that means I'll be publishing an update to americano to have a function entrypoint for prep lib that will read the deps edn from the current directory and use that to trigger the normal compilation.#2021-08-2518:36Alex Miller (Clojure team)it's (effectively) run with -T which requires a fn#2021-08-2519:08Joshua SuskaloThanks. 🙂#2021-08-2610:54henryw374We have this problem where tools.deps is using its cache for :local/root dependencies, but the cache is out of date. This happens sometimes when switching git branch for example
Deleting .cpcache fixes the problem
is there a way we can disable caching of local/root deps, or disable it altogether? (aside from deleting the cache file every time ofc).
thanks#2021-08-2610:54dominicm@henryw374 -Sforce#2021-08-2610:55henryw374thanks. will try that#2021-08-2616:36Eugenhi, I'm trying to know what is the clojure way (deps.edn ?!) of publishing some files (json. data files) to a private maven repository and download that file so we can build a docker image from multiple dependencies.
In gradle fpr example I can set tasks to download a set of dependencies to a specific directory and then process them for use in a zip distribution.#2021-08-2618:29hiredmanyou may need to back up and kind of untangle some things#2021-08-2618:30hiredmantools-deps downloads artifacts (jars) from maven repos and assembles them into a classpath and launches clojure with that classpath#2021-08-2618:30hiredmanmaven repos are for storing artifacts (jars, maybe some metadata about jars) not for storing random json#2021-08-2618:34hiredmanif it isn't random json, but some data some code needs access to, that often lives in resources/ in your project, and gets bundled into your jar along with your code and published to wherever, so you don't need to download it separately, it is baked in#2021-08-2619:17Eugenit's data - queries for a load test - that we generate via an app from the db.
We update them once per quarter and I use them in a locust test (I build a docker image).
They are too big IMO (15MB) to be stored in git so IMO the best place to put them is in a binary repo - we have a private instance.
I don't want to put them in S3 because I will need to give people access there as well and it complicates things.#2021-08-2619:18EugenI'm looking to use tools.build to package them as a zip and probably call maven to upload them to the repo#2021-08-2619:22EugenI did find an issue while following the guide at https://clojure.org/guides/tools_build .
{:paths ["src"]
:deps {org.clojure/clojure {:mvn/version "1.10.3"}
org.clojure/core.async {:mvn/version "1.3.618"}
com.stuartsierra/component {:mvn/version "1.0.0"}
;; io.github.clojure/tools.build {:git/tag "v0.1.9" :git/sha "6736c83"}
}
:aliases
{:build {:deps {io.github.clojure/tools.build {:git/tag "v0.1.9" :git/sha "6736c83"}}
:ns-default ro.ieugen.training.tools-build-guide}}}
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "28414d5d4f4d46684c49475b051c111d"}, :content ("[email protected]")}
and
ls -la src/ro/ieugen/training/tools_build_guide.clj
-rw-r--r-- 1 ieugen ieugen 1299 Aug 26 22:05 src/ro/ieugen/training/tools_build_guide.clj
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "365f534351535876525759451b020f03"}, :content ("[email protected]")}#2021-08-2619:30Eugenif it's usefull, I pushed the code that fails for me https://github.com/ieugen/clojure-training/#2021-08-2619:31Alex Miller (Clojure team)-T will remove your project :paths and :deps, so you'll want to add either :paths ["src"] or :extra-paths ["src"] in your :build alias#2021-08-2619:31Alex Miller (Clojure team)(doesn't really matter which)#2021-08-2619:31Alex Miller (Clojure team)also, btw there is a #tools-build for future q's#2021-08-2619:32Eugenthanks, that's missing in the guide online - so most people following the guide will probably fail like me#2021-08-2619:32Alex Miller (Clojure team)often people put their build script in a different src dir (the project root is automatically included)#2021-08-2619:32Alex Miller (Clojure team)sure is, thanks!#2021-08-2619:33Alex Miller (Clojure team)oh, that's because I'm assuming it's in the root of the project, which is added by -T#2021-08-2619:33Alex Miller (Clojure team)but this is worth explaining for sure#2021-08-2619:34Eugenyes. it would help to explain.
in my case, I'm not in the root of the project ?!#2021-08-2619:34Alex Miller (Clojure team)you have your build script under src/yada/yada#2021-08-2619:35Alex Miller (Clojure team)but if you put your tools_build_guide.clj next to the deps.edn, you could do that instead#2021-08-2619:35Alex Miller (Clojure team)I logged this as https://github.com/clojure/clojure-site/issues/550 so I can improve the next time I do doc work#2021-08-2619:35Eugenok, now I get it now - I was confused a bit#2021-08-2619:36Alex Miller (Clojure team)you're right - it needs explanation! :)#2021-08-2619:37Eugendo you think I can take a shot at that?#2021-08-2620:02Alex Miller (Clojure team)you can, but I'm likely to edit it :)#2021-08-2620:08EugenI don't mind that. I'll see if I get to it before you do.#2021-08-2712:32plexusis there a way to get the aliases that were used when starting clojure at runtime?#2021-08-2712:33plexusI had hoped to find them in clojure.basis but it doesn't look like they're there#2021-08-2712:50Alex Miller (Clojure team)The aliases are not there, just the args that resulted from combining them#2021-08-2712:50Alex Miller (Clojure team)What’s the use case?#2021-08-2717:40Jakub Holý (HolyJak)🙏 What do I need to do to see the resources of a local rot dependency (fulcro) in clojure -Spath ? Currently it only prints its src/main :
> src/shared:/Users/me/Projects/fulcro/src/main:/Users/me/.m2/...
I have modified the https://github.com/fulcrologic/fulcro/blob/develop/deps.edn#L1 to have also resources in paths:
{:paths ["src/main" "resources"]
but it did not help and resources is still not showing up.#2021-08-2717:41dpsuttontry clojure -Sforce to recompute. There's an open ticket (i'll find it in a second) to recognize when local root cpcache is stale#2021-08-2717:42dpsutton-Sforce#2021-08-2717:48Jakub Holý (HolyJak)ah, ok. Thanks a lot!#2021-08-2717:50Alex Miller (Clojure team)maybe some day I'll get around to that :)#2021-08-2721:39Alex Miller (Clojure team)I just pushed an even more pre release of Clojure CLI than usual - 1.10.3.956
• Refine exec exceptions for missing namespace vs missing function in namespace
• Replaced Maven with tools.build#2021-08-2721:40Alex Miller (Clojure team)seems like all the things are in the right places after the switch but if you're feeling experimental, give it a try (ideally it should really just be the same, the bits are just formed in new ways)#2021-08-2721:41Alex Miller (Clojure team)the next step I'm moving towards is to start compiling the internal uber jar and maybe the exec stub jar which should make classpath gen faster#2021-08-2721:47seancorfieldCan you elaborate on Replaced Maven with tools.build?#2021-08-2721:48seancorfield(I'm installing that version now so I can kick the tires)#2021-08-2721:50Alex Miller (Clojure team)not sure how else to say it#2021-08-2721:50seancorfieldI just looked at the commit and now it makes sense, nm.#2021-08-2721:50Alex Miller (Clojure team)all of the jars and tars and zips used to be built with Maven and are now built with tools.build#2021-08-2721:51Alex Miller (Clojure team)I enjoyed the dog food and am hungry for more :)#2021-08-2721:55seancorfieldI look forward to being able to use tools.build to build our uberjars at work but that needs the log4j2 stuff incorporated 🙂#2021-08-2813:34dominicmI immediately thought you meant that you'd replaced the maven fetching with tools.build which baffled me!#2021-08-2821:47niclasnilssonPerhaps a stupid question, but shouldn’t recursive :local/root work? (i.e. a project pointing to a local lib that is pointing ti a local lib). Or does it work and I’m doing something wrong?#2021-08-2822:55seancorfield@niclasnilsson We have plenty of :local/root deps that depend on other :local/root deps so it definitely works.#2021-08-2910:34niclasnilssonI upgraded cljoure from 1.10.1 to 1.10.3 and now it works. Could possibly have been my .cpcache as well (did the mistake of changing two things at once), so now I don’t know which was the solution.#2021-08-2914:52wactbprotWhat is the recommended way to version a project that uses tools.deps? I mean: where should the version number be placed? Thanks in advance.#2021-08-2915:54seancorfield@wactbprot Can you explain what you use the version number for? In the code of the app itself? Just in the pom.xml file? Something else?#2021-08-2915:54seancorfieldAre you building a library? An application?#2021-08-2915:56seancorfieldFor my OSS projects (libraries), I've been using the :version argument of depstar to update the pom.xml file's <version> and <scm><tag> fields up until I switched to tools.build which handles both of those in its write-pom function, and it computes it automatically based on the git commit count.#2021-08-2918:09wactbprotWow, that's a lot of question marks. In pre tools.build times the answer seems to be: it depends. I write oss for a small team (libs and apps). I distribute the code via git. If I interpret your answer correctly, tools.build fills a gap. Thanks for pointing out this new way.#2021-08-2918:41seancorfieldIf the code is on github, folks can depend on it directly from their deps.edn file.#2021-08-3016:05Alex Miller (Clojure team)new CLI prerelease 1.10.3.962 is now available - should be functionally the same, just now compiles the tools.deps and exec stub entry points to be slightly faster when making classpaths if anyone wants to try it. will probably make a new stable in the next day or two#2021-08-3021:12seancorfieldUnsurprisingly, this breaks our exec-command usage at work -- because we can't put an AOT'd exec.jar on the classpath and run (already AOT'd) clojure.core against it @U064X3EF3 so this will prevent us doing any more CLI updates for the time being. Just FYI.#2021-08-3021:15seancorfieldI guess I could vendor in a source version of clojure.run.exec like we had originally... at least until some sort of official support comes for -X invocation via tools.build's process... 😐#2021-08-3021:56Alex Miller (Clojure team)well that was always a strategy with a limited lifetime#2021-08-3021:57seancorfieldHow much effort I invest in this might depend on how close some sort of exec fn support for process might be.#2021-08-3021:59seancorfieldWe can probably just stick with a "working" version of the CLI and not bother updating it until exec fn support appears -- if that is "not too far away". It's just frustrating, given how much of a push the whole -X thing has had, to not have it supported by tools.build yet 😐#2021-08-3022:35Alex Miller (Clojure team)Well, I am waiting for Rich’s input and his availability is unpredictable so I can’t really promise anything#2021-08-3023:21seancorfieldFair enough. Rich can't be rushed.#2021-08-3102:22seancorfieldThis seems to be broken for using -X with any version of Clojure other than 1.10.3:
(! 540)-> clojure -X:test:1.10
... runs tests successfully with 1.10.3 ...
(! 541)-> clojure -X:test:master
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:636).
clojure.core$seq__5419
Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-15548447875655445524.edn
(! 542)-> clojure -X:test:1.9
Exception in thread "main" java.lang.NoClassDefFoundError: clojure/core$seq__5419
at clojure.run.exec$read_args.invokeStatic(exec.clj:105)
at clojure.run.exec$_main.invokeStatic(exec.clj:138)
at clojure.run.exec$_main.doInvoke(exec.clj:138)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
Caused by: java.lang.ClassNotFoundException: clojure.core$seq__5419
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
... 15 more
#2021-08-3102:25seancorfieldThat has nothing to do with my exec-command stuff at work -- that's broken in all of my OSS projects where I do multi-version testing.#2021-08-3103:08Alex Miller (Clojure team)Thanks, I’ll take a look at that!#2021-08-3102:22seancorfieldThis seems to be broken for using -X with any version of Clojure other than 1.10.3:
(! 540)-> clojure -X:test:1.10
... runs tests successfully with 1.10.3 ...
(! 541)-> clojure -X:test:master
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:636).
clojure.core$seq__5419
Full report at:
/var/folders/p1/30gnjddx6p193frh670pl8nh0000gn/T/clojure-15548447875655445524.edn
(! 542)-> clojure -X:test:1.9
Exception in thread "main" java.lang.NoClassDefFoundError: clojure/core$seq__5419
at clojure.run.exec$read_args.invokeStatic(exec.clj:105)
at clojure.run.exec$_main.invokeStatic(exec.clj:138)
at clojure.run.exec$_main.doInvoke(exec.clj:138)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.core$apply.invokeStatic(core.clj:657)
at clojure.main$main_opt.invokeStatic(main.clj:317)
at clojure.main$main_opt.invoke(main.clj:313)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
Caused by: java.lang.ClassNotFoundException: clojure.core$seq__5419
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
... 15 more
#2021-08-3102:24seancorfield(that's with HoneySQL's repo but I can repro this in any project where I try to select a different version of Clojure with -X)#2021-08-3108:55slipsetThere is some traffic on the issues for https://github.com/slipset/deps-deploy I’d be happy to work on it, but it seems a bit of waste if the tools stuff is going to handle deploys. So, while not asking for a timeline nor a roadmap, what would the probability be that tools gets a deploy feature within 2021?#2021-08-3112:26Alex Miller (Clojure team)It’s not something I’m working on or expect to work on soon, but is an area I’ve looked at and would like to have a more comprehensive deps based answer to#2021-08-3113:56slipsetCool. Just for the record. I’d be happy to retire deps-deploy the moment there is support in tools for it.#2021-08-3114:26Alex Miller (Clojure team)new CLI prerelease 1.10.3.967 is now available - undid the compilation of the exec stub, that's just too fragile#2021-08-3116:18seancorfieldConfirmed that works with -X and different versions of Clojure. Thank you!#2021-08-3114:27Alex Miller (Clojure team)I'll probably release to stable tomorrow#2021-08-3114:28borkdude@alexmiller care to elaborate why the exec stub compile was too fragile? is tool.deps still compiled?#2021-08-3114:51Alex Miller (Clojure team)well, it has to mix into anyone's classpath that's using -X or -T so any pre-compilation risks tying you into a Clojure version and I'd rather have as few constraints there as possible. plus, it's just one namespace, so I can't imagine it really matters much anyway#2021-08-3114:52Alex Miller (Clojure team)still compiling the tools.deps entry points - that's all internal to the cli so much safer#2021-08-3114:53Alex Miller (Clojure team)it's not really a huge win (most of the code is calling into stuff like maven which is all compiled Java anyways), but it helps a little#2021-08-3114:55Alex Miller (Clojure team)it's interesting to consider the compilation story for -X and -T calls. you can still use the techniques in https://clojure.org/guides/dev_startup_time but I have spent some time looking at automating it inside the CLI (since it knows the classpath, the entry point, etc - it has all the knowledge). there are some tricky corners though so it's a future area to consider.#2021-08-3114:58borkdudeyes. I think the -X stuff could probably be done in the CLI (in deps.clj this would probably be trivial) and transformed to a program which is then handed off to the JVM.#2021-08-3116:24seancorfield@U04V15CAJ We rely on the exec stuff from our build.clj at work because we have several things we were running with -X before that we now run in process so we build a command-line that we actually run with the exec stub so that it can do EDN parsing of arguments. @alexmiller has indicated that might get lifted out into a library of some sort so it can be reusable (since it's kinda hard to use the exec JAR as part of a classpath).#2021-08-3115:13Alex Miller (Clojure team)how would that be different than what it does now?#2021-08-3115:48borkdudenot sure if that would have any benefits to be honest#2021-09-0111:30kirill.salykinhi,
is there a way to print resolves paths? something like -Sinfo?#2021-09-0111:57delaguardoTry -Spaths#2021-09-0112:04kirill.salykininvalid option#2021-09-0112:12kirill.salykinit is Spath#2021-09-0112:12kirill.salykinthank you#2021-09-0115:31Alex Miller (Clojure team)Clojure CLI https://clojure.org/releases/tools#v1.10.3.967 is now available
• Refine exec exceptions for missing namespace vs missing function in namespace
• Replace Maven-based build process with tools.build
• Compile entry points in tools.deps used for building classpaths for performance
• Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.12.1036
#2021-09-0211:22kirill.salykinhi all
is there a way to ask tools-deps to download all deps but ignore any main options (basically I want just to cached deps)
- clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:test:prod:frontend -P
doesnt work#2021-09-0211:25borkdudeThat's the thing with main opts, I think the clojure CLI should now ignore them with -A but because of historical behavior, it doesn't yet.#2021-09-0211:25kirill.salykinas a workaround i can create empty alias w/o any -m and put it last thing#2021-09-0211:26borkdudePerhaps -P could be made to accept aliases as well#2021-09-0211:36kirill.salykinthis worked for me
clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:test:prod:frontend -Spath > /dev/null
#2021-09-0211:39borkdudenote that -Spath (clj-opt) needs to go before -Aaliases#2021-09-0211:39borkdudealthough this works, it's not guaranteed to work, since the docs recommend otherwise#2021-09-0211:48kirill.salykinah#2021-09-0211:48kirill.salykinthanks#2021-09-0211:48kirill.salykinwhy does
clojure -Sdeps '{:mvn/local-repo ".m2"}' -M:dev:test
download deps every run?#2021-09-0211:50kirill.salykinError building classpath. Could not transfer artifact org.glassfish.jaxb:txw2:jar:2.3.0.1 from/to central (): status code: 416, reason phrase: Range Not Satisfiable (416)
maven misbehaving?#2021-09-0212:04kirill.salykinseems working on CI with older verion of tools-deps
seems like a bug?#2021-09-0212:05kirill.salykin:version "1.10.3.967"
#2021-09-0212:26kirill.salykinfound the solution#2021-09-0212:26kirill.salykinjust deleteing m2#2021-09-0217:04Joshua SuskaloIs there any way to run a tool installed with the -Ttools tool but with the current classpath available? Like if I have a tool installed, can I run it with -X even if it isn't in the current project?#2021-09-0217:09Alex Miller (Clojure team)no, tools definitionally don't run with the project classpath#2021-09-0217:09Alex Miller (Clojure team)you can install that dep as an alias and use -X with it though#2021-09-0217:11Alex Miller (Clojure team)if you want to see the lib/coord for an installed tool, you can do that with:
clj -Ttools show :tool tools
#2021-09-0217:11Alex Miller (Clojure team)(replace the last tools with whatever)#2021-09-0217:17Joshua SuskaloAlright. What I'm thinking about doing (haven't committed to it yet, still exploring how complex an undertaking it will be) is making a tool which can format code in the same way that cider does, but without needing to connect a repl to your entire project.
This is fine to not have the classpath loaded for your project, but an extension that I want for it is to be able to run it inside the classpath for a different operation that instead of formatting the code, it will load the namespaces from your project, explore all the public vars for your namespaces and all the dependencies, record the :style/indent metadata, and write a config file that specifies the indent config for every var that customizes it.#2021-09-0217:18Joshua SuskaloUnfortunately what that means is that I'll either have to have users add this extra alias to their project or their user deps.edn, neither of which is desirable compared to being able to install the tool and then run it with -X, although I can see the reason why it is the way it is.#2021-09-0217:32seancorfield@suskeyhose A tool can read the project deps.edn and find the user's code.#2021-09-0217:32Joshua SuskaloYes, that's what I'll be doing for the formatter tool, but for the configuration generator I need to load both the code and the dependencies.#2021-09-0217:33seancorfieldIf you want it to be able to read code that exists under an alias, your tool would need to accept a list of aliases and use those when computing the basis.#2021-09-0217:33seancorfieldThe tool still does not need to be part of the deps.edn for that. Use tools.deps.alpha as a library to calculate the basis.#2021-09-0217:34seancorfieldThat's how depstar and many other tools work: they use t.d.a to compute the basis.#2021-09-0217:35Joshua SuskaloYes, the easiest way however to do this will be to require the namespaces, inspect them with ns-map, for all vars inspect the metadata, and build up a list that way. It would be preferable to not have to use tools.deps.alpha to procure all the deps, use the basis to find the location of the code, have to deal with source vs jar dependencies, parse/load the code, find metadata, etc.#2021-09-0217:36Joshua SuskaloI suppose I may be able to mess with classloaders if I wanted to in order to load the dependencies at runtime using lambdaisland/classpath, but even that isn't exactly desirable if I can avoid it.#2021-09-0217:37seancorfieldOr just use the add-lib3 branch of t.d.a and add dependencies that way 🙂#2021-09-0217:37Joshua SuskaloI thought there were issues with that branch not working correctly. Has it been fixed, or was I misinformed?#2021-09-0217:39seancorfieldI use it all the time. Not aware of problems. But you would still need to do some classloader stuff to add paths since add-libs doesn't handle that (although I guess you could add :local/root "." to get the current project?)#2021-09-0217:39seancorfield(that wouldn't help with :test deps though -- you'd still need classloader stuff for that)#2021-09-0217:40Joshua SuskaloYeah, loading the current paths can be alleviated by just using the clojure file-seq and load too.#2021-09-0217:40Joshua SuskaloWhich that's what I'd want to do anyway because otherwise there's a chance I don't load all the namespaces which are needed.#2021-09-0217:41seancorfieldYou'd have to figure out the right order to load files in based on their ns deps since you'd need to load from the "bottom" up.#2021-09-0217:41Joshua SuskaloI suppose yeah, but there's already graph libraries and I could just parse the ns decls#2021-09-0217:43Joshua Suskalotools.analyzer provides me with almost everything I need for that#2021-09-0217:44borkdudeClj-kondo can also provide you with that including for CLJS sources #2021-09-0217:46borkdudePotentially it could also just give you that metadata for every var #2021-09-0217:47Joshua SuskaloOooh, if clj-kondo analysis can give me metadata for everything, then I might want to use that. The only problem I might have is for vars defined with macros.#2021-09-0218:13hiredmanif you just create a dummy file, stick all the ns's in a require in there, and load it, it will all load correctly#2021-09-0218:14hiredmanno need to build your own dependency graph to load things correctly#2021-09-0218:19Joshua SuskaloThat's true. clj-kondo would provide the added benefit of not being restricted to jvm clojure though, which is nice.#2021-09-0218:20hiredmanI said the same thing when the compile task in tools.build had an issue with code loading order, and tools.build looks like it got rewritten to build a dependency graph#2021-09-0218:27hiredmanyou could even do the same thing without the temp file with something like
(fn [namespace-names] (binding [*compile-files* true] (run! require namespace-names)))
#2021-09-0218:28hiredman(for compiling)#2021-09-0218:29hiredmanthe issue is just clojure.core/compile calls load-one, which doesn't do any checks to see if a ns has already been loaded, so if you do things in the wrong order you get multiple compiles, if you manually set *compile-files* and use require, it will check before loading#2021-09-0311:22kirill.salykinis there faster way to prefetch deps for all aliases rather then invoke clojure with alias multiple times?
- clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:test -Spath
- clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:prod -Spath
- clojure -Sdeps '{:mvn/local-repo ".m2"}' -A:uberjar -Spath
it is not super long, but still takes around 40 secs (according to ci)#2021-09-0311:46borkdudewhy not combine all those aliases at once?#2021-09-0311:46borkdudenote that combining aliases might result in a different dep resolution than a per alias invocation#2021-09-0311:56kirill.salykinbecause they might influence each other#2021-09-0311:56kirill.salykinexaclty the reason#2021-09-0312:01borkdude@kirill.salykin you could do this in parallel to speed things up. I hope git / mvn can deal with that, but I think it should#2021-09-0312:01kirill.salykinthats an idea, thanks#2021-09-0312:01borkdudeusing babashka tasks this is fairly trivial#2021-09-0312:01kirill.salykintell me more 🙂#2021-09-0312:03borkdudebb.edn
{:tasks {test-deps (clojure "-Sdeps '{:mvn/local-repo ".m2"}' -A:test -Spath")
prod-deps ...
main-deps ...
all-deps {:depends [test-deps prod-deps main-deps]}
And the run bb run --parallel all-deps#2021-09-0312:05kirill.salykinsorry for stupid question, just to ensure, bb is on par with latest tools-deps?#2021-09-0312:05borkdudeyes#2021-09-0312:05kirill.salykinawesome! you are the best!#2021-09-0312:06borkdudeif you don't want to rely on the latest bb clojure, then you can also use (shell "clojure -Sdeps ...")#2021-09-0312:06borkdudeto shell out to the system-installed clojure CLI#2021-09-0312:12kirill.salykinthanks for pointers, need to wrap my head around it#2021-09-0312:12borkdudeSee docs: https://book.babashka.org/#tasks
I also did a talk about it: https://www.youtube.com/watch?v=u5ECoR7KT1Y#2021-09-0312:13kirill.salykinoh, you so nice#2021-09-0312:13kirill.salykinthanks!#2021-09-0312:03kirill.salykinnice, will give it a try!#2021-09-0312:03borkdude(EDIT: added --parallel)#2021-09-0312:04borkdudenote that formally, -Spath must go before the aliases#2021-09-0312:10Alex Miller (Clojure team)You are likely to get different deps from combining than separate#2021-09-0312:11borkdudethat was already clear I think#2021-09-0312:12Alex Miller (Clojure team)We do have some open issues related to parallel downloads too #2021-09-0312:12kirill.salykincare to share the link? I’d like to +1 those#2021-09-0312:12Alex Miller (Clojure team)This is an area of active changes in Maven itself over the last few core releases#2021-09-0312:14kirill.salykingood things#2021-09-0318:31Joshua SuskaloWhere do I go to suggest features for tools deps and the clojure cli? Just http://ask.clojure.org?#2021-09-0318:33seancorfieldYes, that's the best place.#2021-09-0318:39Joshua SuskaloCool, thanks#2021-09-0414:54Joshua SuskaloI see the main thing I wanted to ask about already exists: https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects#2021-09-0421:50Joshua SuskaloDoes it work to specify a default dependency in the root deps.edn and not specify a coordinate in aliases to use it so that you can ensure that all aliases use the same version?#2021-09-0421:53Alex Miller (Clojure team)I'm not sure I'm following the question, can you use a deps.edn to demonstrate the question?#2021-09-0421:54Alex Miller (Clojure team)I think by "root deps.edn", you actually mean the project deps.edn? (the root deps.edn is the one included with tools.deps)#2021-09-0500:59Joshua Suskalo{:deps {some.dep/one {:mvn/version "whatever"}}
:default-deps {other.dep/two {:mvn/version "RELEASE"}}
:aliases
{:some-alias {:extra-deps {other.dep/two nil}}
:other-alias {:extra-deps {other.dep/two nil}}}}
#2021-09-0501:00Joshua SuskaloI know this could be done by specifying three aliases, one which has all the default deps, and the others which then have the actual stuff I want in them, but I was curious if it could be done without a third alias.#2021-09-0517:00jjttjjIs there a way to call a (non -main) function with zero args via an alias? I know this might not make much sense but just wondering if I can convert a project.clj that is doing that via an alias and "-m" without touching the project's code itself. Edit: I guess I can just use "-e" in :main-opts to accomplish this#2021-09-0517:17borkdude@jjttjj doesn't -M -m work as well?#2021-09-0517:36jjttjjSorry not sure what you mean. The -m option just takes a namespace to look for a -main function right?#2021-09-0517:46borkdudeah right#2021-09-0607:10wactbprotHi, I try to build a compiled uberjar application using tools.build.
I follow the https://clojure.org/guides/tools_build
and end up with this error message at clj -T:build uber step:
Execution error (IllegalArgumentException) at repliclj.conf/get-config (conf.clj:12).
Cannot open <nil> as a Reader.
which points to the line:
(get-config (io/resource "config.edn"))
The config.edn file is in the directory resources. I guess
(b/copy-dir {:src-dirs ["src" "resources"] :target-dir class-dir})
copies the file to target/classes/ where io/resource don't find it.
Any advice is very welcome.#2021-09-0607:25hiredmanio/resource should find it fine classes, assuming ./classes/ is on the classpath, which is how things have to be setup for aot compilation anyway#2021-09-0607:26hiredmanDid you call the prep function as shown in the examples?#2021-09-0607:28wactbprotYes, I followed the steps.#2021-09-0607:30hiredmanI would look in ./classes to see if config.edn is in there#2021-09-0607:34hiredmanIf it is in there, it means something is wrong with compiling and ./classes is not on the classpath like if needs to be, if it isn't in there it means resources/ was not properly copied#2021-09-0608:24wactbprotconfig.edn is not in ./classes but in ./target/classes/ . The guide gives (def class-dir "target/classes") So, what can I do to ensure that the file is properly copied?#2021-09-0608:36wactbprotI just noticed that ./classes is not created by the tools.build process, but by Seans -X:uberjar which works btw. This means that I don't have a ./classes folder in the tools.build process.#2021-09-0608:38hiredmanSorry, target/classes is likely correct there for tools.build#2021-09-0608:39hiredmanThere is a #tools-build channel maybe this should move to#2021-09-0716:43bartukais it possible to call clojure -Spath with aliases and get the aliases deps in classpath?#2021-09-0716:54delaguardoclojure -X:alias -Spath should work#2021-09-0716:44bartukaor do I need to stick with tools.deps to get this?#2021-09-0716:56Alex Miller (Clojure team)should work fine#2021-09-0716:59bartukathanks!#2021-09-0807:41Eugencan find-versions also work for non jar artifacts?
I tried to find the versions for a published zip and I get no results.
Checked the source code and it's not supported.
Maven is an artifact repositoy and in essence you can store whatever artifacts there.
Most of them are jars, but they don't need to be.
Also: can it work for classifiers as well ? Don't use them right now, I'm asking for completion sake.#2021-09-0812:22Alex Miller (Clojure team)It is jar only, although you are conceptually correct#2021-09-0812:24Alex Miller (Clojure team)It might work for classifiers, which we do support, can’t actually say that I thought about it. Did you try it?#2021-09-0812:34EugenI did not try with classifier since I don't use it now - I can probably find a dependency online that has classifier and try with that#2021-09-0812:40Alex Miller (Clojure team)The syntax is group/artifact$classifier#2021-09-0807:54Eugendoes tools-deps take into account encrypted server passwords from settings-security.xml ?
Browsed the code and I did not see any references so I imagine it does not.#2021-09-0812:26Alex Miller (Clojure team)We use the Maven apis to access those, so not sure if we are getting that support or not. Probably not would be my guess#2021-09-0809:11EugenIs there a way I can get the latest version of an artifact ?
I tried this but it does not work ( I get the older version)
:deps {com.my.group.example-artifact/load-test-data {:mvn/version "LATEST" :extension "zip"}}}}
~This is perhaps related to ~
This is perhaps related to repository metadata refresh.
The dependencies are available locally so it does not refresh them from upstream.
Any way to force a metadata refresh via resolve-deps ?#2021-09-0812:27Alex Miller (Clojure team)LATEST or RELEASE should resolve metadata#2021-09-0812:27Alex Miller (Clojure team)I think it is subject to the once-per-day update policy though#2021-09-0812:27Alex Miller (Clojure team)No ways to force that other than deleting repo metadata files from m2#2021-09-0812:28Alex Miller (Clojure team)But that :extension is not a thing#2021-09-0812:29Alex Miller (Clojure team)Tools.deps is not a generic interface to Maven, it’s a tool for building classpaths#2021-09-0812:33delaguardoIf I remember correctly updatePolicy can be configured per maven repository in pom.xml#2021-09-0812:35delaguardoSo maybe tools.deps could have a way to configure that in deps.esn#2021-09-0812:36delaguardoSomewhere in :mvn/repos#2021-09-0812:38Alex Miller (Clojure team)Has been requested, and I even wrote the code at one point, but I’m going to resist that for now#2021-09-0812:39Alex Miller (Clojure team)It adds a lot of complexity to other assumptions in the code#2021-09-0812:40ghadiplus LATEST or RELEASE are refs not values, yuck#2021-09-0812:42Eugen@ghadi: that is life 🙂#2021-09-0812:43ghadimy life is based on values 🙂#2021-09-0812:43Eugenif find-versions works with :extension I can avoid using LATEST/RELEASE#2021-09-0812:43EugenI can fetch latest version and use it#2021-09-0812:45Alex Miller (Clojure team)If you want to get non-jars, you’re going to have to tap into a much lower point in the implementation#2021-09-0812:45Alex Miller (Clojure team)This is not part of the public api#2021-09-0913:05golanweissHi,
A small question.
I had some problems with my machine so I had to re-clone everything.
Now when trying to run my Clojure project, deps are being downloaded but it gets stuck.
I tried every possible flag but they all get stuck in the same way
~/dev/git/clj-data/clojure [release/vnext●] (doordawn-eks/data) % clojure -Sforce -Sverbose -Strace -M:1.8:zinc
version = 1.10.3.967
install_dir = /usr/local/Cellar/clojure/1.10.3.967
config_dir = /Users/golanweiss/.clojure
config_paths = /usr/local/Cellar/clojure/1.10.3.967/deps.edn /Users/golanweiss/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/2424673877.cp
Refreshing classpath
DEPRECATED: Libs must be qualified, change incanter => incanter/incanter (/Users/golanweiss/dev/git/clj-data/clojure/zinc/deps.edn)
Downloading: postgresql/postgresql/maven-metadata.xml from madlan-releases
Downloading: social/exif-processor/0.1.2-socialsuperstore-SNAPSHOT/maven-metadata.xml from madlan-releases
Downloading: com/google/guava/guava/maven-metadata.xml from madlan-releases
Downloading: com/google/guava/guava/maven-metadata.xml from madlan-snapshots
Downloading: postgresql/postgresql/maven-metadata.xml from madlan-snapshots
Downloading: org/clojure/clojure/maven-metadata.xml from madlan-snapshots
Downloading: social/exif-processor/0.1.2-socialsuperstore-SNAPSHOT/maven-metadata.xml from madlan-snapshots
Downloading: org/apache/commons/commons-math3/maven-metadata.xml from madlan-snapshots
Downloading: org/apache/commons/commons-math3/maven-metadata.xml from madlan-releases
Downloading: refactor-nrepl/refactor-nrepl/2.0.0-SNAPSHOT/maven-metadata.xml from madlan-snapshots
Downloading: cider/cider-nrepl/0.11.0-SNAPSHOT/maven-metadata.xml from madlan-snapshots
Downloading: refactor-nrepl/refactor-nrepl/2.0.0-SNAPSHOT/maven-metadata.xml from madlan-releases
Downloading: cider/cider-nrepl/0.11.0-SNAPSHOT/maven-metadata.xml from madlan-releases
Downloading: org/javassist/javassist/maven-metadata.xml from madlan-releases
Downloading: org/javassist/javassist/maven-metadata.xml from central
Is there a way to debug and find the problematic dependency (assuming that's the case) without doing it "by elimination"?
(btw I don't have javassist dependency declared directly anywhere)#2021-09-0913:22Alex Miller (Clojure team)is your deps.edn something you can share?#2021-09-0913:27golanweiss#2021-09-0913:28Alex Miller (Clojure team)geez, I think you missed a few deps :)#2021-09-0913:29golanweiss😆#2021-09-0913:30golanweisseventually the culprit was in these
{:deps {amazonica/amazonica #:mvn{:version "0.3.121"},
org.jpmml/pmml-evaluator #:mvn{:version "1.4.3"},
org.jpmml/pmml-evaluator-extension #:mvn{:version "1.4.3"},}
once I changed to latest version problem went away...#2021-09-0913:32golanweissit would be nice if -Sverbose would have print when fetching each direct dependency...#2021-09-0913:32Alex Miller (Clojure team)do you have more :mvn/repos set up in your ~/.clojure/deps.edn?#2021-09-0913:33golanweissyes we use private repo too.#2021-09-0913:25golanweissyeah I can share it
The point is that debugging with -Strace needs to complete the whole deps cycle first.
So it wasn't so helpful to me in finding that problematic dependency.#2021-09-0913:26Alex Miller (Clojure team)yes, I'm aware of that. there is no "as you go" debugging.#2021-09-0913:27Alex Miller (Clojure team)you could try not using your ~/.m2 in case there is something corrupt in there, by adding -Sdeps '{:mvn/local-repo "tmp"}'#2021-09-0913:27Alex Miller (Clojure team)that will download deps anew into the ./tmp directory#2021-09-0913:30golanweissthanks#2021-09-1008:54Ben SlessNot 100% if this should be asked here or in #tools-build as they probably have some overlap, can I use deps to build a project comprised of modules like https://github.com/metosin/reitit?#2021-09-1009:19rickmoynihanYes 🙂#2021-09-1009:21rickmoynihanIIRC juxt’s edge used to demonstrate this kind of setup:
https://github.com/juxt/edge#2021-09-1009:21rickmoynihanI assume it still does; but it constantly evolves#2021-09-1009:23rickmoynihanSee also @U04V70XH6’s blog posts on how to arrange a clj monorepo: https://corfield.org/blog/2021/02/23/deps-edn-monorepo/#2021-09-1009:25Ben SlessI think the difference between Sean's use case and this one is how artifacts are built and distributed. When you have an internal monorepo for building deployable uberjars you don't really care about deploying your jars to some place like clojars, the builds are self contained. In this case I want to distribute the building blocks#2021-09-1009:28Ben SlessMonorepo where the end product is deployable artifacts: I grok
Monorepo where the end product is library modules with inter-dependencies: panik#2021-09-1009:29rickmoynihanFair enough — to be fair I’d like to do that too… Unfortunately some of our components are both open and closed source so it’s not really viable in a mono repo.#2021-09-1009:33rickmoynihanAnyway I think tools-deps and tools-build or babashka tasks or a combination of them will give you much of what you need.
Essentially all you need is a clj mono-repo, with a script that iterates over the modules and deploys them with depstar.
Obviously assembling all that will involve building something somewhat bespoke… though you’d imagine it could be released as a clj build tool.#2021-09-1009:36Ben SlessYes.
I found when I started exploring this problem space that dependencies specified as taken from a :local/root aren't taken into the pom, so I have to figure out something more clever#2021-09-1012:13Alex Miller (Clojure team)Yeah, maven has no way to talk about that (or git deps)#2021-09-1012:21Ben SlessShould they be resolved to an absolute path relative to the project root at pom creation time?#2021-09-1013:09Alex Miller (Clojure team)"they" = ?#2021-09-1013:10Alex Miller (Clojure team)the problem is the pom has no way to talk about "local" artifacts, only things in Maven repos#2021-09-1013:24Ben SlessWhat I mean is, can local/root deps be resolved to where they will be once the artifacts are built and referred to as such in the pom? That way consumers of the module will be able to pull in the dependencies.
Sort of like readlink or dereferencing#2021-09-1013:36Alex Miller (Clojure team)"and referred to as such in the pom" -> no, there is no language for this in pom.xml#2021-09-1013:37Alex Miller (Clojure team)pom.xml specifies deps only as groupId/artifactId/version for artifacts found in maven repos#2021-09-1013:38Alex Miller (Clojure team)so you would need to at least "install" those artifacts to your local repo for them to be addressable by the pom.xml#2021-09-1013:47Ben SlessI was thinking more along the lines of
• I know at build time what the groupID/artifactID/version will be because I can get to the other repo by way of the local/root
• Write those in the pom instead of install them locally
• "eventually consistent"
• ...
• profit?#2021-09-1013:48Ben SlessStop me at any point if it seems unreasonable or like a bad idea#2021-09-1013:54Alex Miller (Clojure team)is something deploying to a maven repo in this process?#2021-09-1013:54Alex Miller (Clojure team)I mean, all of this seems like a bad idea if you're using Maven-oriented tools :)#2021-09-1013:55Alex Miller (Clojure team)Maven does have a whole thing (multi-module projects, pom inheritance, etc) for building and deploying a set of related projects from a single repo#2021-09-1013:56Alex Miller (Clojure team)but you need to be "doing Maven" for that whole thing to make sense#2021-09-1014:07Ben SlessBrian Goetz likes to tell how people start questions with "why don't you just _" when they have a big ask, so I won't ask that, but I would say it would be nice to be able to do that, both for deployment (maven/clojars/artifactory) and for deps, to be able to pull in only parts of a code base which have already been organized as such
Which reminds me of the spec-ulation talk, now#2021-09-1016:30seancorfield@UK0810AQ2 Have you looked at how Polylith structures things? It completely separates the deployable part from the code so you could have a "project" for each thing you want to deploy to Maven/Clojars and separate projects for what your internally deployable stuff is -- and then the code is "assembled" from bases (usually things with a -main) and components (small "libraries").#2021-09-1016:34seancorfieldAs noted in this thread, if you build something that you deploy to Maven/Clojars, then it must declare all of its dependencies as things that are also on Maven/Clojars -- but with a structure like Polylith, you can choose to build different artifacts for different audiences, so you could use git deps with :deps/root and :local/root dependencies and that would work "internally". Polylith's poly tool is a "project" inside the polyfy/polylith monorepo and can be used as a git dep, but they also build other artifacts from it.#2021-09-1016:42Ben SlessYes, but as you noted, it does not solve the dependencies issue, and I wouldn't want to maintain two dependency maps. I would like to programmatically transform the "is" to "ought". It might be easier for maven artifacts than for deps, actually#2021-09-1016:56seancorfieldHmm, then I guess I don't understand what you're trying to do or why Polylith's approach won't solve your problem...#2021-09-1017:00Ben SlessJust take reitit's structure as an example. Can it be done with deps?#2021-09-1017:08seancorfieldI don't use Leiningen so I don't really understand what's going on there -- but superficially it looks very, very similar to a Polylith structure with a workspace-level deps.edn (`project.clj` in reitit) and then each project/base/component having its own deps.edn (`project.clj` in reitit).#2021-09-1017:16Ben SlessYes, but building jars lets you refer to other components as dependencies in maven#2021-09-1017:19dominicm@UK0810AQ2 Author of edge, so another person very familiar with this space. Doing the reitit thing with deps.edn is very easy!
Let's start with our basic layout:
Foo-core/
Foo-plugin/
Each folder will need:
• deps.edn
• pom.xml (to hold version, etc.)
You can see that reitit updates the version everywhere on release: https://github.com/metosin/reitit/commit/8694d312f86d14ac7df3fa431af7cfba6873d8d8#diff-498cb648cb6ba613422c242cc068100c0bafa70b5cc71193570205abdac73b9e but you can maybe automate this with some tooling!
deps.edn really has 2 modes of operation here: release & development. During development, you're looking to depend on the local copies of dependencies (presumably), and during release you're looking to depend on maven central versions of the same version as your version. One way to manage this would be to create a parent pom which has the version in and sets the versions for all the modules in your project. Another option is just to manually maintain which version your module depends on by doing it as part of the update to the pom.xmls in each module.#2021-09-1017:20dominicmOh, so to do development in your deps.edn you should have a :dev alias like so:
:dev {:override-deps {Foo-core/Foo-core {:local/root "../Foo-core"}}}
#2021-09-1017:23Ben SlessCan children poms derive the version from the parent?
Wondering how this will look like with tools build, too#2021-09-1017:24seancorfieldWe use tools.build with Polylith so I'd say that's completely orthogonal.#2021-09-1017:24dominicmtools.build doesn't interact here, particularly. @U04V70XH6 was working on stuff to make builds in other projects work, but that's an optimization that applies fractionally to libraries.#2021-09-1017:26seancorfieldThere's certainly no conflict between having a unified :dev experience with everything using :local/root style deps and then building deployable artifacts that all have Maven-style coordinates and deps.#2021-09-1017:27dominicm@UK0810AQ2 I believe a feature of maven called managed deps enables that, yes.#2021-09-1017:29dominicmI think a question like this has come up before, and I'd like to do something like this for the juxt/yada repo if I get some time. Perhaps a small example repo would be worthwhile.#2021-09-1016:26cap10morganIs it expected behavior to get WARNING: Implicit use of clojure.main with options is deprecated, use -M when running clj --repl or clj -e '(+ 1 2)'? I don't necessarily have any aliases to use in those contexts.#2021-09-1016:27hiredman% clj -M --repl
Clojure 1.10.2
user=>
#2021-09-1016:35cap10morganohhh... interesting. I didn't even realize / try it w/o an alias. hence the bracketed [aliases] in the help output. and the message literally tells me use -M facepalm😆 thanks @U0NCTKEV8!#2021-09-1016:28hiredmanI believe it is, the idea being if you want to invoke clojure.main you need to pass -M#2021-09-1016:28borkdudeclj (without opts) still invokes a REPL though#2021-09-1019:22ChaseI'm trying to deploy to heroku but it seems they use Clojure 1.10.0.411 so they are not liking my up to date depstar invocation. I currently have this in my deps.edn :
:aliases {:uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.297"}}
:exec-fn hf.depstar/uberjar
:exec-args {:aot true}}}
and was calling it with clojure -X:uberjar :jar my-app.jar but it doesn't like -X which I replaced with -A , :replace-deps had to be replaced with :extra-deps and I'm at a loss with what to do for the :exec-fn ... . What would you advise there?#2021-09-1108:04practicalli-johnThe version of Clojure CLI can be specified with the Heroku Config Car CLOJURE_CLI_VERSION
https://practical.li/clojure-web-services/projects/banking-on-clojure/deployment-via-ci.html#heroku-pipeline-configuration#2021-09-1114:40ChaseOh man, I wish I had found this earlier yesterday. That $PORT thing was driving me crazy before I discovered what I was doing wrong.#2021-09-1019:27seancorfieldPetition Heroku to stop using such an ancient version of the CLI? 🙂 I'm a bit surprised you can't specify which CLI version you want, like you can for CircleCI or GitHub Actions...?#2021-09-1019:28seancorfieldGiven that, you'll need to switch to the old -M invocation of depstar (which is no longer documented!)...#2021-09-1019:30seancorfieldYou'll want :main-opts in :uberjar with ["-m" "hf.depstar.uberjar" "-C"] plus whatever additional args you need @chase-lambert#2021-09-1019:32seancorfieldThe old -main entry point doesn't support all the options that the -X entry point does. Here's the excerpt that does the argument parsing:
("-C" "--compile") [{:aot true} (next args)]
("-P" "--classpath") [{:classpath (fnext args)} (nnext args)]
("-D" "--debug-clash") [{:debug-clash true} (next args)]
("-h" "--help") [{:help true} (next args)]
("-J" "--jar") [{:jar (fnext args)} (nnext args)]
("-m" "--main") [{:main-class (fnext args)} (nnext args)]
("-n" "--no-pom") [{:no-pom true} (next args)]
("-S" "--suppress-clash") [{:debug-clash false} (next args)]
("-v" "--verbose") [{:verbose true} (next args)]
("-X" "--exclude") [{:exclude (fnext args)} (nnext args)]
so you'll probably want --jar my-app.jar -m your-main-ns maybe others?#2021-09-1019:32ChaseAwesome, thanks for this.#2021-09-1019:33ChaseYeah I wish the official heroku buildpack was more up to date. It seems they are closing the issues asking for an official deps.edn version. There are various workarounds and community built buildpacks floating out there#2021-09-1019:33seancorfieldAccording to Heroku's own docs you can specify a more modern CLI via the CLOJURE_CLI_VERSION environment variable.#2021-09-1019:34Chasehmm...#2021-09-1019:34Chasewell then. I did not see that solution in the clojure buildpack github repo#2021-09-1019:34seancorfieldSee https://devcenter.heroku.com/articles/clojure-support#setting-clojure-cli-version#2021-09-1019:36ChaseAwesome, I'll just find a way to set that variable then so I don't get future conflicts as I continue to add to my deps.edn#2021-09-1019:37seancorfieldHere's the lines in their build pack that handles arbitrary CLI installation: https://github.com/heroku/heroku-buildpack-clojure/blob/main/bin/compile#L33-L37#2021-09-1019:39seancorfieldSettting "config vars" (aka environment variables I assume): https://devcenter.heroku.com/articles/config-vars#managing-config-vars#2021-09-1019:39seancorfield(I'm guessing here, based on their docs -- I have never used Heroku)#2021-09-1019:56ChaseThat worked! Thanks so much. I feel silly for not checking that option out first#2021-09-1020:28seancorfieldNP. By learning a bit more about Heroku, I'm better able to help others who may have Qs about it, even if I don't use it myself 🙂#2021-09-1108:08practicalli-johnI used to work at Heroku, so am fairly familiar with there tooling.
The buildpack is something I'd like to revisit as my current approach is not as smooth as it should be.
If there isn't a suitable Heroku buildpack, then I'll consider creating a suitable community version.#2021-09-1020:38Mark WardleHello all. Happy user of deps.edn and depstar to build jars and uberjars - should I be thinking to replace my depstar usage with a tools.build build.edn? Any recommendations - I don’t do anything very complex. #2021-09-1020:40seancorfield@mark354 For library JARs, I have switched completely to tools.build -- take a look in projects like next.jdbc or honeysql to see the sort of stuff I'm doing with tools.build nowadays.#2021-09-1020:40seancorfieldEven depstar's own CI pipeline uses tools.build to run tests and build its own JAR! 🙂#2021-09-1020:41seancorfieldFor most folks, using tools.build for uberjars is going to work but there are currently some gaps between uber in tools.build and what depstar does so right now I still recommend depstar for building uberjars.#2021-09-1020:47Mark WardleThat is very helpful thank you - I will keep watching progress! Thank you.#2021-09-1020:44seancorfieldHowever, I'm actively talking with Alex about this and we've been discussing a path forward to help get uber to parity with depstar's uberjar functionality -- mostly around how the latter handles OSS license files in libraries that are pulled into the uberjar and also the thorny problem of log4j2 plugins cache files. My goal is to work with Alex to get tools.build to a point where I can retire depstar completely and recommend everyone use tools.build instead.#2021-09-1020:44seancorfield(also, there's a #tools-build channel for anyone who wants to dig deeper into that stuff)#2021-09-1108:04practicalli-johnThe version of Clojure CLI can be specified with the Heroku Config Car CLOJURE_CLI_VERSION
https://practical.li/clojure-web-services/projects/banking-on-clojure/deployment-via-ci.html#heroku-pipeline-configuration#2021-09-1316:18grzm@seancorfield Did you ever find a satisfactory solution to managed-deps with tools deps? re: https://ask.clojure.org/index.php/9849/teams-common-dependencies-tooling-across-multiple-projects#2021-09-1316:21borkdude@grzm there are various solutions to this, Sean has written a couple of blogs about their mono-repo setup using polylith. At my current job we are using a straightforward approach using babashka and templates that we use to generate deps.edns in all sub-projects from common data / versions.#2021-09-1316:23grzmYeah, I've been following his mono-repo articles: good stuff, but I'm not convinced polylith is the answer for our team. I've been considering the templates/generation method as well.#2021-09-1316:24borkdudeAt least you get a "what you see (generate) is what you get" effect which is easy to follow and debug#2021-09-1316:24grzmAgreed. The drawback is of course caching by another name: are the generated files up to date?#2021-09-1316:25borkdudeI'll share our script#2021-09-1316:25grzmCheers!#2021-09-1316:25borkdudehttps://gist.github.com/borkdude/58f099b2694d206e6eec18daedc5077b#2021-09-1316:26borkdudeso each sub-project has a deps.template.edn which we then use to generate a corresponding deps.edn#2021-09-1316:26borkdudecomments, etc you store in the template, the generate deps.edn are a bunch of sorted maps, to be more friendly with git etc#2021-09-1316:27grzmNice. Thanks!#2021-09-1316:27borkdudeI've hooked this script in the :init phase of bb.edn so any time we execute a task, the deps.edns are synced#2021-09-1316:27grzmWe might have someone on our team using babashka. 😉#2021-09-1316:28borkdudeyou can also hook it to a commit hook, cd etc#2021-09-1316:54seancorfieldPolylith supports :override-deps now but we haven't felt the need for it yet, but it "solves" that issue I'd raised about sharing common deps -- and, to be honest, after I'd had some long, in-depth discussions with Alex about it, shifting everything to essentially a :dev alias in the top-level repo deps.edn file addressed pretty much all my concerns anyway.#2021-09-1316:55seancorfieldIt's still a bit frustrating that :local/root deps can go "stale" -- but we're mostly working with just one .cpcache folder now (at the top of the repo) so -Sforce or just blowing away the folder isn't too much of a burden.#2021-09-1316:57seancorfieldSo, to answer your Q from the channel @grzm, where we are now is a "satisfactory solution to managed-deps", yes.#2021-09-1316:58grzmThanks for the follow-up, @seancorfield#2021-09-1317:04seancorfieldAt this point, we have about a quarter of our codebase migrated to Polylith (in 30+ "bricks" and nearly 20 "projects"), so we have :dev (and :test) aliases in the project deps.edn for Polylith and :everything / :runner aliases for the non-migrated code, and we use :local/root deps for all our subprojects/"bricks". Plus we have a good-sized build.clj file from which we run all our tests and build uberjars etc.#2021-09-1416:14kennyHi. I am updating our CI images to use 1.10.3.967 from 1.10.3.855. After the update, the tests for a project failed by timing out after 10 minutes. The entirety of the test output was as follows.
Downloading: com/datadoghq/dd-trace-api/0.86.0/dd-trace-api-0.86.0.pom from central
Downloading: org/clojure/clojure/maven-metadata.xml from clojars
Downloading: org/clojure/clojure/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from datomic-cloud
I will be rolling back to the earlier version, but I am curious, is this something others have hit? Are there some particular steps I should follow when upgrading from 855 to 967?#2021-09-1416:17Alex Miller (Clojure team)^^ @jaret#2021-09-1416:18Alex Miller (Clojure team)I think this may be specific to the datomic-cloud repo, but maybe should move that to #datomic#2021-09-1416:19kennyInteresting. Was there a change in 855 -> 967 that would affect s3 repos?#2021-09-1416:31Alex Miller (Clojure team)no, I think there may be changes that have happened in one of the datomic repos#2021-09-1416:31kennyAh, ok. Fwiw, I worked it backward. 1.10.3.933 works and 1.10.3.943 does not work.#2021-09-1416:31Alex Miller (Clojure team)we've bumped some aws/s3 deps in that version range, but I have not seen anything like you describe#2021-09-1416:33Alex Miller (Clojure team)hrm, well probably ignore what I said above then#2021-09-1416:33Alex Miller (Clojure team)I've tested s3 stuff but not seen what you're describing#2021-09-1416:34Alex Miller (Clojure team)does it persist with -Sforce?#2021-09-1416:34Alex Miller (Clojure team)if so, do you have a deps.edn that repros that you can give me?#2021-09-1416:35Alex Miller (Clojure team)we did make some changes in the http-client config for s3 in that version range#2021-09-1416:36kennyAll tests I ran are run with -Sforce. I don't have a readily available deps.edn. I also don't think this is related to Datomic. In a test run with 943, the cli output was just this.
Downloading: org/clojure/clojure/maven-metadata.xml from cs-mvn
cs-mvn is a s3 repo.#2021-09-1416:37kennyIt seems like a deadlock, but that would be very speculative. I'm surprised the requests aren't timing out.#2021-09-1416:38kennyIf it helps, the command I'm running.
/home/circleci/clj-1.10.3.943/bin/clojure -Sforce -J-Dclojure.main.report=stderr -J-Xmx3800m -A:test:test-runner -M -m kaocha.runner --reporter kaocha.report/documentation --plugin profiling --plugin kaocha.plugin/junit-xml --junit-xml-file test-results/kaocha/results.xml
#2021-09-1416:39Alex Miller (Clojure team)can you ctrl-\ to get a thread dump?#2021-09-1416:41kenny#2021-09-1416:44Alex Miller (Clojure team)well, that's interesting. don't think it necessarily has anything to do with s3 based on that#2021-09-1416:45kennyYeah. DefaultMetadataResolver-0-1#2021-09-1416:47Alex Miller (Clojure team)it could indeed be a deadlock in the session locks. I'll have to think about this more. you might be able to bypass with -Sthreads 1#2021-09-1416:47kennyI know close to 0 about the maven lib, so take this with a grain of salt... if they are using a fixed size thread pool to get s3 creds in parallel, that could easily result in a deadlock.#2021-09-1416:47kennyI'll try#2021-09-1416:47Alex Miller (Clojure team)there are several layers to this problem, but I don't think it has anything to do with s3#2021-09-1416:48Alex Miller (Clojure team)there have been changes in the session caching I'm doing, and in the underlying maven lib. I suspect my changes at the moment. :)#2021-09-1416:48kennyWith 1.10.3.943, -Sthreads 1 does not have any impact -- still hangs.#2021-09-1416:49Alex Miller (Clojure team)that's probably an important clue :)#2021-09-1416:49Alex Miller (Clojure team)I'm probably not going to get to it today, but I will take a look soon#2021-09-1416:50kennySounds good. I'll leave us at 933 for now.#2021-09-1416:51Alex Miller (Clojure team)I logged as https://clojure.atlassian.net/browse/TDEPS-207#2021-09-1721:49Alex Miller (Clojure team)hey kenny, I have not succeeded in reproducing this or figuring it out, but I did spend some time looking at the various versions of the libs and I think I was using maven resolver libs (1.7.x) that require maven core libs (4.0 alpha+), so I've fallen back to the 1.6.x series there. they have been reworking the concurrency and locking parts of maven in 1.7.x. I don't see that directly implicated but it might be related. But if you wanted to try Clojure CLI 1.10.3.981, it's available.#2021-09-1722:17kennyThanks for digging in. I'll give it a shot, and let you know if it fixes the issue.#2021-09-1800:23kennyJust ran a test with 1.10.3.981, and I'm still getting the same hang.#2021-09-1800:28kennyI can repro locally (not on ci) too, btw.#2021-09-1800:43kennyI tried the work it backwards by commenting things out approach, and I cannot discern any pattern at all. I'll send you the smallest deps.edn I worked it back to. Any other changes, commenting something out or moving local/root deps into this deps.edn, result in no hang. Let me know if there's any info you're interested in.#2021-09-1416:41Alex Miller (Clojure team)I guess if you're not local, you probably can't#2021-09-1416:41kennycircleci ssh access is one of my fav features 🙂#2021-09-1418:43jaretSorry @alexmiller @kenny. Do you still think this is Datomic repo related? I am trying to catch up.#2021-09-1418:44Alex Miller (Clojure team)no#2021-09-1418:44Alex Miller (Clojure team)sorry :)#2021-09-1418:44Alex Miller (Clojure team)I am pretty sure I understand the source of the problem, and I introduced it#2021-09-1418:45Alex Miller (Clojure team)I actually have failing tests for this on CI on newer versions of Java that I had missed#2021-09-1500:56vemvCan these be silenced?
Downloading: io/grpc/grpc-core/maven-metadata.xml from central
Downloading: io/grpc/grpc-api/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from clojars
Downloading: io/grpc/grpc-netty-shaded/maven-metadata.xml from central
Downloading: org/clojure/clojure/maven-metadata.xml from central
(sorry if FAQ)#2021-09-1501:09seancorfield@vemv That indicates something is using either a version range or a symbolic version (LATEST or RELEASE).#2021-09-1501:16vemvTIL, thanks! makes sense.
we're not even using the one dep that is specified at RELEASE so yeah... will zap it ⚡#2021-09-1509:00Carsten BehringSometimes running "clj" downloads a large amount of dependencies.
There is a progress report, but per download only.
But if any single download is very large, we do not see any progress, and clj seems to hang.
"mvn install" does this better, it has a per file donload progress.
Maybe this it could be added to "clj" ?
Sometimes I do a "clj -Spom; mvn install" just to avoid this apparent hanging#2021-09-1512:27Alex Miller (Clojure team)We have a ticket for this, can’t say it’s very high on my priority list#2021-09-1515:12rickmoynihanAre there any known issues with tools.deps and :mvn/repos where tools deps won’t fetch artifacts from a 3rd party repo when you have 2 private s3 repos for your corp (in ~/.clojure/deps.edn ) and then add an extra 3rd party repo for a supplier in a specific projects deps.edn?#2021-09-1515:14rickmoynihanBasically we’re finding that we can’t resolve a bunch of deps that one of our suppliers is hosting on their own maven repo when we include our s3 repos too#2021-09-1515:15borkdudeIs this the s3 concurrency thing again?#2021-09-1515:15rickmoynihanI was wondering that actually#2021-09-1515:16rickmoynihanI’ll try setting -Sthreads 1#2021-09-1515:17Alex Miller (Clojure team)how are you providing creds? ambient AWS or via repo settings?#2021-09-1515:17Alex Miller (Clojure team)doesn't sound like a concurrency thing, sounds like an access thing#2021-09-1515:17rickmoynihanFor the s3 repos I think it’s done via settings.xml — for the supplier repo — there are no creds (it’s all public)#2021-09-1515:21rickmoynihanthe s3 repos are our in-house stuff, and they work in other projects etc fine — it’s only the project which adds a new supplier repo in the project deps.edn that struggles.
It also gets our deps fine; it’s the suppliers deps it doesn’t resolve.#2021-09-1515:23rickmoynihanbut if I remove our s3 repos from ~/.clojure/deps.edn and remove the deps we need from there from the project; so just the suppliers :mvn/repos are active in the projects deps.edn with their dependencies it works.#2021-09-1515:24borkdude@rickmoynihan you can use -Srepro for this#2021-09-1515:24rickmoynihanahh I think I see what @alexmiller is saying — that perhaps the s3 creds are providing creds for the 3rd party repo!? Which it’s then refusing?!#2021-09-1515:27rickmoynihanA colleague has also managed to recreate the same findings (after blowing away her .m2)#2021-09-1515:28rickmoynihan-Sthreads 1 seems to be helping… it looks like it has got further anyway (just taking a long time to fetch)#2021-09-1515:50rickmoynihanok -Sthreads 1 seems to have actually worked — though it took forever#2021-09-1515:51rickmoynihanI’m not sure if the supplier have some dodgy deps either — as literally 1/2 of maven central came down… more than I’d normally expect… e.g. even apache ant! 😕#2021-09-1515:52rickmoynihanat one point I thought it might have been fetching the same deps in a loop -- though could have been deps with similar names#2021-09-1515:53rickmoynihanchecking -Stree output now to see if anything odd is going on#2021-09-1516:00rickmoynihanhmm ok I guess it’s legit though does seem a bit odd:
com.cemerick/url 0.1.1
. pathetic/pathetic 0.5.0
. com.cemerick/clojurescript.test 0.0.4
. org.clojure/clojurescript 0.0-1586
. com.google.javascript/closure-compiler r2180
. args4j/args4j 2.0.16
X com.google.guava/guava 13.0.1 :older-version
X com.google.protobuf/protobuf-java 2.4.1 :superseded
. org.json/json 20090211
. org.apache.ant/ant 1.8.2
. org.apache.ant/ant-launcher 1.8.2
X com.google.code.findbugs/jsr305 1.3.9 :older-version
. com.googlecode.jarjar/jarjar 1.1
. org.clojure/google-closure-library 0.0-2029-2
. org.clojure/google-closure-library-third-party 0.0-2029-2
. org.mozilla/rhino 1.7R4#2021-09-1519:05souenzzoHello
I'm having this error and I can't understand why
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.50 in central ()
This error says that i can't find com.datomic/ion, but it exists in my fs
ls ~/.m2/repository/com/datomic/ion/0.9.50/
ion-0.9.50.jar ion-0.9.50.jar.sha1 ion-0.9.50.pom ion-0.9.50.pom.sha1 _remote.repositories
Also, the same deps.edn, in exacly the same commit works on other machines (CI, other devs, and my machine if I use a time machine to be back in yesterday)
cognitect-dev-tools credentials is present in ~/.m2/settings.xml and :mvn/repos configured in ~/.clojure/deps.edn#2021-09-1519:08souenzzoclj -Strace do not write trade.edn#2021-09-1519:09souenzzoclj -Srepro -Sforce -Sdeps '{}' -Strace do the same error as clj -Spath (in the same directory that contains my deps.edn)
clj -Strace in a empty dir works#2021-09-1519:13Alex Miller (Clojure team)Do you have ambient aws creds with s3 read?#2021-09-1519:19souenzzoi can do aws s3 ls and it list my S3 buckets#2021-09-1519:30souenzzothere is something that I can do to get more debug info?
why trace do not work?
why -Sforce -Srepro -Sdeps {} do not work?#2021-09-1519:31Alex Miller (Clojure team)why are you doing -Sdeps {} ?#2021-09-1519:31souenzzo-Sdeps do not "overwrite" deps.edn?#2021-09-1519:31Alex Miller (Clojure team)no, merge#2021-09-1519:33souenzzooh. ok.
And about trace? Why it not work?
Why/which package is requiring datomic/ion? (that is a huge project, with a lot of local/root)
Why it is trying to find datomic/ion if it exists on disk?#2021-09-1519:33Alex Miller (Clojure team)trace is written at the end, so wouldn't work if you are erroring out#2021-09-1519:34Alex Miller (Clojure team)not sure I can answer why/which is requiring datomic/ion - presumably something that uses datomic#2021-09-1519:34Alex Miller (Clojure team)it might be looking for a different version of datomic/ion than you have#2021-09-1519:35Alex Miller (Clojure team)is this a deps.edn you can share?#2021-09-1519:35souenzzonope 😕
It contians many local/root and git deps.#2021-09-1519:35Alex Miller (Clojure team)I guess it looks like the same version above#2021-09-1519:36Alex Miller (Clojure team)I would certainly try re-running your original command with -Sforce (no -Sdeps)#2021-09-1519:37Alex Miller (Clojure team)other than that, not sure what to suggest other than selectively bisecting your deps to get to something shareable/traceable#2021-09-1519:40souenzzoReproduced with a "inline" deps
https://gist.github.com/souenzzo/6f2143b9247a06a8527422a6ecc50990#2021-09-1519:43Alex Miller (Clojure team)that doesn't look like the right maven repo#2021-09-1519:43Alex Miller (Clojure team)should be "datomic-cloud" {:url ""} I think?#2021-09-1519:45Alex Miller (Clojure team)clj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}} :mvn/repos {"datomic-cloud" {:url ""}}}#2021-09-1519:46Alex Miller (Clojure team)does that work?#2021-09-1519:47souenzzoworked!#2021-09-1519:48Alex Miller (Clojure team)as to why it's downloading when it exists, I'm honestly not sure about that one#2021-09-1519:49Alex Miller (Clojure team)my suspicion is that some of the update checker stuff in maven relies on metadata files the s3 repo does not provide and it's redoing something it shouldn't#2021-09-1519:49Alex Miller (Clojure team)certainly that would make sense to me for SNAPSHOT or RELEASE versions, but in this case it's neither so that seems weird (unless you did have that somewhere)#2021-09-1519:53souenzzoclj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}}}' -Spath
src:....
[0 ~/empty-dir 16:52:23]
vim ~/.clojure/deps.edn ## remove datomic-cloud
[0 ~/empty-dir 16:52:35]
clj -Sdeps '{:deps {com.datomic/ion {:mvn/version "0.9.50"}}}' -Spath
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.50 in central ()#2021-09-1519:57Alex Miller (Clojure team)so in that first case, it's not re-downloading#2021-09-1519:57Alex Miller (Clojure team)you're only running into this when it can't resolve from anywhere#2021-09-1520:02Alex Miller (Clojure team)I logged here https://clojure.atlassian.net/browse/TDEPS-208 and I will take a closer look at this when I have some time#2021-09-1520:04souenzzoShould be related with this file, that as far i know, is a new file
cat ~/.m2/repository/com/datomic/ion/0.9.50/_remote.repositories
#NOTE: This is a Maven Resolver internal implementation file, its format can be changed without prior notice.
#Wed Sep 15 16:44:43 BRT 2021
ion-0.9.50.jar>datomic-cloud=
ion-0.9.50.pom>datomic-cloud=
#2021-09-1520:09Alex Miller (Clojure team)yeah, this is related to the metadata files stuff I mentioned earlier#2021-09-1520:10Alex Miller (Clojure team)I may not be able to "fix" this if it's under the api layer, but need to look at it more closely#2021-09-1519:52Santiagoare there templates for tools.deps like there are for leinigen? wondering if there is a way to bootstrap a pedestal project faster#2021-09-1520:00souenzzocheckout "clj-new"
https://github.com/seancorfield/dot-clojure#2021-09-1520:05souenzzoIt is "deps-new"#2021-09-1520:05Santiagonice thanks!#2021-09-1520:14seancorfieldhttps://github.com/seancorfield/deps-new will let you write a template for a Pedestal project and then reuse that template to create new Pedestal projects faster in future, but I'm not sure if that's what you're actually asking @UFPEDL1LY?#2021-09-1520:16seancorfieldThere's also https://github.com/seancorfield/clj-new which can use Leiningen, Boot, and "clj-templates" to create new projects -- but a lein-template is going to create a lein-based project, not a CLI-based one.#2021-09-1520:16seancorfieldI don't know if there's a Pedestal clj-template...#2021-09-1520:33Santiagothere are some, but they’re all lein -based. I guess I’ll make one for CLI#2021-09-1520:33Santiagothanks!#2021-09-1619:29Carsten BehringRunning by repl, I get repeatedly the same error message about a certain jar:
Error building classpath. Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'
java.lang.IllegalStateException: Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'
#2021-09-1619:33borkdudeconcurrency issue? -Sthreads 1?#2021-09-1619:47Carsten BehringIndeed, this seems to fix it. I remmber that the jars are big, so individaul download takes long.
rw-r--r-- 1 carsten carsten 219M Sep 3 21:35 mkl-2020.3-1.5.4-linux-x86_64-redist.jar
-rw-r--r-- 1 carsten carsten 172M Sep 3 21:28 mkl-2020.3-1.5.4-windows-x86_64-redist.jar
-rw-r--r-- 1 carsten carsten 147M Sep 3 21:34 mkl-2020.3-1.5.4-macosx-x86_64-redist.jar
-rw-r--r-- 1 carsten carsten 137M Sep 3 21:34 mkl-2020.3-1.5.4-linux-x86-redist.jar
-rw-r--r-- 1 carsten carsten 112M Sep 3 21:33 mkl-2020.3-1.5.4-windows-x86-redist.jar
-rw-r--r-- 1 carsten carsten 4.4M Sep 3 21:25 mkl-2020.3-1.5.4-linux-x86_64.jar
-rw-r--r-- 1 carsten carsten 4.1M Sep 3 21:28 mkl-2020.3-1.5.4-linux-x86.jar
-rw-r--r-- 1 carsten carsten 4.1M Sep 3 21:25 mkl-2020.3-1.5.4-macosx-x86_64.jar
-rw-r--r-- 1 carsten carsten 3.3M Sep 3 21:25 mkl-2020.3-1.5.4-windows-x86_64.jar
-rw-r--r-- 1 carsten carsten 2.0M Sep 3 21:28 mkl-2020.3-1.5.4-windows-x86.jar#2021-09-1619:47Carsten BehringSo it seems that clj has a reproducible issue with those.#2021-09-1619:48Carsten BehringI have it inside a Docker, so on an empty maven folder.#2021-09-1619:48Carsten Behringand it happens on every run#2021-09-1619:48Carsten Behring(or docker build, better said)#2021-09-1619:53Alex Miller (Clojure team)I'm going to be looking at some of this stuff tomorrow#2021-09-1620:08Carsten BehringI have it on every run against an empty .m2 folder with this deps.edn
{
:mvn/repos {"clojars" {:url ""}
"central" {:url ""}
"bedatadriven" {:url ""}}
:paths ["src" "resources"]
:deps {
org.clojure/clojure {:mvn/version "1.10.3"}
clj-python/libpython-clj {:mvn/version "2.000"}
scicloj/clojisr {:mvn/version "1.0.0-BETA19"}
scicloj/tablecloth {:mvn/version "6.012"}
scicloj/notespace {:mvn/version "3-beta9"}
scicloj/scicloj.ml {:mvn/version "0.1.0-beta4"}}}#2021-09-1721:52Alex Miller (Clojure team)Hey Carsten, I tried a lot of stuff today and I was able to reproduce this exactly once. The message you were seeing comes out of Maven itself. In a recent lib update I moved the maven resolver libs up to 1.7.x but those versions are not necessarily guaranteed to work with Maven core 3.8.x (they are expecting Maven core 4.0 alphas). Resolver 1.7 has been working on replacing the locking/synchronization infrastructure which is directly in this area. Anyhow, I have fallen back to 1.6.x which should work with the core libs I'm using. If you wanted to try with Clojure CLI 1.10.3.981, I'd be interested to know if that helped or not.#2021-09-2016:10Carsten BehringThis Dockerfile fails for me everytime :
https://gist.github.com/behrica/723cb69c5524497599fb90f7942f153b#2021-09-2016:11Alex Miller (Clojure team)thanks, that will help#2021-09-2016:12Alex Miller (Clojure team)one thing that I think contributes here is a lot of the deps under http://scicloj.ml are really big native libs#2021-09-2016:12Alex Miller (Clojure team)I suspect the size == longer download time which exacerbates the problem#2021-09-2016:12Carsten BehringIyes, the MKL libs are very big#2021-09-2016:12Carsten Behring> several jars of > 100 MB #2021-09-2016:13Carsten Behringerror building classpath. Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'
java.lang.IllegalStateException: Could not acquire write lock for 'artifact:org.bytedeco:mkl:2020.3-1.5.4'#2021-09-2016:13Alex Miller (Clojure team)yeah, that exception is in Maven#2021-09-2016:13Carsten BehringIt works with this clj -P -Sthreads 1#2021-09-2016:14Carsten Behringalways#2021-09-2016:14Carsten Behringi will quickly try with 1.10.3.981,#2021-09-2016:14Alex Miller (Clojure team)they have completely rewritten this area in the newest Maven resolver, but it does not work with the current released version of Maven core#2021-09-2016:15Alex Miller (Clojure team)in the latest I've fallen back to prior Maven resolver version#2021-09-2016:15Carsten Behringdocker build running, it takes some minutes#2021-09-2016:35Carsten BehringIt works with 1.10.3.981#2021-09-2016:38Alex Miller (Clojure team)oh really#2021-09-2016:38Alex Miller (Clojure team)well that took their new code out of the loop, so that's good to know#2021-09-1716:17Kira McLeanI’m getting a similar error as the one @souenzzo mentioned above, but looks like a different cause:
Error building classpath. Failed to read artifact descriptor for org.clojure:data.json:jar:2.4.0
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.clojure:data.json:jar:2.4.0
In CI only (fetching all deps works locally).. wondering if anyone has any clues about why our CI build wouldn’t be able to access the clojure data.json jar? Anyone else having issues today fetching clojure data.json?#2021-09-1716:48souenzzoRemove .m2 dir//maven cache.
Probably an incomplete//corrupee download#2021-09-1716:57Kira McLeanI tried that 😕 .. seems to be the only thing the internet recommends as well. Or at least I believe I’ve tried that, if I’m understanding how invalidating the circle CI cache works. I have had issues with it in the past though!#2021-09-1718:44Alex Miller (Clojure team)can you just clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.4.0"}}}' on the CI machine?#2021-09-1719:25Kira McLeanInteresting.. that gives
> Failed to read artifact descriptor for org.clojure:clojure:jar:1.10.3
maybe there are bigger problems..#2021-09-1719:25Kira McLeanbut also we’re using circle ci so it’s a new machine every time#2021-09-1720:16Alex Miller (Clojure team)that seems bad, maybe some kind of config error?#2021-09-1815:48mike_ananevHello! Can I use Gist in deps.edn with url and full sha? I would like to run my examples from gist.
I'm trying to load my example but clj tools deps fails with message Error building classpath. Library org.rssys/lzw.coder has prefix sha, use full sha or add tag
org.rssys/lzw.coder {:git/url ""
:git/sha "39f95b17f443de026951f97cf5c8f243"}
#2021-09-1815:50Alex Miller (Clojure team)if you are on older Clojure CLI, should be :sha#2021-09-1815:51Alex Miller (Clojure team)or that's not a full sha#2021-09-1815:52mike_ananevWith :sha the same error. Is there any examples to run gist?#2021-09-1815:52Alex Miller (Clojure team)yeah, that's only 32 chars, should be 40#2021-09-1815:52Alex Miller (Clojure team)gists do work, nothing special there#2021-09-1815:54Alex Miller (Clojure team)i see that gists only give you 32 in the ui#2021-09-1815:54Alex Miller (Clojure team)oh wait, that's not the sha#2021-09-1815:55Alex Miller (Clojure team)for that gist, sha is cbb3c6e3abe14c91c6bd19a6c3702a6df31be393#2021-09-1815:55mike_ananevOh, thanks! Where did you find it?#2021-09-1815:55Alex Miller (Clojure team)I cloned it#2021-09-1815:56mike_ananevYeah! It works! Thank you!#2021-09-1815:56Alex Miller (Clojure team)from https://gist.github.com/mikeananev/39f95b17f443de026951f97cf5c8f243/revisions, if you click on the ... for View File, you'll get to this link which has the sha https://gist.github.com/mikeananev/39f95b17f443de026951f97cf5c8f243/cbb3c6e3abe14c91c6bd19a6c3702a6df31be393#2021-09-1815:58Alex Miller (Clojure team)you could also git ls-remote HEAD#2021-09-1815:59Alex Miller (Clojure team)might be useful to have a utility or special support for this#2021-09-1816:01mike_ananev👍#2021-09-2115:53Dimitar UzunovHi! I may have a case of TDEPS-207#2021-09-2115:53Dimitar Uzunovwhat diagnostic info I should grab? I can do CTRL-\ if its useful#2021-09-2115:54Alex Miller (Clojure team)are you using the latest Clojure CLI?#2021-09-2115:55Dimitar Uzunovyes, just gave it a try#2021-09-2115:55Alex Miller (Clojure team)or tools.build v0.5.1 if you're coming to it from there?#2021-09-2115:55Alex Miller (Clojure team)then yes, thread dump would be useful#2021-09-2115:55Dimitar Uzunovposting it now#2021-09-2115:59Dimitar Uzunov@U064X3EF3 here it is#2021-09-2116:05Alex Miller (Clojure team)thx, will add it to the ticket and will look at it the next time I work on this#2021-09-2318:30borkdudeIs this way of checking out the same git lib with multiple roots supported? It works, but I wonder if I can rely on this.
{:deps
{cheshire/cheshire {:mvn/version "5.10.0"}
graal-config/cheshire {:git/url ""
:git/sha "b06e33694d2c878169958f7317ea01d9c0353ab4"
:deps/root "config/cheshire/cheshire"}
graal-config/nippy {:git/url ""
:git/sha "b06e33694d2c878169958f7317ea01d9c0353ab4"
:deps/root "config/com.taoensso/nippy"}}}
The use case is having one repository with lots of native-image configurations that people can include on their classpath#2021-09-2318:31borkdudeas long as you use the same SHA it works, but in other cases it might not?#2021-09-2318:34Alex Miller (Clojure team)looks ok to me, not sure why same sha matters?#2021-09-2318:35borkdudecan you use a different SHA for the same git/url?#2021-09-2318:35Alex Miller (Clojure team)sure#2021-09-2318:35borkdudeif that also works, then this is :thumbsup:#2021-09-2318:35Alex Miller (Clojure team)gitlibs don't care#2021-09-2318:35borkdudeI wasn't sure how they are checked out and if there could be conflicts#2021-09-2318:36Alex Miller (Clojure team)now if you get into wanting to compare the same lib using different url/sha/whatever, things get murkier :)#2021-09-2318:36borkdudethis lib won't have any other deps so I don't think that will be a problem and people should really only include this dep in their final builds#2021-09-2318:37Alex Miller (Clojure team)Like graal-config/nippy exists in the dep expansion at two different shas etc#2021-09-2318:37borkdudeas in: libs should not depend on this lib#2021-09-2318:37borkdudebut only final app users should#2021-09-2318:37Alex Miller (Clojure team)yeah, users always do what you tell them ;)#2021-09-2416:43rgmI have a question about how a lib’s resources get rolled up into an uberjar:
• let’s say I’m beginning a new lib under /lib/foo/ within the project, and it has /lib/foo/deps.edn and /lib/foo/resources/hug.sql
• The outer project is consuming it using :local/root in its deps.edn
• Let’s say using depstar or now tools.build, would the uberjar contain hug.sql (the lib’s resource) by default?#2021-09-2416:45rgmI have a couple plot twists here (eg lein with lein-tools-deps for … reasons), and I don’t think I understand classpath/build stuff all that well.#2021-09-2416:49rgm(suppose now that I’ve boiled it down like that I could check it fairly easily)#2021-09-2416:49rgm(doing that)#2021-09-2416:54Alex Miller (Clojure team)there is no "by default" under tools.build - either can be done, depends what you're doing#2021-09-2417:07rgmah ok I was using the uberjar script in the tools.build guide#2021-09-2417:09Alex Miller (Clojure team)that script is copying things from resources to the classes dir which is included into the uber jar#2021-09-2417:10Alex Miller (Clojure team)so I think the answer is yes, it should be included#2021-09-2417:10rgmmy experiment seems to indicate: yes, the nested resources will get yanked in#2021-09-2417:10rgm#2021-09-2417:11rgm#2021-09-2417:17seancorfieldbuild-clj makes some assumptions about what's "typical": copy src + resources; compile src. But in general you have complete control.#2021-09-2417:18rgmok so the problem I’m having with my local/root lib resources not getting rolled up is probably some mismatch weirdness deep in lein-tools-deps and I can work around it by adding "lib/foo/resources" to the :resource-paths in project.clj#2021-09-2417:21rgm(until such time as I can retire lein out of the project entirely)#2021-09-2417:21seancorfieldlein-tools-deps -- there's your problem 🙂#2021-09-2417:22rgmha well it’s providing a nice stepping stone in the middle of the river#2021-09-2417:22rgmhonestly this has been the only rough edge so far#2021-09-2417:22seancorfieldI archived boot-tools-deps because it only worked in the simplest cases and just caused confusion and frustration beyond those -- and I see a lot of the same around lein-tools-deps (although I think lein is slightly more amenable to such surgery).#2021-09-2417:23seancorfieldIt's been about six years since I last used lein for anything other than helping beginners with problems... 🤷:skin-tone-2:#2021-09-2417:25rgmha well, yeah, I started the project in 2015 so that checks out#2021-09-2417:50rgmverdict: yes, uberjar contains hug.sql from the local/root lib. Committed the experiment in case this is useful to anyone https://github.com/rgm/experiments/tree/master/2021/09-tools-build#2021-09-2418:03seancorfieldSee https://github.com/seancorfield/build-clj#typical-buildclj-with-build-clj for a version with defaults 🙂#2021-09-2418:17rgmoh right on! Thanks for this and, frankly, all you do.#2021-09-2418:22seancorfieldAt work, we build our uberjars like this:
(bb/clean {})
(b/copy-file {:src version-clj
:target "target/classes/ws/uberjar/release.clj"})
(bb/uber {:main main-class :uber-file jar
:compile-opts {:direct-linking true}})
the version-clj file is just a ns we create with a single def that has the git SHA/tag as its value, so we can ask any uberjar what it was built from.#2021-09-2418:22seancorfield(spit version-clj (str "(ns ws.uberjar.release)"
"(def version \"" (git-version) "\")"))#2021-09-2418:22seancorfield(defn git-version
"Return a git version string"
[]
(sh "git" "describe" "--tags" "--match" "b[ui][il]l[di]*"))
#2021-09-2421:02dorabI do something similar for the version. The uber task in my build.clj writes out a version.edn in the resources directory which is then read by the config management library (`aero` in this case) on startup.#2021-09-2417:50rgmverdict: yes, uberjar contains hug.sql from the local/root lib. Committed the experiment in case this is useful to anyone https://github.com/rgm/experiments/tree/master/2021/09-tools-build#2021-09-2418:42john-shafferHow can I work around a transitive dep on a https://ask.clojure.org/index.php/7836/tools-deps-does-not-handle-maven-relocations?show=7836 package? One of tess4j's deps pulls in bouncycastle/bctsp-jdk14 somewhere. How can I find which package pulls it in so I can override it?#2021-09-2418:43Alex Miller (Clojure team)This is not particularly easy right now#2021-09-2418:44Alex Miller (Clojure team)You could exclude it in all of your top level deps, then -Stree I think #2021-09-2418:50john-shafferThanks, I didn't know about -Stree#2021-09-2418:50john-shafferI was always using trace#2021-09-2418:52john-shafferOh, hmm, trace is working okay now too. I think I just messed up the exclusion earlier. Sorry about that#2021-09-2721:35cflemingI’m doing some testing with deps, and I have a bunch of dependencies like this:
intellij/sdk {:local/root "../sdks/213.3714.440-EAP-SNAPSHOT"}
Is there any way I can substitute that version string in the :local/root path?#2021-09-2721:36Alex Miller (Clojure team)no, there is no substitution/replacement functionality in deps.edn#2021-09-2721:36Alex Miller (Clojure team)you can pass -Sdeps on the command line and do it there if that helps at all#2021-09-2721:37cflemingNo, sadly. Ok, thanks, I’ll just have to do it manually.#2021-09-2721:38hiredman(or write some code to generate a per version alias with extra deps)#2021-09-2721:39cflemingInteresting idea, actually. I’m already generating a bunch of these deps files for IntelliJ, which is just a big ball of jars.#2021-09-2903:39cflemingI am trying (once again) to see if I can use deps to manage the dependencies for Cursive, which is tricky because it’s a mixed-language project. I’m almost there, with some fiddling, some babashka scripting, and some hackery. I have one thing blocking me now - creating a dependency between two deps projects in a way that sort-of mimics what Maven would do. The dependency is on a Java-only project, so I need to create a dependency on the compiled class files in some way for compiling the dependent module. Has anyone done anything like this?#2021-09-2903:42lread@cfleming, would a :local/root dep pointing to a jar file work for you here?#2021-09-2903:43cflemingIt would, but it’s not ideal since I’d have to ensure that the dep was built and compiled into a jar in order to be able to build the dependent project. I’m trying to figure out a way that will also work with IntelliJ’s compile process, and that’s tricky.#2021-09-2903:47lreadOh, sorry not familiar with IntelliJ compile process.#2021-09-2903:49cflemingNo worries. If you’re more familiar with Maven, it would be the equivalent of requiring a sub-project to be published into the local Maven repo in order to compile anything using it, which isn’t impossible but is annoying.#2021-09-2903:52lreadWe are using babashka tasks over at graal-build-time as a kind of make. I don't know if that type of thing would hook in at all for you though.
There was a time when I was familiar with Maven, but those memories have faded...#2021-09-2903:53cflemingYes, I’m already using babashka tasks to generate and update all the deps.edn files I’m using. It works great for setting the project up, but I just can’t get this last bit to work.#2021-09-2903:55lreadSorry I can't be of more help, but I do offer you moral support 🙂#2021-09-2903:56cflemingThanks! I have at least a blog post’s worth of material now…#2021-09-2903:57lreadAwesome, I look forward to reading all about it!#2021-09-2906:55borkdude@cfleming there is prep lib now for on the fly compilation, would that help? Exact details should be in the reference (on the phone)#2021-09-3002:17cflemingIn the end I realised that what I needed was an IntelliJ-specific thing, likely only useful to me. So I added support for that, Cursive will now treat dependencies that look like:
cursive/jps-plugin:module {:local/root "jps-plugin"}
(local root with :module suffix on the key) as special, and I just hide those deps in an alias so that only Cursive sees it.#2021-09-3002:19cflemingI’ll document it in case anyone else wants to use deps for managing non-Clojure projects with Cursive, though.#2021-09-2912:28athomasoriginalRE: help/doc
If I have a :build alias in my deps.edn and it has a proper ns doc string should I be able to run clojure -A:deps -T:build help/doc or is this currently strictly only for non-project tools?#2021-09-2912:29Alex Miller (Clojure team)should be able to#2021-09-2912:32athomasoriginalInteresting. When I run clojure -A:deps -T:build help/doc (clj version: 1.10.3.986) I get:
Namespace could not be loaded: help
#2021-09-2917:24seancorfield@tkjone Is it possible you have your own :deps alias, shadowing the root version?#2021-09-2917:28athomasoriginalDoesn’t look like. I actually pulled down next-jdbc github repo and tried running the same command for that lib and I get the same result. I am assuming it works for you though?#2021-09-2917:30seancorfieldWorks for me:
(! 620)-> clojure -A:deps -T:build help/doc
Checking out: at 2ceb95afd9c60072a7a0646a4bb4d53615c6770b
next.jdbc's build script.
clojure -T:build ci
clojure -T:build deploy
Run tests via:
clojure -X:test
For more information, run:
clojure -A:deps -T:build help/doc
-------------------------
build/ci
([opts])
Run the CI pipeline of tests (and build the JAR).
-------------------------
build/deploy
([opts])
Deploy the JAR to Clojars.
-------------------------
build/test
([opts])
Run all the tests.
#2021-09-2917:31seancorfieldAre you sure you don't have a :deps alias in your ~/.clojure/deps.edn file?#2021-09-2917:32athomasoriginallol I 100% do https://github.com/athomasoriginal/dotfiles/blob/master/.clojure/deps.edn#L27. Totally was not thinking about the global one#2021-09-2917:34athomasoriginaland that was indeed the problem facepalm#2021-09-2917:34athomasoriginalThanks, Sean!!#2021-09-2917:48seancorfieldMy dot-clojure used to have a :deps alias too, but I renamed it to :add-libs https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L96#2021-09-3019:37borkdudeConverted a gist into a proper -T / -M tool:
https://github.com/borkdude/api-diff#2021-09-3019:37borkdudeHaven't added any features since the gist, but if desired, happy to receive improvements.
EDIT: I have actually added a new feature now.#2021-09-3021:18lreadVery cool! I might hop on with some features at some later date. For example:
• only look at public API (ignore ^no-docs)
• compare clj to cljs implementation in the same lib
• compare lib a to lib b (eg. rewrite-cljs to rewrite-clj).
#2021-09-3021:19borkdudeyeah, PRs welcome :)
should add some tests too#2021-09-3021:19lread(I did write an api diff thingy a while ago, but... oh boy... way over-complicated! simple_smile)#2021-10-0109:46FiVoSimple question. What do I need to pass to tools.build jar or uber as main class symbol. I just have
(ns scratch
(:gen-class))
(defn -main [& args]
(println "Hello from Clojure!!! " args))
I tried 'scratch/-main but that doesn't seem to work.#2021-10-0109:47borkdude@finn.volkel Just the namespace#2021-10-0109:49borkdudee.g.: https://github.com/clj-easy/graal-config/blob/7a2cc5b1b1652580e26d990cf5bac2ed48247b7e/config/cheshire/cheshire/example/build.clj#L23#2021-10-0109:50FiVothanks#2021-10-0216:25yuhanSlightly confused about clj -T / :tools/usage key in deps.edn - is this deprecated? I get the error
> clj -Ttools list
-T is no longer supported, use -A with repl, -M for main, or -X for exec
#2021-10-0216:29Alex Miller (Clojure team)You need to update your CLI#2021-10-0216:32yuhanoh! so it's a relatively new feature, got it#2021-10-0216:57yuhanAm I right in understanding that clj -T install lib :as name is a sort of UX improvement over adding a :name alias in my ~/.clojure/.deps.edn with some map of :deps and :main-opts as copied from the library's readme?#2021-10-0217:24yuhanSeems like it! I just read the excellent docs over at https://github.com/seancorfield/dot-clojure, hopefully this catches on as an alternative to lein plugins :)
Although I'm still unsure why git deps are being recommended over the more familiar Maven coordinates, both seem to work#2021-10-0220:38seancorfieldGit deps let you depend on libraries that aren't published, or on specific versions that haven't been published yet, such as PRs. For "tools" in particular, there's no point in publishing them (to Clojars) because they are not "libraries" and people aren't building applications that depend on them.#2021-10-0220:40seancorfieldI like git deps a lot. As I noted in another recent thread about this topic, we don't happen to have any git deps in our QA/production artifacts (JARs) but that really is just "coincidence" since we rely on git deps a lot in our dev/test infrastructure (and, in particular, on a lot of dev/test tooling that is not published to Clojars!).#2021-10-0219:04athomasoriginalI’m in the process of moving from depstar to tools.build. Everything seems to be configured correctly, but when I run b/uber I get the following exception:
Cannot write META-INF/license/LICENSE.tomcat-native.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded.
Looking around it seems like this scenario should be covered by the default conflict handlers?#2021-10-0219:46dpsuttonAre you on OS X?#2021-10-0219:48dpsuttonThis might be the issue where there is already a file named license so it cannot make a directory named LICENSE on a case insensitive file system #2021-10-0219:53athomasoriginalYes, OSX. In this case, what is the solution?#2021-10-0219:55dpsuttonadd an exclusion on "LICENSE". Usually there is a file called LICENSE that exists so there is no way to make a folder called "license" to put the LICENSE.tomcat-native.txt file#2021-10-0219:49Alex Miller (Clojure team)It is that#2021-10-0220:54seancorfield@tkjone We ran into that at work and added an exclusion on grpc-netty-shaded and used a different implementation instead (that produced a noticeably smaller uberjar!)... just a sec and I'll pull that up...#2021-10-0220:55seancorfieldcom.google.cloud/google-cloud-vision {:mvn/version "1.103.7"
:exclusions [io.grpc/grpc-netty-shaded]}
io.grpc/grpc-okhttp {:mvn/version "1.41.0"}#2021-10-0220:55seancorfieldThat shaded library is the only one I've run across that causes this problem.#2021-10-0222:15athomasoriginalOutstanding! Did the same thing and it seems to have worked perfectly. For future readers, in my case the solution was:
com.google.firebase/firebase-admin {:mvn/version "8.1.0"
:exclusions [io.grpc/grpc-netty-shaded]}
io.grpc/grpc-okhttp {:mvn/version "1.41.0"}#2021-10-0222:18seancorfieldMy reading suggests that there are three possible implementations that the Google libraries can use, with grpc-netty-shaded just being the default. There's an unshaded version but that can cause conflicts if you also depend on netty-io directly (as we do at work), and then there's the okhttp one which is at least smaller, and seems less problematic from a conflict p.o.v.#2021-10-0222:21athomasoriginalI’m not familiar enough with Java land to even have realized that this is a thing I could look at doing. Glad to see someone else ran into this. It seems off in general because from the look of it, it was just a license file which couldn’t be created. :thinking_face:#2021-10-0222:29seancorfieldI wouldn't have even thought to look for alternatives if I hadn't run over the case-sensitivity conflict (of a LICENSE file in most libraries but a license directory in this one). Since okhttp is substantially smaller, I'm happy with the change. I can't even remember where I found the documentation saying you could swap out the implementations...#2021-10-0222:30athomasoriginalGreat catch!#2021-10-0401:37cflemingIs there a reason why tools.build aliases use :deps instead of :extra-deps in its alias? Does this mean that if the build is invoked using that alias that the deps will be completely replaced rather than augmented as in :extra-deps?#2021-10-0401:42cflemingReading the doc, it looks like it but I’m not sure I’m interpreting that correctly.#2021-10-0401:55cflemingIf so, I don’t understand how this works. Thinking about the t.d.a API, I call (deps/combine-aliases deps-map aliases ) and then call (deps/resolve-deps deps-map alias-args) where alias-args is the result of the first call. But in doing that, I don’t get the dependencies from the :deps entry in the build alias map. I can’t see any way to actually achieve that using the API except manually merging those deps into :extra-deps before calling resolve-deps.#2021-10-0401:58cflemingBacking up a bit for a problem statement, I’m trying to figure out how to make tools.build work with Cursive. Currently Cursive will sync dependencies from the deps.edn file to the project, so that symbols resolve and things generally work. When some of those deps are specified in an alias, the user selects which aliases they would like synced to the IDE, in order to make their dependencies available to Cursive. However this doesn’t work with dependencies under a :deps key, because t.d.a doesn’t return those to me.#2021-10-0402:00cflemingIt seems like the intention is to allow the user to maintain separate classpaths, one for the build tool and another for the project itself. There isn’t a good way to separate those in IntelliJ without some serious gymnastics, so my plan is to just provide a union of the two sets of dependencies that are available everywhere, and the user will have to take care where they use which ones. But for that to happen, I need t.d.a to return the :deps dependencies in addition to the :extra-deps ones.#2021-10-0402:45cflemingIs there any way to use clj to output the classpath of a tool? Using e.g. clj -Spath -T:build gives me some very strange errors.#2021-10-0403:26Alex Miller (Clojure team):deps and :extra-deps are synonyms in aliases#2021-10-0403:26Alex Miller (Clojure team)-T will not include the project :deps and :paths#2021-10-0403:30Alex Miller (Clojure team)what errors are you getting with clj -Spath -T:build ?#2021-10-0403:42seancorfieldI thought :deps and :replace-deps were synonyms, not :extra-deps?#2021-10-0403:44seancorfield@U0567Q30W Can you elaborate on "very strange errors" -- clj -Spath -T:build works and gives exactly the result I'd expect in all the projects I have a build.clj file...#2021-10-0403:45seancorfieldExample (with -Stree which is more readable):
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "e69583878885a6b5838788cb918f88cbd7d7cb8a8796928996"}, :content ("[email protected]")}#2021-10-0403:46seancorfieldhttps://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha.clj#L634 -- this is the context I was thinking of where :dep and :replace-deps are synonyms.#2021-10-0403:48cflemingAh, I see, clj -Spath -T:build works, but clj -Spath -Tbuild tells me: Error building classpath. Bad coordinate for library , expected map: nil#2021-10-0403:49seancorfieldThe latter tries to run a tool called "build" and I doubt you have such a thing installed.#2021-10-0403:49cflemingIf :deps and :extra-deps are synonyms, how should I handle that in my API calls above? Currently I’m not seeing :deps returned.#2021-10-0403:50cfleming@U04V70XH6 I see, I thought the alias was the name of the tool, but I guess I misunderstood.#2021-10-0403:50seancorfield{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "c4b7a1a5aaa78497a1a5aae9b3adaae9f5f5e9a8a5b4b0abb4"}, :content ("[email protected]")}
That's what you'd see with a tool you actually had installed.#2021-10-0403:53cflemingWhat does it mean for a tool to be installed? I just installed one using the example from the doc, and it doesn’t touch the deps.edn. My mental model of this must be wrong, I thought that a tool being installed meant that I had an alias in my deps.edn file which would allow me to execute something.#2021-10-0403:54cflemingI see, they’re installed under .clojure.#2021-10-0403:54Alex Miller (Clojure team)what Sean said above is wrong#2021-10-0403:55seancorfieldWhich part?#2021-10-0403:55Alex Miller (Clojure team)-T:build runs the alias, as a tool. it does not run an installed tool#2021-10-0403:55seancorfieldI never said it ran an installed tool.#2021-10-0403:55Alex Miller (Clojure team)oh#2021-10-0403:56Alex Miller (Clojure team)sorry, I missed "the latter"#2021-10-0403:56seancorfieldColin was confused about -Tbuild vs -T:build -- I was explaining they were different.#2021-10-0403:56cflemingRight, so -T:build runs an alias from my deps.edn file, and -Tbuild runs one I have in my .clojure.#2021-10-0403:56seancorfieldclojure -Ttools install ... is how a "tool" is installed.#2021-10-0403:57seancorfieldAnd I think to answer Colin's other Q (or one of them), there's a tool function in t.d.a that produces the "tool" basis.#2021-10-0403:57Alex Miller (Clojure team)running -Tsomething will find ~/.clojure/tools/something.edn which defines a lib/coord to use as the deps#2021-10-0403:58Alex Miller (Clojure team)the tool function is confusingly different than tool at the CLI#2021-10-0403:59Alex Miller (Clojure team)it pre-dates the existence of -T#2021-10-0404:00cflemingSo is it true that an alias in a deps.edn file designed to be run as a tool will not be used as a normal alias? I don’t think it’s true that :deps and :extra-deps are synonyms, because resolve-deps only looks for :extra-deps.#2021-10-0404:00seancorfieldYeah, I seem to recall using it for quite a while...#2021-10-0404:00seancorfield@U0567Q30W See the link I provided about :dep and :replace-deps (not :extra-deps).#2021-10-0404:00Alex Miller (Clojure team)I am not on a computer with the code atm, but I think it's normalized during canonicalization#2021-10-0404:01seancorfieldIn the confusing tool function:
(defn tool
"Transform project edn for tool by applying tool args (keys = :paths, :deps) and
returning an updated project edn."
[project-edn tool-args]
(let [{:keys [replace-deps replace-paths deps paths]} tool-args]
(cond-> project-edn
(or deps replace-deps) (merge {:deps (merge deps replace-deps)})
(or paths replace-paths) (merge {:paths (vec (concat paths replace-paths))}))))#2021-10-0404:01cflemingThanks Sean, I hadn’t got around to clicking on that, was reading code…#2021-10-0404:01Alex Miller (Clojure team)oh, sorry, yes - it's :deps and :replace-deps that are synonyms#2021-10-0404:02Alex Miller (Clojure team)but in -T there are no project deps so really any of them will add deps#2021-10-0404:02Alex Miller (Clojure team)sorry, I've been Strange Looping for the last week, my brain is out of Clojure world ;)#2021-10-0404:04cflemingOk. So: any tool alias will essentially create its own classpath. Is it true that tool aliases and other aliases are generally always going to be used in different ways? Would you ever have an alias that you would use as both?#2021-10-0404:04cflemingNo worries Alex, I’m jealous 🙂#2021-10-0404:05Alex Miller (Clojure team)there is not a good public api function in tdeps that will do the real equivalent of what -T does on the cli. that stuff is buried in the build-classpath stuff atm. some reasons for that are still supporting some of the deprecated -R etc stuff that make that code not just a simple create-basis call, and some is some work that's just not done yet#2021-10-0404:05cflemingI’m just trying to get straight in my head how to support all this in Cursive, and present it to the user in a way that will make sense.#2021-10-0404:06cflemingOk, thanks Alex. I’ll take a look at the code and see if I can figure it out, and come up with a plan.#2021-10-0404:06Alex Miller (Clojure team)I think we also have a real gap here with getting a repl with the effective classpath for a tool#2021-10-0404:07cflemingThat seems like a reasonable thing to want to do.#2021-10-0404:07Alex Miller (Clojure team)there's no combination of things to actually ask for that on the CLI#2021-10-0404:07Alex Miller (Clojure team)yes, for sure#2021-10-0404:08cflemingSeparating the classpaths in IntelliJ is going to be either really complicated, or a bit crappy.#2021-10-0404:09seancorfield@U0567Q30W For :build, can't you just run it in its own process?#2021-10-0404:09cflemingI’ll play around with it to improve my mental model, and think about it some more.#2021-10-0404:09seancorfieldOr are you trying to do something inside Cursive with build.clj itself?#2021-10-0404:10Alex Miller (Clojure team)it's helpful to separate
• running an installed tool (-Twhatever)
• running a local tool defined by an alias (-T:build)#2021-10-0404:11Alex Miller (Clojure team)those aren't repl things but you might want a different kind of run config to do those#2021-10-0404:11cflemingWell, the way IntelliJ works is by indexing things. In order to do that, it needs to know what the dependencies are, so that it can attach them to the project and index them. The issue is that here those dependencies are hard to determine, and it’s also hard-to-impossible to determine which files should use which dependencies. The easy-but-crappy option is just to have all the tool and project dependencies in one ginormous classpath, and let any file in the project use any of them, and put it on the user to keep that straight in their head.#2021-10-0404:13seancorfieldSo you're trying to compute a classpath for build.clj in IntelliJ's world?#2021-10-0404:13Alex Miller (Clojure team)and separately from above, I really want to get a repl with the classpath being used to run the tools scenarios above, particularly the local tool one, because I want to interactively develop my build#2021-10-0404:13cflemingBut assuming the entirely likely case that the user has a build.clj and a build/something.clj support ns, as well as all their src/**/*.clj files using a different classpath, I don’t think there’s a better option.#2021-10-0404:14cflemingYes, but build.clj is just a convention, right? Any tool can be used to run any clj file in the project root, as well as any other namespaces in subdirectories that they might pull in.#2021-10-0404:14Alex Miller (Clojure team)what does intellij do for something like maven poms with optional deps?#2021-10-0404:15cflemingThe same as what Cursive does right now - you just select the aliases (or profiles) you want synced to the project, and it will sync all of them.#2021-10-0404:15cflemingHowever here you have two distinct classes of dependencies, which ideally wouldn’t mix.#2021-10-0404:16cfleming(it’s really n+1 classes of dependencies though, since each tool can have a totally separate set of deps).#2021-10-0404:16seancorfield@U0567Q30W :build is the relevant alias tho' and it's typically :deps on tools.build (or my build-clj wrapper) and maybe some deploy stuff, and :paths ["."] -- so you already have the case that src.foo.bar inside build.clj is the same as foo.bar in the normal source context 🙂#2021-10-0404:16cflemingYes, but if someone is requiring src.foo.bar for their foo.bar ns, I have no problem telling them they deserve everything they get 🙂#2021-10-0404:17cflemingBut the fact that the alias is :build and the file is build.clj is just pure convention.#2021-10-0404:17Alex Miller (Clojure team)yeah, you can't rely on that#2021-10-0404:17seancorfieldHaha... yeah, that's a pretty pathological case. I was more pointing out that when running -T:build, all of the src and test code is available, just with an extra segment on it.#2021-10-0404:17Alex Miller (Clojure team)but it's not#2021-10-0404:17cflemingNo, it’s not.#2021-10-0404:18Alex Miller (Clojure team)load is not going to succeed as the ns doesn't match the path#2021-10-0404:18cflemingIf you require foo.bar you won’t get that unless src is in your paths.#2021-10-0404:18cflemingRight.#2021-10-0404:18seancorfieldYou can load-file it and then use the namespace.#2021-10-0404:18cflemingUnless you’re using load-file, in which case, see above about deserving what you get.#2021-10-0404:18cflemingHa#2021-10-0404:18cflemingSnap#2021-10-0404:18Alex Miller (Clojure team)but as colin says, you deserve what happens to you at that point#2021-10-0404:19Alex Miller (Clojure team)that code presumably requires other code, which it will not properly be able to load#2021-10-0404:19seancorfieldAnd, to the earlier point, yes, :build could declare any default ns and the file could be anything, not just build.clj#2021-10-0404:20Alex Miller (Clojure team)there seem to be separate problems with paths and with deps#2021-10-0404:20cflemingYes, the path thing is also a problem.#2021-10-0404:20Alex Miller (Clojure team)with deps, it's more about finding the set of deps to "include" in the project right#2021-10-0404:20cflemingYes. But the tool files will need access to their deps too.#2021-10-0404:21Alex Miller (Clojure team)yes, i was including all of the possible sets of deps#2021-10-0404:22cflemingRight. The easy solution is to just mash all the project and tool deps into one, and give all Clojure files access to all of them, and the user will just have to be careful to not use project deps in their build files and vice versa.#2021-10-0404:22cflemingThat’s sounding like my v1 MVP to be honest.#2021-10-0404:23cflemingThe other, slightly more complex option, would be to just mash all the tool deps from the different tools into one, and have a tool classpath and a project classpath, and give the user some way of specifying which files should use which. It sounds horrible but it’s not so bad, I did this recently for Babashka, which is a similar issue.#2021-10-0404:24cflemingThen the worst that might happen is the user might use the deps from one tool while editing another, which is at least slightly less terrible.#2021-10-0404:29Alex Miller (Clojure team)and this is really a problem because you are not in the context of some specific deps execution (which would narrow this for you)?#2021-10-0404:31cflemingUm, I’m not sure what you mean by “in the context of some specific deps execution”. The problem is that for a given Clojure file, I have to determine which dependencies it should be using. IntelliJ has a standard way of doing that, which is to have a classpath per module (which corresponds to a lein/deps/Maven project, i.e. managed by a single deps.edn/project.clj/pom). The issue I now have is that there are multiple sets of dependencies that code in a given module will use.#2021-10-0404:34Alex Miller (Clojure team)right, that's what I'm saying - it's not relative to like a REPL execution that specifies an alias, it's independent of any such context#2021-10-0404:36Alex Miller (Clojure team)seems weird this is not something that has come up elsewhere for intellij#2021-10-0404:44seancorfieldIntelliJ's model does not work well for Polylith-based repos either, which is something else Cursive has problems with because of this need to map to the underlying IntelliJ model. Right Colin?#2021-10-0404:44cflemingLike I say, Babashka is a similar case which I did manage to deal with relatively cleanly. I might be able to do something similar here.#2021-10-0404:45cflemingI’m not sure about that, I still haven’t sat down to look at Polylith seriously. I have some reports of problems, but I don’t think it’s at the level of “I can’t use IntelliJ for this project”.#2021-10-0404:49seancorfieldThere are two separate issues. The only I'm referring to here is that projects all have their own classpath (and the :dev alias is the "development" project which is a separate classpath too)#2021-10-0404:50seancorfield(I don't know how much it interferes with using Cursive with Polylith -- just pointing out it's a similar problem, but it may just not be one you actually need to solve)#2021-10-0404:50cflemingIn Polylith, you mean?#2021-10-0404:51seancorfieldYes. Polylith.#2021-10-0404:51seancorfieldThere is a more problematic issue with Cursive and Polylith but it's not related to this thread.#2021-10-0423:11cflemingStill thinking about this - am I correct in thinking that I really can’t tell if an alias is intended to be used as a tool? I thought the presence of :ns-aliases or :ns-default would do, but that’s used for other things. I could look for :deps or :paths, but I guess a tool could also use :replace-deps or :replace-paths?#2021-10-0423:12cflemingAlso, is a tool required to use one of :ns-aliases or :ns-default?#2021-10-0423:15cflemingLooks like not, since the user can always provide a fully-qualified sym to exec a function.#2021-10-0423:18seancorfieldAn alias can be used for multiple, different contexts. -A, -M, -X, or -T. It's the usage that determines how the contents of the aliases contribute to the classpath. And by the time the process is running, you can't tell how it was invoked.#2021-10-0423:19cflemingRight. So, I really need the user to tell me which aliases are intended to be used as tools.#2021-10-0421:01Andrés RodríguezHey everyone! Got a bit of a messy situation over here. I've got a project with a docker-compose set-up for development with all the basic tools, an exposed REPL and mounts the source code as a volume so that the REPL can be refreshed with code as it's written. As I migrate this project to tools.deps, it has occurred that the .cpcache folder becomes invalid when the container is recreated (not to mention outside of the container) because it lives in the mounted volume but the cached deps live inside of the container's; ergo, recreating the container results in a class not found exception next time things are run.
This would be easy to solve if I could set $CLJ_CACHE to a folder inside of the container but because I am in a project with a deps.edn, tools.deps ignores it, according to the docs in https://clojure.org/reference/deps_and_cli#_classpath_caching and a brief what-if-the-docs-are-wrong experiment. Mounting the host's .m2 cache as a volume is unfortunately out of question because the container has to run in root, plus I'm not sure how cross-plat that is.
Any ideas?#2021-10-0421:08hiredmanthis may not be helpful, but I kind of doubt there is a good reason to run your repl inside of a docker container to begin with, so if you just don't do that then no problem#2021-10-0421:10hiredmanlike, I've get various services I need running for tests running in containers, but my code, repl, and actual development doesn't happen inside a container#2021-10-0421:10borkdudeexactly#2021-10-0421:19Andrés RodríguezI understand your point, but the whole team is used to being ready to work with a single docker-compose up, which prepares everything for frontend projects, Leiningen-based projects, Node.js projects, and I really would like to avoid altering their entire workflow just for one project using tools.deps.#2021-10-0421:20Andrés RodríguezI am making some strides in making that easier via Nix, but still not there 😅#2021-10-0421:43hiredmanin your dockerfile use sed to edit the clojure shellscript#2021-10-0421:53hiredman∴ cat workspace/wsmain/build/clojure/bin/clojure|sed -e 316,+5d -e '315a cache_dir="$user_cache_dir"'|grep -C 3 'Determine whether'
config_paths=("$install_dir/deps.edn" "$config_dir/deps.edn" "deps.edn")
fi
# Determine whether to use user or project cache
cache_dir="$user_cache_dir"
# Construct location of cached classpath file
val="$(join '' ${resolve_aliases[@]})|$(join '' ${classpath_aliases[@]})|$(join '' ${repl_aliases[@]})|$exec_aliases|$main_aliases|$deps_data|$tool_name|$tool_aliases"
#2021-10-0421:53hiredmanif you want to be more flexible use regex addresses instead of line numbers#2021-10-0508:26henrik42I'm using docker in a similar setup. I use a long-living maven-repo volume for ~/.m2 - not a bind-mount. That should be portable. Your colleagues just need to create the volume.#2021-10-0514:08Andrés Rodríguez@U0NCTKEV8 I didn't know the clojure binary was a shell script. That certainly can come in handy either now or later.
@U01GW6NKHP1 I might try this out, I don't know how this interacts with the container's own .m2 the base image already ships with but it sounds like it could be made to work one way or another.
Thank you for the great ideas!#2021-10-0421:06borkdude@hello525 Without going into details because it's getting late here, possibly using -Sforce can be used a temporary workaround, until you have a better solution.#2021-10-0421:17Andrés RodríguezThat's what I'm doing as a workaround, by way of doing rm -rf **/**/.cpcache before execution, but thanks!#2021-10-0515:17mokrHi, just to be 100% certain before I initiate the work:
To use :local/root to depend on the source files of a local project, the required project need to have a deps.edn file, right? In other words a the project.clj file of that lein lib is of no use?
More specifically I got it to work by adding a deps.edn side-by-side with the project.clj, but I’m not sure if that might have some undesirable side-effect, or if I just missed how to “integrate” with Leiningen.#2021-10-0515:20favilahttps://clojure.org/reference/deps_and_cli#_dependencies#2021-10-0515:20faviladeps or pom are supported#2021-10-0515:22mokrThanks, I came across that one, but juts wanted to ask in case there was something else that would help.#2021-10-0515:22favilalein can make a pom for you#2021-10-0515:23favilaiirc it’s just lein pom#2021-10-0515:25mokrHmm, I’m not familiar with pom apart from fiddling with it to get a deps project to create an uberjar a while back.
How often will it have to be updated? During my experiments yesterday, my main project seemed to hot-reload code when the lib code changed. Will that work if using pom?#2021-10-0515:26mokrI could also just change my libraries to be deps.edn based. I’m just not sure how easy it is to generate and deploy compared to lein.#2021-10-0515:27favilais this a cljs project? hot-reloading should be pretty orthogonal to all this#2021-10-0515:28mokrYes, my projects are typically Luminus based cljs + re-frame#2021-10-0518:23seancorfieldNow that tools.build exists, it's pretty easy to create library and application JAR files for deployment, and it provides a whole toolkit of build-related functions.#2021-10-0711:30fadrianI have a project structured as follows:
|
+ --- deps.edn
|
+ --- src <-- Some sources here
|
+ ---+ more-code <-- Library sources here
| |
. + --- module1 <-- Source tree
. + --- module2
. .
.
.
I've tried using :paths ["src" "more-code"] in my deps.edn file (as well as :paths ["src"] deps:["more-code"}), but this doesn't seem to recursively search into the subdirectories of the more-code directory, because when I try to refer to a namespace in src/x.clj which has its namespace defined in (say) module2/foo/src/foo/core.clj, this namespace is not found. The only thing I've found that comes close to working seems to be using :extra-deps {foo/module1 {:local-root "more-code/module1/foo/src"} foo/module2 {:local-root "more-code/module2/bar/src"} ... <for all namespaces referenced in src>} which seems clunky, repetitive, and not maintainable, especially when the namespace structures south of the more-code tree become complex. What am I doing wrong?#2021-10-0802:37souenzzo@U0DTU8J83
• it is :local/root, not :local-root
• :local/root should point to another folder with a deps.edn file.
• you can have many "some sources" folders, like src, you can simply point ["src" "more-code/module1"]
#2021-10-0721:09respatializedis there a way to use an already-defined var for an input argument to a function defined via :exec-fn?
If I have a namespace like this:
(ns foo.bar)
(defn my-func [{:keys [a b]}] (inc a))
(def my-val 3)
and an :aliases map like this:
{
...
:my-func {:ns-default foo.bar
:exec-fn my-func
:exec-args {:a my-val}}
...}
I get the following error:
Execution error (ClassCastException) at foo.bar/my-func
class clojure.lang.Symbol cannot be cast to class java.lang.Number
Is there a way to tell exec-fn to resolve the symbol at execution time, or is this not possible? It doesn't seem like the #' reader macro works in edn, so I can't use that method to ensure the symbol is treated as a var.#2021-10-0721:21Alex Miller (Clojure team)no, you can't do this, although you could wrap my-func in something else that uses my-val#2021-10-0721:32respatializedThat's the solution i landed on, but wanted to confirm. Thanks!#2021-10-0804:21grzmAnyone happen to have (or know of) already-tested clojure.tools.build scripts for publishing to maven central? Probably can adapt (or straight-up use?) @slipset’s deps-deploy#2021-10-0804:50dpsuttonLinked Sean’s helpful stuff but didn’t realize it used that underneath#2021-10-0805:14seancorfieldbuild-clj doesn't do much with deployment except call deps-deploy 🙂 I've no idea if that can deploy to Maven Central... I've only ever deployed to Clojars.#2021-10-0806:48slipsetIt wouldn’t surprise me if it could, as it is a thin wrapper around pomegranate which in turn is a thick wrapper around aether, which basically is the wrapper for communicating with maven.#2021-10-0814:14Darin Douglasswe use it to push to artifactory, i don’t see why it wouldn’t be able to push to maven central#2021-10-1117:11seancorfieldFYI (if you're not in the #releases channel)#2021-10-1118:07pyrHi! I was wondering if there was a repository that could be considered as a good example of doing multiple modules with tools.deps#2021-10-1118:08pyrWe have a lot of repositories that create multiple modules and artifacts (today with a combination of managed-dependencies, lein-sub, and lein-parent)#2021-10-1118:10pyrI'd like to see what a similar approach would look like with tools.deps if there is good prior art#2021-10-1118:58seancorfield@pyr You mean, like Polylith? A monorepo that has code to build various libraries and applications from various combinations of components?#2021-10-1118:59seancorfieldMaybe some of these posts will help too? https://corfield.org/tags/monorepo/#2021-10-1119:00pyr@seancorfield thanks, I'll look into these#2021-10-1119:01pyrwe're a bit weird in the sense that we don't have a large monorepo, rather a handful of large repositories which build multiple modules#2021-10-1119:22borkdude@pyr Note that you can also have multiple deps.edn projects within one git repo. We're doing that here:
https://github.com/clj-easy/graal-config
It's supported by :deps/root . You can consume the same repo with different roots within one deps.edn file.
We don't produce any artifacts because it's not necessary (given that you're using deps.edn).#2021-10-1119:26pyrnice#2021-10-1202:48Michael Stokleyfollowing up from the beginners channel. updating to the latest tools deps resolved my concurrency issue#2021-10-1202:50Alex Miller (Clojure team)thx, we've been updating through various maven lib versions and they've been "working on" this#2021-10-1203:13Michael Stokleythank you!#2021-10-1311:43dangercoderwhen fetching deps through git, is it possible to specify a mirror?#2021-10-1311:45borkdudeyou can specify any git repo + corresponding SHA you want, even a forked/mirrored repo or a SHA in a non-default branch#2021-10-1312:00dangercoderWhat I ment with a mirror was that it goes through a repository-manager like when you use maven and specify mirrors in settings.xml (e.g. jfrog artifactory)#2021-10-1312:10Alex Miller (Clojure team)Not currently but we have a ticket for that#2021-10-1415:13Dimitar UzunovHi! Quick update on what I thought was a case of: https://clojure.atlassian.net/browse/TDEPS-207 Turns out I had the following in the .clojure/deps.edn:
{:mvn/repos {“http://my.datomic.com” {:url “https://my.datomic.com/repo”}
“<no-longer-working” {:url “<<s3://not>|s3://<not> using it anymore>“}}#2021-10-1415:14Dimitar Uzunovthis didn’t use to be a problem but when I updated clojure and tools.deps it got stuck#2021-10-1415:15Dimitar Uzunovso if anyone has this issue they might want to check if they have any issues with their maven repos#2021-10-1416:27Alex Miller (Clojure team)"got stuck" == it hung?#2021-10-1417:53Dimitar Uzunovyes#2021-10-1418:15Alex Miller (Clojure team)if you can get it to that point again, and then take a thread dump (ctrl-\ on *nix, ctrl-break on windows) and dm it to me, that would be extremely useful.#2021-10-1418:59Dimitar UzunovClj-1.10.3.943-dump.txt in here was from me https://clojure.atlassian.net/browse/TDEPS-207#2021-10-1419:00Dimitar Uzunovback when that unused maven repo was in ~/.clojure/deps.edn#2021-10-1419:01Alex Miller (Clojure team)oh ok, thx (too many things to track to remember that)#2021-10-1419:01Alex Miller (Clojure team)then I've got it for when I have time to work on it more :)#2021-10-1419:02Dimitar Uzunovthanks, I hope this info helps 🙂#2021-10-1419:04Alex Miller (Clojure team)yep - the new info above is perhaps useful in combination#2021-12-2319:28Alex Miller (Clojure team)a potential fix for TDEPS-207 is in Clojure CLI prerelease 1.10.3.1053 if you want to give it a try#2021-10-1621:40cflemingIt looks like the specs in the t.d.a repo are fairly out of date now - any chance they could be brought up to date?#2021-10-1621:43Alex Miller (Clojure team)Yes, please file an ask Clojure question #2021-10-1621:47cflemingShould that get filed under Clojure or Contrib libs?#2021-10-1621:48cflemingAh, I see the dropdown under contrib, never mind.#2021-10-1621:51cfleminghttps://ask.clojure.org/index.php/11174/deps-edn-specs-from-tools-deps-alpha-are-out-of-date#2021-10-2220:36Alex Miller (Clojure team)updated in tools.deps.alpha 0.12.1058#2021-10-2221:11cflemingGreat, thank you!#2021-10-2121:15marciolMaybe this question have been made here some at point in the past, but anyways. I assume that -X stands Execute, and -M stands for Main, and -S ? Does it means what exactly?#2021-10-2121:19borkdudeset#2021-10-2121:20seancorfieldI would think "Script" since the -S options affect the CLI script's behavior itself? Roughly. Most of them do not run Clojure stuff or just "make a classpath". Except for maybe -Sdeps.#2021-10-2121:21borkdudeI think I've heard Alex say it means set. But I could be wrong.#2021-10-2121:21seancorfield@alexmiller?#2021-10-2121:23marciolYes, I just assuming that -M is main, but can also mean Module, who knows.#2021-10-2121:23marciolset makes sense as well because you use it to set a path#2021-10-2121:23dominicmhttps://clojurians-log.clojureverse.org/tools-deps/2019-11-18/1574093366.168000#2021-10-2121:23dominicmWhy ask when we can google! 🙂#2021-10-2121:24marciolYes, I googled for it and didn't find @dominicm. May I need to improve my Google search skills 😄#2021-10-2121:24dominicmI used ddg, I lied 🙂 https://duckduckgo.com/?q=site%3Ahttps%3A%2F%2Fclojurians-log.clojureverse.org%2Ftools-deps+meaning+of+S+prefix&t=ffab&ia=web#2021-10-2121:25marciolYes, A good resource is to search across the archive, good point.#2021-10-2121:26borkdudeI just used the zulip search for finding some other thing, also an option#2021-10-2405:52didibusWhat am I supposed to add to git so that I can sync my installed tools across my various machines?#2021-10-2406:00didibusI'm guessing the tools folder?#2021-10-2419:16seancorfieldSee my dot-clojure repo -- I call out tools in the README 🙂#2021-10-2419:23didibusI see, so you commit everything in the tools folder except tools.edn ?#2021-10-2419:30seancorfieldRight, because that is laid down by the CLI tools stuff itself so I don't want it overwritten locally.#2021-10-2419:31didibusNice and simple then, thanks#2021-10-2406:00didibusin ~/.clojure ?#2021-10-2412:38Alex Miller (Clojure team)That should work#2021-10-2612:50rickmoynihanIs the tools folder mechanism documented anywhere? I’ve not seen it mentioned before#2021-10-2613:02mpenetwhat is the tools folder mechanism?#2021-10-2613:05mpenetoh cool#2021-10-2613:13borkdudenote that you can still add a tool to .clojure/deps.edn as well, and invoke it with clojure -X#2021-10-2613:15mpenetI used to just do it at project level#2021-10-2613:16mpenetit's a bit better for "repeatability" for serious stuff (not personal setup)#2021-10-2614:45markaddlemanfyi - just started receiving this message when I invoke brew upgrade:
Warning: Calling bottle :unneeded is deprecated! There is no replacement.
Please report this issue to the clojure/tools tap (not Homebrew/brew or Homebrew/core):
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/clojure/homebrew-tools/Formula/clojure.rb:8
#2021-10-2614:48borkdudefyi @alexmiller you can just remove this line from the brew recipe, I've done this for several of my own already.#2021-10-2620:50Alex Miller (Clojure team)does installing @ versions still work in brew?#2021-10-2620:50Alex Miller (Clojure team)not sure if I'm crazy or they have just broken everything I do again#2021-10-2620:51Alex Miller (Clojure team)brew install clojure/tools/ - no longer finds my formula and installs it#2021-10-2620:52borkdudehmm, should work I think#2021-10-2620:52borkdudelet me try#2021-10-2620:53Alex Miller (Clojure team)you'll need to brew uninstall clojure first#2021-10-2620:53borkdudewhy is that?#2021-10-2620:53Alex Miller (Clojure team)¯\(ツ)/¯#2021-10-2620:53Alex Miller (Clojure team)I just have always needed to#2021-10-2620:55borkdudehttps://gist.github.com/borkdude/87ddedd30ad2e912e04a3d5635463cd5#2021-10-2620:56Alex Miller (Clojure team)well that's better than me#2021-10-2620:56borkdudebefore this I ran brew update#2021-10-2620:56Alex Miller (Clojure team)maybe it's just my system that's weird#2021-10-2620:56borkdudesince I don't auto-update brew since it tends to upgrade stuff that I don't want#2021-10-2620:56Alex Miller (Clojure team)yes, I've done that and brew doctor#2021-10-2620:57borkdudewhat if you completely erase your tap and start over?#2021-10-2620:58borkduderm -rf /usr/local/Homebrew/Library/Taps/clojure
#2021-10-2620:58Alex Miller (Clojure team)trying (I used brew untap though)#2021-10-2620:58Alex Miller (Clojure team)well, that worked#2021-10-2620:58Alex Miller (Clojure team)I guess something was weird in there, dunno#2021-10-2620:59Alex Miller (Clojure team)I did have about 100 versions of Clojure in my caches :)#2021-10-2620:59Alex Miller (Clojure team)thx#2021-10-2621:00Alex Miller (Clojure team)so gross that it warns on those old formulae#2021-10-2614:48borkdudealso see https://github.com/borkdude/homebrew-brew/issues/5#2021-10-2615:13Alex Miller (Clojure team)thx#2021-10-2618:14didibusNow we're just missing an install tool.#2021-10-2620:43aaelonyNot certain the best slack channel to ask this question... In the R language community, there is a cool site to find R packages that might be appropos (https://r-pkg.org/), has someone built this or similar for Clojure?#2021-10-2620:45seancorfieldhttps://www.clojure-toolbox.com/ perhaps (I don't know how up to date it is)#2021-10-2620:45seancorfieldThere's a #find-my-lib channel here that folks use to ask about libraries.#2021-10-2620:45aaelonyoh, that's related but a bit different than search#2021-10-2620:45seancorfieldOtherwise, I think folks just tends to search http://clojars.org which is where most libraries are published.#2021-10-2620:46seancorfieldAlthough that won't find any libraries or tools that are only published in source form -- so you'd have to search https://github.com for those I guess.#2021-10-2620:47seancorfieldhttp://clojars.org is the closest to that R package site I think.#2021-10-2620:47aaelonythe http://clojars.org UI has evolved quite a bit since I last looked. Looks like the search there is quite good too. Aside from fancy sections on featured/most downloaded/trending this week/most depended upon/recently updated categorizations it is pretty much there#2021-10-2620:47seancorfield(although bear in mind that Clojure itself and all the Contrib libraries are published to Maven Central, not Clojars)#2021-10-2620:48seancorfieldThe search on http://clojars.org isn't great but it can be useful sometimes.#2021-10-2620:48aaelonysure. Same in the R world. The http://r-pkg.org site is independent of cran mirrors or github repos#2021-10-2620:48aaelonythanks - looks pretty good to me#2021-10-2620:49aaelonyalthough the "trending" style categorization labels might be cool#2021-10-2620:50seancorfieldI think, generally, you get better answers asking for recommended libraries for problem spaces here in Slack 🙂 In either #find-my-lib or #clojure / #beginners depending on the level of response you want...#2021-10-2620:51aaelonyoh - asking in a more general sense for now. Myself, I like monitoring the categorization labels to see emergent libraries that may be useful in my current suite of projects#2021-10-2620:52aaelonybut maybe the answer you are suggesting is just ask here when needed 😉#2021-10-2620:53seancorfieldAnd maybe follow the project #announcements and #releases channels?#2021-10-2621:22Jeff Evansoh, I suppose I only specified those for the :build alias, not the top level. please disregard#2021-10-2621:30Alex Miller (Clojure team)The https://clojure.org/releases/tools#v1.10.3.998 is now available
• Remove `bottle :unneeded` from brew formulas (no longer needed)
• https://clojure.atlassian.net/browse/TDEPS-209 Include only jar files in classpath from Maven artifacts
• Update to tools.tools v0.2.1 (minor improvements in `clj -Ttools list`)
• Use https://github.com/clojure/tools.deps.alpha/blob/master/CHANGELOG.md 0.12.1058#2021-10-2716:50seancorfieldI figured this was worth sharing from the announcements thread since I feel like not many people know how to get help on "tools"...#2021-10-2716:50seancorfield...and for a specific tool:#2021-10-2716:51seancorfieldAnd for getting help on deps itself:
(! 654)-> clojure -X:deps help/doc
This api provides functions that can be executed from the Clojure tools using -X:deps.
-------------------------
clojure.tools.cli.api/basis
([params])
Create a basis from a set of deps sources and a set of aliases. By default, use
root, user, and project deps and no argmaps (essentially the same classpath you get by
default from the Clojure CLI).
...
#2021-10-2716:57Alex Miller (Clojure team)while that's all true, it's less obvious and discoverable than it should be so my sympathy is with the OP#2021-10-2717:03seancorfieldOh, totally. That's why I figured I'd share it because it isn't "obvious" even though it is mentioned (briefly) in the docs.#2021-10-2717:05seancorfieldI've added a comment to all my build.clj files saying how to get help on the build file and I also add this to the README on my tools (e.g., deps-new). It's super useful. Maybe the install function could print how to get help on a newly installed tool? And maybe list could also mention it?#2021-10-2717:07seancorfieldPerhaps both commands could also give an example of the show function too? I nearly always forget it needs :tool the first time I type it in:
(! 656)-> clojure -Ttools show new
Key is missing value: new
(! 657)-> clojure -Ttools show :tool new
{:lib io.github.seancorfield/deps-new,
:coord
{:git/tag "v0.4.0",
:git/sha "0fb18a62fe068a2b62183c93e5441ababdd7bc6f",
:git/url ""}}
Default namespace: org.corfield.new#2021-10-2717:08Alex Miller (Clojure team)as always, would love any suggestions to go to https://ask.clojure.org#2021-10-2717:08Alex Miller (Clojure team)or to clojure-site issues if doc requests#2021-10-2717:08seancorfieldYup, on it. Will submit an "ask" with the above notes!#2021-10-2717:17seancorfieldhttps://ask.clojure.org/index.php/11214/make-tools-help-more-discoverable#2021-10-2811:54hugodhow does tools.tools determine which namespace to use? e.g clojure -Ttools install runs clojure.tools.tools.ap/install#2021-10-2812:01borkdude@hugod a tool can declare a default namespace#2021-10-2812:02hugodhow?#2021-10-2812:02borkdudehttps://github.com/clojure/tools.tools/blob/739f82822823c72298178a2d961db183acd74e25/deps.edn#L12#2021-10-2812:03hugodah, the missing piece - thank you!#2021-10-2819:56seancorfieldFWIW, it's mentioned here https://clojure.org/reference/deps_and_cli#_other_keys and explained here https://clojure.org/reference/deps_and_cli#_tool_usage and then this section explains how to author a tool, including the default namespace: https://clojure.org/reference/deps_and_cli#_best_practices_for_tool_authors#2021-10-2901:23hugodThanks - I did actually check that page - obviously not well enough#2021-10-2921:30Alex Miller (Clojure team)Clojure CLI prerelease 1.10.3.1007 is now available
• TDEPS-83 - invalidate cp cache when local lib manifests are stale
• -Stree no longer forces cache recompute
• Clean up exception handling for -X
• Update to tools.deps.alpha 0.12.1063#2021-10-2923:02seancorfieldI'm on vacation. Will take a look on Tuesday.#2021-11-0222:58seancorfieldGot 1.10.3.1013 installed today and haven't had any problems so far in dev/test/CI but I haven't really pushed that TDEPS-83 thing yet...#2021-11-0300:26Alex Miller (Clojure team)I've got some more changes for it before I release it#2021-10-2921:32Alex Miller (Clojure team)^^ cc @pez @cfleming the big thing here is the change for TDEPS-83 which will now notice when local manifest files change (deps.edn/pom.xml) and trigger a classpath recompute - this has come up several times with Calva and Cursive, particularly in combination with things like polylith#2021-10-2921:32Alex Miller (Clojure team)so any tire-kicking on this prerelease would be appreciated!#2021-10-2921:34Alex Miller (Clojure team)if things look ok, I will release it next week#2021-10-3004:53cflemingThis basically just adds new conditions to the classpath caching logic, correct? This is all in the script?#2021-10-3009:51borkdudeIt seems to also be in tools.deps since that writes the manifest file given a filename via --manifest-file#2021-10-3009:45borkdude@alexmiller While porting the changes to deps.clj, I noticed that invoking -Stree after the classpath has been cached, has no effect:
$ clojure -Stree
org.clojure/clojure 1.10.1
. org.clojure/spec.alpha 0.2.176
. org.clojure/core.specs.alpha 0.2.44
$ clojure -Stree
... nothing ...
#2021-10-3013:59Alex Miller (Clojure team)Oh, that's bad. #2021-10-3013:59Alex Miller (Clojure team)I forgot that that was why the staleness check is in there!#2021-10-3013:59Alex Miller (Clojure team)I'll have to roll that part back#2021-10-3014:07borkdudeNo worries. I already rolled it back in deps.clj and figured you either have to pre-cache the tree or just leave it like this: I think the latter would be fine.#2021-10-3014:08Alex Miller (Clojure team)Well for now, I will just roll back that part of the change#2021-10-3014:09Alex Miller (Clojure team)Thx for the heads up!#2021-10-3014:09borkdudePerhaps you can cache the tree once it has already been computed in a file in .cpcache#2021-10-3014:11borkdudeTo me personally this is not super important. The manifest checks I'm very happy to have found out about.#2021-10-3015:37lilactownis there a way to consume a subproject of a monorepo via git deps?#2021-10-3015:38lilactowni.e.
project/
sub-project-A/
deps.edn
sub-project-B/
deps.edn
can I have an external project depend on sub-project-A via git deps?#2021-10-3015:44borkdude@lilactown yes: :deps/root "sub-project-A"#2021-10-3015:46borkdudeyou can even refer to the same git repo multiple times with different roots and shas#2021-10-3015:46lilactownah neat. so this would perhaps be the full thing:
{:deps {town.lilac/project-A {:git/url "..." :git/sha "..." :deps/root "sub-project-A"}}}#2021-10-3015:46borkdudecorrect#2021-10-3015:47lilactownOK. in that case, I have more questions 😄 now I want to support this in my monorepo project#2021-10-3015:48lilactownfor development purposes I have sub-project-A depend on sub-project-B via :local/root "../sub-project-B" I'm guessing that won't work for people trying to consume it via git deps like this, right?#2021-10-3015:49borkdudeI'm not sure if local/root works relatively within a library#2021-10-3015:49borkdudebut that's easy to test and see what -Spath returns when you use that library#2021-10-3015:50lilactownOK i'll have to test it more#2021-10-3016:03Alex Miller (Clojure team)It works #2021-10-3016:05lilactownoh neat!#2021-10-3016:05lilactownI'm developing project right now locally so it'll be awhile before I worry about consumers using it via git deps. glad to hear it's likely I won't need to futz too much with the config for them 😄#2021-10-3016:07Alex Miller (Clojure team)There are limits if you get to comparing the same “lib” from within a git dep as that local lib does not retain context with the root git dep#2021-10-3016:24Alex Miller (Clojure team)Clojure CLI prerelease 1.10.3.1013 is now available
• Undo: -Stree no longer forces cache recompute (from last prerelease)#2021-11-0216:09kennyAccidentally ran into https://clojure.atlassian.net/browse/TDEPS-207 again. Is there a workaround to allow us to update to the latest cli version? ATM, we're stuck on 1.10.3.933.#2021-11-0318:35Dimitar Uzunov@U083D6HK9 One thing that worked for me is that I removed unaccessible maven repos from the config file, I had the same symptoms otherwise#2021-12-2319:14Alex Miller (Clojure team)@U083D6HK9 latest CLI prerelease includes a speculative fix for this - would love to have any feedback if you want to try 1.10.3.1053#2021-12-2319:14Alex Miller (Clojure team)sorry for the delay in looking at this#2021-12-2319:22kennyThank you for taking the time Alex. I will take a look later today and let you know. Happy holidays. #2021-12-2319:29Alex Miller (Clojure team)🎅#2021-12-2403:25kennyRan out of time. Will try to get to it tomorrow morning. If not, sometime next week. #2022-12-2816:55kennyAgh, holiday delay... Just tested it out. Still getting a hang.
clojure -version 08:53:05
Clojure CLI version 1.10.3.1053
Then I run:
clojure -A... -Sforce -Stree
DEPRECATED: Libs must be qualified, change deps-deploy => deps-deploy/deps-deploy (/Users/kenny/.gitlibs/libs/com.ardoq/azure-api/4fbac2ddb9eedf0779890ae2c5bb2d1f3dcfdd7f/deps.edn)
DEPRECATED: Libs must be qualified, change deps-deploy => deps-deploy/deps-deploy (/Users/kenny/.gitlibs/libs/com.ardoq/azure-api/4fbac2ddb9eedf0779890ae2c5bb2d1f3dcfdd7f/deps.edn)
DEPRECATED: Libs must be qualified, change clj-http => clj-http/clj-http (/Users/kenny/.gitlibs/libs/com.ardoq/azure-api/4fbac2ddb9eedf0779890ae2c5bb2d1f3dcfdd7f/deps.edn)
DEPRECATED: Libs must be qualified, change clj-time => clj-time/clj-time (/Users/kenny/.gitlibs/libs/com.ardoq/azure-api/4fbac2ddb9eedf0779890ae2c5bb2d1f3dcfdd7f/deps.edn)
Downloading: org/clojure/clojure/maven-metadata.xml from datomic-cloud
And then it hangs. Would another thread dump be useful?#2022-12-2817:28Alex Miller (Clojure team)Sure#2022-12-2818:01kenny#2022-12-2818:27kennyThis is such a weird issue. Something about it is non-deterministic. ~1 out of every 10 runs will succeed.#2022-12-2818:29kennyWe have 2 s3 maven repos in our :mvn/repos. I could see that affecting things for sure.#2022-01-0415:01Alex Miller (Clojure team)I think I have a bit more insight on this and s3 definitely is required to trigger the condition. still thinking about how to fix it but I think I understand it now at least#2022-01-0416:19kennyNice. Let me know if there's any info I can provide. #2022-01-0502:30Alex Miller (Clojure team)@U083D6HK9 with those clues, I was able to make a repro of my own, and there's a new prerelease that you can test with a fix 1.10.3.1058#2022-01-0519:24kennyAwesome! Just tested it and it works. Thank you so much for taking the time to fix this.#2022-01-0519:34Alex Miller (Clojure team)thx for the good reports and retests that helped me fix!#2021-11-0216:18Alex Miller (Clojure team)not sure what you're asking for? not fixed yet#2021-11-0216:23Alex Miller (Clojure team)-207 is relatively rare, are you actually seeing that?#2021-11-0216:30kennyIIRC, I reported it 🙂 Yes, it occurs on every CLI run on any version greater than 1.10.3.933 both locally and on CI. #2021-11-0216:35kennySeveral of the recent tdeps improvements are big QoL improvements for us, so I was hoping to find a way for us to upgrade. #2021-11-0216:47Alex Miller (Clojure team)I can look at it on Friday#2021-11-0216:50kennyOh, awesome. Thank you. #2021-11-0408:13mike_ananev@seancorfield Hello. I'm trying to install clj-new on Linux (CentOS) and get the following error:
➜ ~ clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.362"}' :as clj-new
Checking out: at 3523d8be5ccef13961e49d2abc55a2ed4f18fc69
Error building classpath. Unable to checkout 39663a017d45d05a704113c9c147c7886e66d345
git: 'worktree' is not a git command. See 'git --help'.#2021-11-0408:25lispyclouds@U097654L8 What is your version of Git? worktree is a command introduced in Git 2.5+#2021-11-0408:46mike_ananevOh, thanks. I think this is the error cause.
➜ ~ git --version
git version 1.8.3.1#2021-11-0408:47lispycloudsyep, clojure cli tooling shells out to the system git for cloning repos, hence dependent on that version#2021-11-0409:09dharriganThat's quite an ancient version of git#2021-11-0409:10dharriganReleased sometime in 2012...#2021-11-0410:53mike_ananevyeah, but CentOS 7 from cloud provider is my constraint.#2021-11-0413:02Alex Miller (Clojure team)We require at least git 2.5 #2021-11-0413:30Andrés Rodríguez@U097654L8 As a suggestion: you could use Nix to get yourself more recent tooling or alternatively, use a more modern distro inside of a Docker container.#2021-11-0414:06mike_ananevThanks.#2021-11-0408:14mike_ananevDid not understand how to fix it?#2021-11-0408:15mike_ananev➜ ~ clj --version
Clojure CLI version 1.10.3.998#2021-11-0412:27mpenetwe've been toying with this lib (https://github.com/exoscale/deps-modules) at work, attempting to replace setups involving lein sub in some of our repos, maybe that's of interest here#2021-11-0412:28mpenetit just handles version (and other attributes) propagation in "sub-modules"#2021-11-0412:28mpenetfor the rest tools.deps (and some tools/extensions) seems to do pretty much everything we need I think now#2021-11-0412:32borkdude@mpenet We've done a similar thing with one base/deps.edn which has all the pinned versions and then merging that into each subrepo deps.template.edn just using clojure.edn but using rewrite-clj surely works too#2021-11-0412:33borkdudefwiw, I have an "abstraction" on top of rewrite-clj which lets you manipulate EDN using normal functions like assoc etc: https://github.com/borkdude/rewrite-edn#2021-11-0412:33mpenetsimilar yes, in theory if you fiddle a bit with the options you can do the same#2021-11-0412:33mpenetI saw about rewrite-edn (after the fact), but we're using like 10 lines of rewrite-clj, good enough this way#2021-11-0412:34borkdudealso this deps.edn logic is all compatible with bb as well, so we execute it any time we touch something, since the startup time is neglectible#2021-11-0412:34mpenetyeah we want to avoid to force devs to install anything new#2021-11-0412:35mpenetthe move from lein -> tools.deps is not always welcomed (it's a lot of change), so we want to limit friction#2021-11-0412:35borkdudeyeah of course, but you can have both with the same code#2021-11-0412:35mpenetsame reason why we went with write-in-place vs injecting deps.edn files or alias merging#2021-11-0412:36borkdudewhat I like about this approach is that it's "what you see is what you get"#2021-11-0412:36mpenetbut I suspect that tool could be ported to bb easily in any case#2021-11-0412:36borkdudeinstead of some tool which calculates things at runtime#2021-11-0412:36mpenetit's very little code#2021-11-0412:36mpenetyeah#2021-11-0412:37borkdudefwiw here's "our" version: https://gist.github.com/borkdude/58f099b2694d206e6eec18daedc5077b#2021-11-0412:37mpenetthen you can just do a pedantic style check in CI that fails the run if stuff is not in sync#2021-11-0412:37mpenetand everybody is happy#2021-11-0412:37mpenetyeah I had your gist in memory#2021-11-0412:37mpenetI think I might have starred it#2021-11-0412:38mpenetbut I think keeping comments in particular is really important#2021-11-0412:38borkdudethe comments in "our" case is in deps.template.edn#2021-11-0412:38mpenetwe often have comments indicating typically "this is here because some conflict there bla bla"#2021-11-0412:38mpenetyeah, I went with templates in modules originally#2021-11-0412:39mpenetthen I asked myself, since we rewrite why just not write in the end file#2021-11-0412:39mpenetit's probably visible from the options supported#2021-11-0412:39borkdudeI agree that the rewrite-clj / rewrite-edn approach is nicer. I went with this in neil as well: neil add dep foo/bar adds it to your deps.edn#2021-11-0412:39mpenet:output-deps-edn-file "deps.edn"
:input-deps-edn-file "deps.edn"#2021-11-0412:40mpenetrewrite-clj is really awesome#2021-11-0412:40mpenetbeen using it on other things, quite like it#2021-11-0412:51pyr@borkdude do I get it right that neil is more or less your take on what I tried to do with joker + klein?#2021-11-0412:54borkdudeit's a little bit different. neil just adds things to an existing Clojure project, but doesn't "wrap" or hide any features or tries to make things "easy", it's not a project / task runner. bb already has a task runner which fulfills that part:
https://book.babashka.org/#tasks
All neil does right now is add common "facets" to existing projects, like a test runner, tools.build, etc.#2021-11-0412:54borkdudeI just had to do this manually too often, this is why I wrote this tool#2021-11-0412:57pyrack makes sense#2021-11-0412:58pyrbut as @mpenet points out we're taking the slow road to introduce changes to our tooling, right now we're still in lein land#2021-11-0413:25FiVoI am having trouble loading a native library with cli and java.library.path.#2021-11-0413:25FiVoI am trying
$ clj -J-Djava.library.path=/usr/share/
Clojure 1.10.3
user=> (require '[zeromq.zmq])
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader/loadLibrary (ClassLoader.java:2670).
no jzmq in java.library.path: [/usr/share/]
#2021-11-0413:26FiVoand dpkg returns
$ dpkg -L libzmq-java
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libzmq-java
/usr/share/doc/libzmq-java/AUTHORS
/usr/share/doc/libzmq-java/README.md
/usr/share/doc/libzmq-java/copyright
/usr/share/java
/usr/share/java/zmq-3.1.0.jar
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libzmq-java
/usr/share/maven-repo
/usr/share/maven-repo/org
/usr/share/maven-repo/org/zeromq
/usr/share/maven-repo/org/zeromq/jzmq
/usr/share/maven-repo/org/zeromq/jzmq/3.1.0
/usr/share/maven-repo/org/zeromq/jzmq/3.1.0/jzmq-3.1.0.pom
/usr/share/maven-repo/org/zeromq/jzmq/debian
/usr/share/maven-repo/org/zeromq/jzmq/debian/jzmq-debian.pom
/usr/share/doc/libzmq-java/changelog.Debian.gz
/usr/share/java/jzmq-3.1.0.jar
/usr/share/java/jzmq.jar
/usr/share/java/zmq.jar
/usr/share/maven-repo/org/zeromq/jzmq/3.1.0/jzmq-3.1.0.jar
/usr/share/maven-repo/org/zeromq/jzmq/debian/jzmq-debian.jar
#2021-11-0413:36FiVonevermind was giving it the wrong path#2021-11-0413:57lispers-anonymousI have a coworker who uses a PAT to authenticate with github over https. Looking at https://clojure.atlassian.net/browse/TDEPS-33 it seems like deps.edn would support this type of authentication now. Is that correct? Is there anything special that would need to be done to allow using git coordinates for private repos with https/PAT auth?#2021-11-0414:01Alex Miller (Clojure team)Clojure CLI shells out to git. If it works that way, it probably works with the Clojure CLI.#2021-11-0414:15lispers-anonymousI'll pass along that they probably need to update to the latest clojure cli as well. If it doesn't, switching to ssh is not an issue. Thanks for the info#2021-11-0422:44Alex Miller (Clojure team)Clojure CLI prerelease 1.10.3.1020 is now available:
• Updates to latest tools.deps.alpha (which updates transitive dep versions)
• Modification to prior manifest changes to consider classpath stale if manifest file is missing (local or git dir was deleted)
• Includes alpha version of a new clj -X:deps list - similar to tree but list only the libs on the final classpath (in sorted order), takes all the modifiers from create-basis#2021-11-0422:47Alex Miller (Clojure team)https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/list#2021-11-0511:53EdI want to include a local/root dependency, passing in an alias (so that it includes some extra-paths defined in that project). I recall this being a feature that was talked about but not yet implemented, but I can't find the ticket/ask clojure question/discussion ... Is this done? Can someone point me in the right direction? Or am I barking up the wrong tree?#2021-11-0512:33Alex Miller (Clojure team)Has not been done#2021-11-0512:42Alex Miller (Clojure team)https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects?show=7843#q7843#2021-11-0514:21Edmany thanks ... upvoted 😉#2021-11-0512:59rickmoynihanI’ve often wondered about having something like that.
I guess it might want to be constrained somehow, so a published deps.edn file can state which of its aliases are to be exported?#2021-11-0513:10mpenetI would love to have aero like tag literals. stuff like #include #merge #env #join #profile #ref etc, seems that it could solve 99% of these kind of uses cases where some sort of composition is required#2021-11-0513:18rickmoynihanPersonally I think it makes sense for deps.edn to always be fully resolved, and not require interpretation. Though arguably some of the features like :paths accepting aliases, are implementing a constrained version of #ref
You can of course already use aero to generate deps.edn.
Indeed this is something I was meaning to ask with regards your deps-modules tool… why not just use aero?#2021-11-0513:22mpenetbecause clj / tooling doesn't know how to handle it#2021-11-0513:23mpenetthe ultimate goal was to have the less friction for devs#2021-11-0513:23mpenetthe stuff you generate/modify should be a normal deps.edn file, what you see is what you get#2021-11-0513:24mpenetit's difficult enough to push deps usage when everything works fine (to some extent) with lein#2021-11-0513:25mpenetwysiwyg bit is actually a counter argument to aero style tags 🙂 but if it was part of tools-deps it'd be ok#2021-11-0513:26mpenetone thing for sure, if we have to tell devs who are happy with lein that they have to juggle with tons of aliases to get stuff going they ll be resisting the change#2021-11-0513:27mpenet(personally I just want to speed up dev when you juggle with tons of repos (esp. with deep deps trees), the fact a commit is immediately available from repo a to repo b is the big sell, no need to wait ci/build/deploy from other prs)#2021-11-0513:27mpenetthe rest is sugar on top#2021-11-0513:30rickmoynihanYeah — generating deps.edn from templates certainly isn’t without its problems…
Though arguably tools.build substantially reduces the need to do that#2021-11-0513:30mpenetclearly, I hate it, but it's "the least worst" solution imho so far#2021-11-0513:31mpenetif we had #include/#refs the version pinning problem solved by deps-modules would be solved#2021-11-0513:33mpenetinclude the version file in some internal key in your deps.edn files then #ref to the values you need for coords#2021-11-0513:35mpenetyou could also share aliases from one file to N other files in the "monorepo"#2021-11-0513:35rickmoynihanI think there’s a risk it muddies the water around what subset of deps.edn is available in a local project and at deployment. #include seems dependent on the context.#2021-11-0513:37mpenetI am not sure I follow#2021-11-0513:37Joshua SuskaloWhat impact does putting a provided-scope dependency in the pom for a project and not the deps.edn in order to list an optional dependency? The provided scope dependency in this case would be to allow cljdoc to document my project.#2021-11-0513:51Joshua SuskaloIn particular I'm curious if having a provided dependency in the clojars release will pull in a dependency for anyone depending on my project with tools-deps (or leiningen).#2021-11-0513:37mpenet#include is quite static. If you want to do conditional includes you can use stuff like profiles (in theory), or use aliases 🙂#2021-11-0513:39borkdudeThis has been discussed before btw, you can find some conversation if you search this channel with "interpretation" or so. It was decided against then.#2021-11-0513:39mpenetthe risk is to end up in a situation where you're half way to have "deps.clj" where it's all clojure code anyway, then you can launch a rocket in space and merge your deps#2021-11-0513:39mpenet@borkdude I guess it's still unsolved#2021-11-0513:51Joshua SuskaloIn particular I'm curious if having a provided dependency in the clojars release will pull in a dependency for anyone depending on my project with tools-deps (or leiningen).#2021-11-0513:53borkdude@suskeyhose you can also provide a dependency scope in deps.edn afaik#2021-11-0513:53Alex Miller (Clojure team)no, you cannot#2021-11-0513:54Joshua SuskaloI was looking in the docs and I didn't see that, so I figured I'd have to put it in the template pom.#2021-11-0513:54Alex Miller (Clojure team)provided scope in a clojars dep should not pull in a dep with any tool#2021-11-0513:54borkdudeoh hmm, well if Alex says you cannot, it is true :) perhaps use an alias?#2021-11-0513:55borkdude(I was mixing it up with :exclusions which you can provide)#2021-11-0513:58borkdudewhat's the reason not to do this with an alias...?#2021-11-0514:00Joshua SuskaloBasically I have a project which requires a namespace from a library I don't depend on so that I can optionally extend that library to work with mine. cljdoc chokes on this because it's trying to analyze the namespace with a require for something not on the classpath, but if I specify a provided-scope dependency in the pom then it will include it on the classpath to allow analysis.#2021-11-0514:00Joshua SuskaloI can't do this with an alias because cljdoc is dependent entirely on the maven release and ignores the deps.edn#2021-11-0514:07borkdudeAnother alternative: provide another library which has your namespace#2021-11-0514:07borkdude+ the required library#2021-11-0514:11Joshua Suskalothat's far from ideal when part of the usecase here is that the user shouldn't need to know if the other library is in their transitive dependencies#2021-11-0514:12borkdudeI see yeah.#2021-11-0514:12Joshua SuskaloWhat I'm doing is in the library farolero I'm extending flow's mechanism to not catch the unwinding mechanism. That's something that should be transparent because someone using farolero shouldn't need to know if a library they're using is using flow internally#2021-11-1123:04dominicmyour deps.edn and pom.xml don't need to directly connect.#2021-11-1123:04dominicmIn clip I manually maintain the pom.xml with provided dependencies so that cljdoc will work.#2021-11-0514:04Andreas S.Hello when I'm trying to run "clj" from the CLI I get this:
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main#2021-11-0514:18Alex Miller (Clojure team)platform? version?#2021-11-0514:18Alex Miller (Clojure team)do you have a deps.edn in the current directory or not?#2021-11-0514:19Andreas S.hi alex#2021-11-0514:19Andreas S.im on windows#2021-11-0514:19Andreas S.yes I havea deps.edn in the directory#2021-11-0514:19Alex Miller (Clojure team)if you run from a directory without a deps.edn, does it work?#2021-11-0514:19Andreas S.you mean with lein?#2021-11-0514:19Alex Miller (Clojure team)no, if you run clj from a different dir#2021-11-0514:20Andreas S.no#2021-11-0514:20Andreas S.in all other directories it says the same thing#2021-11-0514:20Alex Miller (Clojure team)is this a new install?#2021-11-0514:20Andreas S.no I think this is quite old#2021-11-0514:20Andreas S.where do I find the latest installer?#2021-11-0514:20Alex Miller (Clojure team)well, probably best to start with a new install#2021-11-0514:20Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/clj-on-Windows#2021-11-0514:24Andreas S.thank you alex that worked!#2021-11-0516:07Alex Miller (Clojure team)if anyone has feedback today on https://clojurians.slack.com/archives/C6QH853H8/p1636065875105900, please let me know!#2021-11-0516:49rickmoynihanOh, forgot to say…
I’ve been using it all day, and it works fine so far :thumbsup:
The -X:deps list feature happened to be exactly what I was wanting to check something#2021-11-0517:44Alex Miller (Clojure team)Me too! :)#2021-11-0518:54borkdudeI'm also happy about the manifest addition: less -Sforce !#2021-11-0519:21Alex Miller (Clojure team)be gentle, kind friends#2021-11-0519:24borkdudeI was trying to give a compliment, did I fail? ;)#2021-11-0519:28Alex Miller (Clojure team)no, I'm agreeing we should "use less force"#2021-11-0519:28Alex Miller (Clojure team):)#2021-11-0519:28borkdudehaha#2021-11-0519:29Alex Miller (Clojure team)make deps, not war#2021-11-0519:29borkdudemake deps, not jar? ;)#2021-11-0519:29borkdudeslogan in favor of git deps ;)#2021-11-0519:30Alex Miller (Clojure team)heh#2021-11-0518:36respatializedis there a way to specify a git SHA for a :local/root dep?#2021-11-0518:38Alex Miller (Clojure team)not as a local dep#2021-11-0518:38Alex Miller (Clojure team)it might be possible to use a git local url and a git sha as a git dep#2021-11-0518:38Alex Miller (Clojure team)but I have not tried it#2021-11-0518:43Alex Miller (Clojure team)if it doesn't work, feel free to file a bug at https://ask.clojure.org#2021-11-0518:49Alex Miller (Clojure team)from a quick perusal of code and a couple tests, I think this won't work right now, but could easily be made to work#2021-11-0518:54respatializedyeah one thing I ran into fairly quickly was the requirement that paths be relative rather than absolute (inherited from clojure.tools.gitlibs based on the stacktrace)#2021-11-0518:55respatializedI can file a full report if you like#2021-11-0518:55Alex Miller (Clojure team)that's really a subset of the problem - the code that translates git urls to cache dirs and libs doesn't handle either raw paths or relative urls#2021-11-0518:55Alex Miller (Clojure team)but yeah, that would be great#2021-11-0518:55respatializedwill do ✔️#2021-11-0519:09respatializedit seems like the URL issue is complected with the SHA issue; I will file separate issues for each#2021-11-0519:21Alex Miller (Clojure team)what's the sha issue?#2021-11-0519:25respatializedhttps://ask.clojure.org/index.php/11250/using-a-git-sha-for-a-local-git-dependency
https://ask.clojure.org/index.php/11251/allow-absolute-paths-for-file-based-git-urls#2021-11-0519:29Alex Miller (Clojure team)yeah, I don't expect to support shas for :local deps#2021-11-0519:30Alex Miller (Clojure team)if you want to treat something as a git dep, it will need to be a git dep#2021-11-0519:31respatializedah, I see - so really the SHA issue takes care of itself if the URL issue is fixed. Feel free to close the first one then.#2021-11-0519:34Alex Miller (Clojure team)yeah#2021-11-0519:37borkdudegit also supports local file support right?
git clone /srv/git/project.git
is that something that can be used in that scenario?#2021-11-0519:37borkdudehttps://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols (see Local Protocol)#2021-11-0519:37Alex Miller (Clojure team)doesn't work#2021-11-0519:38Alex Miller (Clojure team)well, doesn't work with absolute paths or relative .. paths (although it might work in some cases, it's doing unholy things to the gitlibs dir)#2021-11-0519:38borkduderight#2021-11-0519:39Alex Miller (Clojure team)all just a matter of fixing the url -> cache-path function, which is a bit of a hack anyways#2021-11-0519:39borkdudeI see now that is the second issue, sorry for not paying attention#2021-11-0519:39Alex Miller (Clojure team)trying to decide how far I want to go to unhack it :)#2021-11-0820:34Alex Miller (Clojure team)https://clojurians.slack.com/archives/C015AL9QYH1/p1636403633022600#2021-11-0820:35Alex Miller (Clojure team)^^ if you use git deps and like to test things, would be happy for a :thumbsup: if this new CLI prerelease 1.10.3.1029 works as expected#2021-11-0820:36Alex Miller (Clojure team)b/c gitlibs repo dir structure has changed, you should see this as if gitlibs cache is stale (so will see downloads of git repos etc)#2021-11-0820:39seancorfieldI can confirm that it re-downloads my git deps.#2021-11-0820:43deactivateduserJust out of interest, do the gitlibs changes change how checked out git coord directories are exposed up via a lib-map (e.g. returned from clojure.tools.deps.alpha/resolve-deps)?
Background is that some of my code is pulling out the :deps/root key from the lib-map for each dependency, then scanning that directory using standard file I/O. So if that directory itself is still just a vanilla checkout of the git repo my code will be fine (even if it’s now stored within a different parent hierarchy).#2021-11-0820:45Alex Miller (Clojure team)no change#2021-11-0820:48seancorfieldWhile testing this and updating some of my projects, I noticed that the short SHA in the README for the test-runner seems to be incorrect:
io.github.cognitect-labs/test-runner
{:git/tag "v0.5.0" :git/sha "b3fd0d2"}#2021-11-0820:57Alex Miller (Clojure team)that's the sha of the tag I think, which points to the commit (this is an annotated tag)#2021-11-0820:58Alex Miller (Clojure team)yeah#2021-11-0820:59Alex Miller (Clojure team)either is ok as they both unpeel to the same commit#2021-11-0821:00Alex Miller (Clojure team)% clj -Sdeps '{:deps {io.github.cognitect-labs/test-runner {:git/sha "b3fd0d2" :git/tag "v0.5.0"}}}' -Stree
Cloning:
Checking out: at 48c3c67f98362ba1e20526db4eeb6996209c050a
...
io.github.cognitect-labs/test-runner 48c3c67
...#2021-11-0821:00Alex Miller (Clojure team)the way the validation works, it checks that both the sha prefix and the tag unpeel to the same full commit sha, which they do#2021-11-0821:08seancorfieldWeird. Looking on the tags page of the repo, it shows this short SHA: {:git/tag "v0.5.0" :git/sha "48c3c67"} which is what I've been using.#2021-11-0821:19Alex Miller (Clojure team)and that's not wrong#2021-11-0821:19Alex Miller (Clojure team)annotated tags are objects in the git store, just like commits and that tag object itself is b3fd0d2, which points to commit 48c3c67#2021-11-0821:20Alex Miller (Clojure team)lightweight tags are NOT objects in the git store and won't have their own sha#2021-11-0821:20Alex Miller (Clojure team)they are just name refs that refer to a commit#2021-11-0821:23Alex Miller (Clojure team)b/c of how I usually look up the sha, I'm probably usually getting the tag sha, rather than the commit sha, which admittedly might be confusing#2021-11-0821:28Alex Miller (Clojure team)usually I do git rev-parse --short v0.5.0#2021-11-0821:28Alex Miller (Clojure team)but I can probably do git rev-parse --short v0.5.0^{commit} to get the commit tag instead and then it will match what people see on gh#2021-11-0821:29Alex Miller (Clojure team)(the latter is closer to what gitlibs is doing)#2021-11-0821:32seancorfieldAh, interesting. Thanks.#2021-11-0913:45souenzzoanyone using git deps with a private aws codecommit repository?!
when I tried to do that, tools-deps was implemented via jgit, that do not support the auth method required by aws codecommit
Now that it is using git CLI, I would like to know if anyone is using it.#2021-11-0914:05Alex Miller (Clojure team)What's the auth method in question? #2021-11-0915:09souenzzoThere is many ways to configure it. most of them suppose that you will use git command line.
https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html
https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-without-cli.html
I tried many paths
I give up was when I found that my ~/.gitconfig has this section:
[credential]
helper = !aws codecommit credential-helper
And JGit do not supprt ! syntax.#2021-11-0915:15Alex Miller (Clojure team)in general, if it works using git from the command line, it will probably work with the CLI#2021-11-0915:15Alex Miller (Clojure team)as that's all the CLI is ultimately doing now#2021-11-0915:16Alex Miller (Clojure team)there are some helper env vars at https://github.com/clojure/tools.gitlibs/#configuration that can be useful to debug as well if needed#2021-11-1603:37Alex Miller (Clojure team)it's built in#2021-11-1902:39seancorfieldI'm trying to debug a tools issue in CI. I'm seeing
clojure -Ttool list
Error building classpath. Bad coordinate for library , expected map: nil
The CLI is 1.10.3.933 The deps.edn file in the project seems valid (`clojure -Stree` works). I've also tried updating to 1.10.3.1029. The failed CI is https://github.com/rm-hull/nvd-clojure/runs/4259660700?check_suite_focus=true -- you can see the output of clojure -Sdescribe and the root deps.edn just as a sanity check...#2021-11-1903:15Alex Miller (Clojure team)I think that should be -Ttools ?#2021-11-1903:59Alex Miller (Clojure team)I added a check for unknown tool name with a better error message in a couple places#2021-11-1904:00seancorfieldEr, yes, OK, that doesn't really explain what we were seeing, but I'll go back to the PR and see what's up.#2021-11-1904:02seancorfieldHere's the earlier failure:
clojure -Ttools install nvd-clojure/nvd-clojure '{:mvn/version "RELEASE"}' :as nvd
Error building classpath. Bad coordinate for library , expected map: nil
and that definitely has -Ttools @alexmiller#2021-11-1904:03seancorfieldI ended up "solving" it by writing the tools/tools.edn file manually -- so it's something to do with clojure not laying that down when it first runs...#2021-11-1904:06Alex Miller (Clojure team)so there was no tools.edn there?#2021-11-1904:08seancorfieldCorrect, here's another run:
+ clojure -Ttools list
Error building classpath. Bad coordinate for library , expected map: nil
+ ls -l /home/runner/.config/clojure/
total 4
-rw-r--r-- 1 runner docker 1490 Nov 19 04:07 deps.edn
+ ls -l /home/runner/.config/clojure/tools/
ls: cannot access '/home/runner/.config/clojure/tools/': No such file or directory
+ mkdir /home/runner/.config/clojure/tools
+ echo '{:lib io.github.clojure/tools.tools :coord {:git/tag "v0.2.2" :git/sha "e1febed7ddaa5be15721255c13eb68e11bbbb398"}}'
+ clojure -Ttools list
(works now)#2021-11-1904:09seancorfieldTOOL LIB TYPE VERSION
tools io.github.clojure/tools.tools :git v0.2.2
+ clojure -Ttools install nvd-clojure/nvd-clojure '{:mvn/version "RELEASE"}' :as nvd
Installed nvd
+ clojure -Ttools list
TOOL LIB TYPE VERSION
nvd nvd-clojure/nvd-clojure :mvn 1.9.0
tools io.github.clojure/tools.tools :git v0.2.2#2021-11-1904:09Alex Miller (Clojure team)hmm, maybe something with using a special config dir#2021-11-1904:09seancorfieldWhat makes clojure decide whether to lay down the tools stuff?#2021-11-1904:09Alex Miller (Clojure team)if it's missing or the one in the install is newer than the one there#2021-11-1904:10seancorfieldAny suggestions for debugging why it isn't doing it (above)?#2021-11-1904:11Alex Miller (Clojure team)the logic is in the clojure script, so hack away#2021-11-1904:11seancorfieldHmm, OK. Will take a deeper look tomorrow, when I'm back at my main system...#2021-11-1904:13Alex Miller (Clojure team)what's your clj -Sdescribe?#2021-11-1904:13Alex Miller (Clojure team)if [ "$install_dir/tools.edn" -nt "$config_dir/tools/tools.edn" ]; then
mkdir -p "$config_dir/tools"
cp "$install_dir/tools.edn" "$config_dir/tools/tools.edn"
fi
is the whole of the relevant code#2021-11-1904:14Alex Miller (Clojure team)maybe that condition needs to explicitly check for missing, although I know I've tested this several ways#2021-11-1904:16seancorfieldJust a sec... I had clojure -Sdescribe in an earlier CI script...#2021-11-1904:17seancorfieldHere's what appeared for 933 (which failed the same way):
{:version "1.10.3.933"
:config-files ["/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-933/x64/lib/clojure/deps.edn" "/home/runner/.config/clojure/deps.edn" "deps.edn" ]
:config-user "/home/runner/.config/clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-933/x64/lib/clojure"
:config-dir "/home/runner/.config/clojure"
:cache-dir ".cpcache"
:force false
:repro false
:main-aliases ""
:repl-aliases ""}#2021-11-1904:20seancorfieldHere's the output from 1029:
{:version "1.10.3.1029"
:config-files ["/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure/deps.edn" "/home/runner/.config/clojure/deps.edn" "deps.edn" ]
:config-user "/home/runner/.config/clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure"
:config-dir "/home/runner/.config/clojure"
:cache-dir ".cpcache"
:force false
:repro false
:main-aliases ""
:repl-aliases ""}#2021-11-1904:24seancorfieldThis is on Ubuntu 20.04.3 LTS according to that CI log.#2021-11-1904:31Alex Miller (Clojure team)I repro'ed it when there is no tools dir#2021-11-1904:32Alex Miller (Clojure team)actually, no I didn't, it's fine#2021-11-1904:33Alex Miller (Clojure team)well I'm sure it's something in that area, can't really be much else#2021-11-1914:09seancorfield@alexmiller Should I create a JIRA for this or are you already on it?#2021-11-1914:11Alex Miller (Clojure team)if you have a repro for it....#2021-11-1914:12Alex Miller (Clojure team)otherwise not sure what I would do with it#2021-11-1914:12Alex Miller (Clojure team)I tested a few things, haven't been able to repro#2021-11-1914:14seancorfieldYeah, it's a weird one. I'd never seen it before and the only repro I have is that CI pipeline...#2021-11-1914:16Alex Miller (Clojure team)does it fail like that occasionally or always?#2021-11-1914:20seancorfieldAlways. And I think I know why:
+ ls -lR /opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure
/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure:
total 12
-rw-r--r--+ 1 runner docker 512 Nov 19 14:17 deps.edn
-rw-r--r--+ 1 runner docker 1490 Nov 19 14:17 example-deps.edn
drwxr-xr-x+ 2 runner docker 4096 Nov 19 14:17 libexec
/opt/hostedtoolcache/ClojureToolsDeps/1.10.3-1029/x64/lib/clojure/libexec:
total 15752
-rw-r--r--+ 1 runner docker 16125828 Nov 19 14:17 clojure-tools-1.10.3.1029.jar
-rw-r--r--+ 1 runner docker 3636 Nov 19 14:17 exec.jar
There is no tools.edn in the installation. So this is a bug with the DeLaGuardo/setup-clojure Action I think? @delaguardo?#2021-11-1914:23seancorfieldLooks like https://github.com/DeLaGuardo/setup-clojure/blob/master/src/cli.ts doesn't copy the tools.edn file into the tool cache.#2021-11-1914:26seancorfieldhttps://github.com/DeLaGuardo/setup-clojure/issues/34#2021-11-1914:27Alex Miller (Clojure team)sounds right#2021-11-1914:31delaguardoRight, this is a problem of the action. Will fix it later today#2021-11-1914:35seancorfield@delaguardo Given the caching of the clojure setup, will you be able to fix it for existing versions, or will this only affect new CLI versions going forward?#2021-11-1914:37delaguardoShould not be a problem. I will check it as well#2021-11-1914:38seancorfieldCool. I just read the issue that mentioned caching which was what made me think about it...#2021-11-1914:38seancorfieldsetup-clojure is awesome BTW -- thank you for your work on it!#2021-11-1914:52borkdudewhy isn't this using the bash script on linux and macos but it is using the powershell script on Windows?#2021-11-1919:52delaguardoI don't have much experience with windows and rely heavenly on premade script. For other platforms it seems like a better approach to control the resources because of few reasons:
• Better control what should be cached (unfortunately this hit the problem of very strange cache behavior of GitHub actions, main reason why I don't yet reply in the corresponding issue)
• I don't like the way how installation script works in general. It follows some "best practices" which i don't understand personally. Maybe it is the time to reconsider, but before that i would like to have a reply to the patch i proposed for packaging scripts.#2021-11-1914:36seancorfield(and I guess it will need to be conditional on tools.edn existing -- that only came in with 1.10.3.933 right @alexmiller?)#2021-11-1914:37Alex Miller (Clojure team)yep#2021-11-1915:43rickmoynihan@dominicm I have a project which still uses pack.alpha to create build artifacts; unfortunately the project has now acquired a dependency that uses the new :git/sha key, and not the old :sha key.
This causes pack.alpha to choke… I tried bumping the tools.deps.alpha version it uses to the latest; but it seems there have been some breaking changes for you in that dep. Specifically clojure.tools.deps.alpha.reader/merge-deps install-deps and slurp-deps have now moved/vanished.#2021-11-1915:45Alex Miller (Clojure team)they're in clojure.tools.deps.alpha (that change happened long long ago)#2021-11-1915:46rickmoynihanI guess I’m probably better migrating this to use tools.build uber instead; though I was using the skinny jar functionality to preserve the true classpath; rather than munge it#2021-11-1915:46rickmoynihan@alexmiller: so they’ve just moved ns? Or are they removed entirely now?#2021-11-1915:48rickmoynihanjust trying to figure out what the quicker fix will be — migrating or patching a fork of pack.alpha#2021-11-1915:48Alex Miller (Clojure team)moved, and I think the install one changed names to root-deps#2021-11-1915:48Alex Miller (Clojure team)the merge stuff might be factored slightly differently too, don't remember now#2021-11-1915:49Alex Miller (Clojure team)the same stuff is all there, just might be slightly different api#2021-11-1915:50rickmoynihanyeah — I’ll see if I can patch it#2021-11-1915:50rickmoynihanthanks again for pointing me in the right direction 🙇#2021-11-1915:51Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/commit/3dec3cd5c47f3eb7ef0b7e3a77087dc3f19d11d5#2021-11-1915:51Alex Miller (Clojure team)(June 2020)#2021-11-1915:52rickmoynihanperfect that’s the commit I’m looking for 👌#2021-11-1916:13rickmoynihan@alexmiller thanks, that was exactly what I was looking for…
I’ve patched up the skinny feature of pack.alpha here: https://github.com/juxt/pack.alpha/pull/94#2021-11-1919:04mkvlrwhen generating a jar, do all deps need to be available on maven as well and I can’t depend on :git/sha deps? Seeing the following warning and indeed valuehash is missing from the jar:
$ clj -T:build jar
Producing jar: target/clerk-0.3.265.jar
Skipping coordinate: {:git/sha ff1d4b7f1260daf41c786a61cb45d02871b7baf9, :git/url , :deps/manifest :deps, :deps/root /Users/mk/.gitlibs/libs/io.github.arachne-framework/valuehash/ff1d4b7f1260daf41c786a61cb45d02871b7baf9, :parents #{[]}, :paths [/Users/mk/.gitlibs/libs/io.github.arachne-framework/valuehash/ff1d4b7f1260daf41c786a61cb45d02871b7baf9/src]}
I guess it makes sense that this doesn’t work. Any recommended way to bring this in (I can’t find a jar on clojars or central)? I guess https://jitpack.io should work.#2021-11-1919:14seancorfield@mkvlr That message comes from tools.deps when it is building the pom.xml file -- and, yes, you can't have a JAR on Maven/Clojars that depends on git deps because consumers generally couldn't resolve that.#2021-11-1919:15mkvlr@seancorfield makes sense, would copying the sources into my jar be a bad idea?#2021-11-1919:16seancorfieldPotentially a bad idea, yes. Users of your JAR would "unexpectedly" get all those other nses and that might cause conflicts.#2021-11-1919:18mkvlrso better to fork & publish under a different name with different namespace root?#2021-11-1919:18mkvlrjitpack doesn’t work, or only partially, master seems to work but the lastest sha which master points to doesn’t. Seems to be bad to depend on a mutable name for sure.#2021-11-1919:21seancorfieldThis was a possibility with depstar -- by default it copied local and git deps into the JAR (since consumers of JARs couldn't consume those as deps in the pom) but there were caveats in the docs about it, and an option to not do that. tools.build by default does the latter -- not copying them (which is the better default behavior). Hard to say what you should do without knowing more about your project...#2021-11-1919:21seancorfieldYou're depending on Arachne but that doesn't have published artifacts?#2021-11-1919:22seancorfieldIn which case, anyone using your library would also have to depend on Arachne via git anyway -- so why publish your library at all? Why not let people depend on it via git, like they would have to do for Arachne?#2021-11-1919:23seancorfield(git deps are "contagious" in that respect: once you depend on them, your "customers" have to depend on them as well...)#2021-11-1919:28mkvlrI want consumers of our lib (https://github.com/nextjournal/clerk) to use it via maven as well (if they’re using lein or whatever). In general I’d like to make consumption easy. Can’t find a published artifact of https://github.com/arachne-framework/valuehash which I’d like to depend on, I’ll open a ticket with them.#2021-11-1919:34Alex Miller (Clojure team)I'm not sure Luke is still doing anything with that#2021-11-1919:35Alex Miller (Clojure team)so you might need to fork and publish if you want it out there#2021-11-1919:36mkvlr@alexmiller I’ll ask and do that if I don’t hear back, thank you. Might have been once published to http://maven.arachne-framework.org/artifactory/arachne-dev which shows up in some repos but http only is probably also not a good idea?#2021-11-1919:37Alex Miller (Clojure team)not even supported by most recent maven#2021-11-1919:37Alex Miller (Clojure team)subject to MITM attacks#2021-11-2121:48vlaaadI have an idea for a library that fits in a tweet, and tweets are immutable, can we have twitter procurer?#2021-11-2123:54hiredmanhttps://erlang.org/pipermail/erlang-questions/2011-May/058768.html#2021-11-2200:01dpsuttonI wrote one one time. But just fetching the tweet leaves longer messages truncated so I abounded it#2021-11-2200:06dpsuttonSince we have unlimited scroll back at the moment https://clojurians.slack.com/archives/C03RZGPG3/p1578369151371700#2021-11-2200:19hiredmanI mean, for clojure, since it is already expression oriented, you could do something like a reader tag that loads an expression from Twitter(or wherever) instead of via the deps machinery#2021-11-2207:15vlaaadWoah nice#2021-11-2207:15vlaaadI had no idea it was tried already!#2021-11-2212:15vlaaadI made a fun library that defines code from tweets: https://github.com/vlaaad/tweet-def#2021-11-2212:35vlaaad@U064X3EF3 I currently implemented tweet deps as a library ^^^, but I think it makes more sense to have tweet deps as a part of tools-deps toolchain… do you think Clojure tools could support it?#2021-11-2213:21Alex Miller (Clojure team)Dep sources are an open extension point - you have the power to adapt it into tools.deps :)#2021-11-2213:22Alex Miller (Clojure team)clojure.tools.deps.alpha.extensions #2021-11-2213:25Alex Miller (Clojure team)What doesn't exist at the moment is the ability to install and use an extension with the CLI#2021-11-2213:25vlaaadYeah, that's what I was thinking about..#2021-11-2213:26Alex Miller (Clojure team)Well if you did the other work I'd think more about that :)#2021-11-2213:45vlaaadgotcha!#2021-11-2121:54vlaaadMaybe as April Fool's special...#2021-11-2211:33Jelle LichtHow would one list all the transitive dependencies + licenses of a deps.edn project?#2021-11-2320:06deactivateduserThe following tool.build library will do the latter: https://github.com/pmonks/tools-licenses#2021-11-2320:08deactivateduserIt shouldn’t be hard to:
a) turn this into a “tool”, independent of tools.build
b) have it emit the dependency tree (right now it simply groups dependencies into “direct” or “transitive”, without showing the actual dependency graph - I figured folks would use clj -Stree or whatever in conjunction with this tool)#2021-11-2320:08deactivateduserIf either of those things interest you, issues and/or PRs would be very welcome!#2021-11-2612:19Jelle LichtLate reply, but thanks for the pointer. No promises on any useful contributions on my end 😉#2021-11-2622:43deactivateduserYou already exceeded your own expectations - thanks for the issue (now fixed)! 😄#2021-11-2213:31Alex Miller (Clojure team)clj -X:deps list (in latest) will give you the list#2021-11-2213:32Alex Miller (Clojure team)I think there is a project that tells you more about licenses but I don't remember the names. I would be interested in integrating that info into the list function too#2021-11-2302:23Ian Fernandez~/deptest/
.
├── b
│ ├── deps.edn
│ └── pom.xml
└── deps.edn
In this folder I have b folder with a deps.edn, how can I make ./deps.edn resolve deps of ./b/deps.edn when I have:
{:paths ["."]
:deps {b/b {:local/root "./b"}}}#2021-11-2302:24Ian FernandezI want to
clj -Stree to resolve even the deps that ./b/deps.edn has#2021-11-2302:25seancorfield@d.ian.b I would expect it to -- but unless you are using a very recent version of the CLI, :local/root deps can get "stale" and you need -Sforce to get them picked up.#2021-11-2302:25Ian Fernandezhum nice, it got#2021-11-2302:25Ian Fernandezthanks @seancorfield#2021-11-2302:26Ian Fernandezand if I want to have the test aliases paths/devs for ./b/deps.edn?#2021-11-2302:26seancorfieldWhat does clojure -version say? The most recent version is 1.10.3.1029#2021-11-2302:26seancorfieldYou can't get aliases from dependencies.#2021-11-2302:27Ian Fernandez1.10.3.1020, upgrading now#2021-11-2302:27Ian Fernandezi think i've missed to add some deps#2021-11-2302:27Ian Fernandezthanks 🙂#2021-11-2302:27Ian FernandezI'm taking a look at your posts#2021-11-2302:27Ian Fernandez😄#2021-11-2302:28Ian Fernandezright now I'm working for a company that uses boot-clj and I'm using macbook m1#2021-11-2302:28seancorfieldThere are a lot of issues with even how it would work to process aliases from dependencies. Lots of unanswered questions.#2021-11-2302:31Ian Fernandezif it got a keyword-name-namespaceish for a project module and aliases got resolved like:
clj -M:b/test#2021-11-2302:31seancorfieldI'm not getting into it -- that's something complex for the core team to figure out.#2021-11-2302:31seancorfieldNearly every simple, obvious suggestion is wrong here 🙂#2021-11-2302:32Ian Fernandezhahahahhaha yeah, I was just asking why something like couldn't get to be possible#2021-11-2302:33seancorfieldWhat if multiple dependencies provided the same alias?#2021-11-2302:33seancorfieldWhat if you had a user-level alias and you add a dependency to a project -- which should have priority?#2021-11-2302:35seancorfieldRight now, we have three well-defined contexts in a direct progression: project deps take precedence over user deps which take precedence over root deps.#2021-11-2302:36seancorfieldIf aliases can start to come from dependencies, there's a whole tree of places for them to start coming from...#2021-11-2302:29Ian Fernandezboot doesn't work on it hahaha, I'm trying to migrate stuff to deps, trying to have a deps.edn root for every git repo in the company#2021-11-2302:29Ian Fernandez😄#2021-11-2302:30seancorfieldOur build.clj at work reads the deps.edn from subprojects -- using tools.deps.alpha -- and merges in aliases from them and builds a "basis" for operations (such as running tests).#2021-11-2302:31Ian Fernandezif it got a keyword-name-namespaceish for a project module and aliases got resolved like:
clj -M:b/test#2021-11-2302:32Ian Fernandez@seancorfield thanks for helping 🙂#2021-11-2302:36Ian Fernandezthere's a way to convert build.boot file to a deps.edn file?#2021-11-2302:36Ian Fernandezsome project for it?#2021-11-2302:36seancorfieldNot that I know of.#2021-11-2302:36seancorfieldWhat we did with boot was move our dependencies to deps.edn files and then add code to boot to read those files. And then we migrated to the CLI 🙂#2021-11-2302:37seancorfieldBut that was three and a half years ago I think.#2021-11-2302:37Ian Fernandezhum ok, then it need some hard work 😅#2021-11-2408:23cflemingclojure.tools.deps.alpha/make-classpath is marked as deprecated, but there’s no mention of why, or suggestions for an alternative. What should I be doing instead?#2021-11-2408:30borkdude@cfleming I'm guessing create-basis which returns a map with contains the classpath#2021-11-2408:31cflemingSounds plausible, or even just the current implementation, which is a one-liner and doesn’t use anything deprecated.#2021-11-2408:31cflemingIt would be nice to understand the thinking a little more though.#2021-11-2408:32borkdudecreate-basis is pretty much the primitive used all over in tools build#2021-11-2413:25Alex Miller (Clojure team)make-classpath-map is the replacement, but certainly create-basis is often a better thing to use #2021-11-2413:25Alex Miller (Clojure team)It depends what you're trying to do of course :)#2021-11-2419:46cflemingIn my case, get the individual elements of the classpath as strings. Previously I called make-classpath then used str/split.#2021-11-2420:37cflemingIn the end I just inlined the existing impl and removed the string join, so I don’t need to re-split it.#2021-11-2620:32ChaseI've found I need to use (defn foo [& args]...) when evoking a function with clj -X core/foo , even if I don't intend on that function taking any args, or else I get a wrong number of args passed in error. Should I approach this in a different way?#2021-11-2620:34ChaseI'm mostly using that functionality when I want to test side effect stuff like user input. Should I be approaching that differently too you think? That's the only time I feel the jvm startup pain#2021-11-2620:39borkdudeif it's solely for testing, you can also do -M -e "((requiring-resolve 'core/foo))"#2021-11-2620:43vlaaad@chase-lambert -X always provides exactly one argument (a map)#2021-11-2620:44vlaaadif you want to test stuff, just use REPL, why do it through cli...#2021-11-2700:05Chasetbh, I need to find out how to use things like (read-string) and take user input from the repl. I'm using nvim w/ conjure and I'm not sure it can do that.#2021-11-2817:12hugodI’ve been working on an installable tool for running build tasks. The tasks can be run using clj -T or by using a babashka script (works for most tasks). The tasks optionally use a project.edn file to provide the project name and version.
The project is alpha at the moment.
Thanks to @borkdude for helping to get babashka to be able to run tools.build tasks.
https://github.com/hugoduncan/makejack#2021-11-3003:58CnlyHi, I’m not sure if this is the most suitable channel but I’m trying to figure out the differences between clj -T:something and clj -Tsomething. They seem to give different output:
$ clj -Ttools list
TOOL LIB TYPE VERSION
tools io.github.clojure/tools.tools :git v0.2.2
$ clj -T:tools list
Unqualified function can't be resolved: list
Could anyone give me a hint? Thanks!#2021-11-3004:05Alex Miller (Clojure team)you came to the right place, and yes they are different#2021-11-3004:06Alex Miller (Clojure team)-T:something says to use the alias :something in your deps.edn
-Tsomething says to use the installed tool named something#2021-11-3004:06Alex Miller (Clojure team)both will remove your project deps/paths and add :paths ["."]#2021-11-3004:07Alex Miller (Clojure team)in the case above, tools is the name of a pre-installed tool (you can find the guts of it at ~/.clojure/tools/tools.edn - which is really just a lib/coord that will be added to your deps)#2021-11-3004:07Alex Miller (Clojure team):tools is nothing, which is why it errors#2021-11-3004:08Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#tool_install is more reference material on this#2021-11-3004:41CnlyThanks for the info! So tools in -Ttools is the name of a built-in, and deps in -X:deps is the alias of a built-in. I see how I got confused.#2021-11-3004:43Alex Miller (Clojure team)Yep#2021-11-3014:41dharriganI notice that running clj -Mantq and clj -M:antq work, is the first form wrong?#2021-11-3014:41CnlyI think the : is just omitted in the first form…#2021-11-3014:42Alex Miller (Clojure team)the latter is correct#2021-11-3014:42Alex Miller (Clojure team)the former is (sloppily) allowed#2021-11-3014:42Alex Miller (Clojure team)really just a by-product of how the parsing is implemented#2021-11-3014:44borkdudewasn't the first one the default syntax in the very beginning and eventually -A:foo:bar became the standard? By the time -M came it was an established idiom?#2021-11-3014:46Alex Miller (Clojure team)no, the first one has never been a documented thing#2021-11-3014:46Alex Miller (Clojure team)it has always been concatenated keywords#2021-11-3014:46borkdudeah I see :)#2021-11-3016:14dharriganta 🙂#2021-11-3020:49Jeff EvansAnyone know what might be causing this? When running clojure -X:deps prep
clojure --help | grep Version
Version: 1.10.3.1029
Prepping metabase/java-deps in /Users/<person>/repos/metabase_master/java
Error compiling Java sources: nil
{:via
[{:type java.lang.NullPointerException,
:at
[clojure.tools.build.tasks.javac$javac
invokeStatic
"javac.clj"
28]}],
:trace
[[clojure.tools.build.tasks.javac$javac invokeStatic "javac.clj" 28]
[clojure.tools.build.tasks.javac$javac invoke "javac.clj" 21]
[clojure.lang.Var invoke "Var.java" 384]
[clojure.tools.build.api$javac invokeStatic "api.clj" 229]#2021-11-3020:52borkdude@jeffrey.wayne.evans What do you get when you evaluate
(javax.tools.ToolProvider/getSystemJavaCompiler)
#2021-11-3020:57Jeff Evansnil#2021-11-3021:00borkdudeSo this is what causes the NullPointer exception.#2021-11-3021:03borkdudeI have no idea why it returns nil in your case, but I'm not that familiar with the javax.tools.Provider API to be honest#2021-11-3021:05borkdudePerhaps you are using an JRE as opposed to a JDK?#2021-11-3021:06borkdudeThis may help: https://stackoverflow.com/a/10944833/6264#2021-11-3021:11Alex Miller (Clojure team)JRE would be my guess too#2021-11-3021:57Jeff Evansthanks, folks. this is for a colleague who is just trying to iterate without becoming an expert in Clojure, so I’ll double check to ensure the environment is completely sane. I think we had some recent confusion around the whole Temurin project switchover stuff#2021-12-0100:18Jeff EvansTo close the loop, that is indeed what the issue was (JRE as opposed to JDK)#2021-12-0100:56Alex Miller (Clojure team)Probably something that can be better reported#2021-11-3021:16vemvCloning:
I see this on every Circle run for a Tools-based job. Is there a recommended way to cache it?
Using circleci/clojure:openjdk-17-tools-deps-buster-node btw.#2021-11-3021:17Alex Miller (Clojure team)it's a bug in the clojure action#2021-11-3021:17Alex Miller (Clojure team)I think there's an issue for it#2021-11-3021:18Alex Miller (Clojure team)https://github.com/DeLaGuardo/setup-clojure/issues/34#2021-11-3021:18Alex Miller (Clojure team)at least I assume that's the same cause#2021-11-3021:19vemvdoes the Circle image use setup-clojure?#2021-11-3021:26seancorfieldThis is how nvd-clojure works around it for now: https://github.com/rm-hull/nvd-clojure/blob/master/.github/integration_test.sh#L14-L16#2021-11-3021:29seancorfield(although that doesn't help with the caching option, it could at least get it into your cachable folder if you then ran clojure -P ... afterward)#2021-11-3021:42vemvnot sure if the GH Actions and Circle thingies are the same issue, although if you're sure I'll take your word about it
from my side other than the lack of caching I'm not experiencing an issue in Circle#2021-11-3021:47seancorfieldAre you caching ~/.gitlibs?#2021-11-3021:47seancorfieldSorry, I sort of assumed you already would be...#2021-11-3021:48seancorfieldHere's what I cache in my GH Actions jobs: https://github.com/seancorfield/next-jdbc/blob/develop/.github/workflows/test.yml#L21-L29#2021-11-3021:49seancorfield(I've stopped using CircleCI at this point)#2021-11-3021:50vemv> Are you caching `~/.gitlibs`?
yes but ~/.clojure ~/.cpcache seem needed as well, thanks!#2021-11-3021:50seancorfieldAh, yes, otherwise it will want to recompute dependencies.#2021-11-3021:52seancorfield(make sure you're on a non-XDG system -- on XDG systems, it's ~/.config/clojure not ~/.clojure)#2021-11-3021:53seancorfieldI mention that because the nvd-clojure integration test seems to be on an XDG system -- note the path for tool/tools.edn -- so ~/.clojure wouldn't help you there.#2021-11-3022:04borkdudeNo, CircleCI doesn't use setup-clojure - that one is specific for Github Actions (as always: I reserve the right to be wrong)#2021-12-0100:25seancorfieldGood to know but caching ~/.gitlibs and ~/.clojure is the important part, right? Even for CircleCI.#2021-11-3022:08borkdude@seancorfield don't you need to cache $PWD/.cpcache as well as the clojure CLI writes cp caches there too? Or does setup-clojure behave otherwise?#2021-12-0100:21seancorfieldTrue, you could cache that to avoid recomputes for the classpath for running tests, which would speed up repeated runs where the classpath hasn't changed. And it's probably not worth caching ~/.cpcache to be honest. My main concern was preventing unnecessary git checkouts and Maven/Clojars downloads.#2021-12-0100:24seancorfieldI'm not entirely sure what $PWD would be for GH Actions. I think ~ is /home/runner and I think $PWD might be /home/runner/work/<repo>/<repo> based on some of what I'm seeing in the logs but I haven't confirmed that... Not sure if how to specify that in the cache action without hard-coding the path, per repo tho'?#2021-12-0105:31Alex Miller (Clojure team)just released Clojure CLI prerelease 1.10.3.1036 which adds license info to clj -X:deps list, give it a try if you like#2021-12-0105:32Alex Miller (Clojure team)Here's an example output in tools.deps.alpha itself#2021-12-0105:36Alex Miller (Clojure team)works with Maven deps, pom-based file deps, local jars, but not with deps-only local/git deps#2021-12-0106:22seancorfieldWhat causes this limitation @U064X3EF3?#2021-12-0112:42Alex Miller (Clojure team)I'm pulling the pom license info, deps projects don't have that in a known place. Can potentially fill that gap.#2021-12-0118:58seancorfieldI would imagine that 90% of projects that have an explicit license have a LICENSE file at the top level? GitHub does that by default when you create a new repo and select a license, and both lein new and deps-new (and clj-new) create LICENSE (EPL, but with a note in the README that you can change licenses). boot-new too, not that anyone uses that these days 🙂#2021-12-0119:26Alex Miller (Clojure team)for sure, but what do I do with a blob of text in a LICENSE file?#2021-12-0119:27Alex Miller (Clojure team)I have not spent the time yet to devise a strategy to go from that to "license name"#2021-12-0119:29Alex Miller (Clojure team)maybe just the first line is often sufficient#2021-12-0119:30seancorfieldAh, right, I hadn't thought that through very much...#2021-12-0119:31Alex Miller (Clojure team)it seemed worth releasing it without that before I figured that part out :)#2021-12-0119:32seancorfield...looking across a bunch of LICENSE files here, I suspect you could pattern match common ones from the first non-blank line of the file (which looks to be the first line in nearly all cases).#2021-12-0119:32Alex Miller (Clojure team)yeah, that's probably a decent hack#2021-12-0119:32seancorfield(! 647)-> head -1 */LICENSE
==> build-clj/LICENSE <==
Apache License
==> clj-new/LICENSE <==
Eclipse Public License - v 1.0
==> deps-new/LICENSE <==
THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC
==> expectations/LICENSE <==
THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC
==> next.jdbc/LICENSE <==
THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC
==> polylith/LICENSE <==
Eclipse Public License - v 1.0
==> sean-polylith/LICENSE <==
Eclipse Public License - v 1.0
(just what I have on hand on this machine)#2021-12-0119:39seancorfieldLooking across my whole machine, a lot of LICENSE files begin with a copyright line. Nearly all of those have this text, whatever license this is:
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
~BSD maybe?~ Nope.#2021-12-0119:40seancorfieldMIT.#2021-12-0122:21deactivateduserFYI it would be better to adopt existing best practice in this area i.e. https://spdx.dev/license-list/matching-guidelines/#2021-12-0122:23deactivateduserThe folks involved in that initiative have spent a LOT of time considering this problem, including “corner cases” that are a lot more common than you might think (e.g. dual-or-more licensed software where the author has concatenated all of the licenses’ texts into a single ./LICENSE or ./COPYING or whatever file).#2021-12-0122:26Alex Miller (Clojure team)I'm not going to do text matching of any license text as I don't want to "pay" for maintaining or including or downloading the text to match against#2021-12-0122:26deactivateduserRight. Just pointing out best practice, based on years of experience (others’, to clarify, not mine).#2021-12-0122:27Alex Miller (Clojure team)I linked this site in the prior discussion, I am aware of it#2021-12-0122:27deactivateduserYou’ll also hit the issue that many MANY Maven artifacts don’t include license information at all, either in their pom.xml or as a LICENSE or COPYING file. In fact a bunch of Clojure’s own JARs are like this too.#2021-12-0122:27Alex Miller (Clojure team)actually, Maven central requires license declaration in the pom and virtually all libs do include it#2021-12-0122:28Alex Miller (Clojure team)Clojars does not but in practice seems most I've looked at do include it#2021-12-0122:28Alex Miller (Clojure team)so just using pom info yields a pretty high success rate#2021-12-0122:28deactivateduserThat may be a recent thing, as there certainly are a large number of popular Maven-hosted artifacts that don’t.#2021-12-0122:28Alex Miller (Clojure team)it has always been a requirement on central#2021-12-0122:29Alex Miller (Clojure team)or "always" since I started deploying stuff there 10-15 yrs ago#2021-12-0122:29Alex Miller (Clojure team)other repos are more lax#2021-12-0122:29seancorfieldRemember: perfect is the enemy of good. So unless you're going to write a patch for tools.deps.alpha, what we have is "good enough" for a lot of people I suspect 🙂#2021-12-0122:30Alex Miller (Clojure team)I am happy to get about two standard deviations of "good"#2021-12-0122:31deactivateduserPretty sure you’re incorrect about that @U064X3EF3 - here’s an example POM published today that does not contain any license information: https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.12.122/aws-java-sdk-core-1.12.122.pom#2021-12-0122:31Alex Miller (Clojure team)if there is another hack or two to cover the deps license files cases, at least for the ~5 most common Clojure libs, I'd take that (and want to consider a way to let people say this more deterministically without a pom)#2021-12-0122:32Alex Miller (Clojure team)it's in the parent pom#2021-12-0122:32Alex Miller (Clojure team)https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-pom/1.12.122/aws-java-sdk-pom-1.12.122.pom#2021-12-0122:32deactivateduserNot that it’s complete (at all), but https://github.com/pmonks/tools-licenses at least attempts to look for license texts inside deps (both Maven and git coord).#2021-12-0122:32Alex Miller (Clojure team)I'll get those too#2021-12-0122:32deactivateduserIt uses a similar strategy to the one you and @seancorfield were discussing above (simple text matching the header of the license), which SPDX does not recommend.#2021-12-0122:33Alex Miller (Clojure team)% clj -X:deps list :extra '{:deps {com.amazonaws/aws-java-sdk-core {:mvn/version "1.12.122"}}}'
com.amazonaws/aws-java-sdk-core 1.12.122 (Apache-2.0)
com.fasterxml.jackson.core/jackson-annotations 2.12.3 (Apache-2.0)
com.fasterxml.jackson.core/jackson-core 2.12.3 (Apache-2.0)
com.fasterxml.jackson.core/jackson-databind 2.12.3 (Apache-2.0)
com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.12.3 (Apache-2.0)
commons-codec/commons-codec 1.15 (Apache-2.0)
commons-logging/commons-logging 1.2 (Apache-2.0)
joda-time/joda-time 2.8.1 (Apache 2)
org.apache.httpcomponents/httpclient 4.5.13 (Apache-2.0)
org.apache.httpcomponents/httpcore 4.4.13 (Apache-2.0)
org.clojure/clojure 1.10.3 (EPL-1.0)
org.clojure/core.specs.alpha 0.2.56 (EPL-1.0)
org.clojure/spec.alpha 0.2.194 (EPL-1.0)
software.amazon.ion/ion-java 1.0.2 (The Apache License, Version 2.0)#2021-12-0122:34Alex Miller (Clojure team)^^ works just fine on the one you mentioned#2021-12-0122:35Alex Miller (Clojure team)https://central.sonatype.org/publish/requirements/#license-information#2021-12-0122:35Alex Miller (Clojure team)they have rules that check this ^^#2021-12-0122:36deactivateduserAnother example (from 2018): https://repo1.maven.org/maven2/javax/mail/javax.mail-api/1.6.2/javax.mail-api-1.6.2.pom#2021-12-0122:37deactivateduser(and yes, I realise there is license text in an XML comment at the top)#2021-12-0122:37Alex Miller (Clojure team)again ,parent poms have the license info#2021-12-0122:37Alex Miller (Clojure team)% clj -X:deps list :extra '{:deps {javax.mail/javax.mail-api {:mvn/version "1.6.2"}}}'
javax.mail/javax.mail-api 1.6.2 (CDDL/GPLv2+CE)
org.clojure/clojure 1.10.3 (EPL-1.0)
org.clojure/core.specs.alpha 0.2.56 (EPL-1.0)
org.clojure/spec.alpha 0.2.194 (EPL-1.0)#2021-12-0122:38Alex Miller (Clojure team)no license text matching stuff is needed imo for Maven artifacts, it's there for 99.9% of things without that#2021-12-0122:39deactivateduserRight, which leaves other Maven repos (Clojars in particular), and git coords.#2021-12-0122:40Alex Miller (Clojure team)Clojars seems to be relatively high even without a requirement, so not worried about that either#2021-12-0122:40Alex Miller (Clojure team)mostly trying to look at local/git deps projects#2021-12-0122:41Alex Miller (Clojure team)but I'd rather solve this by creating a well-known place to list the equivalent to the pom fields#2021-12-0122:41deactivateduserFor that you’d likely have to rely on LICENSE or COPYING files.#2021-12-0122:41deactivateduserWhich is why https://github.com/pmonks/tools-pom exists…#2021-12-0122:41deactivateduserThough because it can take its input from anywhere, it’s not helpful for determining licenses.#2021-12-0122:42deactivateduser(also because tools.build task names are not standardised, so there’s no “API” for a license determination task to invoke to generate a POM and then parse that)#2021-12-0122:44Alex Miller (Clojure team)sure there is :)#2021-12-0122:44Alex Miller (Clojure team)tools.deps and tools.build can both generate and parse poms :)#2021-12-0122:44deactivateduserSure, but only incomplete ones.#2021-12-0122:45Alex Miller (Clojure team)I don't understand what we're even talking about now. to the problem hand, I can just define a well known attribute in the deps.edn to pull this from#2021-12-0122:45deactivateduserThere’s no way (as of about a month ago, when I last looked into this in some detail) to tell tools.deps / tools.build about “extra” pom.xml elements that I want included in the poms it generates*.
*short of checking in an incomplete “template” pom.xml in my repo, which creates other problems for downstream tooling#2021-12-0122:45Alex Miller (Clojure team)you can give it a base pom and sync into it#2021-12-0122:46deactivateduserWhich messes with downstream tooling that now thinks my project is buildable with Maven.#2021-12-0122:46Alex Miller (Clojure team)and eventually I'll get around to making something broader there#2021-12-0122:46Alex Miller (Clojure team)you can call it something that's not pom.xml#2021-12-0122:46deactivateduserHow many steps am I away from solving my business problem now? 😉#2021-12-0122:47Alex Miller (Clojure team)well like I said, none of this has anything to do with the problem I'm trying to solve#2021-12-0122:47deactivateduserI just wanted to make sure my project isn’t violating my organisation’s open source consumption policies, and now I’m creating thisisnotapom.xml files and learning how to tell my build tooling to consume it. 😉#2021-12-0122:48deactivateduserThat’s the thing though - these two areas are intertwined (for better or worse - I’m not advocating on behalf of how Maven pom.xml and repositories work - I’m simply stating how they work).#2021-12-0122:48Alex Miller (Clojure team)well I don't know what you're doing, this has nothing to do with providing a license name for a deps.edn project#2021-12-0122:49deactivateduserSure it does - as you yourself said, putting license information in a pom.xml file makes that use case easy. How do I do that with tools.deps / tools.build?#2021-12-0122:49Alex Miller (Clojure team)put it in the deps.edn file, and give it to tools.build from there?#2021-12-0122:49deactivateduserOh yeah - I don’t. Since that’s not readily supported yet (without checking in pom.xml fragments in my repo).#2021-12-0122:49Alex Miller (Clojure team)ok, bye#2021-12-0122:50deactivateduser“Those who cannot remember the past are condemned to repeat it.” ¯\(ツ)/¯#2021-12-0122:54seancorfield"Which messes with downstream tooling that now thinks my project is buildable with Maven." -- that's just not true. The source file for tools.build to generate pom.xml from does not need to be called pom.xml nor does it need to be in the root of the project.#2021-12-0122:54seancorfield(I do this in some of my projects -- which is why I jumped on your comment as blatantly false @deactivateduser10790)#2021-12-0122:55deactivateduserYet that’s the default?#2021-12-0122:55deactivateduserOthers here have commented on this in the past, and I don’t have much interest in parroting their (well-founded, IMO) arguments in this regard.#2021-12-0122:55seancorfieldAnd yet you do.#2021-12-0122:58deactivateduserNot at all. I jumped into this thread to share my experience with license detection, both as someone who had responsibility for it professionally (at an open source Foundation), and as someone who recently scratched an itch because this didn’t previously exist in the tools.build ecosystem.#2021-12-0122:58deactivateduserIf you’re not interested in learning from experience, just say so. There’s no need for you or @U064X3EF3 to be dicks about it.#2021-12-0215:28richhickey@deactivateduser10790 As frustrated as you might be by the conversation, please avoid ad hominem rhetoric. We don’t talk about other people like that in the Clojure community. https://clojure.org/community/etiquette#2021-12-0106:19seancorfieldVery nice! Just ran it on our code base and it starts with
bidi/bidi 2.1.6 (MIT)
camel-snake-kebab/camel-snake-kebab 0.4.2 (EPL)
cfml-interop/cfml-interop 0.3.0 (EPL)
cheshire/cheshire 5.10.0 (MIT)
clj-antlr/clj-antlr 0.2.5 (EPL)
clj-commons/fs 1.6.309 (Eclipse Public License - v 1.0)
clj-stacktrace/clj-stacktrace 0.2.8 (MIT)
clj-tuple/clj-tuple 0.2.2 (MIT)
clojure-csv/clojure-csv 2.0.2
clojure.java-time/clojure.java-time 0.3.3 (MIT)
clout/clout 2.2.1 (EPL)
com.adobe.xmp/xmpcore 6.1.11 (The BSD 3-Clause License (BSD3))
com.braintreepayments.gateway/braintree-java 2.104.0 (MIT license)
com.cognitect/anomalies 0.1.12 (Apache-2.0)
com.cognitect/http-client 0.1.106 (Apache-2.0)
com.cognitect.aws/api 0.8.524 (Apache-2.0)
com.cognitect.aws/endpoints 1.1.12.69 (Apache-2.0)
com.cognitect.aws/s3 814.2.991.0 (Apache-2.0)
com.corundumstudio.socketio/netty-socketio 1.7.17 (Apache v2)
com.drewnoakes/metadata-extractor 2.16.0 (Apache-2.0)
com.fasterxml.jackson.core/jackson-annotations 2.8.11 (Apache-2.0)
...
#2021-12-0106:20seancorfield(I have no idea where that Adobe dependency is coming from 😆 )#2021-12-0115:57Alex Miller (Clojure team)https://clojurians.slack.com/archives/C06MAR553/p1638374097097000#2021-12-0116:00dpsuttonhow do you identify licenses? And do you preserve copyright notices when using :full?#2021-12-0116:02dpsuttonoh full vs short is “Eclipse Public License 1.0” vs “EPL-1.0"#2021-12-0116:02Alex Miller (Clojure team)"all" published maven deps have a license section so I'm taking from there. :full gives you whatever the project lists (this is license, not copyright) whereas :short is a canonicalized shortened form, trying to use https://spdx.org/licenses/ ids#2021-12-0116:03Alex Miller (Clojure team)with :format :edn you get both full name + license url#2021-12-0116:05Alex Miller (Clojure team)the "shortener" is literally text matching but I seeded it based on real data (and if not found, just falls back to full)#2021-12-0116:06Alex Miller (Clojure team)so it's super dumb, but works about 95% of the time#2021-12-0116:47Joshua SuskaloI'm seeing some very strange behavior in my current system. I'm setting up tools build to be used, and as a part of it I'm using the merge-edns function from tools deps. I have the latest version of tools deps and tools build on the classpath, but this function is not accessible. The only accessible functions are combine-aliases, lib-location, make-classpath, print-tree, and resolve-deps. Is this some strange behavior where a very old version of tools deps is being loaded first on the classpath and as a result the version I have at the root of my deps tree is being overriden?#2021-12-0116:48Alex Miller (Clojure team)sounds like it#2021-12-0116:48Joshua SuskaloI do see that a different alias put a different version of tools deps as a root dependency. What's the way that the cli chooses which to use?#2021-12-0116:48Alex Miller (Clojure team)usually it will take the newest but hard to say without more detail#2021-12-0116:49Joshua Suskalowell I had 0.8.695 in an alias that came first on the comamndline and version 0.12.1084 on an alias that came later.#2021-12-0116:49Joshua Suskaloso I'm guessing that older version is what got loaded, after changing it everything loads fine#2021-12-0116:50Joshua SuskaloBut I was expecting the latest to be used, not a first-one-wins#2021-12-0116:52Alex Miller (Clojure team)well, without specifics, hard for me to say more#2021-12-0116:52Joshua SuskaloWhat more information would be helpful? I unfortunately can't just share the project as it's proprietary.#2021-12-0116:53Joshua SuskaloI could build a minimal repro deps edn though#2021-12-0116:57Joshua Suskalo;; In deps.edn
{:aliases {:dev {:extra-deps {org.clojure/tools.deps.alpha {:mvn/version "0.8.695"}}}
:build {:deps {org.clojure/tools.deps.alpha {:mvn/version "0.12.1084"}}}}}
$ clj -A:dev:build
Clojure 1.10.3
user=> (require '[clojure.tools.deps.alpha :as t])
nil
user=> t/merge-edns
Syntax error compiling at (REPL:0:0).
No such var: t/merge-edns
#2021-12-0116:57Joshua SuskaloThis is a minimal reproduction case#2021-12-0116:57Joshua Suskalolooking at it I expect it might have something to do with the build alias using :deps while the dev alias uses :extra-deps#2021-12-0116:58Joshua Suskalobut this is more or less intended because normally the build alias will be invoked with -T, but in this case it's added with -A for a devtime repl with access to both the development code and also the build script.#2021-12-0117:00Alex Miller (Clojure team)aliases will merge in order for identical deps like that, so this is the expected behavior#2021-12-0117:00Alex Miller (Clojure team)so clj -A:build:dev would presumably have worked#2021-12-0117:03Joshua SuskaloOkay, cool. Good to know this is expected behavior.#2021-12-0117:03Joshua SuskaloThanks for the help!#2021-12-0118:37kingcodeHigh-level question: is it possible to create a custom name and refer to it elsewhere? E.g.
{:custom-name <some-long-path>
{:aliases {:a {:extra-deps [:custom-name,....]}
:b { blah-blah :custom-name etc..}}}}
Intent is to avoid duplicating a long path b/w different aliases.
Or even to refer to another alias within an alias?
Thx for any comment...#2021-12-0118:44borkdudethis is already what aliases are for. in some cases aliases can be used as a reference to data, but I'm not sure where it is supported...#2021-12-0118:45kingcodeOK thx @borkdue. I just found a doc segment addressing that. If I understand correctly I can refer to anything inside the aliases section, and then use it from anywhere…however I want to refer to an alias from within the aliases section.#2021-12-0118:46kingcode…and since custom names are declared (only?) in the aliases section it seems non-trivial as a deps.edn newbie..#2021-12-0118:48borkdudelots of things are non-trivial to implement, but not non-trivial in the sense of added complexity in the long run :)#2021-12-0118:49borkdudesome people (including me) have a deps.edn-generation-from-templates approach for some projects.#2021-12-0118:49kingcodeI can rephrase my question more clearly: can an alias refer to another?#2021-12-0118:50kingcodeI’m sure it’s possible, just haven’t seen an example yet..#2021-12-0118:50kingcodeas in my example above#2021-12-0118:50dpsuttonhttps://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases?show=10564#q10564 That does not exist but you can upvote this feature request on http://ask.clojure.org#2021-12-0118:51kingcodeSounds good - thx! Will do…#2021-12-0118:51dpsuttonthat’s why @borkdude was saying he generates deps.edn to simulate this.#2021-12-0118:53borkdude(for some projects)#2021-12-0118:53kingcodeIndeed, I can see why now. I would have thought this a great selling feature for deps.edn. @borkdude, maybe you should work your magic to generate a parser which inserts custom declarations 🙂#2021-12-0118:54kingcodeI meant a neat tool would parse custom tags and generate a compliant deps.edn#2021-12-0118:56borkdudethis is pretty trivial to do using e.g. a script that uses selmer and then spits out the real deps.edn.
usually you want to preserve whitespace. selmer is included in babashka so you could write a script using that and hook it up to a git hook or so#2021-12-0118:56kingcodeThanks @dpsutton - I did upvote the request#2021-12-0118:56borkdudeyou could also use rewrite-clj / rewrite-edn to update the deps.edn#2021-12-0118:56kingcodeOK…thx for the suggestion.#2021-12-0118:56kingcodeCool stuff. thx#2021-12-0118:57borkdudefor a tool that uses rewrite-clj / rewrite-edn you can check this as an example:
https://github.com/babashka/neil#2021-12-0118:57kingcodeIndeed it sounds like a simple thing to implement - will look at babashka/neil 🙂#2021-12-0118:58kingcodeIt looks like this is what I was looking for. Thanks again!#2021-12-0118:59borkdudeThe selmer approach is probably easier to start with#2021-12-0119:01kingcodeWill look at both. I just though of another approach, dont’ know if it’s possible as is: merging aliases? E.g.
clj -A:a:b:c#2021-12-0119:02borkdudethis is certainly possible#2021-12-0119:03borkdudein fact, that is what they are for :)#2021-12-0119:03kingcodeAh!? Didn’t realize this. As per my example command above I guess?#2021-12-0119:04kingcodeMaybe I’ll just try that…#2021-12-0119:04kingcodeI guess it merges nested keys as well?#2021-12-0119:05borkdude{:aliases {:a {:extra-deps {}} :b {:extra-deps {}}
clj -A:a:b
gives you a REPL with both the deps from a and b#2021-12-0119:06kingcodeFantastic - thank you! So one can write mergeable aliases without resorting to custom names this way.#2021-12-0119:06borkdudeyes - it wasn't even obvious to me that this is something you didn't know yet - sorry for that :)#2021-12-0119:07kingcodeNot at all - I am sorry for not being clearer. Thx!#2021-12-0119:07kingcodeI always wanted to learn deps.edn and cli tools, but keep getting lost in the weeds and forget/re-learn every time.#2021-12-0119:11seancorfield@kingcode Here's how I start my REPL (as an example of merging aliases):
SOCKET_REPL_PORT=5000 clojure -J-Dlog4j2.configurationFile=log4j2-sean.properties -M:rebel:classes:reflect:jedi-time:portal:everything:dev:test:runner:build:dev/repl
#2021-12-0119:12seancorfield(`:dev` and :test come from the Polylith portion of our project, :everything and :runner are the equivalents for the non-Polylith "legacy" portion, :build is so I can work on our build.clj script in the REPL, the rest come from my user ~/.clojure/deps.edn file which is up on GH)#2021-12-0119:14kingcodeThx @seancorfield, nice example. Do you need to worry about a profile over-riding another during the merge?#2021-12-0119:14kingcode(or a nested value thereof..)#2021-12-0119:15dpsuttonoften you rely on later things shadowing earlier. and nested values have different ways of being merged with an appropriate accumulator. ie, extra deps don’t clobber, they aggregate#2021-12-0119:16kingcodeThx @dpsutton, what I was hoping for 🙂#2021-12-0119:17dpsuttonappend-unique, merge, last-wins, etc#2021-12-0119:25seancorfieldYeah, the rules are pretty straightforward, once you know them 🙂#2021-12-0119:55kingcodeI still see a need for being able to refer to other aliases, as some tools require specific aliases and won’t allow the user to pick several of them at once. So having a single alias referring to a bunch of others (unless one is ready to copy reams of path data across aliases) would be handy in this one use case.#2021-12-0119:56borkdude@kingcode can you give an example of such tools? I think such tools should allow a seqable of aliases#2021-12-0120:02kingcodeYes indeed - I am currently following setup instructions to make CIDER work with shadow-cljs. I haven’t gone through the flow yet, but Cider expects :cider-clj and :cider-cljs aliases, and I currently have a :dev alias with :extra-deps, :extra-paths inside it. So I will need to copy/merge whatever is in :dev alias within :cider-clj/s.
What I would like to do is declare :dev-base-extra-deps, :dev-base-extra-paths aliases, and inside e.g. :cider-clj, have {:extra-deps [:dev-base-extra-deps, <local extra deps>]}#2021-12-0120:04kingcodeIf custom names were allowed outside of :aliases this woud be trivial, as shown in the https://clojure.org/reference/deps_and_cli#_paths#2021-12-0120:05borkdude@kingcode both cider and shadow accept more than one alias?#2021-12-0120:05borkdudeJust use C-u M-x cider-jack-in#2021-12-0120:05borkdudeand then you can specify your extra aliases#2021-12-0120:05kingcodeok thx…sorry to take your time - will try that. In previous attempts CIDER would only give single choices.#2021-12-0120:06dpsuttoncider doesn’t expect any aliases. it constructs its own with its required dependencies and adds them in#2021-12-0120:06borkdudeand in shadow-cljs.edn you can specify {:deps [:foo :bar :baz]}#2021-12-0120:06kingcode?? And in the above example :foo, :ba.. are aliases?#2021-12-0120:06borkdude{:deps {:aliases [:cljs]}
:builds ...}
sorry, had to look that up#2021-12-0120:07kingcodeIf so then inter-alias references are allowed then.#2021-12-0120:07kingcodeWill play with this and get back to you for results 🙂 thx!#2021-12-0120:08seancorfieldThose tools read deps.edn and resolve aliases themselves.#2021-12-0120:08seancorfieldWe do this at work -- treating aliases as data -- so we can have a generic build.clj script in our Polylith monorepo and it can build any of the projects within it.#2021-12-0120:09seancorfieldExample in thread...#2021-12-0120:09seancorfield(doseq [p projects]
(println "\nRunning: uber on" p)
(let [project-dir (str (System/getProperty "user.dir") "/projects/" p)
aliases (with-dir (io/file project-dir) (get-project-aliases))]
(binding [b/*project-root* project-dir]
(bb/clean {})
(bb/uber (assoc (:uberjar aliases)
:compile-opts {:direct-linking true}))
(bb/clean {}))))
pulls the :uberjar alias out of each project's deps.edn file.#2021-12-0120:10seancorfieldand that alias is just data that specifies the :main namespace so compile-clj and uber know what to do in tools.build#2021-12-0120:11kingcodeOK thx! Will look at this in my context and see how I can use it.👍#2021-12-0120:11seancorfield:uberjar
{:uber-file "../../../build/uberjars/api-1.0.0.jar"
:main api.main}
That's the alias from projects/api/deps.edn so the generic build can compile api.main and declare it as the Main-Class in the uberjar and it knows where to create the uberjar file.#2021-12-0120:11seancorfield(defn- get-project-aliases []
(let [edn-fn (juxt :root-edn :project-edn)]
(-> (t/find-edn-maps)
(edn-fn)
(t/merge-edns)
:aliases)))#2021-12-0120:12seancorfieldt is tools.deps.alpha, b is tools.build, bb is build-clj.#2021-12-0120:12seancorfieldAnd with-dir comes from t.d.a in another ns.#2021-12-0120:12seancorfieldLMK if you have further Qs.#2021-12-0120:12kingcodeIs that a public project?#2021-12-0120:13seancorfieldNo, that's from our Polylith monorepo at work.#2021-12-0120:13kingcodeOK np - many thanks for your help!#2021-12-0120:13seancorfieldBut I think Polylith itself has a similar build.clj if I recall correctly from the PR I sent them...#2021-12-0120:14seancorfield...yeah, it does: https://github.com/polyfy/polylith/blob/master/build.clj#2021-12-0120:14kingcodeAh ok I get it - Polylith is public. Am looking at it..#2021-12-0120:14seancorfieldIt has generic jar and uberjar tasks that read alias data from the projects/*/deps.edn files.#2021-12-0120:15seancorfieldI modeled the PR on how we do it at work.#2021-12-0120:15seancorfieldI suspect that may be one of the most "powerful" build.clj scripts out there in OSS land?#2021-12-0120:16borkdudeI think shadow just reads the aliases from the config file and then calls clojure -Spath -A:alias1:alias2 to get the classpath and then continues its course#2021-12-0120:18kingcodeThx @borkdude, will see if Cider can do the same. Thank you both!#2021-12-0120:19borkdudeI looked into the source. It constructs a command line call to invoke a Clojure main program with the aliases in place.#2021-12-0120:20borkdudeI wonder if tools.build changes how we build CLJS programs in the future#2021-12-0215:28richhickey@deactivateduser10790 As frustrated as you might be by the conversation, please avoid ad hominem rhetoric. We don’t talk about other people like that in the Clojure community. https://clojure.org/community/etiquette#2021-12-0202:30jeff.terrellI think I must be missing something obvious here.
I have the "JPam" jar on my classpath:
$ clj -Spath | perl -pe 's/:/\n/g' | grep jpam-1.1
/home/jeff/.m2/repository/net/sf/jpam/jpam/1.1/jpam-1.1.jar
...and in that jar is a net.sf.jpam.Pam class:
$ jar tvf /home/jeff/.m2/repository/net/sf/jpam/jpam/1.1/jpam-1.1.jar | grep Pam.class
3520 Thu Apr 16 11:30:28 EDT 2009 net/sf/jpam/Pam.class
...yet when I try to use it, I get a NoClassDefFoundError:
$ clj
Clojure 1.10.3
user=> (require '[clj-pam.auth :as auth] :reload-all)
nil
user=> (auth/simple-auth "jeff" "hello")
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader/loadLibrary (ClassLoader.java:2429).
no jpam in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
Relevant code (slightly simplified) is:
(ns clj-pam.auth
(:import net.sf.jpam.Pam))
(defn simple-auth [username password]
(.authenticateSuccessful (new Pam) username password))
Am I missing something obvious?#2021-12-0202:33Alex Miller (Clojure team)Looks like jpam is a native lib#2021-12-0202:34Alex Miller (Clojure team)That java.library.path is not classpath, that's native libs#2021-12-0202:38jeff.terrellI do see the word "native" in places that suggest you're right, but I don't really know what that means with respect to the JVM finding what it needs to find.#2021-12-0202:39jeff.terrellThere is a dependency that has "native" in the name:
$ clj -Spath | perl -pe 's/:/\n/g' | grep native
/home/jeff/.m2/repository/org/clojars/canweriotnow/jpam-native-deps/1.1.1/jpam-native-deps-1.1.1.jar
...and that jar has native-looking things in it:
$ jar -tvf /home/jeff/.m2/repository/org/clojars/canweriotnow/jpam-native-deps/1.1.1/jpam-native-deps-1.1.1.jar
0 Thu Mar 07 17:32:46 EST 2013 native/
0 Thu Mar 07 17:32:04 EST 2013 native/linux/
0 Thu Mar 07 17:36:30 EST 2013 native/linux/x86_64/
33015 Fri Jun 08 01:19:18 EDT 2007 native/linux/x86_64/libjpam.so
0 Thu Mar 07 17:36:56 EST 2013 native/linux/x86/
29446 Sat Jun 09 03:32:24 EDT 2007 native/linux/x86/libjpam.so
0 Thu Mar 07 17:32:26 EST 2013 native/macosx/
0 Fri Mar 08 16:58:52 EST 2013 native/macosx/x86_64/
14884 Fri Mar 08 16:58:52 EST 2013 native/macosx/x86_64/libjpam.jnilib
0 Thu Mar 07 17:37:26 EST 2013 native/macosx/x86/
45348 Sat Jun 09 03:50:24 EDT 2007 native/macosx/x86/libjpam.jnilib
How do I get Clojure/the JVM to find these native things?#2021-12-0203:01phronmophobicusually, it can find the right native shared library in the jar based off the OS and architecture. what OS and architecture are you using?#2021-12-0203:19jeff.terrelluname reports x86_64 GNU/Linux, and there is a native/linux/x86_64/libjpam.so file in the JAR. I would think it would be able to find it too, but apparently not?#2021-12-0203:35phronmophobic~oh weird. I know it sometimes checks a few places, but for my native library, the path in the jar is ~ . Actually, ignore this. The path I use is from JNA.#2021-12-0203:36phronmophobicI know there are some setups and builds that will do fancy things like finding and extracting the jar for you.#2021-12-0203:40phronmophobicIs this the library? https://github.com/gregrluck/jpam/blob/master/Linux_x86_64.properties
It looks like they expect some specific setup, https://github.com/gregrluck/jpam/issues/1#2021-12-0203:49phronmophobicActually, I think the way to do it is to extract the shared library from the jar to somewhere on your library path.#2021-12-0203:55phronmophobicfwiw, I've been using JNA via https://github.com/Chouser/clojure-jna/ to interface with c libraries and the c interface for many libs is better than the java wrappers.#2021-12-0216:33jeff.terrellYes, that's the library. Ah, that's very helpful, thanks! I can work with this. Just hadn't come across any native stuff before.#2021-12-0202:40jeff.terrellMaybe I need to go learn more about what exactly "native" means in a Java context...#2021-12-0203:23jeff.terrellTurns out, if I extract the relevant libjpam.so for my architecture and point the java.library.path property to the containing directory, it works. Example:
clj -J-Djava.library.path=.
Seems like there should be a better way. Oh well.#2021-12-0206:23hiredmanMany java projects that use native code do that automatically via a static initializer that writes the shared library to a temp file and loads it from there#2021-12-0206:27hiredmanThis sort of thing https://github.com/xerial/sqlite-jdbc/blob/master/src/main/java/org/sqlite/SQLiteJDBCLoader.java#L189#2021-12-0216:35jeff.terrellThat's very helpful, thanks!#2021-12-0209:07Mateusz MilianHi, a question regarding dependencies:
I have some local libs (jars), which I want to include into my classpath. I do not want to add them one by one (is too many). Is there any way to include all jars from the folder using deps.edn? Something like:
{:deps {lib/lib {:local/root "local/lib/*}}
I know, I can merge jars and create fatJar, but I'm wondering if there is a better option, similar to:
java -cp "local/lib/*" -jar app.jar
#2021-12-0209:18hiredmanThat kind of thing is problematic because clj tries to avoid launching multiple jvms by generating the classpath once, and caching it based on deps.edn contents#2021-12-0209:19hiredmanAnd wild cards are weird there#2021-12-0209:20hiredmanAssuming it is a fairly static list of jars, just write a clojure program to list them and generate a deps.edn containing them all#2021-12-0209:33Mateusz Milianok, thx. This will solve my issue.#2021-12-0213:26pithylessWhen I've got an error like this: Error building classpath. Unable to compare versions for borkdude/sci (one is from git, another from maven); how can I find the projects that are bringing in the dependencies? Normally, I would use -Stree , but that fails in this case.#2021-12-0213:27Alex Miller (Clojure team)Don't know that I have a good answer other than commenting stuff out in the deps#2021-12-0213:28Alex Miller (Clojure team)You can also just add sci as a top level dep#2021-12-0213:28Alex Miller (Clojure team)That overrides stuff below#2021-12-0213:28Alex Miller (Clojure team)Would at least let you run tree#2021-12-0213:29borkdude@pithyless fwiw SCI moved to babashka/sci now as well (to make matters more complicated ;))#2021-12-0213:32pithylessInteresting; adding it as a top-level dep allowed me to override it, but -Stree is no longer returning the other versions so still not sure which one was conflicting#2021-12-0213:33pithylessBut it did push me over the wall, thanks!#2021-12-0215:15Joshua SuskaloThis new license detector is really cool @alexmiller, thanks for the work on that!
Do you think there would be value to adding a complementary command that would list out every dependency for which a license can't be found to make it easier to manually validate them?#2021-12-0215:24Alex Miller (Clojure team)I was debating whether to put something in there (other than nothing) like "unknown" or "not detected" or something, wasn't sure if that was needed or not#2021-12-0215:25Alex Miller (Clojure team)I mean you could :format :edn, slurp, and use filter to list those#2021-12-0215:25Joshua SuskaloThat's true, building a tool that does this that's external isn't hard. I just was thinking it might fit well in the existing tool. I think you could also work out a grep expression to do the same thing.#2021-12-0215:26Joshua SuskaloSo yeah, it's not necessarily a hard problem, it's just up to if you think it fits into the tooling.#2021-12-0215:26Alex Miller (Clojure team)I guess I'd rather wait and see how big an issue it is#2021-12-0215:26Joshua Suskalofair enough#2021-12-0215:28dpsuttonwe recently built this at metabase. It was right before our switch to deps.edn so it is still based on a classpath rather than a basis map. But it is open source so have at it: https://github.com/metabase/metabase/blob/master/bin/build-mb/src/build/licenses.clj#2021-12-0215:31dpsuttonlooks in jars for license information. but i may have to revisit it since it doesn’t find it in Clojure’s own jar. But we have a backfill to list licenses for ones not detected. And in CI we have a test that fails if it finds any that it doesn’t not classify#2021-12-0215:35Alex Miller (Clojure team)yeah, I don't think we want to do any of that, thanks. I suspect you're missing a lot of license info by not using a real pom model that would pull that from parent poms (which the deps one will find)#2021-12-0215:37dpsuttonYeah. I was surprised to hear Clojure’s jar had a license information in it as we had to backfill it. So i’ll revisit that next week. I wasn’t suggesting deps.edn include this but for others who needed a bit more from licensing checks#2021-12-0215:41Alex Miller (Clojure team)clojure has a parent pom, that has license data#2021-12-0215:42dpsuttonI see. I’ll check out how tools.deps uses those classes and adjust accordingly. I also go digging through for license files in the jar and it would be nice if i could abandon that#2021-12-0215:42Alex Miller (Clojure team)actually, no it's in the published pom. is it not in the pom inside the jar?#2021-12-0215:43dpsuttonI think it is in the actual jar. i’m not sure why i miss it navigating through the xml#2021-12-0215:43dpsuttonI cheat and look for a pom adjacent to the jar and then dig in the jar itself. So i should see it in the published pom#2021-12-0215:45Alex Miller (Clojure team)yeah, it's in the pom inside the jar#2021-12-0215:45Alex Miller (Clojure team)so you should find it in either of those places#2021-12-0218:19Jeff EvansI found this old thread, but just to ask again, has anyone managed to implement something akin to Maven dependencyManagement in deps.edn?
The scenario is that we have a number of plugins, each of which might have its own dependencies. But ultimately, these plugins (which are each an uberjar in their own right) are loaded dynamically into the root application, and therefore they all end up in the same classloader. If one plugin depends on a different version of Jackson than another, there could be a clash when both plugins are shoveled into the same classloader.
In Maven-land, I would simply have a parent pom.xml that declares dependencyManagement and puts a Jackson version in there, and each child module can inherit from that pom.
https://clojurians.slack.com/archives/C6QH853H8/p1595987442250200#2021-12-0219:09mpenetyou can achieve this with that tool : https://github.com/exoscale/deps-modules/#2021-12-0219:09mpenetit's updating the deps.edn files from a version file#2021-12-0219:31Jeff Evansthank you! will have a look at it#2021-12-0218:29Alex Miller (Clojure team)there's not really anything akin to that yet. there are various workarounds.#2021-12-0218:29Alex Miller (Clojure team)like generating your deps.edn or overloading parts of it at run time#2021-12-0310:16dergutemoritzHeya, can't seem to find anything in the reference regarding "pom only" maven dependencies (not sure what the proper term is). Specifically, I'm looking for the tools.deps equivalent of this Leiningen dependency: [com.kohlschutter.junixsocket/junixsocket-core "2.4.0" :extension "pom"].#2021-12-0310:19dergutemoritzApparently, the maven equivalent is <type>pom</type>#2021-12-0310:19dergutemoritzAha, just found https://ask.clojure.org/index.php/10892/add-support-for-bom-bill-of-materials-dependencies#2021-12-0310:20dergutemoritzSo that's what these things are called 😅 TIL#2021-12-0310:22dergutemoritzPhew there is a workaround at least. Alrighty then, consider this case closed everyone 🙏#2021-12-0421:20Oliver GeorgeI'm learning about GitHub Actions at the moment. This https://github.com/tonsky/datascript/blob/master/.github/workflows/build_test.yml#L7 taught me that both java and lein are present on ubuntu-latest container. Mentioning as it might be worth petitioning to get the Clojure CLI tools on there too. Here's the details of what they have https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md (a lot).#2021-12-0421:31dharriganHave you seen https://hub.docker.com/_/clojure/?#2021-12-0421:32dharriganI use that in my github actions to build our uberjars#2021-12-0421:32dharrigan(which is fed into docker for containerisation)#2021-12-0421:32dharrigan- name: Setup Clojure
uses: DeLaGuardo/#2021-12-0421:32dharrigan- name: Build the UberJAR
run: |
cd ${{ env.APPLICATION }}
clojure -T:build uberjar#2021-12-0421:33dharriganPerhaps that may be of help?#2021-12-0423:32lread@U11EL3P9U this is probably also of help to CircleCI users, since CircleCI has deprecated their clojure convenience docker image.#2021-12-0508:29dharrigan:thumbsup:#2021-12-0601:53Oliver GeorgeI'm using DeLaGuardo/setup-clojure for the first time. Seems great. Docker images are super useful too (esp. now I know about multi-stage builds). Perhaps these are superior in that we're in better control of what versions we're using. Less chance of stubbing toes on some old stable version of tools deps or feeling blocked when a bug isn't fixed or features slow to appear.
Anyway, just wanted to get the topic on the radar of those who might care and be in a position to act.#2021-12-0510:53dominicmCan I use a basis to get a list of jvm-opts/main-opts? I can see they're under :resolve-args and :classpath-args , but those don't have the right names & make-classpath2 does something very different to get what it calls "exec-args".#2021-12-0514:22Alex Miller (Clojure team)Not currently but this is something I'm going to work on. There are a couple of tools.build tickets where this needed too. (It's incidental that they are under the other keys and you should not rely on that)#2021-12-0514:28dominicmI thought it might be. I'm going to rely on it for now 🙈 with a mind that pack controls the tools.deps version anyway.#2021-12-0511:57dominicmThe naming of clojure.tools.cli.api in tools.deps is a bit confusing, it seems like it should be part of https://github.com/clojure/tools.cli on name alone!#2021-12-0511:59dominicmAre there any naming conventions appearing around tools, apis, etc?
It seems like <project>.api is for functions which take a basis
<project>.cli.api is for functions which can be used from -T, and therefore don't take a basis
Is that right?#2021-12-0514:23Alex Miller (Clojure team)Yes, that's what we've been trying to do#2021-12-0514:25dominicm@alexmiller I shall copy that then! New pack version coming out soon with the new tools.deps, tools.build support etc.#2021-12-0621:14hugodIs it expected behaviour that a top level :jvm-opts in deps.edn is ignored?#2021-12-0621:16Alex Miller (Clojure team)yeah, that's not a feature#2021-12-0621:17Alex Miller (Clojure team)there is an ask question at https://ask.clojure.org/index.php/7834/support-default-jvm-opts?show=7834#q7834 about this if you want to vote on it. (but I think it's unlikely that that will be the final answer)#2021-12-0621:36hugodOK, thanks Alex.#2021-12-0621:36hugodIs it possible to introspect the command line arguments from a running JVM?#2021-12-0622:01Alex Miller (Clojure team)generically, not that I'm aware of#2021-12-0622:02jumarIt's probably better to use an external tool for that - either checking the cmd line or using something like jcmd, JMC, etc.#2021-12-0622:07jumarSince JDK 9, you can also use ProcessHandle if this is what you are after:
(-> (java.lang.ProcessHandle/current) .info .commandLine .get)
EDIT: or this to get only the args
(-> (java.lang.ProcessHandle/current) .info .arguments .get)
#2021-12-0814:09hugodthanks @U06BE1L6T - that’s useful to assert that the jvm options are getting set#2021-12-0621:36borkdudeThere is also a clj-kondo issue about it: https://github.com/clj-kondo/clj-kondo/issues/1269
Feel free to upvote using a thumbs up.#2021-12-0621:37dominicmJust wanted to drop a note that the new :classpath-roots and :classpath apis are fantastic, I've been able to simplify a bunch of code AND better align with how tdeps behaves 🙂#2021-12-0709:11eelkeHello, I am having trouble building an uberjar with clojure.tools.build.api. Do you have an example of a build that includes ClojureScript, js, and css?#2021-12-0709:14eelkeMy build file looks like:#2021-12-0709:15eelke#2021-12-0709:17eelkeI compile all the clj, js into the target folder. Then the jar does seem to contain all the necessary files, but I just do not see the files being loaded into the browser when running the jar#2021-12-0709:17eelkeAny ideas?#2021-12-0710:37borkdudeIn which phase are you running the CLJS build?#2021-12-0710:37borkdudeIf you do that prior, then clean will wipe out all of those compiled files again#2021-12-0711:06eelkethat is not the issue since I can see all the compiled files in de jar. The clean happens before running the cljs build#2021-12-0711:09eelkeHere is ahttps://gist.github.com/eelkevanfoeken/3ef0a6f4519f34956d63ecd0ef38137aof the build.clj#2021-12-0711:09eelkeit is probably something simple I am missing#2021-12-0711:15eelkeI also cannot find a single example of using this method of creating an uberjar that combines Clojure and ClojureScript#2021-12-0711:15borkdudeSo you want to put the uncompiled ClojureScript files into the uberjar?#2021-12-0711:16borkdudeHow does that help you with a browser ?#2021-12-0711:16borkdudeI do not see where you are doing the CLJS build#2021-12-0711:23eelkeok, I will provide you full details at a later moment.#2021-12-0712:20Rachel WestmacottDoes anyone know if it's possible in project A to reference a local project B as a local dependency of A while targeting an alias in the local project B?#2021-12-0712:23borkdudenot possible#2021-12-0713:24Rachel Westmacottnice to have that confirmed - I couldn't see a way. Thank you!#2021-12-0714:37dvingoThere is a question here about it https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects#2021-12-0722:44dominicmNot sure if there's anything to learn here, but I ended up adding :paths ["."] to my :build alias so I could do -M:build and start a REPL.#2021-12-0722:48Alex Miller (Clojure team)yeah, this is a known problem that has been nagging at me. eventually I will get around to tackling it seriously#2021-12-0723:52seancorfieldOur "solution" to that is:
$ clj -M:build -i build.clj -r
#2021-12-0723:53seancorfield(for a REPL that is specifically for running build tasks -- which is how I normally run them -- but I also have :build on my primary development REPL command so I can develop and test build.clj inside my editor -- using the editor's "load file" on build.clj to get it loaded into the REPL)#2021-12-0723:54seancorfieldMy development REPL is started via:
SOCKET_REPL_PORT=5000 clojure -J-Dlog4j2.configurationFile=log4j2-sean.properties -M:rebel:classes:reflect:jedi-time:portal:everything:dev:test:runner:build:dev/repl
#2021-12-0807:42borkdudeI guess one could also just place the file in src?#2021-12-0810:54dominicmThen it wouldn't work with -T#2021-12-0811:01borkdudeand also it would be pulled in if you copied everything from src to your uberjar, it would require the build deps if you were to compile everything in a dir, etc. yeah, that's not good.#2021-12-0813:20borkdudeI think I found an edge case in the clojure CLI (actually via deps.clj because I ported it exactly that way).
Given deps.edn:
{:deps {exoscale/coax {:git/url ""
:git/sha "0d4212af7c07e4f05f74186f05df8a97777b43fe"
:deps/manifest :deps}}}
This library doesn't have a deps.edn in their repo, but also not a pom. But adding :deps/manifest :deps works because the lib has no deps.
Running clojure -Spath doesn't actually cache the classpath since it will check if /Users/borkdude/.gitlibs/libs/exoscale/coax/0d4212af7c07e4f05f74186f05df8a97777b43fe/deps.edn exists (it is in the list of manifest files to check) and if not, it will declare the caching as stale. But this deps.edn file will never be there, because it isn't in their repo.#2021-12-0813:23borkdudeSo instead of:
if [[ ! -f "$manifest" || "$manifest" -nt "$cp_file" ]]; then
stale=true
break
fi
perhaps it should be:
if [[ -f "$manifest" && "$manifest" -nt "$cp_file" ]]; then
stale=true
break
fi#2021-12-0813:25borkdudeAlternatively the :deps/manifest :deps could mandate that the remote repo should have a deps.edn and not accept one without it. (I prefer the first option I think)#2021-12-0813:28Alex Miller (Clojure team)well people sometimes do this to force projects with no deps to work so don't think we want to do the last one#2021-12-0813:28borkdudeexactly#2021-12-0813:29Alex Miller (Clojure team)but also don't think we want to do the first one - this is a helpful check in the case of gitlibs cache going stale (sha deleted etc)#2021-12-0813:29Alex Miller (Clojure team)I think I'd tend toward fixing tdeps to not return the non-existent manifest#2021-12-0813:29borkdudeperhaps the file should not be added to the manifest list then, if it doesn't exist in the repo - indeed!#2021-12-0813:30borkdudedoes tdeps have that info when it compiles this list? then this would be the preferred option for me as well, so far#2021-12-0813:38Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/commit/c815c440e0521647a590a700cf08d6eb9ad8c1f8#2021-12-0902:56dvingoI'm not sure if this is the correct place to ask but...
I just spent over an hour trying to figure out why I got this:
$ clj
Error building classpath. class clojure.lang.Symbol cannot be cast to class clojure.lang.Keyword (clojure.lang.Symbol and clojure.lang.Keyword are in unnamed module of loader 'app')
java.lang.ClassCastException: class clojure.lang.Symbol cannot be cast to class clojure.lang.Keyword (clojure.lang.Symbol and clojure.lang.Keyword are in unnamed module of loader 'app')
at clojure.lang.Keyword.compareTo(Keyword.java:114)
at clojure.lang.Util.compare(Util.java:153)
at clojure.lang.APersistentVector.compareTo(APersistentVector.java:439)
at clojure.lang.Util.compare(Util.java:153)
at clojure.core$compare.invokeStatic(core.clj:842)
at clojure.core$compare.invoke(core.clj:833)
at clojure.lang.AFunction.compare(AFunction.java:51)
at java.base/java.util.TimSort.binarySort(TimSort.java:296)
at java.base/java.util.TimSort.sort(TimSort.java:239)
at java.base/java.util.Arrays.sort(Arrays.java:1441)
at clojure.core$sort.invokeStatic(core.clj:3103)
at clojure.core$sort.invokeStatic(core.clj:3090)
at clojure.tools.deps.alpha$sort_paths.invokeStatic(alpha.clj:565)
at clojure.tools.deps.alpha$flatten_libs.invokeStatic(alpha.clj:573)
at clojure.tools.deps.alpha$make_classpath_map.invokeStatic(alpha.clj:592)
at clojure.tools.deps.alpha$calc_basis.invokeStatic(alpha.clj:738)
at clojure.tools.deps.alpha.script.make_classpath2$run_core.invokeStatic(make_classpath2.clj:128)
at clojure.tools.deps.alpha.script.make_classpath2$run.invokeStatic(make_classpath2.clj:170)
at clojure.tools.deps.alpha.script.make_classpath2$_main.invokeStatic(make_classpath2.clj:224)
at clojure.tools.deps.alpha.script.make_classpath2$_main.doInvoke(make_classpath2.clj:191)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.main$main_opt.invokeStatic(main.clj:514)
at clojure.main$main_opt.invoke(main.clj:510)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
and was digging through tools.deps sources which led me here: https://github.com/clojure/tools.deps.alpha/blob/a4aa25a54b6bc7bb0b87e7e553276cd5b0463417/src/main/clojure/clojure/tools/deps/alpha.clj#L569
and then found out that one of my my libspecs had a : in front of it... and thus that sort was being invoked with something like:
(sort '[[:com.wsscode/pathom-viz lilactown/helix cljs-bean/cljs-bean] [nubank/workspaces cljsjs/highlight])
So the request would be for some better error/sanity checking when building the classpath map, to confirm the data is valid.#2021-12-0902:58hiredmanThere are tools that do linting of deps.edn files already#2021-12-0902:59hiredmanhttps://practical.li/clojure/continuous-integration/github-actions/clj-kondo-lint.html#2021-12-0903:49dvingoThanks, I'm aware of kondo's ability to lint deps files. Personally I experienced performance problems when I used it last (integrating with cursive), but I'll try it again.
I would say though, because t.d.a is such an essential tool when working with clojure I think it makes sense to add basic sanity checking to the config map. Imagine someone new to the language who has a single character typo and as a result sees that stacktrace which gives no clue as to how to proceed or where to look. Seems like a great opportunity to improve the user experience of using clojure.#2021-12-0905:13Alex Miller (Clojure team)thanks, we have specs for the deps file but for historical reasons, they're not being used yet, thanks for the bump#2021-12-0913:38jmayaalvhi! is there a way to set the jvm version clojure cli uses? didn’t manage to find anything in the docs.#2021-12-0913:39Alex Miller (Clojure team)It uses whatever Java version you have on the path #2021-12-0913:40Alex Miller (Clojure team)So totally up to you :)#2021-12-0913:41jmayaalvgot it! thank you alex!#2021-12-0921:51Kira McLeanCan I use a project that uses leiningen as a local dep (i.e. is there some way to tell tools.deps the manifest type is a project.clj file?#2021-12-0921:54Alex Miller (Clojure team)currently, no but there is a pending ticket/patch for this#2021-12-0922:02Kira McLeancool! thanks 🙏#2021-12-0922:46borkdude@kiraemclean A workaround could be to add the deps of that project to your project and then include that project using {:git/url ... :sha ... :deps/manifest :deps}#2021-12-0922:46borkdudeor :local/root as you asked#2021-12-0922:47borkdudebut since you already have that project locally, you can also use lein pom to write a pom.xml and then tools.deps will understand it without any other changes#2021-12-0923:26Kira McLeanah amazing, this is very helpful, thanks 🙂 relatedly.. need to learn better debugging skills for Clojure. I’m trying to debug a weird issue and want to dig around the source of a library but finding it hard with just my editor 😭#2021-12-1012:06rickmoynihanJust give me a shout if you want some hands on help or a pairing session#2021-12-1012:10rickmoynihanIn light of the Log4shell zero-day ( https://www.lunasec.io/docs/blog/log4j-zero-day/ )
Does anyone have any advice on getting automated CVE dependency checks for tools.deps based projects?#2021-12-1012:17pavlosmelissinoshttps://github.com/rm-hull/nvd-clojure is quite nice (formerly lein-nvd)#2021-12-1018:06seancorfieldI have it installed as a CLI "tool" so I can run it on any project: https://github.com/seancorfield/dot-clojure#basic-tools#2021-12-1101:30ChaseIs there a way to send something like a watch command to the cognitect-labs test-runner using the cli so that whenever I save code it runs the tests automatically in the background?#2021-12-1102:43seancorfieldIt would be fairly easy to create a watcher in the REPL that could kick off tests...
$ clojure -Sdeps '{:deps {juxt/dirwatch {:mvn/version "RELEASE"}}}' -M:rebel
[Rebel readline] Type :repl/help for online help info
user=> (require '[juxt.dirwatch :refer [watch-dir]] '[ :as io])
nil
user=> (watch-dir println (io/file "."))
#object[clojure.lang.Agent 0x32f5ecc4 {:status :ready, :val #object[sun.nio.fs.LinuxWatchService 0x7ef8eda7 "#2021-12-1102:45seancorfieldThe Cognitect test-runner has a programmatic -X entry point that you could invoke from a watcher.#2021-12-1102:47ChaseOk, cool, I'll look into that option. Looks like koacha has a watch command too if I want to explore that one. These exercism exercises come with the cognitect one though so I figured I would explore that#2021-12-1102:49seancorfieldYou'd probably want to run it in a thread and keep track of when it is running (so you don't run tests in parallel) but also keep track of changes so you know to run it again if changes came in while the previous run was in progress...#2021-12-1102:50seancorfieldReally though you probably want incremental testing to make watching worthwhile...#2021-12-1102:52seancorfieldFrankly, I don't like that workflow. I prefer to run tests via hotkeys in my editor all the time I'm editing (and evaluating) code. I have a hot key bound to, essentially, "find the current ns's tests and run them" and another hot key bound to "run the tests in the current ns" (to allow for whether I'm in a code file or a test file.#2021-12-1102:53seancorfieldBut my workflow is all in the REPL. I don't switch to a terminal for any of that. Edit, eval, edit, eval, test. A lot of it without saving files 🙂#2021-12-1102:54ChaseThat's what I've been currently doing as well and it works great. But I like to tinker with my tooling and workflow every now and then#2021-12-1103:09ChaseBy that I mean I was somehow getting too lazy to even type ,tn lol#2021-12-1215:03practicalli-john@U9J50BY4C I've added a :test/watch alias that runs koacha in watch mode (with fail-fast) to https://practical.li/clojure/clojure-tools/install/community-tools.html, so I can run quickly run koacha with any Clojure project.
clojure -M:test/watch
I like to run koacha before pushing commits to a project as a sanity check
https://practical.li/clojure/testing/test-runners/kaocha-test-runner.html#2021-12-1102:07Alex Miller (Clojure team)Nothing in test-runner but you can use os watch stuff for that#2021-12-1214:48weavejesterA quick question: use of -m has been deprecated, but what’s the replacement? -M only seems to work with aliases, so I’m not sure how to idiomatically specify an explicit main function at the command line.#2021-12-1214:51Alex Miller (Clojure team)-M -m#2021-12-1214:52Alex Miller (Clojure team)clj -M -m my.main#2021-12-1214:52weavejesterAh, thanks!#2021-12-1314:41vemvHi,
one thing that I "know"/assume is that clojure -Ttools execution is not affected by any deps.edn file present in the $PWD, is that correct?
i.e. any dependency or other piece of data in $PWD/deps.edn should not affect the classpath that clojure -Ttools will use
If so, can one count on that as contractual behavior?#2021-12-1314:42vemvContext: I want to make -Ttools one of the very few recommended ways to run https://github.com/rm-hull/nvd-clojure which is very much classpath-sensitive#2021-12-1314:47Alex Miller (Clojure team)if you use -T, that will remove the :deps and :paths from the deps.edn in building the classpath#2021-12-1314:49Alex Miller (Clojure team)-Ttool runs with a classpath that is based on :paths ["."] and :deps {<the-tool-as-dep>}#2021-12-1315:06vemvYes it's in the right direction
paths . might be not exactly ideal though, as it seems to augment the classpath. I just want a program that never has extra deps/paths, regardless of $PWD.
i.e. its classpath should be always the same.
Would it be possible to not set the :paths at all? With an option perhaps?#2021-12-1315:07Alex Miller (Clojure team)it's not right now#2021-12-1315:10vemvAlright, in that case for the time being, -Ttools will be one of the recommended APIs for nvd-clojure but not the only one.
Would gladly revisit the topic at a later point (and if you are so generous, would appreciate being pinged about it if there are any changes in this area)#2021-12-1315:31Alex Miller (Clojure team)yeah, I'm not going to remember that sorry :)#2021-12-1314:49Alex Miller (Clojure team)so, I believe this is what you want#2021-12-1407:38vemvDifferent topic, any user of cider-nrepl (whether it's from Emacs or e.g. Calva, vim-iced) would like to take on a smallish isolated task?
I would be super grateful, it escapes a little bit my area of expertise / what I can QA myself.
It boils down to wrapping up (perhaps slightly reworking) this PR, its author is on PTO (for a good reason 🎉) https://github.com/clojure-emacs/enrich-classpath/pull/8
Can explain context/requirements as needed.#2021-12-1412:19practicalli-johnI assume the -P flag is exactly the same as using -X:deps prep argument with the clojure command. Are there any differences?
Does -X:deps prep work with other aliases just as -P does (I should probably test this myself anyway 🙂 )#2021-12-1413:15Alex Miller (Clojure team)No, those are different things entirely#2021-12-1413:16Alex Miller (Clojure team)-P does not prep dependent libs (although it will fail and tell you to prep them)#2021-12-1414:37practicalli-johnBy dependent libs, do you mean the libraries that a dependency entry in a deps.edn file depends upon?
(a library that is not explicitly defined in the deps.edn file, but is a dependency of a library that is defined as a dependency)?
I feel I am still missing something in my understanding, especially the purpose / value of the -P flag.#2021-12-1414:40Alex Miller (Clojure team)maybe more clearly, -P does not prep libs#2021-12-1414:42Alex Miller (Clojure team)-P predates that and was designed to be a flag you could prefix to whatever command you plan to run (say, in a container) to trigger all of the download and classpath construction without actually running the command#2021-12-1414:44practicalli-johnand hopefully my last annoying question, what does it mean to prep libs?
Is that in relation to buildng jars from Git repos and local/root dependencies? I am not sure otherwise.#2021-12-1414:45Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#prep#2021-12-1414:45Alex Miller (Clojure team)preparing source-based libs to be on the classpath if needed#2021-12-1414:49practicalli-johnOkay, that makes sense, thanks for clearing that up. Seems like there is lots more to read up on that page too 🙂#2021-12-1414:55Alex Miller (Clojure team)this particular point is certainly confusing and still not entirely resolved from a design perspective#2021-12-1415:03practicalli-johnMy initial confusion was that they were both described as Prepare in the help and I didn't realise what an unprepared lib actually was (until now).#2021-12-1415:06Alex Miller (Clojure team)yes, that is the problem with using the same word for different things :)#2021-12-1415:18Alex Miller (Clojure team)but "prepare" is such a useful word :)#2021-12-1416:56practicalli-johnI just wasnt prepared for that 🙂#2021-12-1521:18Karol WójcikHow can I set a different JVM version to use with clojure command?#2021-12-1521:33Karol WójcikTHX!#2021-12-1521:23hiredmanyou can see how it finds java here https://github.com/clojure/brew-install/blob/1.10.3/src/main/resources/clojure/install/clojure#L179-L190 basically in order it looks at JAVA_CMD, java in your PATH, and JAVA_HOME#2021-12-1721:27kennyFollow up on TDEPS-207: I switched my environment from Ubuntu to Mac (originally reported the thread dump from Ubuntu) and am still experiencing a deadlock on clojure executions on v1.10.3.1040.#2021-12-1721:33hiredmanthis is almost certainly not it, but you could try a require of the s3 transport at the top like here https://github.com/clj-easy/graal-build-time/blob/b91cd78a977fdf81758705668f44b0d2ddad4bdc/build.clj#L1-L2#2021-12-1721:33hiredman(I guess that assumes tools.build, since it goes at the top of your build.clj, and maybe you aren't using that)#2021-12-1721:52Alex Miller (Clojure team)I think that's a different issue. I haven't done anything to address TDEPS-207 yet, so not surprising that it's still not working. I've got some time next week so will probably be doing some work on tdeps then.#2021-12-1815:43practicalli-johnI'm writing a few introductory articles on Clojure CLI tools to help people get started (and as a refresh for myself)
I would appreciate any https://github.com/practicalli/blog/pull/105 before I publish them to the production blog
https://practical.li/blog-staging/posts/clojure-cli-tools-understanding-aliases/
https://practical.li/blog-staging/posts/clojure-which-execution-option-to-use/ (I have a few tweaks to do with the section on -T tools)#2021-12-1918:38ChaseSeems like good intros to me. The current simple-api-server project is bit out of sync with the docs but it's super minor (the article describes using depstar to build an uberjar but the project template has now switched to build-clj). Maybe a mention that the created template project shows examples of how to use the aliases in the created readme would help some who are hands on explorers.#2021-12-1918:38ChaseMaybe an explanation of what to do if your -main function takes any arguments would be nice.#2021-12-1918:38ChaseI'm also still unclear on what :main-opts could be.#2021-12-1918:40ChaseSo any function called using -X or with :exec-args has to take a map in as it's only argument right? So is there anyway to run an arbitrary function without that requirement?#2021-12-2314:58Alex Miller (Clojure team)sorry, just getting around to reading these. some notes on https://practical.li/blog/posts/clojure-which-execution-option-to-use/:
• -A is not deprecated (it's the only way to provide aliases when starting a repl)
• git-resolve-tags should be considered deprecated (doesn't really make sense with the new git support), might just be better not to mention it as I plan to remove it
• In addition to clojure --help you can also man clojure - over time man is likely to grow more expansive than the short help
• "directive providing a specific name for the too" - typo at the end
• clojure -M:project/hotload:env/dev:lib/cider -P - this won't work, -M -X and -T all scarf all subsequent arguments, so you need clojure -P -M:project/hotload:env/dev:lib/cider#2021-12-2315:04Alex Miller (Clojure team)filed an issue with the above for tracking#2021-12-2413:39practicalli-johnFixed, thank you.
I was a little confused about -A, but then though of an example, so added a specific section about -A to clarify
I use Rebel Readline to run an interactive REPL in a terminal, so haven't used the -A option in a long time.#2021-12-2013:38pavlosmelissinosIs there a way to tell the Clojure CLI tools to start a REPL and run some code in it as soon as the REPL is up?
-X runs a function but starting REPLs is associated with the -A flag, so I'm not sure what the right approach is here.#2021-12-2013:38pavlosmelissinossome context:
I'm not sure how common my workflow is but when working with clj+cljs, I start two REPLs and then I have to go to the clj REPL, load the dev namespace and run something like (dev/go) in it to start the server#2021-12-2013:41Michael WI do that with shadow-cljs, and it has a setting: :builds {:main {:modules {:main {:init-fn some.namespace/some-fn}}}}#2021-12-2013:51pavlosmelissinosOh so the frontend triggers the backend? Sounds a bit unorthodox to be honest but I'll consider it if there's no CLI tools-related solution, thanks.#2021-12-2013:52Michael WI'm not sure about how common it is but I like not having to start 2 repls, shadow-cljs does it for me, and with vscode and calva I can switch between the clj and cljs repls easily.#2021-12-2110:19practicalli-johnClojure code can be run when starting a REPL by providing a user.clj file with user namespace on the classpath.
https://practical.li/clojure/clojure-tools/projects/configure-repl-startup.html#2021-12-2013:58Alex Miller (Clojure team)You can make your own main that does whatever you want then start the repl#2021-12-2013:59borkdude$ clj -M -e "(clojure.main/repl :init (fn [] (require '[clj-kondo.core :as clj-kondo])))"
#2021-12-2013:59Alex Miller (Clojure team)https://insideclojure.org/2020/02/11/custom-repl/ is a different but similar use case#2021-12-2014:01borkdudeor: $ clj -M -e "(require '[clj-kondo.core :as clj-kondo])" -r#2021-12-2014:36pavlosmelissinosRight, that makes sense. I'll read up a bit, thanks!#2021-12-2110:19practicalli-johnClojure code can be run when starting a REPL by providing a user.clj file with user namespace on the classpath.
https://practical.li/clojure/clojure-tools/projects/configure-repl-startup.html#2021-12-2223:35favilaCan anyone point me to some build tools in the tools.deps.edn ecosystem that prepares “lib directory of jars”-style deployment artifacts instead of an uberjar?#2022-12-2721:04slimslenderslacks@U09R86PA4 I've been working on a tool for creating docker images that uses the lib directory of jars approach (https://github.com/atomisthq/jibbit). I think of this as the approach that I first saw with https://github.com/orb/lein-metajar where all the jars were unpackaged, and then a ClassPath manifest was created. I did something similar to what Alex Miller mentioned: https://github.com/atomisthq/jibbit/blob/main/src/jibbit/core.clj#L85-L114#2022-12-2721:05slimslenderslacksIterate over the :classpath-roots in the basis to copy the files, and also to produce the Class-Path manifest entry.#2022-12-2721:07slimslenderslacksI keep the libs and aot compiled parts on separate layers because the libs tend to change at a slower rate, and I can often cache that layer for quite a while.#2022-12-2816:01favila@U0143KP47M4 thank you. My goals are along the same lines, not for docker per se but for smaller cacheable units for faster builds and deploys, and possibly also sharing aot-ed code for faster local dev startup. Also I’ve hit many edge cases related to uberjarring (resource merging, sealing, multi-release jars, etc) and I am just done fighting that.#2022-12-2818:10slimslenderslacksYa, the resource-merging problem had been a real headache for me too. Thanks for bringing up the multi-release and sealed jar issues too. I hadn't considered those cases but that's really interesting.#2021-12-2300:06Alex Miller (Clojure team)can you explain what that means?#2021-12-2300:07Alex Miller (Clojure team)just collecting all the transitive jars without rebundling them?#2021-12-2300:08favilaYes#2021-12-2300:09Alex Miller (Clojure team)if you use create-basis, it has paths to all the libs in it, you could combine that with copy-file#2021-12-2300:09hiredmanhttps://gist.github.com/hiredman/d68cafb6aa8cea563c7b77d54f522421 does something like that. Uses sftp to copy jars and dirs (git deps) to a remote system and generates a premade classpath file, but it was never fully developed and is based on an older tools.deps#2021-12-2300:11hiredmanOh, and it hashes everything and skips copy anything that already exists#2021-12-2300:16Alex Miller (Clojure team)(let [target "target/libs"]
(doseq [[lib coord] (-> (b/create-basis nil) :libs)]
(doseq [path (:paths coord)]
(b/copy-file {:src path
:target (str target "/" (.getName (jio/file path)))}))))
something like that#2021-12-2300:17Alex Miller (Clojure team)if you care about local/git libs, you'd want to be a little more careful what you copy out of :paths#2021-12-2315:00markaddlemanI'm using https://github.com/seancorfield/build-clj (thanks, @seancorfield!) to create an AOT uberjar. I'm using clojure 1.11.0-alpha3. When I compile the uber jar, I get a couple of reflection warnings and several warnings like WARNING: update-keys already refers to: #'clojure.core/update-keys in namespace: clojure.tools.analyzer, being replaced by: #'clojure.tools.analyzer.utils/update-keys. I'm not concerned about any of these warnings. When I run my application using java -jar uber.jar , I get the same set of warnings as the application starts up. This surprises me. I expect the warnings during AOT compilation but I do not expect them when running as a set of AOT classes. Maybe my understanding of how executing an AOT jar is wrong. Is this behavior expected?#2021-12-2315:14borkdude@markaddleman I think there is a new version of tools.analyzer which solves these warnings. they are caused by same-named vars in clojure.core in clojure 1.11#2021-12-2315:14borkdudethey should not affect the behavior of your program though#2021-12-2315:15markaddlemanThanks. I think analyzer is getting pulled in transitively (probably through core.async but I haven't investigated)#2021-12-2315:16markaddlemanBut, that's not really my worry. I'm wondering why I'm getting these warnings when executing an AOT jar. I expected that these warnings occur when loading the namespace for compilation but the resulting bytecode would not re-evaluate.#2021-12-2315:18borkdudevars in clojure are still dynamically created at load time, even when source has already been AOT-ed#2021-12-2315:19markaddlemanOh, duh, of course. The warnings are not about the namespace loading per se, they are about creating the vars. Thanks!#2021-12-2315:25Alex Miller (Clojure team)that is all correct, and most recent core.async / tools.analyzer.jvm will fix the warnings#2021-12-2319:10Alex Miller (Clojure team)https://clojurians.slack.com/archives/C015AL9QYH1/p1640286580108400#2021-12-2319:12Alex Miller (Clojure team)(note: version above was supposed to be 1.10.3.1053)#2022-12-2612:23practicalli-johnFrom a user point of view, I have a feeling I am missing some understanding in regards to -T tools approach.
I appreciate that tools makes it more explicit that the tool is separate from any Clojure project configuration, so an advantage for tool developers.
However, the benefit seems less obvious for the user over using -X with a user wide alias or -T with the same user wide aliases.
Using an alias in the user wide deps.edn file, I can run a tool without having to install it (although I do have to create the alias initially). I can create explicit names for specific functions from the tool and define my own defaults.
I can version the user wide deps.edn file and so is easy to have a wide range of tools immediately available with a clone command.
I can also update the library versions for these tools with antq. This seems to save a lot of time installing and updating individual tools.
Perhaps its the install aspect of tools I am unclear on.
I can list tools that are installed and use my own name for a tool (although I still have to specify the specific function to use from that tool.
If I put the .clojure/tools directory under version control, is this enough to 'install' tools if I clone this onto another computer? Does clojure do anything else when installing a tool?
It only seems possible to set default values for a tool in an alias (assuming the tool itself doesn't have the defaults I want). For example, I may want to set a default template when using clj-new or deps-new.
I tried adding :exec-fn and :exec-args to the .clojure/tools/toolname.edn file for an installed tool, but this does not seem to be passed to a tool. Maybe its tool specific
I wonder if tools have a more obvious advantage if they have a number of functions that can be called. The more functions a tool has, the more need for a flexible command line approach.
For example, if I have a tool called project-automation that I install as projects
Then I can call new, test, build, outdated-deps, uberjar, deploy-jar, etc. However, this is a composite of multiple tools each of which are currently installed using an individual name for each tool.
At the very least it seems prudent to update the practicalli/clojure-deps-edn documentation to use -T for those aliases that should be run as an isolated tool, so the do not pick up project paths and deps.
Any further insight or documentation I should read is appreciated. Thanks.#2022-12-2613:38Alex Miller (Clojure team)The only thing install does is put the tool file there#2022-12-2613:39Alex Miller (Clojure team)I think the part you're missing is that tools can publish some config in their own deps.edn under :tools/usage#2022-12-2613:40Alex Miller (Clojure team)Right now, only :ns-default and :ns-aliases#2022-12-2613:41Alex Miller (Clojure team)But that may expand in the future#2022-12-2620:04seancorfield@jr0cket I have my tools folder under version control - it's in my dot-clojure repo, and that's how I keep installed tools in sync across my machines. Nothing else is needed.#2022-12-2709:44pezIs there a way I can find all the command line options and all the config options for tools.deps? Specifically I am looking for some way to add repl-opts in an alias config, rather as command line arguments. My google fu does not suffice.#2022-12-2718:07seancorfield@U0ETXRFEW Can you elaborate? What are you looking for that isn't in the docs? And do you mean t.d.a or the CLI?#2022-12-2817:02practicalli-john@U0ETXRFEW There is a summary and example here (assuming I understand the question). I am not sure what you specifically mean by repl-opts, as far as I know that doesnt exist
https://practical.li/blog/posts/clojure-cli-tools-understanding-aliases/
The full extent of arguments that can be passed also depends on what Clojure code you are calling#2022-12-2718:49vinnyataidehello, I'm not sure this is the correct channel but I'm receving this message when I add tools.namespace to my deps version 1.2.0.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See for further details.
#2022-12-2718:53vinnyataidethe strange thing is this started out of nowhere, I was using it before normally#2022-12-2719:09vinnyataidefixed by adding com.fzakaria/slf4j-timbre {:mvn/version "0.3.21"} to my deps#2022-12-2816:57practicalli-john@vinnyataide If you dont need to use timbre, then you can explicitly add the following dependency to your project deps.edn file to set the no-op(eration) logger implementation
org.slf4j/slf4j-nop {:mvn/version "1.7.32"}#2022-12-2920:27vemvare -Ttool s best referred to as "tools", "named tools" (per https://clojure.org/reference/deps_and_cli#_using_named_tools), sth else...?#2022-12-3016:18Darin Douglasswhat’s best-practice on clojure(script) as :deps in libraries? we’ve run into a case where, because clojurescript is packaged in the library’s pom, data.json breaks when serializing instants:
❯ clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.4.0"} com.yetanalytics/colossal-squuid {:mvn/version "0.1.3"}}}' ;; this includes both `clojure` and `clojurescript` (with the latter being the problem) in its `:deps`
Clojure 1.10.3
user=> (require '[clojure.data.json :as json])
nil
user=> (json/write-str (java.time.Instant/now))
Execution error at clojure.data.json/write-generic (json.clj:385).
Don't know how to write JSON of class java.time.Instant
user=>
i started looking around at clojure/script libraries a bit and didn’t see anything super conclusive, though of the cljc libraries i looked through, colossal-squuid is the only one with a hard dep on clojurescript.
should clojure(script) be treated as “provided” deps in libraries and just assumed to provided by the app? should apps (per colossal-squiid’s recommendation) just :exclusion what it doesn’t need?#2022-12-3016:53favilaBack when I was maintaining cljc libs, I would build different artifacts (maybe using classifiers? don’t remember) for each. This was definitely not a mainstream approach though, so for other people’s libs I just checked the transitive dependencies carefully and used exclusions.#2022-12-3016:54favilaI think the fundamental lesson I took away from that experience was that even though you share a src folder doesn’t mean you share a deps.edn/project.clj/pom.xml#2022-12-3016:54favilathe same code has different sets of dependencies. Since dependency declaration in the maven system is tied to artifacts, that means different artifacts is the least-friction approach#2022-12-3017:08Darin Douglassya, when the different flavors of the library have completely different dependency sets it makes sense to have different artifacts#2022-12-3016:36Alex Miller (Clojure team)I assume this is because of the bundled older data.json in Clojurescript? I'd say that's really the core problem and both those suggestions are valid workarounds. Really cljs should probably be shading that dep to avoid this issue.#2022-12-3016:38Darin Douglassthat’s kinda what i thought as well, though -Stree shows that old version being dropped
❯ clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.4.0"} org.clojure/clojurescript {:mvn/version "1.10.879"}}}' -Stree
org.clojure/clojure 1.10.3
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56
org.clojure/data.json 2.4.0
org.clojure/clojurescript 1.10.879
. com.google.javascript/closure-compiler-unshaded v20210505
. org.clojure/google-closure-library 0.0-20201211-3e6c510d
. org.clojure/google-closure-library-third-party 0.0-20201211-3e6c510d
X org.clojure/data.json 0.2.6 :use-top
. org.clojure/tools.reader 1.3.3
. com.cognitect/transit-clj 0.8.309
. com.cognitect/transit-java 0.8.332
. com.fasterxml.jackson.core/jackson-core 2.8.7
. org.msgpack/msgpack 0.6.12
. com.googlecode.json-simple/json-simple 1.1.1
. org.javassist/javassist 3.18.1-GA
. commons-codec/commons-codec 1.10#2022-12-3016:39Darin Douglassalso in the repl, the clojure/data/json.clj resource apparently points at the right 2.4.0 jar ¯\(ツ)/¯#2022-12-3016:39Alex Miller (Clojure team)Clojurescript bundles it in its jar so tools.deps can't know about it or make decisions about it#2022-12-3016:39Darin Douglassah#2022-12-3016:39Alex Miller (Clojure team)Depending how the classpath sorts, you can see either#2022-01-0420:27hlshipAre the rules for mapping project names to GitHub URLs extensible? A lot of our libraries are in an internal GitHub repo, and it would be nice to omit the :git/url tag, as it could maybe be computed from the fully qualified artifact name.#2022-01-0420:52pavlosmelissinosI'm pretty sure it doesn't matter that the repos are private as long as they're on http://GitHub.com or on any other of the supported hosts/domains. :thinking_face:
Or are you asking a different question?#2022-01-0420:37hlshipThat is, a way to extend the rules listed here: https://clojure.org/reference/deps_and_cli#_coord_attributes#2022-01-0420:59Alex Miller (Clojure team)no#2022-01-0423:04kennytiltonHmmm. I am used to project.clj where I can do a lein deps to check and pull down dependencies. Now working from a deps project I am in deps land, where I do npm i and npm start. That does not seem to pull down the jar I referenced in deps.edn: Clojure says it cannot find a certain path: The required namespace "tiltontec.model.core" is not available, it was required by "app/core.cljs".
That path is my own project, so I can lein install and all goes well.
My build approach just running npm must be daft. This is my deps.edn:
{:paths ["src" "src/demo" "src/mxreact"]
:deps {lilactown/helix {:mvn/version "0.1.1"}
binaryage/devtools {:mvn/version "0.9.7"}
thheller/shadow-cljs {:mvn/version "2.15.1"}
tiltontec/matrix {:mvn/version "4.1.7-SNAPSHOT"}
cljs-http/cljs-http {:mvn/version "0.1.46"}}}
Is there a deps counterpart to lein deps?#2022-01-0423:09slimslenderslacksAre you building a clojurescript project here?#2022-01-0423:13seancorfieldclojure -P should be all you need.#2022-01-0423:14seancorfield(but make sure you're using a recent enough version of the CLI -- check clojure -version and then check https://clojure.org/releases/tools )#2022-01-0423:14seancorfieldNot sure why you're referring to npm here tho' @U0PUGPSFR?#2022-01-0423:17slimslenderslacksIf you're using shadow-cljs to start up the process to compile your clojurescript, maybe your sample project is using deps.edn to manage your clojurescript deps. So you might have both a shadow-cljs.edn and a deps.edn in your project?#2022-01-0423:19kennytiltonYep.
;; shadow-cljs configuration
{:deps true
:builds
{:app {:target :browser
:output-dir "js"
:asset-path "/js"
:modules {:app {:entries [app.core]}}
:devtools {:http-root "."
:http-port 8888
:reload-strategy :full
:preloads [devtools.preload
demo.dev]}}}}#2022-01-0423:22kennytiltonHeh-heh, @U04V70XH6: monkey see, monkey do. I think I started with a Helix ReactNative demo, converted that into a Matrix RN project, and it went so well I am also converting to a Matrix+React (web) project.#2022-01-0423:23seancorfieldclojure -P = lein deps; clojure -Stree or clojure -X:deps tree = lein deps :tree (I believe)#2022-01-0423:24seancorfield(but, like I say, check you're using an up-to-date version)#2022-01-0423:24kennytiltonI just installed CLI fresh because I upgraded to OS X Monterey and when I tried building sth it said "Go to Hell, you need all new everything." That was three hours ago. But I will double check the version. Thx!#2022-01-0423:26kennytiltonOK. let me blast Matrix out of .m2 and see how far I get. Delighted btw that the Clojars deploy no longer requires all that GPG stuff. That had me kneecapped for months.#2022-01-0423:26hiredmanif you are doing stuff with npm, you aren't doing stuff with tools.deps except via a third party tool, which would be shadow js here#2022-01-0423:27hiredmanso your issue is going to be with shadow js, so you should find the shadow js channel and ask there#2022-01-0423:33kennytiltonClojure CLI version 1.10.3.1040 Trying clojure -P next. But, yeah, @U0NCTKEV8, I will check with #shadow-cljs to see what I am misisng. 🙏#2022-01-0423:37kennytiltonYer a genius, @U04V70XH6!
$ clojure -P
Downloading: tiltontec/matrix/4.1.7-SNAPSHOT/maven-metadata.xml from clojars
Downloading: tiltontec/matrix/4.1.7-SNAPSHOT/matrix-4.1.7-20220104.224346-2.pom from clojars
Downloading: tiltontec/matrix/4.1.7-SNAPSHOT/matrix-4.1.7-20220104.224346-2.jar from clojars
And after npm start... [:app] Build completed. (177 files, 14 compiled, 0 warnings, 6.88s)
Hellasweet. Now let's see how I can tweak my Shadow config to do that.
Thx all! 🙏#2022-01-0423:11hiredmannpm has nothing to do with tools deps and deps.edn#2022-01-0500:10kennytiltonI started from https://github.com/lilactown/helix-todo-mvc, @hiredman. That manages to put up a web page supported by deps.edn and shadow-cljs.edn via npm i and npm start. It also works now that I have ripped out Helix and substituted Matrix. npm start seems to lead to npx shadow-cljs watch app via package.json:
{
"private": true,
"scripts": {
"start": "npx shadow-cljs watch app"
},
"dependencies": {
"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-refresh": "^0.8.1",
"react-router-dom": "^5.1.2",
"react-slider": "^1.3.1",
"shadow-cljs": "^2.8.94",
"todomvc-app-css": "^2.4.1",
"todomvc-common": "^1.0.5"
}
}
I wonder if I should have add the Matrix dependency there? ^^^ Though I should think shadow would find its way to deps.edn thanks to the :deps true in shadow-cljs.edn. I will see what the shadow crew thinks. Thx! 🙏#2022-01-0500:20seancorfield@hiskennyness package.json is going to be just NPM dependencies, not Clojars/Maven. deps.edn is going to be just Clojars/Maven (unless addition repos are declared in the deps.edn file).#2022-01-0500:22seancorfield(as for what, if anything, Shadow-cljs does with the deps.edn file, you'll have to ask in #shadow-cljs)#2022-01-0500:35slimslenderslacksfwiw, I think npx shadow-cljs watch app will start up a jvm pretty much like what you'd get with clj -M:app and an alias of
{:aliases
{:app
{:extra-deps {org.clojure/clojure {:mvn/version "1.10.3"},
org.clojure/clojurescript {:mvn/version "1.10.866"},
thheller/shadow-cljs {:mvn/version "2.14.2"}},
:main-opts ["-m" "shadow.cljs.devtools.cli" "watch" "app"]}}#2022-01-0506:06thhellerthat is indeed very similar with the exception of server mode as described here https://code.thheller.com/blog/shadow-cljs/2017/11/18/the-many-ways-to-use-shadow-cljs.html. clj will always start a new JVM and never take advantage of an already running instance.#2022-01-0506:07thhellernote that nowadays the shadow-cljs command just uses a precompiled AOT dependency and does not AOT on demand anymore. you could get the same with clj using thheller/shadow-cljs$aot I believe#2022-01-0511:21scarytomI have a question about packaging libraries, and I hope this is the right channel to ask it in.
Say I am building a Clojure library that happens to have some Java bits (e.g. some interfaces or some AOT compiled classes). My build process will do the compilation steps and create a Jar containing the Java classes and Clojure clj files. I will then push that to Clojars or Maven Central, and other people can depend on my library. If my lib depends on other libraries, expressed as :mvn/version deps in my deps.edn file then, when I build my library's jar, I create a pom file that expresses these dependencies and downstream users of my lib get these "transitive" deps pulled in automatically.
Now what if my library has a dependency on something that is only packaged as a git ref, described in my deps.edn file as, for example {io.github.clojure/tools.build {:git/tag "v0.7.4" :git/sha "ac442da"}. How does this get expressed in my pom file? How do downstream users of my lib get that dependency automatically?#2022-01-0511:24scarytomI should add that the downstream consumers of my lib might be Java codebases using maven not Clojure codebases using tools-deps#2022-01-0511:44rickmoynihanEssentially you can’t.
If you’re publishing artifacts to maven all transitive dependencies MUST also be published in maven.
However tools.build is a build tool; so downstream users typically won’t need that as a dependency… i.e. that’s normally something only you need to build/deploy your project. Consumers won’t typically need it; unless you were also distributing build tools.#2022-01-0511:47rickmoynihanThere are some workarounds, e.g. you can package the git/deps into your libraries jar…
However you need to be very careful if you do that, as it could cause classpath hell for consumers if someone else were to also do the same, as it would be hidden from tools.deps classpath resolution, and it would be hard for consumers to control which version they got.#2022-01-0512:19scarytomtools.build was just an example dep, sorry if that was distracting.#2022-01-0512:19scarytomthat workaround sounds undesirable to me#2022-01-0512:22scarytomI guess the other alternative is to package the git/dep into a jar myself and publish to my own maven repo under the same group/artifact, so that resolution conflicts between that artifact and other direct use of the git/dep are highlighted.#2022-01-0512:22scarytombut that doesn't work if I want to make my lib public#2022-01-0512:23scarytomI think this is a fairly horrible consequence of using git/deps. If more people choose to package their clojure libs in this way it will make it very hard to create java-friendly libs downstream.#2022-01-0512:28rickmoynihanYes; the best thing to do is either:
1. contribute a maven deployment to the upstream project
2. fork it and release under a coordinate you control; but also repackage the namespaces into a new top-level ns to avoid collisions with the original.
I don’t think it’s that big a deal… git/deps let you get started quickly, and are good for sharing top-level deps immediately across a team/org without having to deal with extra infrastructure; and they’re also good for tooling, where you already know people are using tools.deps.
If you want to make them a proper library for others to consume, you should release them as a maven lib, as that has the broadest audience for consumption.#2022-01-0512:30rickmoynihantools.build is also getting easier. If people start from something like a clj-new template etc, they’ll be taken 80% of the way there.#2022-01-0512:31scarytomso why is tools.build not doing the latter then? It is a proper library, and it is legitimate for me to depend on it if I am building my own tool that wishes to make use of its utility functions for compilation etc.
I notice the deps.edn for tools.build expresses a dependency on tools.deps, which uses a maven artifact not a git sha (presumably to solve this problem).
Is there a plan for tools.build to be published as a maven artifact?#2022-01-0513:01rickmoynihanYou’d need to ask @U064X3EF3 but whilst I agree it’s legitimate to use it as a library, I think it’s also legitimate for tools.build to assume you’re not going to be consuming from maven, but via tools.deps and the clojure CLI tools.
I’m struggling to think of use cases where you wouldn’t be doing that.#2022-01-0513:04rickmoynihanThe same is true of @U04V70XH6’s build-clj which also uses tools.build as a library, and doesn’t ship a maven artifact.
I’d expect 99.99% of possible users who have this as a dep will be consuming it in this manner; so I suspect there’s little point in doing it.#2022-01-0513:34Alex Miller (Clojure team)As a build time tool, you don't really need a maven artifact, so I haven't published one. Presumably a lib depending on it would also be used at build time and can also use a git dep. I'm not opposed to doing so and may in the future if there is a need.#2022-01-0513:51scarytomthanks both for your answers. I'm building an internal library that depends on tools.build, and we do most of our library publishing from a monorepo, so it doesn't make sense to package using git deps (especially considering all our current tooling works around maven deps). I'm not trying to fight the tide here, but it would be awfully convenient if tools.build were available as a maven dep too.#2022-01-0513:53rickmoynihanWill that library not also always be run via tools.deps / clojure CLI? What is consuming it?#2022-01-0514:06scarytomit will, but I don't want everyone to have to specify the library AND ~tools.deps~ tools.build itself in their deps.edn. The latter should come transitively by virtue of using the library.#2022-01-0514:07scarytomThis ensures the library comes with a compatible version of ~tools.deps~ tools.build.#2022-01-0514:07rickmoynihanyou mean tools.build right?#2022-01-0514:07scarytomyes, sorry#2022-01-0514:11rickmoynihanwhat is “the library” doing here? If it’s truly a build tool, doesn’t :deps/prep-lib sufficiently solve the problem of creating those interfaces/classes for you?#2022-01-0514:11rickmoynihani.e. why can’t the outermost library dep, be a git/dep?#2022-01-0514:15scarytomthe repo for the library contains other things too, so it is hard to publish it as a git/dep. We publish things as jars internally.#2022-01-0514:16rickmoynihando you know about :deps/root?#2022-01-0514:19scarytomyes, but that's a smell because it distributes knowledge about the packaging of your library amongst all downstream consumers.#2022-01-0514:21rickmoynihanarguably it’s just part of the coordinate#2022-01-0514:21rickmoynihanas the root belongs to the sha#2022-01-0514:21scarytomyou could make that case I guess.#2022-01-0514:29scarytomTo take a step back here, and attempt to summarise, there are basically two ways of distributing deps in clojure now. Some people will want to continue to use mvn deps and others will use git deps. For those using the former, they will find themselves unable to use the latter without taking the packaging task on themselves.
I think your statement way back at the start of this thread is insightful:
> If you want to make them a proper library for others to consume, you should release them as a maven lib, as that has the broadest audience for consumption.#2022-01-0514:32rickmoynihanThere are trade offs between the two. I use both, as do many others.#2022-01-0514:34scarytomIn the meantime. I have packaged tools.build v0.7.4 as a jar and pushed it into our internal mvn repo. This solves my immediate problem, but will make it harder to keep up-to-date with new tools-build releases. I hope that an official tools.build jar will be made available at some point, but I accept that this probably isn't a priority for the maintainers.#2022-01-0514:35scarytomI want to say a big thank you to you for engaging on this @U06HHF230 -- It has been a very helpful conversation.#2022-01-0514:36Alex Miller (Clojure team)I'd appreciate it if you made a request for this at https://ask.clojure.org and then I will (eventually) turn that into a ticket etc#2022-01-0514:36scarytomI will do that @U064X3EF3, thanks.#2022-01-0515:02scarytomhttps://ask.clojure.org/index.php/11433/will-tools-build-be-released-as-a-jar#2022-01-0517:17seancorfield@UCQV87CN7 Despite your folks using Maven for "everything", I assume your projects still have deps.edn under git and therefore also your build.clj file -- and you have one or more git repos that are accessible to everyone? So there are two options for you: 1) if you have a monorepo containing all the projects this might affect, your build library will be in that repo and you could use a :local/root dep to depend on it (which would pull in any deps it uses, such as tools.build via git) since you're using the CLI to run the build or 2) put your build library under git internally and update your deps.edn to refer to it. If you were using my wrapper, you'd depend on that from GitHub and it would pull in tools.build via git -- and it can rely on that because the CLI is being used to drive the build.#2022-01-0517:17seancorfield(I'd had requests to release build-clj as an artifact but pushed back because tools.build is not available as an artifact)#2022-01-0515:24rickmoynihanI guess clj -X:deps tree can’t really be a -T tool as tools run outside of the projects classpath, but “deps tree” needs to run within it… Is that correct?
I just wondered for a second why it was -X not -T…#2022-01-0515:30Alex Miller (Clojure team)well, it pre-dates -T#2022-01-0515:30Alex Miller (Clojure team)it could be a tool - it doesn't use the project classpath (like -Stree does)#2022-01-0515:36rickmoynihanahh cool, ok, that was what made me wonder…
In that case would it not make sense to document it as a tool instead? My first thought was because it predates tools.#2022-01-0515:37rickmoynihanI appreciate it doesn’t make a lot of difference; it just might be more consistent with usage now we have tools.#2022-01-0515:47Alex Miller (Clojure team)just haven't gotten around to it#2022-01-0518:17Alex Miller (Clojure team)https://clojurians.slack.com/archives/C06MAR553/p1641406221150600#2022-01-0520:00Joshua SuskaloHow does tools-deps handle git submodules when procuring a git dep?#2022-01-0520:01Alex Miller (Clojure team)badly#2022-01-0520:01Alex Miller (Clojure team)will not work correctly right now#2022-01-0520:01Joshua Suskaloalright, good to know#2022-01-0520:02Alex Miller (Clojure team)submodules are deeply weird and I did hack on it for a while to try to get them to work properly but the working tree strategy that gitlibs uses is I think fundamentally incompatible with submodules#2022-01-0520:02Joshua Suskaloalright, so if I need a submodule then I should procure it myself as a part of the deps/prep step?#2022-01-0520:03Alex Miller (Clojure team)I think you are likely to be affecting all other working trees of the repo in that case#2022-01-0520:03Joshua Suskaloah, that's no fun#2022-01-0520:03Alex Miller (Clojure team)so it might appear to work until it is spectacularly broken#2022-01-0520:03Joshua SuskaloI'll try to solve this with just a straight extra git dep then#2022-01-0520:03Alex Miller (Clojure team)I did spend a day or so trying to making this work and eventually shelved it, it is not easy#2022-01-0520:04Joshua SuskaloI believe you. I think in this case I can get away with just an extra git dep.#2022-01-0520:04Joshua SuskaloI think the main real usecases for it would also be solved by aliases on deps#2022-01-0520:10Joshua Suskalolike this: https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects?show=7843#2022-01-0520:10Joshua Suskalosince the only reason I can imagine to prefer submodules to a git dep is to allow you to specify which paths to include based on aliases#2022-01-0520:15Alex Miller (Clojure team)Understood#2022-01-0713:09socksyHi there, I don't have JIRA access so I can't comment on the issue, but I can report that at Pitch we are still hitting https://clojure.atlassian.net/browse/TDEPS-153 regularly, with version 1.10.3.967 (the version that is supposed to fix it?). Specifically on CircleCI with the runner using this docker image: circleci/clojure:openjdk-8-tools-deps-1.10.3.967-node .
We are just now implementing -Sthreads 1 as a workaround, but figure that the bug report might still be helpful#2022-01-0713:24Alex Miller (Clojure team)There was at least one instance of this fixed after this version but I would expect this to be fixed on the current version (1.10.3.1058). If you have any repro with that version, please file a new issue at https://ask.clojure.org with the stack trace#2022-01-0716:39socksyThanks for the heads up, we'll check it out next week, and of course add an issue as instructed (if necessary) :)#2022-01-0716:40rickmoynihanWhat is the best way to debug why deps aren’t being fetched from private s3 buckets?#2022-01-0716:41rickmoynihanI have added my new :mvn/repos in my deps.edn and have a corresponding ~/.m2/settings.xml#2022-01-0716:42rickmoynihanI should add this works for some s3 buckets I already have… I’m just adding another (so I essentially just copy/edited the config of the previous ones)#2022-01-0716:44rickmoynihanAlso all of the 3 repositories are actually in the same s3 bucket; with the same creds… there are just different top level folders in them partitioning the deps as part of a promotion strategy…#2022-01-0716:46rickmoynihanand the original 2 work — but the new 3rd one doesn’t seem to.
I should add I deployed the deps via the feature I added last year to deps-deploy… so I may have just misconfigured that somehow; though I can confirm that the dep is in the bucket in what looks to be the right place; but it would be nice to debug it to see which end is at fault (the deployment or the consumption / auth etc.#2022-01-0716:51Alex Miller (Clojure team)can you fetch the dep outside the Clojure CLI using aws?#2022-01-0716:53rickmoynihanyes#2022-01-0716:53Alex Miller (Clojure team)and what happens when you use the cli? error?#2022-01-0716:55rickmoynihanclojure -P
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.pom from swirrl-jars-releases
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.pom from swirrl-jars-branch-builds
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.pom from swirrl-jars-snapshots
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.jar from swirrl-jars-releases
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.jar from swirrl-jars-branch-builds
Downloading: com/swirrl/auth0/0.3.build-improvements_local-94a8b78/auth0-0.3.build-improvements_local-94a8b78.jar from swirrl-jars-snapshots
Error building classpath. Could not find artifact com.swirrl:auth0:jar:0.3.build-improvements_local-94a8b78 in central ()
NOTE the other thing I was going to double check is that the version of this dep is slightly unusual…. i.e. not semver.#2022-01-0716:56rickmoynihanI’m assuming that is allowed… I’ve done stuff like that in the past#2022-01-0716:57rickmoynihanoh actually I pasted a slightly different one there#2022-01-0716:57rickmoynihanthat was me testing it via a dirty local build (just installed into .m2)#2022-01-0716:57Alex Miller (Clojure team)it's certainly not sorted in a way you'd expect for version comparison#2022-01-0716:58rickmoynihantrue#2022-01-0716:59rickmoynihanbasically the setup is that branch builds are building immutable artifacts like this#2022-01-0716:59Alex Miller (Clojure team)maven version is MAJOR.MINOR.INCREMENT-QUALIFIER - if any of the first 3 is not a number, the whole thing is alpha sorted#2022-01-0716:59rickmoynihanok so I might need an increment too?#2022-01-0716:59Alex Miller (Clojure team)and minor and increment are optional#2022-01-0717:00rickmoynihanthe version has major minor#2022-01-0717:00Alex Miller (Clojure team)so just changing the .build to -build would turn the last part into a qualifier#2022-01-0717:00rickmoynihanah gotcha#2022-01-0717:00Alex Miller (Clojure team)but I'm not sure that would affect the issue#2022-01-0717:00rickmoynihanme neither — but good to know#2022-01-0717:00rickmoynihanand it’s something I will fix#2022-01-0717:01Alex Miller (Clojure team)in short, I don't know. I would debug it by using tools.deps.alpha and hitting the maven procurer and/or s3 transporter directly#2022-01-0717:02rickmoynihanyeah I could certainly do that#2022-01-0717:03rickmoynihanis there any internal logging that you’re aware of in the java deps, that I could switch on to get an idea?
Oh wait IIRC it now uses the cognitect aws stuff is that right?#2022-01-0717:04Alex Miller (Clojure team)don't know, never do that#2022-01-0717:04Alex Miller (Clojure team)maven stuff certainly has some logging stuff but I have never bothered to configure it#2022-01-0717:07rickmoynihanAh ok you wrap the cognitect s3 stuff into reified aether classes?
https://github.com/clojure/tools.deps.alpha/blob/452b3f215b8f624d17a99ecab22e9e8673e700fc/src/main/clojure/clojure/tools/deps/alpha/util/s3_transporter.clj#2022-01-0717:07rickmoynihanthat should certainly make it a little easier to debug at a repl#2022-01-0717:23rickmoynihanRunning this at a REPL:
(clojure.tools.deps.alpha/resolve-deps
{:deps {'com.swirrl/auth0 {:mvn/version "0.3.0-build-improvements-4302ec2"}}
:mvn/repos {"swirrl-jars-branch-builds" {:url ""}}} nil)
raises this exception:
1. Unhandled clojure.lang.ExceptionInfo
Could not find artifact
com.swirrl:auth0:jar:0.3.0-build-improvements-4302ec2 in
swirrl-jars-branch-builds ()
{:lib com.swirrl/auth0,
:coord
{:mvn/version "0.3.0-build-improvements-4302ec2",
:deps/manifest :mvn,
:parents #{[]}}}
maven.clj: 167 clojure.tools.deps.alpha.extensions.maven/get-artifact
maven.clj: 155 clojure.tools.deps.alpha.extensions.maven/get-artifact
maven.clj: 178 clojure.tools.deps.alpha.extensions.maven/eval7338/fn
MultiFn.java: 244 clojure.lang.MultiFn/invoke
alpha.clj: 465 clojure.tools.deps.alpha/download-libs/fn/fn
concurrent.clj: 35 clojure.tools.deps.alpha.util.concurrent/submit-task/task
AFn.java: 18 clojure.lang.AFn/call
FutureTask.java: 266 java.util.concurrent.FutureTask/run
ThreadPoolExecutor.java: 1149 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 624 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 748 java.lang.Thread/run#2022-01-0717:28rickmoynihanLooks like I’m running out of time today… I’ll need to pick this up again next week…
Many thanks for your sage advice @alexmiller 🙇#2022-01-0717:33Alex Miller (Clojure team)you want to close the gap from there to what it's looking for in aws#2022-01-0717:33rickmoynihanyes definitely#2022-01-0717:38rickmoynihanI can see it’s calling s3-get-object and it’s raising an exception from in there… looks like I mistyped the artifact id#2022-01-0717:39rickmoynihanyep that’s it… the real artifact was called com.swirrl/swirrl-auth0 facepalm#2022-01-0717:39rickmoynihanthanks for the help… it’s so much easier to debug the s3 stuff now it’s in clojure! 🙇#2022-01-1004:29jasonjcknIs there a github example repository of this blog post anywhere https://corfield.org/blog/2021/02/23/deps-edn-monorepo/#2022-01-1005:08seancorfield@jasonjckn Probably the closest is this https://github.com/seancorfield/usermanager-example/tree/polylith but it hasn't been updated to reflect the latest changes in Polylith (or what we're doing at work). Very happy to talk about our monorepo tho' -- it's not all Polylith so it applies beyond that.#2022-01-1006:30seancorfield@alexmiller Is it documented/supported anywhere that :exec-fn can be a vector of function names to execute? I just discovered that works (based on the (loop [fns f...] ...) code here https://github.com/clojure/brew-install/blob/1.10.3/src/main/clojure/clojure/run/exec.clj#L176#2022-01-1013:34Alex Miller (Clojure team)No, not documented or supported#2022-01-1013:34Alex Miller (Clojure team)Just an accident of implementation#2022-01-1020:06seancorfieldThanks. I was just really surprised it worked 🙂#2022-01-1106:22didibusThat could be a neat feature if it also like had the output of the prior be the input of the next, or something like that#2022-01-1117:32seancorfieldThat's exactly how it works (well, "worked"):
(! 736)-> clojure -X:exec :foo 13
a {:foo 13}
b {:foo 13, :a :called}
Tue Jan 11 09:30:46
(sean)-(jobs:0)-(~/clojure/fresh)
(! 737)-> cat src/exec.clj
(ns exec)
(defn a [m] (println 'a m) (assoc m :a :called))
(defn b [m] (println 'b m) (assoc m :b :called))
Tue Jan 11 09:31:07
(sean)-(jobs:0)-(~/clojure/fresh)
(! 738)-> cat deps.edn
{:aliases {
:exec {:exec-fn [a b] :ns-default exec}}}#2022-01-1120:46didibusNice.
I think an alternative could also be that it passes the map to the first, and then merges the return back onto the map and passes that to the next.
That said, in that case, each exec-fn would need to take a map and return a map.
Whereas I think now in theory you could have one return a string for example and the next one take a string as input no?#2022-01-1120:48didibusBut in any case, I see why they "don't support" this anymore.
I think control on class path is important to have on a per-exec-fn basis. And this functionality everything is ran in the same context.
When I think of this feature, I'm automatically thinking build task chaining, which it can't do because of lack of class path control.
So it makes sense to push people to build.clj instead and chain things using the shell or babashka, etc.#2022-01-1120:54seancorfieldIt's why I generally run my build.clj tasks from a REPL so I can thread several at once.#2022-01-1106:32Alex Miller (Clojure team)we implemented that at one point but then decided not to keep it and this is a vestige of that#2022-01-1216:37weavejesterCan a tool dependency easily access its own version data? As git deps appear to need a SHA, I can’t hardcode the version.#2022-01-1216:38weavejesterFor context, I’m building a tool that bootstraps a bb.edn file, which itself needs a dependency.#2022-01-1217:01Alex Miller (Clojure team)access from where?#2022-01-1217:02Alex Miller (Clojure team)as in, you are running code from the tool and want to know what version of itself that is?#2022-01-1313:35weavejesterYep, that’s right. It’s so I can tell Babashka via bb.edn that this is the version of the library it should reference.#2022-01-1217:08Alex Miller (Clojure team)there is not a public api (yet!) but this hack will work:
(-> (System/getProperty "clojure.basis") slurp clojure.edn/read-string :libs (get 'org.clojure/clojure) :mvn/version)#2022-01-1313:50weavejesterThanks! That appears to work!#2022-01-1217:08Alex Miller (Clojure team)or at the end, check :git/tag , :git/sha etc#2022-01-1222:52seancorfieldNot sure if this is "known issue" or not: in CI (and pretty much only in CI), we are seeing intermittent "Failed to read artifact" errors from the aether stuff in the CLI (stacktrace in thread). Is this the multi-threading/race condition issue?#2022-01-1222:52seancorfieldError building classpath. Failed to read artifact descriptor for ring:ring:jar:1.9.4
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for ring:ring:jar:1.9.4
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:259)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$read_descriptor.invokeStatic(maven.clj:115)
at clojure.tools.deps.alpha.extensions.maven$fn__1127.invokeStatic(maven.clj:143)
at clojure.tools.deps.alpha.extensions.maven$fn__1127.invoke(maven.clj:143)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__770$fn__772$fn__773.invoke(alpha.clj:406)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__479.invoke(concurrent.clj:35)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact ring:ring:pom:1.9.4 from/to clojars (): /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:244)
... 13 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact ring:ring:pom:1.9.4 from/to clojars (): /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:628)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:514)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:402)
... 16 more
Caused by: java.io.FileNotFoundException: /root/.m2/repository/ring/ring/1.9.4/ring-1.9.4.pom.part (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:163)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:252)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:482)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
... 21 more
#2022-01-1222:55seancorfieldRe-running the failed step nearly always succeeds (and the "failed" artifact is random from build to build).#2022-01-1222:57hiredmanIs our m2 saved between builds?#2022-01-1222:57seancorfieldMostly, yes, but it drifts as we update deps, and then BitBucket deletes and recreates the cache about every two weeks (to make it fresh again).#2022-01-1223:12Alex Miller (Clojure team)Don't think that matches anything I've seen#2022-01-1223:12Alex Miller (Clojure team)ring stuff will be coming from clojars right? #2022-01-1223:15Alex Miller (Clojure team)It looks like that is in the maven stuff that downloads then copies into place and it is missing the download. Is it possible bitbucket deletes out from under you?#2022-01-1223:18seancorfieldNo. This all happens in a single step. I don't know that it is only Clojars deps -- I think I've seen it from Maven deps too -- but I'll keep a closer eye on that in future.#2022-01-1223:19seancorfieldGood to know this isn't a known-but-supposedly-fixed issue 🙂#2022-01-1223:49Alex Miller (Clojure team)there are general issues with Maven concurrent downloads that they've been working on for Maven 4#2022-01-1223:49Alex Miller (Clojure team)is there any chance of concurrent processes hitting this same repo or it for sure just 1?#2022-01-1223:51seancorfieldI'm not using -Sthreads 1 but otherwise it is just this one process.#2022-01-1223:54hiredmando we know how bitbucket shares the m2 cache? if all the builds should have access to m2, what happens when multiple builds are running at once?#2022-01-1223:57hiredman(found the docs, directory is just and restored after a successful build, no shared filesystem or anything like that)#2022-01-1222:53seancorfieldWe're on the latest CLI (1058).#2022-01-1501:03Reily SiegelIs it possible to build tools.deps.alpha without a transitive dependency on clojurescript? I recently pacakged clojure-tools for Guix, which requires that all pacakges be built from source. However, building ClojureScript (and it's dependencies) from source is quite the undertaking. To avoid this, I simply strip out the S3 transporter, which is the only part of tools deps that depends transitively on clojurescript. However, this is for obvious reasons not really ideal in the long run. Is there a way to break this dependency chain somewhere? Currently we have tools.deps -> cognitect.aws -> core.async -> clojurescript.#2022-01-1501:05Reily SiegelFor context, here is how I strip s3-transporter (in scheme, for version 0.12.1104):
(lambda _
(for-each delete-file
(list
(string-append
"src/main/clojure/clojure/"
"tools/deps/alpha/util/s3_aws_client.clj")
(string-append
"src/main/clojure/clojure/"
"tools/deps/alpha/util/s3_transporter.clj")
(string-append
"src/test/clojure/clojure/"
"tools/deps/alpha/util/test_s3_transporter.clj")))
(substitute*
"src/main/clojure/clojure/tools/deps/alpha/util/maven.clj"
(("clojure.tools.deps.alpha.util.s3-transporter")
"")))#2022-01-1501:32seancorfieldcore.async does not depend on ClojureScript:
org.clojure/core.async 1.5.648
. org.clojure/tools.analyzer.jvm 1.2.2
. org.clojure/tools.analyzer 1.1.0
. org.clojure/core.memoize 1.0.253
. org.clojure/core.cache 1.0.225
. org.clojure/data.priority-map 1.1.0
. org.ow2.asm/asm 9.2
. org.clojure/tools.reader 1.3.6#2022-01-1501:33seancorfieldtools.deps.alpha does not depend on ClojureScript either:
clojure -Sdeps '{:deps {org.clojure/tools.deps.alpha {:mvn/version "RELEASE"}}}' -Stree|fgrep script
<empty result>#2022-01-1501:34seancorfieldHere's the list of org.clojure deps that get pulled in for tools.deps.alpha:
(! 755)-> clojure -Sdeps '{:deps {org.clojure/tools.deps.alpha {:mvn/version "RELEASE"}}}' -Stree|fgrep org.clojure
org.clojure/clojure 1.10.3
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56
org.clojure/tools.deps.alpha 0.12.1109
. org.clojure/data.xml 0.2.0-alpha6
. org.clojure/data.codec 0.1.0
. org.clojure/tools.gitlibs 2.4.172
. org.clojure/tools.cli 1.0.206
. org.clojure/data.json 2.4.0
. org.clojure/tools.logging 1.2.1
. org.clojure/core.async 1.5.644
. org.clojure/data.xml 0.2.0-alpha6
. org.clojure/core.async 1.5.644
. org.clojure/tools.analyzer.jvm 1.2.1
. org.clojure/tools.analyzer 1.1.0
. org.clojure/core.memoize 1.0.253
. org.clojure/core.cache 1.0.225
. org.clojure/data.priority-map 1.1.0
. org.clojure/tools.reader 1.3.6#2022-01-1501:35seancorfieldWhat makes you think you have a dependency on ClojureScript @U6NJBB596?#2022-01-1501:37seancorfield(if you built clojure-tools from source, that depends on t.d.a so you would have already run into this issue?)#2022-01-1501:47Alex Miller (Clojure team)I second Sean’s confusion. core.async has a project.clj that depends on clojurescript for testing but that's not the project file we use to build core.async (built with Maven from the pom.xml)#2022-01-1501:48Reily SiegelAh, I found the problem. It was attempting to compile macros for clojurescript, which depend on cljs.analyzer. removing macro namespaces from the list of aot compiled namespaces should fix this.#2022-01-1501:48Alex Miller (Clojure team)Again, there should be no clojurescript anywhere in this deps tree#2022-01-1501:49Reily SiegelThe problem here stemming from the fact that guix's clojure-build-system tries to aot compile every clj file.#2022-01-1501:50Reily SiegelGuix doesn't use maven for dependency resolution, it uses its own packages, and trying to aot ClojureScript macros (in clj files) without ClojureScript failed.#2022-01-1501:51Reily SiegelThank you for helping me realize something obvious after staring at what was ultimately a stupid problem for way too long.#2022-01-1501:51seancorfieldWhich namespaces are you talking about here?#2022-01-1501:52Reily Siegelcljs.core.async.impl.ioc-macros#2022-01-1501:52Alex Miller (Clojure team)Well, guix’s system is wrong to do that #2022-01-1501:53seancorfieldYeah, definitely. And packaging a library as AOT'd code is likely to cause horrible problems for consumers of that library.#2022-01-1501:54Alex Miller (Clojure team)Lots of Clojure libs have clj files that are not part of the lib#2022-01-1501:56Reily SiegelDo you think it would be worh it to get that default changed in guix? Currently (afaict) it calls clojure.core/compile on every clj file that isn't excluded by default, unless specifically disabled. I am not super familiar with guix clojure-build-system, just trying to get a few things packaged. Currently the number of Clojure libraries packaged is abysmally small.#2022-01-1501:56seancorfield@U6NJBB596 You're aware that lots of Clojure libraries are packaged as JAR files with no compilation of source code?#2022-01-1501:58seancorfield"Packaging" Clojure libraries seems kinda wrong on several levels. The normal way to get Clojure libraries is as dependencies in a project -- fetched by the Clojure tooling itself, when it is run.#2022-01-1501:58Reily SiegelI am aware of this, yes. I am neither familiar with the details of maven distribution, nor why guix's clojure-build-system was written like that.#2022-01-1502:00seancorfieldI can sort of understand wanting the clojure/`clj` tool itself as a package that one of these package managers can fetch and install for you, but there's no reason to package Clojure libraries in general.#2022-01-1502:02Alex Miller (Clojure team)I personally just don't get the “rebuild” everything from source obsession#2022-01-1502:02Reily SiegelYeah, that is the normal way, which is why my first target for packaging was clojure-tools, to allow that use case. However guix likes to have everything packaged in its own system to facilitate reproducible builds of everything. And because guix as a package manager follows GNU guidelines on only providing software built from source, we get into situations where (possibly pre-compiled) maven artifacts are an issue.#2022-01-1502:02Alex Miller (Clojure team)What could be more reproducible than using the exact globally published bits everyone else uses?#2022-01-1502:03seancorfieldIt's like these package managers are stuck in the 1990s before the Java ecosystem appeared...#2022-01-1502:03Reily SiegelI don't necessarily disagree, it /is/ a GNU project...#2022-01-1502:05Reily SiegelBut packaging clojure-tools solves things for 99% of cases, and for when people want to integrate a Clojure codebase with guix tools, they can package libraries themselves.#2022-01-1502:05seancorfieldSeems like GNU's guidelines are at odds with how the Java ecosystem works. Does Guix has brew as a package (linuxbrew) that can be installed via the package manager?#2022-01-1502:06seancorfield(b/c once you have brew you can use that to install the Clojure tools directly from the Clojure repos)#2022-01-1502:07Reily SiegelIt does not. My patch for clojure-tools (sans S3 transport) was already accepted, Ill just have to send a new patch lifting that restriction once I sort the AOT issues#2022-01-1502:23seancorfieldI'm curious about the process for upgrading stuff? It seems like packaging clojure-tools relies on community volunteers and that means they have to repackage every time there's a new release of the CLI?#2022-01-1502:25seancorfieldWhat happens if a community volunteer moves on, or even is off on vacation for a while? Doesn't that just block people from upgrading their clojure-tools in a timely manner?#2022-01-1502:26seancorfieldWhat I like about brew on Linux is that I can install any version of the Clojure CLI, as soon as it is released by the core team -- and I can install prerelease versions just as easily as stable versions, and I can switch back and forth between them easily.#2022-01-1502:30Alex Miller (Clojure team)This is all basically same as Debian, that redoes all this same work with their own slightly different build system that makes slightly different versions of libs too. They chunk it up into periodic release trains#2022-01-1502:32Alex Miller (Clojure team)But what seems weird to me is that once you start using the custom Clojure cli, you are back again to using maven libs etc so …#2022-01-1718:50Jelle LichtFwiw, reproducible builds are perhaps not the best name for the concept: verifiable builds would perhaps be a better description. But it's a short ddg search away if you want to read up on it https://reproducible-builds.org. If a person does not experience (or care about) a problem, naturally they won't appreciate the trade-offs made to solve the problem. Edit: which is not a dig at anyone or any point made here, but just an almost tautological observation. #2022-01-1621:02emccueCan i call create-basis from multiple threads? Looking to provide info like “If you run with these aliases you get …” and thinking of resolving that in parallel#2022-01-1622:19Alex Miller (Clojure team)You might run into problems with maven or git downloading#2022-01-1703:35emccueIf i have a deps.edn that looks like this
{:deps {a/b {...}}
:aliases {:build {:deps {c/d {...}}}}}
Is there a way to create a basis that just has the deps from the build alias, and not a/b
(to my understanding this is what clj -T … does)#2022-01-1704:44Alex Miller (Clojure team)yes, but it will not be identical to -T without a small bit of modification.
not sure which api you're using to create the basis (`tools.deps.alpha/create-basis`, tools.cli.api/basis, or tools.build.api/create-basis - they are all slightly different), but they can all take an :aliases [:build] argument#2022-01-1704:46Alex Miller (Clojure team)the one difference from -T is that -T adds "." to the :paths . you can either make this explicit by putting it in the :build alias :paths, or you can also pass a :extra {:paths ["."]} to any of those api calls above too#2022-01-1706:18emccuetools.deps.alpha/create-basis#2022-01-1706:20emccueand it turns out the reason i was so confused is that i was passing the aliases as strings, which resolved to nothing, and then i saw the root classpath#2022-01-1707:53zackteoIs there a way I can add a truststore as a JVM parameter? In deps.edn? Am getting sun.security.validator.ValidatorException which I understand I need to doing something like -Djavac.net.ssl.trustStore=.... but I'm not sure how to go about it for deps edn cause previously when using lein I could set my env var of JVM_OPTS#2022-01-1707:59dharriganCan you pass it though, i.e., clojure -.ssl... -M:alias1:alias2....?#2022-01-1708:01dharriganFor example, I have an shell alias that starts up clojure in debug...`clojure -J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -J-Dclojure.core.async.go-checking=true -M:foo:bar`#2022-01-1708:09zackteoOooo! That works I was putting it in as a vector so that was the problem 😅#2022-01-1708:11zackteoBut now I need to make sure it integrates the editors my team uses :x the jvm-opts part that is#2022-01-1708:16zackteo@dharrigan do you know of anyway to get configure deps.edn apply the jvm-opts based on an env variable etc#2022-01-1708:18dharriganLike doing "${SOME_ENV}" in the deps.edn? If so, no, I don't believe the deps.edn is run through any shell expansion.#2022-01-1708:22zackteoHmmm more like how lein would use JVM-OPTS and apply that to whatever java command is ran when running lein#2022-01-1708:24dharriganNo, I don't believe there is any functionality for that.#2022-01-1708:26zackteoI'm reading the reference docs and might this be what I want? JVM arguments may either be passed on the command line (with -J) or by using data stored in an alias under :jvm-opts and passed with -X or -A or -M#2022-01-1708:27zackteoI dont quite understand what does the latter mean exactly#2022-01-1708:29dharriganThe first example is the one I gave above, i.e., to pass in the JVM parameters as the clj process is created, the second option, here is an example :repl/nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "0.9.0"}}
:main-opts ["-m" "nrepl.cmdline"]}
This works when I do clojure -J-Djavax.... -M:repl/nrepl#2022-01-1708:34zackteoBut when I modify it
:repl/nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "0.9.0"}}
:jvm-opts ...
:main-opts ["-m" "nrepl.cmdline"]}
clojure -M repl/nrepl doesn't work. It seems like the jvm-opts isn't being applied#2022-01-1709:22dharriganI think it's being applied for me, ...#2022-01-1709:22dharrigan:local-dev {:extra-deps {local-dev/local-dev {:local/root "/home/david/.clojure/libs/local.dev" :deps/manifest :deps}}
:jvm-opts ["-Djavax.net.ssl.trustStore=/home/david/foo"]
:main-opts ["-m" "local.dev.main"]}#2022-01-1709:22dharrigan#2022-01-1709:23dharrigan#2022-01-1709:23dharriganSo, I launched a Main, passing in jvm-opts, then used visualvm to confirm that the JVM had started with the -D set.#2022-01-1712:36zackteoThanks for your help! Let me look into it again :)#2022-01-1719:26emccueterminology question - why alias instead of profile#2022-01-1719:44vlaaadBecause it aliases a piece of data with a name#2022-01-1719:46vlaaadI remember hearing a sentiment from @U064X3EF3 that aliases might contain completely arbitrary data that can be passed to clojure programs#2022-01-1719:47vlaaadIt seems there isn't much support for that now, but maybe it's coming? :thinking_face:#2022-01-1720:12Alex Miller (Clojure team)This is the correct sense of it, tools.deps happens to be the most prominent user of this data#2022-01-1720:13Alex Miller (Clojure team)Support for this is partially there, more to come#2022-01-1802:12slimslenderslacksWhile building a -T:tool style tool, I also wondered about whether users would expect a tool's config map to be passed as an alias - I ended up just reading the config from a seperate .edn file. If I needed different configurations for different "profiles", there didn't seem to be a way to use the cli to select one of them ( clojure -T:tool -A:config2 or something like that).#2022-01-1803:13Alex Miller (Clojure team)It is kind of possible now by reading the injected basis, but I think this is an area where more could be done by referencing aliases#2022-01-1801:46lispers-anonymousIs it possible to use git repositories hosted in source hut with the new tools functionality? I'd like to be able to do
clojure -Ttools install '{:git/tag ...}' :as tool
Looking through the source of tools.deps, there is a hardcoded list of hosted https://github.com/clojure/tools.deps.alpha/blob/dce6bcba266e9261ed671f3f594d7b122f761bf1/src/main/clojure/clojure/tools/deps/alpha/extensions/git.clj#L17-L20 that seems to be parsing the git repo, but sourcehut is absent. I was hoping there would be a way to just provide a url directly#2022-01-1802:03lispers-anonymousI think I have answered this myself (by looking more closely at the code). I can pick a meaningful name and provide a :git/url in the argument map. So this works
clojure -Ttools install '{:git/tag ... :git/url ""}' :as tool #2022-01-1803:15Alex Miller (Clojure team)Yes, support for most git url types is supported that way, including git file repos#2022-01-1803:25lispers-anonymousDo you know if it's possible to do this with the clojure -X:deps find-versions tool as well?#2022-01-1812:52Alex Miller (Clojure team)Should be#2022-01-1812:53Alex Miller (Clojure team)It takes an optional :git/url iirc#2022-01-1812:55Alex Miller (Clojure team)Actually it doesn't, but it easily could#2022-01-1813:57lispers-anonymousThat's what I thought, I had poked around but couldn't get it working. Is there a place I can open a ticket for that?#2022-01-1814:32Alex Miller (Clojure team)https://ask.clojure.org#2022-01-1803:42emccueIs there any reason that a basis doesn’t convey which aliases were activated to form its classpath roots? (or is there and I’m just missing it)#2022-01-1811:13viestiHmm, can :mvn/local-repo be put into an alias, in order to override the cache path in CI (Gitlab) while keeping the default path locally?#2022-01-1811:14viestiah, by adding
-Sdeps '{:mvn/local-repo ".m2"}
#2022-01-1811:15viestididn't realise that this still reads the deps.edn, thought it would override it 🙂#2022-01-1820:03Aleedwhat’s the equivalent of a peer dependency with tools.deps?
i.e. if I’d like to ensure a library is included but don’t want to put constraints on the version or where it’s loaded from#2022-01-1820:42hiredmanThat is bad, don't do it#2022-01-1820:45hiredmanYou have a version you tested with, you should depend on that version#2022-01-1820:48seancorfield(a user of your library can always specify a different version if they want -- but with the usual "not tested against that version" caveats)#2022-01-1820:53hiredmanhttps://research.swtch.com/npm-colors someone involved with golang tools writing about a recent npm issue with packages not depending on specific versions#2022-01-1821:53Aleedfor this use-case it’s actually for a monorepo. I was looking for a way to specify a global dependency for the project, rather than listing it out in each subproject#2022-01-1821:54Aleedthis is for a front-end app, where you wouldn’t want two versions of React running, for example#2022-01-1923:32matthaveneris it possible to run an arbitrary function with arbitrary arguments from clj ?#2022-01-1923:33Alex Miller (Clojure team)You can use -M -e to run any Clojure expression#2022-01-2210:53pavlosmelissinosI think this belongs here. Please bear with the wall of text...
I'm trying to understand how dependencies are resolved, so I'd like some help with this 🙂 (questions in thread)#2022-01-2210:54pavlosmelissinos1. For obvious reasons you can't refer to a library twice using a single symbol in the same deps.edn. You can "trick" clojure.tools.deps to include different versions of the same library by using different symbols for each, as long as the namespaces are different (e.g. use honeysql v1 and honeysql v2 at the same time). Is my understanding correct?
2. How about different versions of the same library against transitive uses of it (top-level vs transitive or transitive vs transitive) when the same symbol is used? E.g. if dependency A has library X v1.4 in its deps.edn and dependency B has library X v1.5 is it possible to actually include both versions of library X in the deps tree?
3. If not, how does tools deps deal with the fact that libraries are generally tested against the specific versions of their dependencies? If, say, a top-level dependency of your project overrides every transitive use of it (like the docs mention, "top dep versions always win") then you might end up with unexpected results, even in the code of transitive deps. In a perfect world, this wouldn't be an issue because there would be no breaking changes but in our flawed reality how does Clojure handle this situation, especially in projects with large dependency graphs?
4. If Clojure doesn't try to handle it, what's driven that decision?
5. Is there a place (book, docs, whatever) where I can read more about the above? I've already read the very helpful https://clojure.org/reference/dep_expansion but I'm still not 100% sure how dep resolution works and why it's like that 🙂
Thanks!#2022-01-2214:41Alex Miller (Clojure team)1. You could, except ultimately you're making a classpath that contains an ordered set of locations to look and whatever version is earlier in the classpath will be used as its found first. Having two different versions of the same lib in your classpath should generally be considered an error.
2. No - only one version of a lib will be selected
3. This is the world we live in - libs are tested against some version of a dep but the application using the lib has full control over what version is used and it might not work. The application developer has to handle this, ultimately by selecting a set of compatible versions at the top if the selected versions don't work together (either by expliciting pinning versions or excluding transitive deps). There is no guarantee that there is a compatible solution (although this happens surprisingly rarely)
4. Doesn't handle it, and we inherited this from the Java world.
5. Anything you read about the Java / Maven ecosystem is likely applicable, not sure I have a good pointer.#2022-01-2214:44Alex Miller (Clojure team)Another factor to be aware of is that Maven/Leiningen use a different version selection algorithm than tools.deps/Clojure CLI. Maven uses the first version it encounters (top-down, breadth first walk) and tools.deps will generally select the newest version it can. In practice, this matters less than you'd expect.#2022-01-2219:29pavlosmelissinosAlright, that was illuminating. I think I'll let the whole thing sink a bit before I ask any follow-up questions. Thanks so much for the authoritative answer!#2022-01-2222:40seancorfield"(e.g. use honeysql v1 and honeysql v2 at the same time)." -- just to be clear here, HoneySQL v1 and v2 are essentially different libs: they use different Maven coords and different namespaces, precisely so you can depend on both and use them side-by-side when you are migrating. That's unusual -- most libraries that have v1 and v2 force you to choose, because they generally use the same coordinates (and usually the same namespaces too).#2022-01-2510:55Jakub Holý (HolyJak)Hi! Do you know, is there a GitHub Action for running an arbitrary Clojure CLI tool on a project?#2022-01-2511:20dharriganMoved here. We do something very similar to this at work:#2022-01-2511:21dharrigan- name: Setup Java
uses: actions/#2022-01-2614:53emccuePerhaps a silly question, but why do npm and rust make lockfiles but maven, gradle, lein, deps, etc don’t?
I assume its some limitation/property of the ecosystem, I just can’t intuit what#2022-01-2615:44Aleednpm is also more flexile when it comes to declaring which version to use. e.g. you have the ^ prefix to specify any version greater than.
i’m not sure either as to why other strategies don’t require lockfiles at all though#2022-01-2615:47pavlosmelissinos> npm is also more flexile when it comes to declaring which version to use. e.g. you have the `^` prefix to specify any version greater than.
Not a fan of that kind of "flexibility"
I just had a deployment fail because of a breaking change between two consecutive patch versions of a library (which I could have prevented if I didn't use the caret)#2022-01-2615:53Aleedi’m not a fan either, just noting the difference above
seems like you didn’t have a lockfile lol#2022-01-2615:58pavlosmelissinosYeah, lockfile wasn't on version control so it was partly my fault but still I don't see why you need to specify an approximate version#2022-01-2616:17seancorfieldMaven lets you specify version ranges and also supports RELEASE and LATEST but they are all discouraged.#2022-01-2616:20emccueright but transitive dependencies are always kinda implicitly on a version range as to what gets resolved/picked#2022-01-2616:21emccueso it still feels like there is a benefit?#2022-01-2617:01pavlosmelissinospackage.json,`deps.edn` , etc are contracts. They communicate to the user which versions your library has been tested against. (in applications this doesn't matter as much because end users care less about your dependencies)
So using version ranges in the way ^ does means either:
• "I've tested every released version under the vX major version, including future minor/patch versions" which isn't very honest or
• "I trust that every minor/patch version that will come out in the future will be compatible with my code" which is naive
Is there a third interpretation?#2022-01-2617:04seancorfield@U3JH98J4R In Clojure, for a given set of specific top-level deps, everything is fixed -- all your transitive dependency versions are fixed too (you may not be able to predict them ahead of time but they won't change). If you want to force a specific version of something that is pulled in transitively, you just put it in your deps.edn file.#2022-01-2617:05seancorfieldWell, as long as none of the deps you pull in behave badly and actually have a range of versions specified, of course.#2022-01-2813:48slipsetOn ranges and such, I made this a while ago https://github.com/slipset/get-profiles#2022-01-2614:59Alex Miller (Clojure team)not a silly question, these are different strategies for managing versions#2022-01-2616:27hlship> :default-deps provides a set of default coordinate versions to use if no coordinate is specified. The default deps can be used across a set of shared projects to act as a dependency management system:#2022-01-2616:29hlship(from the manual). How would this work in practice? Is there a way to get clj to read from deps.edn and from another file to provide those defaults? Our monorepo has 47 sub-modules and it's a pain to keep some things synchronized. Failing that, I'm working on a tool to make global updates to deps.edn files across the monorepo.#2022-01-2616:30Alex Miller (Clojure team)not really, this is kind of an unfinished idea#2022-01-2616:30Alex Miller (Clojure team)I mean, you can read from ~/.clojure/deps.edn but that's probably not really a good match for this need#2022-01-2616:31Alex Miller (Clojure team)I think most people doing this kind of thing are doing some form of programmatic rewrite#2022-01-2616:32Alex Miller (Clojure team)another dodge if they are all shared is to locate them in one common dep that the others all depend on but I assume they vary across your modules#2022-01-2616:32hlshipRight. I'm debating whether to a) walk all the deps.edn files and make updates or b) generate the deps.edn files from a template (e.g., template has dependency on org.clojure/core.async, and a master file identifies the actual coord to use).#2022-01-2616:33Alex Miller (Clojure team)I believe I've seen people doing both. I think the benefit of a) would be having the deps.edn files there and available to use with other tooling#2022-01-2616:34hlshipRight, otherwise you are either a) having to check in genrated files to Git or b) having to generate the deps.edn files all the time in tooling or CI.#2022-01-2616:34borkdude@hlship For the programmatic rewrite, here's some examples:
• https://gist.github.com/borkdude/58f099b2694d206e6eec18daedc5077b
• https://github.com/exoscale/deps-modules
#2022-01-2616:34emccue@hlship You can also do my idea which no one else likes and just have a singular local dependency per external dep#2022-01-2616:36emccuelibraries/whatever
{:deps {org.apache/thing {:mvn/version "x.y.z"}}}
libraries/other
{:deps {org.apache/other {:mvn/version "x.y.z"}}}
module/in-your-project
{:deps {libraries/other {:local/root "../../libraries/other"}}}#2022-01-2616:37emccueyes, this leads to the world’s dumbest libraries folder#2022-01-2616:37emccueand no, we don’t do this at work, we don’t have nearly enough modules to justify it#2022-01-2616:38emccuebut… it does solve the version pinning problem#2022-01-2616:38borkdudePerhaps polylith can also be an option, @seancorfield is probably shortly going to mention that ;-)#2022-01-2617:14seancorfieldI wasn't going to mention Polylith specifically but I was going to point @hlship at https://corfield.org and say "Take a look at my blog series about our monorepo and how we've evolved our use of deps.edn over time." 😛#2022-01-2617:16seancorfield(we started with a repo-based "user" deps.edn, switched to generated deps.edn files, and then switched to something fairly close to how Polylith does things -- and now we're actively migrating to Polylith proper)#2022-01-2618:52Aleed@hlship I tried to address shortcomings with monorepo usage a while back, in a library called Interdep, but then took a hiatus from programming.
Recently got back to it and did another iteration of way it could work.
I’m still experimenting with ideal use, but here it is in case you’re looking for an example to reference https://github.com/rejoice-cljc/interdep
We unify all subdeps into a single config and then pass that as an “-Sdeps” argument. This approach is most transparent as you can keep the deps.edn files as is and you can run sub-projects independently without extra work.
@borkdude the improvements in Babashka have made this a lot easier to manage 🙂#2022-01-2618:55AleedRight now I force aliases to be namespaced as it makes them easier to merge, but I’m considering just handling merging as it’d make it less restrictive and possibly be a simpler way to run multiple aliases (as right now we’re doing this via profiles as described in docs)#2022-01-2619:05borkdudeWhich improvements specifically? Thanks for the feedback#2022-01-2619:19Aleedbb.edn and the built-in clojure command
before I was running my own script and starting a clojure process myself
now I can just plug this into the bb config
run (clojure (interdep/enhance-args *command-line-args*)
and do bb run … to start my program with custom deps and aliases#2022-01-2621:09seancorfield@alexmiller Do I take this "Add check to error on invocation of multiple exec functions" to mean that :exec-fn [f1 f2] is now detected/disallowed? 🙂#2022-01-2621:10seancorfieldDoes it specifically check it is a symbol (qualified or simple)?#2022-01-2621:19Alex Miller (Clojure team)yes#2022-01-2621:51borkdudeI guess we could add that to clj-kondo as a check, but not sure how common that mistake is made (perhaps only by early adopters from when it was supported)#2022-01-2622:05seancorfieldI didn't even know it was possible until very recently... ¯\(ツ)/¯#2022-01-2622:08Alex Miller (Clojure team)I don't think it's worth adding a check for it#2022-01-2622:08Alex Miller (Clojure team)and maybe some day it will actually be a thing#2022-01-2622:12borkdudeAgreed#2022-01-2700:12hlshipSo I'm in the middle of converting all our internal dependencies from :mvn/version to :git/url (this would only make sense inside Walmart) and I"m having problems:
> clj -X:dev:test
Error building classpath. Unable to compare versions for com.walmartlabs.clojure/util: {:mvn/version "1.3.0", :deps/manifest :mvn} and {:git/tag "v1.4.0", :git/sha "925c4a5c6984ec641c3fa0f786ec7e48da5e7c64", :git/url "
The issue is that I have, somewhere, a transitive dependency that uses the :mvn/version coordinate, and this library now uses a :git/url coordinate. I'm OK with the tool complaining that you can't mix and match this way BUT the above message does not help me hunt down which transitive dependency itself has the :mvn/version coordinate. Is there a tool option that helps here? Is there additional logging that can be enabled? -Strace doesn't seem to work; this error seem to prevent that from being written.#2022-01-2700:15hiredmanmaybe try clj -X:deps tree I forget what the exact command is, I think that should print out a tree of deps#2022-01-2700:19hiredmanhmmm, that likely won't work either 😕#2022-01-2700:28hiredmanok, if you temporarily add the problematic dep to your project's deps.edn, you can run clj -X:deps tree, and it will show all the places that dep is depended on, and show them all as overridden at the top level#2022-01-2700:29hiredmanthen you can find the problem, fix it, and remove the temporary addition to your project's deps.edn#2022-01-2700:40hlshipRight; I'm trying adding a :local/root coord and running clj -Stree.#2022-01-2700:42hlshipNo dice; same message.#2022-01-2700:45hiredmanif you add a dependency to the deps.edn of the project where you are invoking clj on the problematic dependency com.walmartlabs.clojure/util it will override all the other deps transitive deps (closer to the root of the tree wins) and let you run tree#2022-01-2700:46hlshipI thought so too. Let me try with :git/url instead of :local/root.#2022-01-2700:46hiredmanI say it will do this because I tried it locally#2022-01-2700:47hlshipAh, bad map nesting.#2022-01-2700:47hiredmanI am in project A which depends on projects C and D, projects C and D both have deps on tools.deps.alpha, one with maven and one with git#2022-01-2700:47hiredmanah#2022-01-2700:50hlshipThink I found it.#2022-01-2700:50hlshipThanks!#2022-01-2817:07imreIs there a way to force clojure -Spath to return absolute paths even for project paths like str:test?#2022-01-2817:12Alex Miller (Clojure team)No - what problem are you trying to solve?#2022-01-2817:14imremerge classpaths from multiple clojure projects in subfolders#2022-01-2817:14imrefor some analyzer step in our ci#2022-01-2817:15imrebut that's fine, I can work around it, just wanted to check if there was a way#2022-01-2817:22imreone more thing: will Clojure always treat the deps.edn in the current directory the Project source or is there a way to override that? like clojure -Spath -depsedn=./subfolder/deps.edn?#2022-01-2817:23imreI'm guessing not, based on the reference#2022-01-2817:25imreyep, appears so, nevermind 🙂#2022-01-2817:29seancorfield@U08BJGV6E If you don't mind omitting the user deps.edn, you could point the CLJ_CONFIG env var at the subfolder and clojure would treat that as your user deps.edn instead. It would still then also include the project deps.edn however.#2022-01-2817:30seancorfieldYou could also do -Sdeps '{:deps {subfolder/code {:local/root "subfolder"}}}' -Spath which would give you a combined classpath of that subfolder and the project deps.edn files.#2022-01-2817:31imreThank you I figured it is actually easier to just go (cd subfolder && clojure -Spath ) instead#2022-01-2817:31imrethose are nice tricks though#2022-01-2817:31seancorfieldIf you want "multiple clojure projects in subfolders", the :local/root deps might be the way to go there? If you only want one subfolder, then cd is definitely easier.#2022-01-2817:32imrebut does carry the caveats of considering any deps.edn in the cwd - my intention is to examine projects in multiple subfolders in isolation#2022-01-2817:32imreso what I ended up doing is looping over subfolders and going cd in parentheses#2022-01-2817:33imrethat keeps it clean and doesn't mix .cpcaches either#2022-01-2819:02borkdudeAnother trick:
clojure -Spath -Sdeps '{:deps {current/deps {:local/root "."}}}'
and strip the leading relative directories#2022-01-2821:10souenzzoclj -Spath | tr ':' '\n' | while read i; do realpath "$i"; done | tr '\n' ':'#2022-01-2821:16imre@U2J4FRT2T thank you, a version of that is what I ended up arriving at!#2022-01-2821:20borkdudeSlightly longer version :)
clj -Spath | bb '(as-> *in* $ (slurp $) (str/trim $) (str/split $ #":") (map fs/real-path $) (str/join ":" $))'
#2022-01-2822:09imrefind . -name deps.edn \
| while read p
do
dirname "$p"
done \
| while read dp
do
realpath "$dp"
done \
| sort \
| while read dp
do
( cd "$dp" && clojure -Spath -Sforce -Srepro \
| tr ":" "\n" \
| while read p
do
realpath "$p"
done
)
done \
| sort \
| uniq \
| tr "\n" ":"#2022-01-2901:57Lukas Domagala> merge classpaths from multiple clojure projects in subfolders
It feels like I’ve seen this request a few times in the last months, spread over different channels. Would it make sense to build some version of it into the cli?#2022-01-2907:49nivekuil$ clj -Sdeps "{:deps {org.clojure/clojure {:mvn/version \"1.11.0-alpha4\"}}}"
Clojure 1.10.1
user=>
running this inside a project dir, and it's starting the older clojure version instead of 1.11. works as expected anywhere else. is there some cache I have to clear? cli version is 1.10.3.1069#2022-01-2908:02hiredmanTry with -Stree at the end#2022-01-2908:08nivekuiltree has the expected version
org.clojure/clojure 1.11.0-alpha4
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
#2022-01-2908:09dpsutton❯ echo '{:deps {org.clojure/clojure {:mvn/version "1.9.0"}}}' > deps.edn
❯ clj -Sdeps "{:deps {org.clojure/clojure {:mvn/version \"1.11.0-alpha4\"}}}"
Clojure 1.11.0-alpha4
user=>
❯ clj
Clojure 1.9.0
user=>
i recreated this with a deps.edn with an explicit version and the Sdeps version still won#2022-01-2908:11nivekuilyeah, it should be merged last and works in other projects, but for some reason this one really likes 1.10.1#2022-01-2908:15nivekuilok, I bisected my deps.edn. turns out it will only load 1.10.1 if meander/epsilon {:mvn/version "0.0.643"} is there#2022-01-2908:15nivekuilif I comment that line out I can load any clojure version.. odd#2022-01-2908:37dpsuttonlooks like they've got a clojure in the jar:
clojure/core$inc.class#2022-01-2908:38dpsuttonmeander 239 doesn't have this issue#2022-01-2908:39nivekuilI bumped it to 650 and it's fine there too :)#2022-01-2908:40dpsuttonyeah. they just had a bad build i guess#2022-01-3110:14imreFound this today. It appears -X:deps gets tricked by a user.clj on the path that requires a namespace that is on the project classpath but not on the classpath that -X:deps itself uses?
Is this expected? Is there any better way to handle this than adding something like -Sdeps '{:aliases {:no-paths {:replace-paths []}}}' -A:no-paths?
; # enviroment
;
; clojure -Srepro -version
Clojure CLI version 1.10.3.1069
;
; # deps & source
;
; echo '{:paths ["src"] :deps{org.clojure/java.classpath {:mvn/version "1.0.0"}}}' > deps.edn
; mkdir src && echo '(ns user (:require clojure.java.classpath))' > src/user.clj
;
; # launch repl
;
; clj -Srepro
Clojure 1.10.3
user=> (find-ns 'clojure.java.classpath)
#object[clojure.lang.Namespace 0x21325036 "clojure.java.classpath"]
user=>
; # check deps
;
; clojure -Sforce -Srepro -X:deps list :license :none
Exception in thread "main" Syntax error compiling at (user.clj:1:1).
at clojure.lang.Compiler.load(Compiler.java:7652)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:368)
at clojure.lang.RT.maybeLoadResourceScript(RT.java:364)
at clojure.lang.RT.doInit(RT.java:486)
at clojure.lang.RT.init(RT.java:467)
at clojure.main.main(main.java:38)
Caused by: java.io.FileNotFoundException: Could not locate clojure/java/classpath__init.class, clojure/java/classpath.clj or clojure/java/classpath.cljc on classpath.
at clojure.lang.RT.load(RT.java:462)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__6856.invoke(core.clj:6115)
at clojure.core$load.invokeStatic(core.clj:6114)
at clojure.core$load.doInvoke(core.clj:6098)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5897)
at clojure.core$load_one.invoke(core.clj:5892)
at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
at clojure.core$load_lib.invokeStatic(core.clj:5936)
at clojure.core$load_lib.doInvoke(core.clj:5917)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$load_libs.invokeStatic(core.clj:5974)
at clojure.core$load_libs.doInvoke(core.clj:5958)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$require.invokeStatic(core.clj:5996)
at clojure.core$require.doInvoke(core.clj:5996)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at user$eval138$loading__6737__auto____139.invoke(user.clj:1)
at user$eval138.invokeStatic(user.clj:1)
at user$eval138.invoke(user.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.eval(Compiler.java:7170)
at clojure.lang.Compiler.load(Compiler.java:7640)
... 6 more
;
; # check deps with workaround
;
; clojure -Sdeps '{:aliases {:no-paths {:replace-paths []}}}' -A:no-paths -Sforce -Srepro -X:deps list :license :none
org.clojure/clojure 1.10.3
org.clojure/core.specs.alpha 0.2.56
org.clojure/java.classpath 1.0.0
org.clojure/spec.alpha 0.2.194#2022-01-3110:45imreasked at https://ask.clojure.org/index.php/11522/x-deps-might-fail-when-user-clj-requires-something too#2022-01-3113:48Alex Miller (Clojure team)You could use -T:deps instead here#2022-01-3115:02imreooh let me read up on that#2022-01-3115:09imrethat's actually proving to be a challenge to google for, do you happen to have a link to where I can get more info?#2022-01-3116:01Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_aliased_tool_execution#2022-01-3116:13imreThank you!#2022-01-3116:14Alex Miller (Clojure team)but fyi, I have fixed the :deps alias for next release so -X:deps will be fine#2022-01-3116:19imreSaw the commit, thank you for that!#2022-01-3116:19imreReminds me of this one I opened some time ago btw: https://ask.clojure.org/index.php/9947/could-replace-deps-imply-replace-paths-by-default#2022-01-3115:23lread@alexmiller cljdoc CI https://app.circleci.com/pipelines/github/cljdoc/cljdoc/280/workflows/4e0f40cf-1b73-48a9-9f35-2ab8d0096466/jobs/3337?invite=true&invite=true#step-109-4 issue https://clojure.atlassian.net/browse/TDEPS-153 using v1.10.3.1058. But without a reproducible use case, I’m not sure how much value that is to you.#2022-01-3115:55lreadInteresting maybe @alexmiller: https://app.circleci.com/pipelines/github/cljdoc/cljdoc/282/workflows/829be0b5-171f-48fb-9228-6cadb5f476ad/jobs/3348?invite=true#step-108-4 under same environment.#2022-01-3115:57Alex Miller (Clojure team)not sure the second is any more interesting :)#2022-01-3116:00lreadHa! Maybe not. Have not experienced this issue myself yet, and seeing it twice so close together made me wonder. Could be just sheer luck.#2022-01-3116:04Alex Miller (Clojure team)yes, it's just a timing thing. buried in the guts of the maven stuff there is an unsynchronized collection being updated. if you happen to hit it in two threads simultaneously, it will blow up. mutable shared data is no fun and no fun to fix.#2022-01-3115:55Alex Miller (Clojure team)it's useful thanks, I've reopened TDEPS-153 and I'll take a look when I get a chance#2022-02-0103:34Patrixhey all, I’m looking for some pointers here.. Any help appreciated:
I’m looking at https://kit-clj.github.io/ , and want to use the 2 commands on this page:
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.381"}' :as new
clojure -Tnew create :template io.github.kit-clj :name yourname/guestbook
This is what I get with the first one:
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.381"}' :as new
Error building classpath. Unknown tool: tools
clojure --version
Clojure CLI version 1.10.3.1069
I’m definitely on the latest… not sure what I did wrong. I swear I ran these 2 commands a few weeks ago when kit was first announced, deleted the folder, trying again now..
I’m on macos 12.3 (beta), but behavior is the same on 12.2 (not beta). clojure is installed from MacPorts.#2022-02-0103:49Alex Miller (Clojure team)We don't release anything on macports so that's probably not a good install #2022-02-0103:50Alex Miller (Clojure team)The brew install will copy in the tools definition to -/.clojure/tools/tools.edn#2022-02-0105:32seancorfieldFWIW someone showed the guix Clojure CLI installer and that's also missing the tools stuff. I think these installers were based on pre-tools CLI and they are do custom installs of specific files and they seem to update the JAR version but don't revisit the actual files in more recent versions. The GitHub Actions installer was like that until the author reverted to using the official Linux installer to set things up. That's why these unofficial installers are really problematic 😞 @patrix#2022-02-0114:24Alex Miller (Clojure team)I would like to make the Linux installer work seamlessly on Mac too (I don't think it does right now)#2022-02-0105:47Patrixthat would all make sense, then.. I don’t have brew on my Macs because it annoys me 😉
I’ve managed to find the tools.edn and copy it to my ~/.clojure/tools, and it does appear to work now. And I’ve checked in the macports install, no tools.edn there… 😞
thanks for all the info! I probably should submit a bug to macports or something…#2022-02-0105:53phronmophobicI'm also on macports 😕 Here's the process I ended up using in case it's helpful:
1. Find url from https://github.com/clojure/homebrew-tools (eg. https://github.com/clojure/homebrew-tools/blob/master/Formula/clojure.rb)
2.
wget ""
tar xvf clojure-tools-1.10.3.1069.tar.gz
cd clojure-tools
sudo HOMEBREW_RUBY_PATH=`which ruby` ./install.sh /usr/local
#2022-02-0105:54Patrixoh, I tested it just now and seems to work:
I extracted the tools.dep from that file and put it in /opt/local/lib/clojure#2022-02-0105:55Patrixit probably won’t persist the next time macports updates clojure, we’ll see..#2022-02-0105:58phronmophobicyea, the method I used puts clojure outside of macport's management#2022-02-0106:00phronmophobicI'd rather macports be completely in charge of the dependency or do it myself rather than trying to share responsibility for the install.#2022-02-0106:02seancorfieldFWIW, I ended up using brew on macOS (despite its quirks) and then linuxbrew on Windows/WSL2 so I at least have the official Clojure tap/install going on. I don't use brew for anything else.#2022-02-0106:03seancorfieldThat way I can easily upgrade stable releases any time and I can also easily install any specific version (I have dozens of versions installed on both macOS and Ubuntu so I can run regression tests).#2022-02-0106:07phronmophobicI would love to use an officially supported install method, but as far as I can tell, homebrew and macports are incompatible#2022-02-0106:12seancorfieldAs problematic as Homebrew can be, I've had far worse problems with MacPorts over the years, TBH. Hence why I stick to Homebrew just for Clojure.#2022-02-0106:12seancorfield(and I don't use MacPorts for anything at all)#2022-02-0106:22PatrixI guess mileage does vary..#2022-02-0106:24phronmophobicI use macports for lots of things and it's non-trivial to switch. At some point, I looked into whether it would be worth switching and found that homebrew wouldn't work for me 🤷#2022-02-0106:26PatrixI was in the middle of fixing a production issue and needed to install a piece of software - everything else related, I had installed with brew. brew install xxxxxx ended up updating half my packages because of one minor python update, wasting valuable time.. that very same day I switched to macports#2022-02-0106:26Patrixanyway for now having tools.edn in /opt/local/lib/clojure and in ~/.clojure/tools/ will work#2022-02-0106:28Patrixthanks @U04V70XH6, @U064X3EF3, @U7RJTCH6J!#2022-02-0106:30seancorfieldYeah, I def. wouldn't install stuff generally with brew -- their belief that everything should always depend on the very latest version of every package is just insanity. IMO.#2022-02-0106:30seancorfieldIt's why Alex switched to a Clojure-specific tap and removed the dependency on the JDK (because brew would "helpfully" install whatever it thought was the latest JDK and override your chosen installation).#2022-02-0106:31seancorfieldAside from apt, I don't rely on package managers for anything normally. The maintainers often have very peculiar ideas about "how the world should be"...#2022-02-0106:57Patrixah makes sense why clojure is using a tap in brew, indeed#2022-02-0106:58Patrixyeah more and more I just want to run things in Docker, as much as I dislike it, just to avoid packagers assuming a million things about my system. But that brings its own pains#2022-02-0106:58Patrixespecially on a Mac haha#2022-02-0107:09seancorfieldI run Docker (on both my Mac and on Windows/WSL2) to provide Percona, Elastic Search (two instances), and Redis -- but I always run Clojure on the bare metal.#2022-02-0107:10seancorfieldBut I only develop in Clojure so I have no need of anything but a JDK installed (well, I have several versions, but JAVA_HOME controls those).#2022-02-0107:13PatrixMakes sense lol#2022-02-0114:18Alex Miller (Clojure team)Presumably someone did some work on macports for Clojure to be available at all there and it could be updated to work#2022-02-0200:03PatrixYes I suppose so, soon as I convince myself to create a Trac account on https://macports.org I’ll submit a bug report..#2022-02-0208:53Ben SlessNix / guix support?#2022-02-0218:59phronmophobicI'm embarrassed to say that I hadn't considered trying to submit a patch to the clojure Port for macports. It's been updated fairly recently. Just manually comparing the https://github.com/macports/macports-ports/blob/master/lang/clojure/Portfile and https://github.com/clojure/brew-install/blob/1.10.3/src/main/resources/clojure/install/install.sh, it seems like the only thing missing is copying tools.edn.
I'll https://guide.macports.org/#project.github and see what happens.#2022-02-1520:21phronmophobicbetter late than never: https://github.com/macports/macports-ports/pull/14016
hopefully, I followed the PR process correctly 🤞#2022-02-1520:52Alex Miller (Clojure team)thanks!#2022-02-1621:44phronmophobiclooks like my patch is now included 🎉
$ sudo port install clojure
---> Fetching archive for clojure
---> Attempting to fetch clojure-1.10.3.1075_1.darwin_21.noarch.tbz2 from
---> Attempting to fetch clojure-1.10.3.1075_1.darwin_21.noarch.tbz2.rmd160 from
---> Installing clojure @1.10.3.1075_1
---> Deactivating clojure @1.10.3.1075_0
---> Cleaning clojure
---> Activating clojure @1.10.3.1075_1
---> Cleaning clojure
---> Scanning binaries for linking errors
---> No broken files found.
---> No broken ports found.
---> Some of the ports you installed have notes:
clojure has the following notes:
Install the rlwrap port and invoke `clj` for more a more ergonomic REPL.
$ ls /opt/local/lib/clojure/
deps.edn example-deps.edn libexec/ tools.edn
$ rm ~/.clojure/tools/*
$ clojure -Sforce -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.381"}' :as new
Installed new
$ ls ~/.clojure/tools/
new.edn tools.edn #2022-02-1621:54Alex Miller (Clojure team)nice work!#2022-02-2303:01Patrixthanks @U7RJTCH6J!#2022-02-0113:36vlaaadAmazing that this git dep works:
io.gitlab.arbetsformedlingen/taxonomy-dev/backend/jobtech-taxonomy-common {:git/tag "v1.12" :git/sha "e5556b7"}#2022-02-0113:37vlaaadcan I rely on that, or should I specify :git/url and use saner lib name?#2022-02-0113:37vlaaad@cfleming also would be nice if Cursive didn’t complain about symbols with multiple slashes…#2022-02-0120:59cflemingIs that even a thing? I didn’t think that was legal.#2022-02-0121:00cflemingActually, I just read the rest of the conversation, never mind…#2022-02-0114:23borkdudesymbols with multiple slashes are invalid. https://clojure.org/reference/reader#2022-02-0114:23Alex Miller (Clojure team)I would not rely on that and use a git url instead#2022-02-0114:30vlaaadThanks!#2022-02-0114:32vlaaadI think there was some law of someone that states that with time all observable behaviors of a dependency/system will be relied upon#2022-02-0121:19cflemingThat’s the law of anyone developing tooling.#2022-02-0114:33borkdudehyrum#2022-02-0117:55vlaaadWouldn't it be nice to have svg "shields"/"badges" for readmes that show latest git dep versions? Shame that GitHub doesn't allow selecting text in these svgs...#2022-02-0414:13vlaaadI found there is already a thing for github versions, e.g.:
https://badge.fury.io/gh/cognitect-labs%2Ftest-runner.svg#2022-02-0120:44hlshipQ: Is there a reason why the tools can't identify the source/line/column of errors, e.g.:
~/workspaces/walmart/push-services/push-load-test-service > clj -X:dev:test
Error building classpath. Coord of unknown type: {}#2022-02-0120:44hlshipIt would helpful if it said "In file deps.edn, line 18" or something, especially if it's from a deps.edn from a dependency.#2022-02-0120:48Alex Miller (Clojure team)the point where that is detected is many steps away from when the deps is read (and may not even be from a file), and has already gone through multiple steps of merging/manipulation#2022-02-0120:49Alex Miller (Clojure team)not impossible, but it's probably a fair amount of work to get to that point#2022-02-0120:49Alex Miller (Clojure team)would probably be easier to catch with more validation during the read#2022-02-0122:57borkdudeclj-kondo does offer linting for deps.edn - feel free to suggest more checks in #clj-kondo @hlship#2022-02-0122:58borkdudeit already lints that example:#2022-02-0123:02Alex Miller (Clojure team)depending on where that's coming from above, you wouldn't see it outside your own deps.edn though#2022-02-0123:03borkdudetrue#2022-02-0414:13vlaaadI found there is already a thing for github versions, e.g.:
https://badge.fury.io/gh/cognitect-labs%2Ftest-runner.svg#2022-02-0312:53teodorluWhat's a good way to look up the latest stable coordinate for a package?
$COMMAND_OR_SCRIPT org.clojure/clojure
# Not by manually browsing maven central, looking at clojars, or copying from Github READMEs
1.10.3
#2022-02-0312:55teodorluso .. if I had actually read the docs, I would have found:
clj -X:deps find-versions :lib clojure.java-time/clojure.java-time
{:mvn/version "0.1.0"}
{:mvn/version "0.2.0"}
{:mvn/version "0.2.1"}
{:mvn/version "0.2.2"}
{:mvn/version "0.3.0"}
{:mvn/version "0.3.1"}
{:mvn/version "0.3.2"}
{:mvn/version "0.3.3"}#2022-02-0313:00teodorluWith
alias clj-dep-versions="clj -X:deps find-versions :lib"
, we can
clj-dep-versions org.clojure/clojure
{:mvn/version "1.0.0"}
{:mvn/version "1.1.0"}
{:mvn/version "1.2.0"}
{:mvn/version "1.2.1"}
{:mvn/version "1.3.0-alpha5"}
...#2022-02-0313:00teodorluBut I think I'll go with neil.#2022-02-0312:55borkdude@teodorlu I have a babashka script for this called neil, which currently just add the lib directly to your deps.edn:
$ neil add dep :lib org.clojure/clojure :deps-file foo.edn
$ cat foo.edn
{:deps {org.clojure/clojure {:mvn/version "1.11.0-beta1"}}
:aliases {}}#2022-02-0312:59teodorluooh, that's even nicer. And I persume faster startup than JVM.#2022-02-0312:59teodorluThank you 🙏#2022-02-0313:00borkdudehttps://github.com/babashka/neil#2022-02-0313:01borkdudeYou can also use it to add a tools.build config + file to your repo:
neil add build
#2022-02-0313:03teodorluProvided NixOS package works great :thumbsup:#2022-02-0313:39Godwin KoWhat’s the difference between clj -Stree & clj -X:deps tree? In one of my project using clj-http, the first command runs fine but the second one throw exception of could not locate file on classpath :thinking_face:#2022-02-0313:51Alex Miller (Clojure team)The first runs in your project classpath, the latter runs outside of it, but if you're just running those commands they should basically be the same. Can you share your clj version and the error?#2022-02-0314:29Godwin Koclj version is 1.10.3.967, here’s the output for clj -Stree
org.clojure/data.json 1.0.0
org.clojure/clojure 1.10.3
. org.clojure/spec.alpha 0.2.194
. org.clojure/core.specs.alpha 0.2.56
org.clojure/core.logic 1.0.0
org.clojure/tools.namespace 1.1.0
. org.clojure/java.classpath 1.0.0
. org.clojure/tools.reader 1.3.4
instaparse/instaparse 1.4.10
metosin/malli 0.2.1
. borkdude/dynaload 0.2.2
. borkdude/edamame 0.0.11-alpha.15
X org.clojure/tools.reader 1.3.2 :older-version
. org.clojure/test.check 1.1.0
org.clojure/math.combinatorics 0.1.6
org.clojure/core.match 1.0.0
org.clojure/data.priority-map 1.0.0
http-kit/http-kit 2.5.3
and here’s for clj -X:deps tree
Exception in thread "main" Syntax error compiling at (user.clj:1:1).
at clojure.lang.Compiler.load(Compiler.java:7652)
at clojure.lang.RT.loadResourceScript(RT.java:381)
at clojure.lang.RT.loadResourceScript(RT.java:368)
at clojure.lang.RT.maybeLoadResourceScript(RT.java:364)
at clojure.lang.RT.doInit(RT.java:486)
at clojure.lang.RT.init(RT.java:467)
at clojure.main.main(main.java:38)
Caused by: java.io.FileNotFoundException: Could not locate org/httpkit/client__init.class, org/httpkit/client.clj or org/httpkit/client.cljc on classpath.
at clojure.lang.RT.load(RT.java:462)
at clojure.lang.RT.load(RT.java:424)
at clojure.core$load$fn__6856.invoke(core.clj:6115)
at clojure.core$load.invokeStatic(core.clj:6114)
at clojure.core$load.doInvoke(core.clj:6098)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5897)
at clojure.core$load_one.invoke(core.clj:5892)
at clojure.core$load_lib$fn__6796.invoke(core.clj:5937)
at clojure.core$load_lib.invokeStatic(core.clj:5936)
at clojure.core$load_lib.doInvoke(core.clj:5917)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$load_libs.invokeStatic(core.clj:5974)
at clojure.core$load_libs.doInvoke(core.clj:5958)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$require.invokeStatic(core.clj:5996)
at clojure.core$require.doInvoke(core.clj:5996)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at user$eval138$loading__6737__auto____139.invoke(user.clj:1)
at user$eval138.invokeStatic(user.clj:1)
at user$eval138.invoke(user.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.eval(Compiler.java:7170)
at clojure.lang.Compiler.load(Compiler.java:7640)
... 6 more
#2022-02-0314:30Godwin KoI switch from clj-http to http-kit to see if that’s a library specific issue, but seem not……#2022-02-0314:31Godwin Komy deps.edn is a very simple one, nothing special I guess
{:paths ["src" "resources"]
:deps {http-kit/http-kit {:mvn/version "2.5.3"}
instaparse/instaparse {:mvn/version "1.4.10"}
metosin/malli {:mvn/version "0.2.1"}
org.clojure/core.logic {:mvn/version "1.0.0"}
org.clojure/core.match {:mvn/version "1.0.0"}
org.clojure/data.json {:mvn/version "1.0.0"}
org.clojure/data.priority-map {:mvn/version "1.0.0"}
org.clojure/tools.namespace {:mvn/version "1.1.0"}
org.clojure/math.combinatorics {:mvn/version "0.1.6"}}
:aliases {:nrepl {:extra-deps {cider/cider-nrepl {:mvn/version "0.25.5"}
refactor-nrepl/refactor-nrepl {:mvn/version "2.5.1"}}
:main-opts ["-m" "nrepl.cmdline"
"--middleware" "[cider.nrepl/cider-middleware,refactor-nrepl.middleware/wrap-refactor]"
"--interactive"]}}}#2022-02-0314:33Alex Miller (Clojure team)ah, so the difference is in your user.clj being included - there is actually a fix for this in the Clojure CLI released yesterday - 1.10.3.1075#2022-02-0314:49Godwin Koconfirm that 1.10.3.1075 fixed the issue, thx a lot :man-bowing::skin-tone-2:#2022-02-0314:52Alex Miller (Clojure team)happy coincidence :)#2022-02-0317:30seancorfieldFigured I'd move the conversation here from #announcements -- I'd lean toward just removing it as a file at this point but I don't know whether there are still tools out there relying on the files instead of using t.d.a as a library.
I think there might also be a little confusion if clojure -Sdescribe stops showing the root deps.edn file but the docs all continue to talk about merging three deps.edn files?
Updating the brew-install version to "match" the t.d.a resources' version is the path of least resistance but that redundancy is definitely starting to bother me.#2022-02-0317:44Alex Miller (Clojure team)there are still tools (notably the datomic stuff) that I believe are still pulling the file via -Sdescribe so I'm going to continue letting it drag along for a while#2022-02-0317:45Alex Miller (Clojure team)I have filed prs/issues on projects I'm aware of still using old APIs using this older stuff to get it cleaned up but for now, I'm ok being lazy about cleaning it up#2022-02-0317:52seancorfieldOK, so just update the brew-install deps.edn to avoid confusion? (and to help those legacy tools work "properly")#2022-02-0317:55Alex Miller (Clojure team)I did that#2022-02-0317:55Alex Miller (Clojure team)I will probably also start just extracting it from tdeps so there is only one thing to update#2022-02-0320:51simonkatzIf I have an alias in a project-level deps.edn file, is there something I can add to my user-level deps.edn to add some dependencies to that alias? (Maybe I have :dev aliases in multiple projects, and I want to add some dev dependencies to all projects.) (If I have a :dev alias in both places, it seems that the user-level one is ignored rather then merged.)#2022-02-0320:54Alex Miller (Clojure team)you can use multiple aliases, so user deps.edn could have shared stuff and project level could have per-project#2022-02-0320:55Alex Miller (Clojure team)then clj -A:common:dev ... or whatever#2022-02-0321:00simonkatz@alexmiller OK, that makes sense. Thanks.
The problem (if that’s what it is) is that maybe I have the :dev alias shared across my team, and maybe shared scripts and/or a shared Emacs .dir-locals file specifying (cider-clojure-cli-aliases . "dev") , so that eg starting a REPL from Emacs does nice things. So it would be nice to be able to piggyback onto the :dev alias in my user-level deps.edn.
Maybe I’m just doing it wrong.#2022-02-0321:06simonkatzHmmm, I could agree with my team to have a :user-level-dev alias, and our shared scripts etc could refer to that.#2022-02-0414:08practicalli-john@nomiskatz user level aliases can be used in the Emacs .dir-locals.el configuration
I put more general aliases in the user level and them complement or override with specific configuration in the project deps.edn
So you could have (cider-clojure-cli-aliases . “:env/dev:project/dev”) where :env/dev is user level and :project/dev is project dep.edn
There are some more examples here https://practical.li/spacemacs/clojure-projects/project-configuration.html#2022-02-0414:30simonkatz@U05254DQM Thanks. I think that’s where I had got to with my :dev and :user-level-dev (now renamed to :user/dev) aliases. It’s good to know that seems to be the right approach.
I’ve added your linked-to page to my big list of deps.edn-related things to look at.#2022-02-0414:51jjttjjIf there's a library A I depend on with a git dep, has in its own dependencies a git dependency on library B with an wrong :git/sha and :git/tag combination, i get an error:
Error building classpath. Library <lib B> has sha and tag that point to different commits
This happens even if I add and :exclusions for lib B in my own deps.edn. First: is that supposed to happen? I think that it is based on https://clojure.org/reference/dep_expansion (it's an error in the expansion step which happens before exclusions). Second: is there a way to somehow get around this?#2022-02-0414:55seancorfieldInstead of trying to exclude B from A, have you tried putting the correct dependency on B directly into your deps.edn as a top-level dep?#2022-02-0414:56Alex Miller (Clojure team)That's a good idea#2022-02-0414:57Alex Miller (Clojure team)But I'll make a ticket for this, maybe it's something that could be handled with warning/continue somehow#2022-02-0414:57seancorfieldIf the tag and the SHA disagree, which should it pick though @U064X3EF3?#2022-02-0414:57Alex Miller (Clojure team)Neither#2022-02-0414:58Alex Miller (Clojure team)But if same dep exists elsewhere in the tree could use that#2022-02-0414:59seancorfieldAh, gotcha. So it would warn about the mismatch and drop that library...#2022-02-0414:59Alex Miller (Clojure team)You can't drop the lib but you could not consider that version#2022-02-0414:59jjttjj> have you tried putting the correct dependency on B directly into your deps.edn as a top-level dep?
Yeah that definitely be the ideal answer and in my case I can, just had me thinking it would be more annoying to have to do if I didn't control library B and have to fork it just for that. Not that that would be tools dep's fault/responsibility, just curious if it had a way to mitigate it.#2022-02-0415:01jjttjjOh wait I read the thing I quoted wrong. I thought you meant fix it in B#2022-02-0415:01jjttjjI did in fact already have the corrected dependency on B in my own deps.edn when it was giving me the error#2022-02-0415:48Alex Miller (Clojure team)well, that seems like a bug. I added https://clojure.atlassian.net/browse/TDEPS-217 to track this. needs more detail but placeholder for now.#2022-02-0415:52jjttjjThanks, I could try to get a minimal repro over the weekend if that'd be useful#2022-02-0415:55Alex Miller (Clojure team)probably not needed, I should be able to repro easily#2022-02-0417:14Adam Helins(Sorry if it's a common request, I have found much references about it)
I often feel like being able to provide aliases when using a :git/sha dep would provide quite a lot of flexibility. Has it been considered more thoroughly?#2022-02-0417:41Alex Miller (Clojure team)We have a ticket for it, have not thought about it deeply#2022-02-0418:36Adam HelinsA use case is being able to consume from monorepo-like projects (e.g. several libs organized in a single repo)#2022-02-0420:41borkdude@adam678 This is already possible using :deps/root#2022-02-0420:42borkdudeExample:
{:deps
{cheshire/cheshire {:mvn/version "5.10.0"}
com.taoensso/nippy {:mvn/version "3.1.1"}
com.github.clj-easy/graal-config-cheshire {:git/url ""
:git/sha "b06e33694d2c878169958f7317ea01d9c0353ab4"
:deps/root "config/cheshire/cheshire"}
com.github.clj-easy/graal-config-nippy {:git/url ""
:git/sha "b06e33694d2c878169958f7317ea01d9c0353ab4"
:deps/root "config/com.taoensso/nippy"}}}#2022-02-0508:28Adam HelinsYes but this works only when each "subproject" has its own deps.edn, but not for setups having just one root deps.edn#2022-02-0508:45borkdudetrue#2022-02-0420:53Ian FernandezHow can I debug this error?
Caused by: org.apache.http.client.HttpResponseException: status code: 400, reason phrase: Repository version policy: SNAPSHOT does not allow version: 0.7.0 (400)
#2022-02-0420:54Ian Fernandezlike, I'm trying to add inside :mvn/repos key, {"my-repo" {:url ,,, :snapshots false}}#2022-02-0420:55Ian Fernandezthis keyword works?#2022-02-0420:55Ian FernandezI'm translating a project.clj to a deps.edn and this got into my way#2022-02-0421:11Alex Miller (Clojure team)this is not currently supported for deps.edn#2022-02-0421:19Ian Fernandez:thinking_face: there's another way to correct this error that I'm having?#2022-02-0421:23Ian Fernandezgot working adding another repo for it#2022-02-0421:23Ian Fernandezthanks#2022-02-0500:47hlshipWhat's the thinking on having one alias "depend" on another; for example, in our libs, we tend to have a :dev alias that defines test namespaces and dependencies, and then a :test alias that brings in the test-runner dependency and exec-fn; we end up having to execute clj -X:dev:test. What would be the repercussions of something like {:depends-on [:dev]} to incorporate aliases. I have to imagine this is something you've already considered.#2022-02-0508:46practicalli-johnMy own thoughts are about the complexity this will bring, especially if one alias depends one another which then depends on another…
At present aliases are defined without dependencies on other aliases, so I consider them akin to pure functions. The results of composing aliases, :dev:test, it therefore simple to understand
Adding dependencies between aliases feel like it will easily break this simple model and make aliases more complex.
I am sure we appreciate the circular dependencies challenges that arise from library dependencies when each library includes a range of other libraries.
I assume alias dependencies would also increase the complexity of the tooling that needs to processes those alias dependencies.
I therefore value keeping aliases simple and self-defined#2022-02-0501:11seancorfield@hlship I think that's probably one of the most requested features in one form or another... There's at least one Q on http://ask.clojure.org about (which may have a JIRA link).#2022-02-0501:13seancorfieldHere's one of them https://ask.clojure.org/index.php/9302/allow-transitive-deps-edn-dependencies-to-inherit-aliases#2022-02-0501:14seancorfieldThis is another related Q https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects#2022-02-0501:15seancorfieldAnd this one is also related https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases#2022-02-0501:17seancorfieldThe latter would allow you to have a :test alias that expanded to :dev:runner for example.#2022-02-0501:18seancorfield(the first two are more about aliases in projects you depend on -- your :depends-on threw me for a second there, until I re-read it and posted that third link!)#2022-02-0508:46practicalli-johnMy own thoughts are about the complexity this will bring, especially if one alias depends one another which then depends on another…
At present aliases are defined without dependencies on other aliases, so I consider them akin to pure functions. The results of composing aliases, :dev:test, it therefore simple to understand
Adding dependencies between aliases feel like it will easily break this simple model and make aliases more complex.
I am sure we appreciate the circular dependencies challenges that arise from library dependencies when each library includes a range of other libraries.
I assume alias dependencies would also increase the complexity of the tooling that needs to processes those alias dependencies.
I therefore value keeping aliases simple and self-defined#2022-02-0508:51practicalli-johnTaking my user level aliases as an example, I create many aliases with a specific purpose which can mostly be used by themselves. This makes the Clojure command simple and more deterministic
https://github.com/practicalli/clojure-deps-edn#2022-02-1120:56jasonjcknAnyone have tips for managing env vars in a project , so cider-jack-in has the right env vars, etc. Having multiple configurations dev, prod, etc. I’m testing direnv right now.#2022-02-1121:27Darin DouglassI like the defaults to work with local development then override the needful vars during deployment#2022-02-1212:10practicalli-johnI use juxt/aero for managing environment variables in Clojure projects.
They work well with component lifecycle services (Integrant, Component, etc)
https://practical.li/clojure-web-services/repl-driven-development/integrant-repl/#aero-tag-literals#2022-02-1212:12practicalli-johnSpacemacs (and Doom Emacs I think) have .env files that can be reloaded.
However, it the values of an environment variable change, then I believe the REPL will need to be restarted to pick them up.#2022-02-1203:14Alex Miller (Clojure team)https://clojurians.slack.com/archives/C015AL9QYH1/p1644635680901549#2022-02-1203:16seancorfieldInstalling now, not sure how much testing it'll get over the weekend as we have a lot of chores to do but will report back once I've played with it.#2022-02-1203:33seancorfieldA quick test of tree and list in our work repo with :aliases looks good.#2022-02-1516:58cjsauerHow would I pass a JSON string as an arg to a clojure -X:bench execution when that JSON is generated by a sub-command? Example:
clojure -X:bench :json "$(my_cmd --blah)"
I keep getting errors like:
Unreadable arg: "{\"blah\": 1}"
Which suggest to me that the quoting is wrong#2022-02-1516:58cjsauerI can't seem to find the magic incantation that gets the quoting right#2022-02-1517:00Alex Miller (Clojure team)probably tricky#2022-02-1517:00Alex Miller (Clojure team)you need the outer double quotes for the inner to be evaluated#2022-02-1517:00cjsauerI can pass the JSON in manually like this:
clojure -X:bench :json-plan '"{\"blah\": 1}"'
And it works, but the single quotes seem to preclude the sub-command#2022-02-1517:00Alex Miller (Clojure team)but you also need explicit double quotes#2022-02-1517:00borkdudemaybe "\"$(...)\""?#2022-02-1517:00Alex Miller (Clojure team)so yeah, something like that#2022-02-1517:00borkdudeor just use -M -m :)#2022-02-1517:01Alex Miller (Clojure team)or pipe into stdin and read that :)#2022-02-1517:01borkdudeor god forbid, a REPL#2022-02-1517:01cjsauer"\"$(...)\"" results in clojure seeing only the opening { character...#2022-02-1517:01cjsauerYea, I may have to switch my approach here 😉#2022-02-1517:01dpsuttonor put it in a file and pass the file as the arg. would be helpful when keeping interesting test cases around#2022-02-1517:02borkdude^ what he said#2022-02-1517:02cjsauerYep, I'll just do that#2022-02-1517:02cjsauerthanks all#2022-02-1517:02cjsauerWas breaking my fingers in the name of concision haha#2022-02-1517:02cjsauerit must fit on one line#2022-02-1517:03cjsauerCan -X functions read from stdin?#2022-02-1517:04dpsuttonwe have an interesting test runner that can take a namespace, a list of namespaces, or a directory and run all tests contained in those. You could do similar logic so you can pass a file, a list of files, or a directory. clj -X:bench -json examples/weird/funky.json or clj -X:bench -json examples/stress or what have you. Or you can put different keys for the different types#2022-02-1517:05cjsauerNice. Actually it looks like I can just use (read-line) to grab the JSON from stdin!#2022-02-1517:07borkdudein case it's multi-line: (slurp *in*)#2022-02-1517:08cjsauerThanks, that's probably safer#2022-02-1517:43seancorfield@cjsauer Try '"'$(my_cmd --blah)'"' -- although that may still not work with the internal quoting.#2022-02-1517:44seancorfield(but, yeah, reading JSON from stdin is likely to be easier than wrangling shell quoting)#2022-02-1518:17borkdude@seancorfield The outer single quotes will prevent the $() from being "eval-ed"#2022-02-1518:31seancorfieldNo it won't.#2022-02-1518:31seancorfield(! 678)-> echo '"'$(clojure -Spath)'"'
"src:/Users/sean/.m2/repository/org/clojure/clojure/1.11.0-beta1/clojure-1.11.0-beta1.jar:/Users/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/Users/sean/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar"
#2022-02-1518:33seancorfield(but it does not handle quotes in the content unfortunately)#2022-02-1518:33seancorfield(! 754)-> echo '"'$(cat deps.edn)'"'
"{:aliases { :hiccup {:extra-deps {hiccup/hiccup {:mvn/version "RELEASE"}}} :exec {:exec-fn [a b] :ns-default exec} :build {:extra-deps {io.github.seancorfield/build-clj {:local/root "/Developer/workspace/build-clj"}}} :build-slim {:extra-deps {io.github.seancorfield/build-clj {:local/root "/Developer/workspace/build-clj/slim"}}} :dev {:jvm-opts ["-Dclojure.compile.warn-on-reflection=true"]}}}"
so you'd need to pipe it through some sed escaping or some such...#2022-02-1518:34seancorfield(! 755)-> echo '"'$(cat deps.edn|sed 's;";\\";g')'"'
"{:aliases { :hiccup {:extra-deps {hiccup/hiccup {:mvn/version \"RELEASE\"}}} :exec {:exec-fn [a b] :ns-default exec} :build {:extra-deps {io.github.seancorfield/build-clj {:local/root \"/Developer/workspace/build-clj\"}}} :build-slim {:extra-deps {io.github.seancorfield/build-clj {:local/root \"/Developer/workspace/build-clj/slim\"}}} :dev {:jvm-opts [\"-Dclojure.compile.warn-on-reflection=true\"]}}}"#2022-02-1521:18borkdude@seancorfield Ah, I see!#2022-02-1602:12cjsauer@seancorfield nice. I went with the stdin approach given that JSON is going to be nye impossible to trust as a bash arg.#2022-02-1608:09borkdude@cjsauer In bash’s defence, it should work fine when passing the arg as JSON to a -M -m invocation. The extra quoting required by the EDN parsing of -X is what makes this hard. In Clojure CLI’s defense, it can make other things easy ;-).#2022-02-1608:51borkdudePerhaps the clojure CLI can have some convention to prevent the argument from being parsed as EDN. E.g.:
clojure -X foo/bar :foo 1 :bar @$(.. some json here ...)
An @ isn't valid in EDN anyway and it's just one character to check to see if the arg must be handled as a literal string instead.#2022-02-1614:18Alex Miller (Clojure team)you know where to make requests... :)#2022-02-1614:40borkdudeBefore making requests, hearing a firm :face_vomiting: , no, or such can save me time ;)#2022-02-1614:40borkdudeBut a "hmm, not a bad idea, will consider, could you make an ask" will be enough for me to create one :)#2022-02-1614:41borkdudeI'll take your reply as the latter.#2022-02-1614:43Alex Miller (Clojure team)willing to consider :)#2022-02-1614:45borkdudeAsked:
https://ask.clojure.org/index.php/11585/consider-convention-for-bypassing-edn-parsing-in-x-usage#2022-02-1614:47borkdudeI would be super happy if this got addressed!#2022-02-1614:57lreadYeah, like most folks, I too prefer to ping on Slack for a :face_vomiting: or facepalm reaction before investing time in crafting a more formal issue or request.#2022-02-1616:50borkdudeI may have found a nicer convention as the leading character:
$ clojure -X foo/bar :foo /dude :bar '"dude"' ;;=> (foo/bar {:foo "dude" :bar "dude"})
A leading slash. Normally this would result into an invalid symbol (or something else invalid) when reading EDN, so it would probably not be breaking anything. It's kind of irrelevant if we go with / or @ but since @ in command line args often denotes reading a file's contents, maybe it's better to reserve that one for later.#2022-02-1616:51borkdudeI'm going to try both on Windows, to see if it's going to behave correctly#2022-02-1616:52Alex Miller (Clojure team)can you put suggestions on the ask question please?#2022-02-1616:53borkdudeI will.#2022-02-1616:55borkdudeJust experimenting, I will paste the outcome in the issue.
After trying bash, cmd.exe and powershell, I found that / may work better than @ since @ seems to not play well in powershell.#2022-02-1616:59borkdudeIssue updated.#2022-02-1706:20practicalli-johnNot sure I would feel secure pushing dynamically generated JSON directly as an argument to a Clojure function on the command line.
I would be much happier if the command pushed JSON to a file and then that filename passed as an argument to the Clojure function.
As well as being simpler, it would also keep a separate record of what JSON was passed to the function.
Or at least pass the generated JSON through a tool to convert to EDN safely and then pass as an argument.
Adding an escape character / for certain argument values feels a bit cryptic in terms of usability, but admittedly I don't have any use cases that would take this approach anyway.#2022-02-1707:52borkdudeIt's not just about JSON, it's about having to quote arguments if you just want the vanilla cmd line string in general. A workaround is to parse symbols and then convert them to strings again, but this can result in invalid symbol reads. Quoting in shells is just hard and also makes it a bit more verbose. Having a way to bypass this is imo useful. Having a less cryptic way: all for it! I'm curious about other alternatives :).#2022-02-1713:08rickmoynihanhmm… You can always use *command-line-args* but you’d probably want to switch from -X to -M for invocation in order to stop that parsing altogether. For the JSON case you’d almost certainly be better passing it via stdin or a file though.
Regarding the separator / it also strikes me as confusing because it’s the unix path separator and iirc it used to be a /arg separator in DOS — not used Windows for over 20 years though, so no idea if that’s still relevant 😆#2022-02-1713:12rickmoynihanIntroducing a special escape character to simplify escaping rules also feels like a bit of a contradiction#2022-02-1713:15rickmoynihan@borkdude: Also one question; in your JSON usecase how would such a thing know to escape strings in the JSON from both the clojure reader and the shell?#2022-02-1713:17rickmoynihanI definitely get the motivation for such a thing, I’m just not sure there’s going to be a good solution :thinking_face:#2022-02-1713:42borkdude@rickmoynihan I know command-line-args and -M, that's what I've been recommending when people run into this, but some of the tooling went the -X route and even deprecated -M usage, so you will run into this no matter what.#2022-02-1713:44rickmoynihanyeah though arguably that’s a problem for the tools… I mean they should pick the flag/approach for the job.#2022-02-1713:46Alex Miller (Clojure team)for me, the open question is whether this is something that should be annotated on the function or at the CLI#2022-02-1713:51rickmoynihanyeah I was thinking you’d want to opt certain keys out of the processing outside of this… perhaps as some kind of args data… maybe some kind of :tools.deps/dont-read #{:arg1 :arg2} structure??#2022-02-1713:51rickmoynihanin :exec-args though that might be a level too deep ??#2022-02-1713:56Alex Miller (Clojure team)I was actually thinking about on the function itself, but that's another option#2022-02-1713:56Alex Miller (Clojure team)it needs a design sheet to compare, I'll get around to it eventually#2022-02-1713:56rickmoynihanyeah though I recall a bunch of discussions from years back about how vars are overloaded etc 🙂#2022-02-1715:12borkdudeI'll document these alternatives in the issue#2022-02-1715:22borkdudeI think the var metadata approach is a nice one: then it will work regardless of configuration and it won't clutter the invocation#2022-02-1715:25borkdude(defn foo
{:tools.deps/exec-skip-edn-read #{:a}}
[{:keys [a b c]}]
...) #2022-02-1713:46rickmoynihanI think the problem is that affordances of -X and build.clj are too good 🙂#2022-02-1713:48rickmoynihanwhich meant perhaps people switched from -M too soon…
I speak as someone who doesn’t enjoy maintaining both -M and -X interfaces — but I can’t shake the fact that there are trade offs between them#2022-02-1713:46Alex Miller (Clojure team)well, that's the first question :)#2022-02-1717:55Matheus BernardesWhen I try to download all version of com.google.javascript/closure-compiler-unshaded t.d.a returns nil, digging in the problem I saw that this https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/extensions/maven.clj#L190 is being responsible to the problem since it’s only matching versions starting with number and all the versions of com.google.javascript/closure-compiler-unshaded starts with a letter v.
Is it supposed to work like that or is it a bug?#2022-02-1717:57Alex Miller (Clojure team)Maven version components are (usually) numbers#2022-02-1717:57Alex Miller (Clojure team)Can you back up to what you're actually calling?#2022-02-1717:58Matheus Bernardesclj -X:deps find-versions :lib com.google.javascript/closure-compiler-unshaded#2022-02-1717:59Matheus Bernardeshttps://mvnrepository.com/artifact/com.google.javascript/closure-compiler-unshaded#2022-02-1718:08Alex Miller (Clojure team)Thx, that helps#2022-02-1718:09Alex Miller (Clojure team)That is definitely a presumption baked into that program. Can you put this in an https://ask.clojure.org question for me?#2022-02-1718:17Matheus Bernardeshttps://ask.clojure.org/index.php/11588/t-d-a-does-not-support-mvn-versions-starting-with-letters#2022-02-1718:37Alex Miller (Clojure team)Thanks!#2022-02-1812:14Adam HelinsI'm currently converting a project to use tools.build
I'm getting several license conflicts when uberjaring, aren't those resolved by default?
Execution error (ExceptionInfo) at clojure.tools.build.tasks.uber/explode (uber.clj:172).
Cannot write META-INF/license/LICENSE.boringssl.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded.
#2022-02-1812:20delaguardoNo, they don't. This most likely happens because one of your dependencies brings META-INF/license file#2022-02-1812:22delaguardoI solve it by adding this
:exclude [#"(?i)^META-INF/license/.*"
#"^license/.*"]
As an option for uber function#2022-02-1812:29Adam HelinsThanks, does the job perfectly!#2022-02-1812:48Adam Helins(sorry for posting in the wrong channel!)#2022-02-1818:43seancorfieldWe switched from grpc-netty-shaded to grpc-okhttp to avoid that license conflict -- it should be a drop-in replacement for most uses -- and it shrank our JAR files by about 10MB too, as I recall.#2022-11-0415:44genekimThank you to @U04V4KLKC and @U04V70XH6 — I had the same problem with this GCP library: build.clj that used https://github.com/seancorfield/build-clj with the following actions:
(ns build
(:require [clojure.tools.build.api :as b]
[org.corfield.build :as bb]))
(def lib 'genek/pubsub-web)
(defn uber2 "build uber JAR locally." [opts]
(-> opts
(assoc :lib lib :version version
:exclude [#"(?i)^META-INF/license/.*"
#"^license/.*"])
(bb/uber)))
#2022-02-1814:00Adam HelinsWondering about the status and urgency of https://clojure.atlassian.net/browse/CLJ-2638
Historically, I've always found the idea very useful in the context of monorepos. Now, we are in the process of merging repos at work and once again, this idea popped-in and we were agreeing it would save us from a lot of troubles.#2022-02-1814:08borkdudePerhaps until that's there, this could be worth a look. https://github.com/exoscale/deps-modules#2022-02-1814:23Alex Miller (Clojure team)Looks like that ticket is in the wrong place, I'll move it...#2022-02-1814:24Alex Miller (Clojure team)don't think I'm imminently going to look at it though#2022-02-1818:44seancorfield@UCFG3SDFV We have a pretty big monorepo at work and, to be honest, we do not find this to be an issue. What sort of alias issues are you running into?#2022-02-1819:07Alex Miller (Clojure team)rather than answering here, it would be far more helpful to me to collect it at https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases#2022-02-1909:01Adam HelinsI just did, hopefully I made my case clear 🙂#2022-02-1902:55cflemingHi all, I see t.d.a uses requiring-resolve, so I guess it’s a thing that deps requires Clojure 1.10+?#2022-02-1902:57cflemingI’m seeing errors in my exception tracker like: Unable to resolve symbol: requiring-resolve in this context. This executes deps using the user’s version of Clojure, so I’m guessing that they’re using an older Clojure.#2022-02-1904:31Alex Miller (Clojure team)“This” = what?#2022-02-1904:34Alex Miller (Clojure team)And yes, tools deps requires 1.10#2022-02-1905:08seancorfield@U0567Q30W I'm a bit puzzled why you're trying to execute t.d.a. in the context of the user's Clojure setup/context?#2022-02-1907:47cflemingSo that was one of my more terrible problem reports, since it was vague and also wrong. Revisiting the code, I’m not using the user’s context at all. I either use the uberjar distributed with the CLI, or I use t.d.a downloaded from Maven. So I’m not sure why users would be seeing that error, since they should be using the correct version of Clojure. Looking at the stacktrace for the root exception (below), it seems like it’s probably failing when loading clojure.tools.deps.alpha.util.s3-transporter via clojure.tools.deps.alpha.util.maven. Is there anything else that might cause that to happen?
java.lang.RuntimeException: Unable to resolve symbol: requiring-resolve in this context
at clojure.lang.Util.runtimeException(Util.java:221)
at clojure.lang.Compiler.resolveIn(Compiler.java:7299)
at clojure.lang.Compiler.resolve(Compiler.java:7243)
at clojure.lang.Compiler.analyzeSymbol(Compiler.java:7204)
at clojure.lang.Compiler.analyze(Compiler.java:6752)
at clojure.lang.Compiler.analyze(Compiler.java:6729)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3813)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7005)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.analyze(Compiler.java:6729)
at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3813)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7005)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.access$300(Compiler.java:38)
at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6368)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7003)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6991)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.analyze(Compiler.java:6729)
at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:6100)
at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5460)
at clojure.lang.Compiler$FnExpr.parse(Compiler.java:4022)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7001)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6991)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.access$300(Compiler.java:38)
at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:595)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7003)
at clojure.lang.Compiler.analyze(Compiler.java:6773)
at clojure.lang.Compiler.analyze(Compiler.java:6729)
at clojure.lang.Compiler.eval(Compiler.java:7066)
at clojure.lang.Compiler.load(Compiler.java:7514)
at clojure.lang.RT.loadResourceScript(RT.java:379)
at clojure.lang.RT.loadResourceScript(RT.java:370)
at clojure.lang.RT.load(RT.java:460)
at clojure.lang.RT.load(RT.java:426)
at clojure.core$load$fn__6548.invoke(core.clj:6046)
at clojure.core$load.invokeStatic(core.clj:6045)
at clojure.core$load.doInvoke(core.clj:6029)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5848)
at clojure.core$load_one.invoke(core.clj:5843)
at clojure.core$load_lib$fn__6493.invoke(core.clj:5888)
at clojure.core$load_lib.invokeStatic(core.clj:5887)
at clojure.core$load_lib.doInvoke(core.clj:5868)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$load_libs.invokeStatic(core.clj:5925)
at clojure.core$load_libs.doInvoke(core.clj:5909)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$require.invokeStatic(core.clj:5947)
at clojure.core$require.doInvoke(core.clj:5947)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at clojure.tools.deps.alpha.util.maven$eval715$loading__6434__auto____716.invoke(maven.clj:9)
at clojure.tools.deps.alpha.util.maven$eval715.invokeStatic(maven.clj:9)
at clojure.tools.deps.alpha.util.maven$eval715.invoke(maven.clj:9)
...etc etc...
#2022-02-1918:48Alex Miller (Clojure team)the only other thing I could think of would be the parallel loading issues we used to see in the s3 transporter thread loading#2022-02-1918:50Alex Miller (Clojure team)that should be fixed in recent versions but maybe if you have an older version of tools.deps#2022-02-1919:55cflemingYeah, could be that the user just hasn’t updated their deps install. I won’t worry about that one unless it keeps happening then. Thanks!#2022-02-2110:05arohnerI’m calling tools.deps as a library, and occasionally seeing this exception in CI:
Exception in thread "main" java.lang.ClassCastException: class java.util.HashMap$Node cannot be cast to class java.util.HashMap$TreeNode (java.util.HashMap$Node and java.util.HashMap$TreeNode are in module java.base of loader 'bootstrap')
at java.base/java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1900)
at java.base/java.util.HashMap$TreeNode.treeify(HashMap.java:2016)
at java.base/java.util.HashMap.treeifyBin(HashMap.java:768)
at java.base/java.util.HashMap.putVal(HashMap.java:640)
at java.base/java.util.HashMap.put(HashMap.java:608)
at java.base/java.util.HashSet.add(HashSet.java:220)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:848)
at org.apache.maven.model.validation.DefaultModelValidator.validateId(DefaultModelValidator.java:825)
at org.apache.maven.model.validation.DefaultModelValidator.validateEffectiveModel(DefaultModelValidator.java:341)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:501)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:437)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:252)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:297)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$fn__771.invokeStatic(maven.clj:132)
at clojure.tools.deps.alpha.extensions.maven$fn__771.invoke(maven.clj:122)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__464$fn__466$fn__467.invoke(alpha.clj:406)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__187.invoke(concurrent.clj:35)
at clojure.lang.AFn.call(AFn.java:18)
StackOverflow suggests a race condition is likely#2022-02-2110:06borkdude@U050R7ECY This is a workaround I applied based on information from an ask issue:
https://github.com/clj-easy/graal-build-time/blob/b91cd78a977fdf81758705668f44b0d2ddad4bdc/build.clj#L1-L2#2022-02-2110:06borkdudeIt usually happens when you calculate more then one basis I think#2022-02-2110:08arohnerOh, so then upgrading to TDEPS 0.12.1098 might fix it?#2022-02-2110:08borkdudeWorth a shot!#2022-02-2110:16arohnerHrm, that stacktrace doesn’t match the one I see#2022-02-2110:41arohner0.12.1098 also contains a bump to maven-core 3.8.4, which contains a fix for https://issues.apache.org/jira/browse/MNG-7285 which might fix#2022-02-2113:34Alex Miller (Clojure team)I believe, even on the latest version, this can still happen. Deep inside the bowels of Maven there is some unsynchronized state held in either the system or session that I'm sharing across expansion threads that clearly maven is not expecting to be shared.#2022-02-2113:38Alex Miller (Clojure team)I have not had time to look at it more closely yet but I suspect the real solution is to not share stuff because Maven is dumb. Unfortunately, that means it will probably also be slower because I have to redo more work. So, this will need some time to figure out the tradeoff there. If you really want to avoid it, you can set -Sthreads 1 on the CLI to avoid multiple threads.#2022-02-2113:39Alex Miller (Clojure team)This whole general area of concurrent use is an area of active work in Maven land. Maven core 4.x is redoing a lot of stuff, but it's not done yet so I've been holding off#2022-02-2114:12arohnerThanks!#2022-02-2110:24Carsten BehringI was quite often bitten by a deps.edn and using somehow wrong (wrong spelled for example) git/sha coordinates.
Example:
{:deps
{org.clojure/data.csv {:git/url ""
:sha "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}}}
This result in a un-informative exception, NPE:
It would be nice to have something more informative.
(specially which coordinate is wrong, in case of many it saves some try-and-error)
Error building classpath.
java.lang.NullPointerException
at clojure.tools.deps.alpha.util.dir$canonicalize.invokeStatic(dir.clj:30)
at clojure.tools.deps.alpha.extensions.deps$fn__1467.invokeStatic(deps.clj:32)
at clojure.tools.deps.alpha.extensions.deps$fn__1467.invoke(deps.clj:32)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__770$fn__772$fn__773.invoke(alpha.clj:406)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__479.invoke(concurrent.clj:35)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
#2022-02-2111:49rickmoynihanyeah I’ve seen this too in dev after depending on a sha in a PR/branch for a dep which you later force push a fix to#2022-02-2114:09Alex Miller (Clojure team)I think this particular problem is fixed in the latest release actually - what version of CLI are you on?#2022-02-2110:56borkdudeI think clj-kondo could help here as well if there is a way to see that it's an invalid SHA. Is that possible to check using a pure function?#2022-02-2111:07Carsten BehringIt can go wrong as well in other ways, even with a valid sha, example: (forgetting to add the :git/url gives as well a NPE (and no way to see which coordinate is not working)
{:deps
{org.clojure/data.csv
{ :sha "e5beccad0bafdb8e78f19cba481d4ecef5fabf36"}}}#2022-02-2114:11Alex Miller (Clojure team)Any time you find a bad error message, please file a question on https://ask.clojure.org with the repro and your version, happy to work on these as they come in#2022-02-2217:29Carsten Behringhttps://ask.clojure.org/index.php/11599/non-informative-error-message-dependency-deps-cannot-found#2022-02-2111:10vlaaadI just had wild idea: for -X invocations, tdeps could provide an extra flag to fn to signify if this function is called as an entry point.
Use case: changing the behavior depending on whether it’s known to be an entry point or whether it’s a part of a larger program, e.g.:
• calling shutdown-agents at the end if it’s an entry point
• calling (Platform/exit) for JavaFX UIs if it’s an entry point
• rebinding System.{out,err} to special output if it’s an entry point
Makes sense to create an ask?#2022-02-2114:15Alex Miller (Clojure team)Put it on ask please. There are a lot cases here and we've made a bunch of delicate choices in this area already#2022-02-2119:51vlaaadhttps://ask.clojure.org/index.php/11598/allow-configuring-behavior-for-when-its-invoked-entry-point#2022-02-2217:37Carsten BehringI am experimenting running Clojure REPL in a podmain container which has a kind of "restricted" $HOME directory.
(read-only and a lot of environment variables are not set).
I saw that starting clojure clojure CLI tries to download clojure jars from maven, and tries to write them into /.m2/
I could not find anywhere what controls the place of writing of the jars. It is not HOME, neither M2_REPO environment vars because changing them does not make a difference.
Is there a way to configure this location ?
For reference the exception is:
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.clojure:clojure:pom:1.10.3 from/to central (): /.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.pom.part.lock (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.
#2022-02-2217:41ghadihttps://clojure.org/reference/deps_and_cli#_maven#2022-02-2217:42Carsten BehringThanks, saw it now.#2022-02-2218:04Carsten BehringHmm. It uses the "tilde" which I cannot change.
And inside the container there is no ".clojure/deps.edn" file (yet)
which I could change.#2022-02-2218:07Carsten BehringSo in some form the clojure CLI cannot run
in an environment (container) where the "current user" has no home directory assigned in the /etc/passwd file
(and this is the source for the ~ expansion)
Even Though the place for the .clojure directory can be changed, the place for m2 cannot, at least not "in one go".#2022-02-2218:11Carsten BehringTo have this automated, would require a 4 step process:
1. export HOME=/tmp
2. clojure -e ":ok" -> will create /tmp/.clojure/deps.edn
3. some command line magic to set :mvn/local-repo inside /tmp/.clojure/deps.edn#2022-02-2218:12Alex Miller (Clojure team)you can use -Sdeps '{:mvn/local-repo "wherever"}'#2022-02-2218:12Alex Miller (Clojure team)it's unclear to me if that addresses everything you need#2022-02-2218:13Alex Miller (Clojure team)you don't need a .clojure/deps.edn (using -Srepro will skip it, but I'm not sure that's even needed here)#2022-02-2218:14Carsten Behringyes, it does. That was the trick.
Thanks.#2022-02-2219:01orestisIf I see . com.google.guava/guava 31.0.1-jre :newer-version in the output of clojure -Stree, should I expect that I'm using this version of guava? And if so, why does putting an explicit dep of that same version at the top level of my deps.edn fix my issue?#2022-02-2219:02orestisUsing Clojure CLI version 1.10.3.1040, trying now with also 1.10.3.1075#2022-02-2219:04seancorfieldt.d.a prefers top-level versions over transitive deps by design so you can easily control this sort of thing.#2022-02-2219:05seancorfieldThe . means "using this". If it had X it would mean "not using this".#2022-02-2219:06orestisAh never mind, I was running my -Stree with slightly different aliases.#2022-02-2219:07orestisNever mind, I was running -Stree with an extra alias that was bringing in this version, but under normal circumstances it wasn't. Sorry for the noise.#2022-02-2614:19borkdudeWorkaround for getting a REPL for -A with :main-opts invocation:
clj -Scp $(clojure -Spath -A:babashka/dev)
#2022-02-2614:47Alex Miller (Clojure team)What are you working around?#2022-02-2616:17borkdudeNot invoking the main-opts thing#2022-02-2619:23practicalli-johnI define a separate alias that doesn't include the main-opts, such as :lib/kaocha (in practicalli/clojure-deps-edn) for including Kaocha test runner as a library rather than calling it as a test running.
Not sure I understand any use cases that would require a workaround#2022-02-2620:20borkdudeDefining a separate alias is also a workaround ;)#2022-02-2620:22borkdudeMy use case is that 99% of the time I am using -M:alias to invoke the main function, but today I needed to invoke a REPL with that dependency, which was unusual, and I thought I'd share my workaround.
Which brings me to the question: when will -A + :main-opts be really deprecated so it will invoke a REPL? I think the deprecation warning has been there for long enough now?#2022-02-2708:25practicalli-johnI agree that the -A execution option does not work consistently (has side effects) as described in the documentation.
It's also unclear about its future. If it's deprecated, typically that means it will be removed at some point (which I still feel would the correct approach)
Including an -L flag for libraries, which only read :extra-deps (ignores :main-opts, :exec-fn, etc) would feel far more consistent to the other flags -X -M -P -T which are very representative of their purpose#2022-02-2710:12borkdude-A is for adding aliases to the classpath. Originally -A + :main-opts would execute the program. That behavior is deprecated, not -A itself. For invoking main-opts you should use -M. It would be more consistent now if the main function would not be invoked with -A and the REPL would be started with the alias deps.#2022-02-2714:23Alex Miller (Clojure team)Maybe I'll finally do this when we switch to 1.11 version#2022-02-2713:40practicalli-johnCurrently -A, -M and -X all add aliases to the classpath, although each execution operation should result in different outcome (otherwise why would an execution option be there). It is unfortunate that -A behaves almost like -M with a warning added to use -M
Defining -A as adding aliases inst really elaborating on its purpose, although that seems to have been the original concept before all of the other flags were introduced.
Chaining aliases with -M is my typical way to include multiple aliases with :main-opts, with only the last alias passing the :main-opts to clojure.main. That is documented and works consistently for me. It was strange to adapt to initially though.
Using a :lib/thing alias naming is a meaningful convention use in the practicalli/clojure-deps-edn user level configuration to clarify the use of the alias, aiming to make the overall clojure command easier to understand. A :lib/ and :src/ qualifier help distinguish adding an extra library or source with other aliases that will run Clojure code
Using -A execution option remains a source of confusing to me, especially when used with -M, as it seems to run the :main-opts from the -A alias on the few occasions I've used it.
I haven't felt the need to use -A in a long time and replace it with -M almost automatically now. Where I see -A used with -M I also tend to combine the -A alias into the -M alias chain.
Hence the opinion -A should be deprecated. O at least replaced with something that has a clearer purpose that is not complected with the past.
This is only my opinion.#2022-02-2714:25Alex Miller (Clojure team)-A is the only way to pass aliases when starting a repl, so it's not going away#2022-02-2717:50practicalli-johnSorry, I am unsure I understand this fully. Are you talking about an implementation detail? Otherwise, I am confused.
If running the clojure or clj command, then I can pass an alias to the REPL using the -M execution argument and the REPL will start so long as the alias does not contain a :main-opts key and value.
I use the Rebel Readline termnal UI for a REPL I regularly pass aliases in with the -M flag also and so long as repl/rebel alias is last in the chain all the aliases will load their :extra-paths and :extra-deps
I am concerned I am missing something fundamental in the use of Clojure CLI now#2022-02-2717:55Alex Miller (Clojure team)while you can start a repl that way (as a particular execution of clojure.main via -M), we have stated that using clojure without a "mode" (-M / -X / -T) is how you start a repl with the CLI#2022-02-2717:56Alex Miller (Clojure team)if you want to also supply aliases when starting a repl, you do that with -A#2022-02-2717:56Alex Miller (Clojure team)from clj -h:
Usage:
Start a REPL clj [clj-opt*] [-Aaliases] [init-opt*]
Exec fn(s) clojure [clj-opt*] -X[aliases] a/fn? [kpath v]* kv-map?
Run tool clojure [clj-opt*] -T[name|aliases] a/fn [kpath v] kv-map?
Run main clojure [clj-opt*] -M[aliases] [init-opt*] [main-opt] [arg*]
Prepare clojure [clj-opt*] -P [other exec opts]
#2022-02-2717:57Alex Miller (Clojure team)while clj is the same as clj -M -r right now, it may not always be so#2022-02-2717:58Alex Miller (Clojure team)and clj w/o -M etc may in the future take other (new) arguments#2022-02-2718:42practicalli-johnSo -A is the official way to pass an alias when running a REPL with clojure or clj. Other approaches are not guaranteed to work
Okay, that makes sense.
However, I never use clj or clojure to start a REPL by themselves, or with -A.
I always go through -M, for both rebel readline and CIDER jack-in (and running a REPL with nREPL for cider-connect)
I will go away and think about this a lot more as I think there is something that is not sinking in yet.
Thanks again for the explanation.#2022-02-2719:04Alex Miller (Clojure team)> I always go through -M, for both rebel readline and CIDER jack-in
why?#2022-02-2719:04Alex Miller (Clojure team)(one reason would be using -i or -e args)#2022-02-2719:06Alex Miller (Clojure team)being able to pass those to clj without -M is deprecated and will eventually go away (which highlights that this is different than -M -r)#2022-02-2809:00practicalli-johnWhy use -M
CIDER jack-in uses :main-opts to run a REPL, I believe to support nREPL. I also use -M to run a REPL with nREPL (headless or with figwheel-main) via a separate command line for CIDER connect
Figwheel-main also seems to require :main-opts to run, so -M is used
I haven't looked at the inner workings of the projects for a while, so don't have more details at hand
I haven't used -A since the depreciation warning always added
I haven't really used -i or -e flags, except for abandoned experiments.
If I should be doing something else, it's really not obvious to me, sorry.#2022-02-2813:23Alex Miller (Clojure team)If you're using main-opts then you should be using -M (but I suspect you are really running a program that happens to implement the client side of a remote repl) vs using clojure.main’s to start a terminal repl in these cases, which is the point of the non -M repl#2022-03-0110:54practicalli-johnnREPL can provide a client to a REPL process when run in interactive mode. I assume this is what CIDER is doing or it has implemented its own client that communicates over nREPL.
Rebel readline replaces the client ui that would otherwise be created with clojure , to provide additional features
I do not use the specific case of running a REPL process and prompt with the clojure binary (or clj wrapper). So it would seem -A is not currently part of my workflow as I dont run a REPL prompt in that way.
Once the -A flag removes :main-opts support, then it would seem only the -M execution option is the only way nREPL & CIDER will work as the aliases include :main-opts. This assumes -M does not change in a way that breaks nREPL & CIDER.
If this is incorrect, I am afraid I do not know what is correct.
I am sure this is taking up too much of your time, so unless there is an alternative I'll stick with -M until I have more time to investigate this. Thank you.#2022-03-0112:37Alex Miller (Clojure team)That all seems correct#2022-02-2714:23Alex Miller (Clojure team)Maybe I'll finally do this when we switch to 1.11 version#2022-02-2719:49danielcomptonHas the Clojure homebrew tap moved recently? The docs (and our internal scripts) reference
brew install clojure/tools/clojure
but the repo is at https://github.com/clojure/homebrew-tools.
I'm having trouble getting Homebrew to realise that the repo has moved, even after removing the old tap it still rewrites clojure/homebrew-tools/clojure to clojure/tools/clojure when using a Brewfile
e.g.
tap "clojure/homebrew-tools"
brew "clojure/homebrew-tools/clojure"
prints
$ brew bundle
Using clojure/tools
Using clojure/tools/clojure
Makes me think there is some other config or cache somewhere that Homebrew is still holding on to.#2022-02-2720:05borkdudeThe repo hasn't moved. a tap clojure/tools translates into a git repo named clojure/homebrew-tools. the homebrew- prefix is required by homebrew in the git repo but does not appear in the tap name.#2022-02-2720:32danielcomptonAh, thanks! Didn't realise it did the translation. Root of the issue for us is that brew bundle doesn't automatically create taps for clojure/tools/clojure, but brew install does, so our Brewfile was partly broken if you hadn't installed some of the taps yourself previously.#2022-02-2721:31steveb8nQ: is it possible to use an alias from a dependency libs deps.edn? Can’t find any way to do this except duplicate the aliases in the top level deps.edn#2022-02-2721:31steveb8nseems like it would be a useful feature so I assumed it was possible but I can’t figure out how#2022-02-2721:33steveb8nhave been searching https://clojure.org/reference/deps_and_cli#_aliases but can’t find any clue there#2022-02-2721:36steveb8nI use Cursive and it allows a project to enable aliases from libs but, both in cursive and from the cli, when I print via -Sdeps tree, extra deps enabled by the alias are not present. Pretty sure this is a tools.deps question but I suppose it’s related to Cursive as well since it exposes the aliases in its UI @U0567Q30W#2022-02-2721:38steveb8nI’m testing without Cursive, just in terminal. when doing that, it reports that the alias is “WARNING: Specified aliases are undeclared and are not being used”#2022-02-2722:03seancorfieldNo, you can't invoke aliases from dependencies. But it is a frequently requested feature.#2022-02-2722:03seancorfieldhttp://ask.clojure.org should have a few issues around that which you could go vote for.#2022-02-2722:41steveb8nthanks. that’s good to know. @U0567Q30W why does Cursive expose aliases from deps libs?#2022-02-2800:54cfleming@U0510KXTU You mean in the Deps toolwindow? That’s because you can only enable the aliases project-wide there to control the synchronisation of the whole project (project in the IntelliJ sense, probably made up of lots of deps modules). No-one so far has requested being able to control that on a per-module basis. FWIW that’s what Maven does with profiles, too.#2022-02-2800:56steveb8nok. maybe my workflow is unique but I don’t think so. when I see aliases from dependencies and I enable them, I see the jars appear in the classpath which is expected. but when I try to load them in the repl, they are not available. that is confusing. am I thinking about this wrong?#2022-02-2800:57steveb8n(yes deps tool window)#2022-02-2800:58steveb8nI don’t think I need “per module” features. Just finding the UX confusing#2022-02-2800:59steveb8nfrom Seans response I know that I need to duplicate any aliases in the top level project. that will mean I see 2 aliases for each I’m using in the deps window as well#2022-02-2801:00steveb8nhappy to demo if I’m not explaining well#2022-02-2811:59Eugenhi, I was using this function to get the local path for a jar (zip file).
But now resolve-deps (latest version) does not fill the path no more .
(defn jar-path
"Find the path on disk where resolve-deps downloads the file.
Example: (jar-path res ')"
[resolve-deps-result lib]
(first (:paths (get resolve-deps-result lib))))
I get :
deps (tools/resolve-deps (:ontolog-ui-deps basis) {:trace true})
#:{ontolog-ui {:mvn/version 0.1.13174, :extension zip, :deps/manifest :mvn, :parents #{[]}, :paths nil}}#2022-02-2812:04Eugenseems like I don't have the zip locally.
I suspect this is the cause.
However I am expecting that resolve-deps should download the dependency.
The zip is in the maven repo.#2022-02-2812:06Eugenreverting back to version 0.12.1036 works#2022-02-2812:10EugenI found the issue for this in rlease 0.12.1058 - TDEPS-209 Include only jar files in classpath from Maven artifacts#2022-02-2812:13Eugen@alexmiller: I am affected by this change.
I am using the functionality to download a zip file (clojurescript UI app) from maven repository .
Any way I can download a sepcific maven artifacts to a directory ?#2022-02-2813:25Alex Miller (Clojure team)tools.deps is a library whose primary focus is building classpaths from deps, not to download arbitrary artifacts from maven#2022-02-2813:26Alex Miller (Clojure team)It is possible to use some of the more internal functions that do this, rather than the top level apis#2022-02-2816:47Eugenthanks, I managed to find a lower level function and make it work.
Hopefully it will remain stable 🙂 .
It think it would be beneficial to have a way to set the download
;; give a dep, download just that dep (not transitive - that's handled by the core algorithm)
(let [repos (maven/active-repositories (maven/read-settings) {:additional-profiles ["drevidence"]})]
(ext/coord-paths 'com.drevidence.docsearch/ontolog-ui {:mvn/version "RELEASE"
:extension "zip"} :mvn {:mvn/repos repos}))
#2022-02-2816:47EugenI'm using deps-deploy/maven-settings :as maven#2022-02-2817:52Alex Miller (Clojure team)I'd be surprised if that works given the check in that method#2022-02-2817:52Alex Miller (Clojure team)(when (contains? #{"jar"} extension) ...#2022-02-2818:22Eugenyeah, you are right.
I tested with the old version - ahh#2022-02-2818:25Eugenget-artifact is private so not much I can do there, will check deeper#2022-02-2818:26Eugenmaven/coord->artifact + ArtifactRequest. might do the trick#2022-02-2818:40Alex Miller (Clojure team)at some point, just using pomegranate might be easier for this#2022-02-2818:43Eugenthanks, I will take a look at that as well#2022-02-2818:38seancorfield@alexmiller Are there still some known race conditions in downloading dependencies in the very latest t.d.a.? (details in 🧵 )#2022-02-2818:38seancorfieldI just updated Component to 1.1.0 and we use it in a lot of local subproject deps.edn files and got this:
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Downloading: com/stuartsierra/component/1.1.0/component-1.1.0.pom from clojars
Error building classpath. Failed to read artifact descriptor for com.stuartsierra:component:jar:1.1.0
org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.stuartsierra:component:jar:1.1.0
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:259)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at clojure.tools.deps.alpha.extensions.maven$read_descriptor.invokeStatic(maven.clj:115)
at clojure.tools.deps.alpha.extensions.maven$fn__1127.invokeStatic(maven.clj:143)
at clojure.tools.deps.alpha.extensions.maven$fn__1127.invoke(maven.clj:143)
at clojure.lang.MultiFn.invoke(MultiFn.java:244)
at clojure.tools.deps.alpha$expand_deps$children_task__770$fn__772$fn__773.invoke(alpha.clj:405)
at clojure.tools.deps.alpha.util.concurrent$submit_task$task__479.invoke(concurrent.clj:35)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.stuartsierra:component:pom:1.1.0 from/to clojars (): /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:244)
... 13 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact com.stuartsierra:component:pom:1.1.0 from/to clojars (): /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:52)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:628)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:514)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:402)
... 16 more
Caused by: java.io.FileNotFoundException: /Users/sean/.m2/repository/com/stuartsierra/component/1.1.0/component-1.1.0.pom.part (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:163)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:252)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:482)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
... 21 more
#2022-02-2818:39seancorfieldFour concurrent downloads of the POM, from the look of it, and one of them seems to have deleted a partial download out from one of the others maybe?#2022-02-2818:40seancorfield(running the CLI command again worked just fine the next time)#2022-02-2818:40Alex Miller (Clojure team)yes, still known issues#2022-02-2818:41Alex Miller (Clojure team)really, the problems are in Maven land and are being worked on for maven 4, but it's not done yet#2022-02-2818:41Alex Miller (Clojure team)"fixing" it in tda means sharing less stuff, and probably being slower, but that's probably what I'll have to do#2022-02-2818:42Alex Miller (Clojure team)when you build your stack on a pile of mutable stuff, well ... these things are hard :)#2022-02-2818:44seancorfieldNice to know the central pillar of the Java library ecosystem is so solid 😉#2022-02-2819:45Alex Miller (Clojure team)given how much I've already jettisoned from the Maven stack, I sometimes dream about getting rid of the whole thing#2022-02-2818:38Alex Miller (Clojure team)yes#2022-02-2820:16Alex Miller (Clojure team)https://clojurians.slack.com/archives/C06MAR553/p1646079361736239#2022-03-0620:12seancorfieldhttps://github.com/seancorfield/build-clj -- Common tools.build tasks abstracted into a library, building on the examples in the official tools.build guide.
• v0.8.0 9bd8b8a -- 2022-03-06
• Fix https://github.com/seancorfield/build-clj/issues/18 by using resolve-path from tools.build.
• This project is now tracking tools.build version numbers (v0.8.0) [I'll figure out a version schema for any releases I make between #2022-03-0719:15danielcomptonHow do I tell whether tools.deps has detected the right :deps/manifest or if I need to override it myself? Running clj -Stree just shows me the path to my source tree#2022-03-0719:20Alex Miller (Clojure team)there's not really much magic here - it looks for deps.edn, then pom.xml, that's really all there is to it#2022-03-0719:21Alex Miller (Clojure team)you can provide :deps/manifest to override but the main two cases for that are:
• you have both, but want pom.xml
• you have neither#2022-03-0719:27Alex Miller (Clojure team)you could tell what it found by looking at the .lib file created in .cpcache (either by timestamp or by using -Sverbose to determine which hash it's looking at) - the coord should have a :deps/manifest entry#2022-03-0801:33danielcomptonNeat, thanks!#2022-03-0815:41Shuky BadeerHi guys! I'm using presto SQL CLI to query Datomic from the terminal using sql.
Now the output here makes some sense. One thing i noticed here is that it appends "v1/statement" to the original server url (as shows in the first line of the error). I'm not sure what I'm doing wrong but would appreciate some feedback.#2022-03-0815:50Alex Miller (Clojure team)I don't think this is really related to the Clojure CLI - probably #datomic is the best place for this#2022-03-0815:53Shuky BadeerRoger that! Big thanks Alex#2022-03-0819:10souenzzoThere is no mention to CLJ_CONFIG at clj --help and at man clj#2022-03-0819:13souenzzoAnd at https://clojure.org/reference/deps_and_cli#_deps_edn_sources, that seems to be the unique doc reference about CLJ_CONFIG, it is not clear if CLJ_CONFIG should be a path to a file /custom/deps.edn or a path to a directory, that contains a deps.edn file#2022-03-0819:57Alex Miller (Clojure team)it's a path to a dir#2022-03-0819:57Alex Miller (Clojure team)but I can clarify that in the docs#2022-03-0819:57Alex Miller (Clojure team)the files are always expected to be called deps.edn#2022-03-0909:30mkvlrI’m setting :https://github.com/nextjournal/clerk/blob/9a62fe7dd40850cd20b9b1afb7c78058da6de600/deps.edn#L43. And I’m getting a https://github.com/nextjournal/clerk/runs/5476651483?check_suite_focus=true. Anybody know how to fix this? Relevant line is :jvm-opts ["-Dclerk.resource_manifest={\"/js/viewer.js\" \"\"}"]#2022-03-0912:43Alex Miller (Clojure team)You probably need different embedded escaping/quoting on windows#2022-03-0912:50Alex Miller (Clojure team)That opt gets put into a .jvm file in .cpcache, then pulled out of the file and put on the command line, where you might need “”” around the embedded opts#2022-03-0913:43mkvlrembedded opts is the edn map, ie the value of the prop? tried various things so far without sucess#2022-03-0913:43mkvlrwas able to get CI green by switching to bb clojure though cc @U04V15CAJ#2022-03-0910:35bnstvnHi! Basic question here.
I understand I can get the classpath from the top level :deps with clj -Spath. Also I can get the classpath from an alias with clj -Spath -A:my-alias, and can merge these with clj -Spath -A:my-alias1:my-alias2.
How can I merge the classpath from the top level :deps with a classpath from an alias?#2022-03-0912:41Alex Miller (Clojure team)Depending what's in the alias that is what you get - aliases modify the deps classpath. Can you give more detail about what's in the aliases, what you get, and what you want?#2022-03-0913:17bnstvnThanks!
Considering this deps.edn:
{:deps {dep/dep-root {:local/root "dep-root"}}
:aliases {:my-alias1 {:deps {dep/dep-1 {:local/root "dep-1"}}}
:my-alias2 {:deps {dep/dep-2 {:local/root "dep-2"}}}}}
When I’m running this clj -Spath -A:my-alias1:my-alias2, I can get a classpath with dep-1 and dep-2.
I would like to get a classpath which has dep/dep-root and eg dep/dep-1#2022-03-0916:53seancorfield@UGSM2S2CS Use :extra-deps in your aliases instead of :deps. The former is additive. The latter is not.#2022-03-0918:40bnstvnThanks! In that way, can I get a classpath with the deps only from the alias?#2022-03-0918:54seancorfieldNo. You'd need two versions of the alias, I think, one with :extra-deps and one with :replace-deps (or just :deps, but I think the former is clearer).#2022-03-0918:54seancorfieldI guess my question would be: Why? What problem are you really trying to solve?#2022-03-0918:54seancorfieldPerhaps there's a better approach...#2022-03-0918:55seancorfieldYou could have :deps {} and then under :aliases have :default {:extra-deps {dep/dep-root ...}} etc and explicitly specify the :default alias when you need those "root" deps.#2022-03-0921:03bnstvnyes, that’s what I ended up with, it works.
[well. I have an ancient Java project using fairly complex ant build with xdoclet/ejb2/custom codegen/etc where the classpath is hardcoded into the build files in various places using only local jars.
I wanted to start to use some Clojure here.
I didn’t want to/cant replace the whole build and I don’t want to duplicate the dependencies, so I figured that in the ant config wherever a custom classpath is used, i outsource to deps.edn with an alias and call clj -Spath -A:alias from ant to calculate the classpath.]#2022-03-0916:33danielcomptonIs there a way to have :override-deps at the top level of my deps.edn file? It seems like it only works when it's put inside an alias?
I'm trying to override Netty dependency versions (ultimately I'd like to use the Netty BOM) but for now I can just write out each dependency. But I'd prefer not to have all of our clojure commands have to include a -A:defaults alias on every command. It feels like it would be quite easy for people to accidentally get the wrong Netty version.#2022-03-0916:36Alex Miller (Clojure team)no, that's not a feature#2022-03-0916:37Alex Miller (Clojure team)if you're trying to override the version of a transitive dep, you can add :exclusions to the top-level dep, then add the transitive dep at the top with whatever version you want#2022-03-0916:38Alex Miller (Clojure team)lmk if that doesn't make sense#2022-03-0916:40favila> if you’re trying to override the version of a transitive dep, you can add :exclusions to the top-level dep, then add the transitive dep at the top with whatever version you want#2022-03-0916:40favilaIf this isn’t the problem :override-deps is meant to solve, what problem is it meant to solve?#2022-03-0916:41Alex Miller (Clojure team)it's meant to solve that problem when you apply an alias#2022-03-0916:42Alex Miller (Clojure team)that is, a conditional override#2022-03-0916:42Alex Miller (Clojure team)if I'm reading the original request right, the goal is non-conditional override#2022-03-0916:42danielcomptonThere are a bunch of different libraries all with dependencies on various Netty components. I want to specify that if any of them pull in any of the Netty dependencies they should get the version I selected, without actually adding a dependency on any extra Netty JARs that aren't needed#2022-03-0916:43Alex Miller (Clojure team)yeah, so there is no place to put that right now - you need to exclude on each top-level dep that includes those transitive deps#2022-03-0916:43danielcomptoni.e. if our Redis library adds a new dependency on netty-buffer in a new version, I want to have already specified which version of netty-buffer will get pulled in, without actually including it in the JAR#2022-03-0916:43Alex Miller (Clojure team)this is something that's been requested (global exclusions / global overrides)#2022-03-0916:44danielcomptonWhen searching I found I'd asked the same question a few years ago: https://ask.clojure.org/index.php/8440/equivalent-of-leiningens-managed-dependencies-in-deps-edn#2022-03-0916:44Alex Miller (Clojure team)still haven't decided how this should best be done in deps.edn#2022-03-0916:44favilaFWIW @U051KLSJF we ourselves just eat the annoyance of having a required alias#2022-03-0916:45favilait’s a far better world to live in than exclusion wack-a-mole to address every CVE#2022-03-0916:45Alex Miller (Clojure team)I read that ask question as being slightly different (managed deps being - what dep should I use if version is not supplied)#2022-03-0916:45Alex Miller (Clojure team)vs force the version to be something regardless of what transitive deps use#2022-03-0916:45danielcompton:managed-dependencies in Leiningen provides overrides for version resolution. I don't think it relies on the version not being provided in the main :dependencies vector, but that is a related feature#2022-03-0916:47favilaThat’s my understanding also, and what I used it for when using lein.#2022-03-0916:49favilaI think it’s a direct analog to maven/pom dependencyManagementhttps://maven.apache.org/pom.html#dependency-management#2022-03-0916:50Alex Miller (Clojure team)well, I may misunderstand either maven's dependencyManagement or lein's expression of it but I think of managed deps as default versions that can be overridden#2022-03-0916:50favilaall of these kind of combine the tasks of “override” and “default”#2022-03-0916:50danielcomptonhttps://github.com/technomancy/leiningen/blob/4d8ee78018158c05d69b250e7851f9d7c3a44fac/doc/MANAGED_DEPS.md#managed-dependencies#2022-03-0916:50danielcompton> 1. It does not actually introduce any dependencies to your project. It only says, "hey leiningen, if you encounter one of these dependencies later, here are the versions that you should fall back to if the version numbers aren't explicitly specified."
#2022-03-0916:51danielcomptonI think this means if the version number isn't explicitly specified in your :dependencies file. Which could either be you included the dependency with no version number, or it was a transitive dependency#2022-03-0916:52favilaIt does in fact prefer :managed-dependencies versions to those of transitive deps, because the effect is seen in the resulting classpath. This is why this is perfect for addressing CVEs#2022-03-0916:53favilaor, at least better than exclusions + add a false direct dep#2022-03-0916:55Alex Miller (Clojure team)well, I'll think about it more and maybe look at it Friday, thx for the poke#2022-03-1021:06mruzekwHi there! Is there a way to install a named tool in a deps.edn so the dev doesn't need to install it manually?#2022-03-1021:11Alex Miller (Clojure team)yes, but exactly how will depend on the tool a bit#2022-03-1021:12Alex Miller (Clojure team)using an installed tool is really just specifying a top level dep so you can put that in the alias. the extra bit is that the tool can declare some usage info in their deps.edn that you will also need to copy into the alias. but at that point, it's basically the same - just -T:alias instead of -Ttool#2022-03-1021:13Alex Miller (Clojure team)if you had an example, I could be more specific#2022-03-1021:21mruzekwThanks, sure. I'm looking to integrate https://github.com/atomisthq/jibbit#2022-03-1021:24Alex Miller (Clojure team)so you should be able to make an alias for that like:
:jibbit {:replace-deps {io.github.atomisthq/jibbit {:git/tag "v0.1.12" :git/sha "db2adb8"}}
:replace-paths ["."]
:ns-default jibbit.core
:ns-aliases {jib jibbit.core}}#2022-03-1021:24mruzekwClojure CLI version 1.10.3.1087
deps.edn alias
:jib {:extra-deps {io.github.atomisthq/jibbit {:git/tag "v0.1.12"
:git/sha "db2adb8"}}
:exec-fn jibbit.core/build}
Command and Result
❯ clj -T:jib
WARNING: Specified aliases are undeclared and are not being used: [:jib]
No function found on command line or in :exec-fn
#2022-03-1021:24mruzekwGotcha. Thanks. I'll try that#2022-03-1021:25mruzekwIt'd be really nice if the CLI (or another tool) installed the alias#2022-03-1021:25Alex Miller (Clojure team)that first warning sounds like it didn't find the alias so I'd check your full deps.edn there#2022-03-1021:26Alex Miller (Clojure team)in general, we've decided not to modify your deps.edn for you#2022-03-1021:26mruzekwFair enough.#2022-03-1021:26Alex Miller (Clojure team)borkdude does have a tool for that though :)#2022-03-1021:27Alex Miller (Clojure team)I can't remember what it's called @U04V15CAJ#2022-03-1021:28mruzekwI'll peak at his github 🙂#2022-03-1021:28mruzekw❯ clj -T:jib build
Unqualified function can't be resolved: build
❯ clj -T:jib jib/build
Namespace could not be loaded: jib
❯ clj -T:jib jibbit.core/build
Namespace could not be loaded: jibbit.core
#2022-03-1021:28mruzekwTook your alias and just did :jibbit -> :jib#2022-03-1021:29Alex Miller (Clojure team)doesn't seem like it's finding/using the alias - can you share the whole deps.edn?#2022-03-1021:29mruzekwSure#2022-03-1021:30mruzekwOh, you know what, I see an extra } one sec#2022-03-1021:30mruzekwGetting closer
❯ clj -T:jib build
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.git/eval5816$fn (git.clj:76).
Library io.github.atomisthq/jibbit has coord with missing sha
Full report at:
/var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/clojure-7416877088757369135.edn
#2022-03-1021:31Alex Miller (Clojure team)I ninja edited above - you'll need the :git/sha too#2022-03-1021:31Alex Miller (Clojure team)I forgot that when I first posted#2022-03-1021:32mruzekw#2022-03-1021:32mruzekwHere's the full deps.edn#2022-03-1021:40mruzekwMaybe this is related?
https://clojureverse.org/t/how-to-depend-on-tools-build/7916#2022-03-1021:41Alex Miller (Clojure team)nah#2022-03-1021:42Alex Miller (Clojure team)I'm not sure why you're seeing that, give me a few to look at it#2022-03-1021:42Alex Miller (Clojure team)I can repro locally#2022-03-1021:43mruzekwNo rush, thanks#2022-03-1021:44borkdude@U064X3EF3 Did you mean rewrite-edn?
https://github.com/borkdude/rewrite-edn#2022-03-1021:47lreadOr maybe https://github.com/babashka/neil?#2022-03-1021:55Alex Miller (Clojure team)that's the one I was thinking of#2022-03-1021:55Alex Miller (Clojure team)I'm pretty sure there's a bug here, but I haven't spotted it yet :)#2022-03-1021:58Alex Miller (Clojure team)oh, this message is coming out of jibbit's use of tools.build and tools.deps, not out of the CLI at all!#2022-03-1021:58Alex Miller (Clojure team)as much as people complain about stack traces, they have like ... answers in them#2022-03-1022:03mruzekwhaha#2022-03-1022:04mruzekwThat makes more sense given it downloads jibbits dependencies#2022-03-1022:06mruzekwThe sha doesn't seem to match the tag for clojure/tools.build#2022-03-1022:06mruzekwclojure/tools.build {:git/tag "v0.7.5" :git/sha "2526f58"}#2022-03-1022:06mruzekwv0.7.5 on GH -> 34727f790d0b4d568fe3b5699ce6ebfcc1f55d39#2022-03-1022:07mruzekwHow is this working in the clj -Tjib ... case then?#2022-03-1022:08Alex Miller (Clojure team)those shas are actually the same - one is the annotated tag that points to the commit, the other is the commit (tools.deps chases both to the commit)#2022-03-1022:09Alex Miller (Clojure team)the error is:
"Library io.github.atomisthq/jibbit has coord with missing sha",
:data
{:lib io.github.atomisthq/jibbit, :coord {:git/tag "v0.1.12"}},#2022-03-1022:09mruzekwOh :thinking_face:#2022-03-1022:09Alex Miller (Clojure team)which, I'm not sure where that's coming from#2022-03-1022:10Alex Miller (Clojure team)the alias has the sha, the basis written for this -T call has the sha, the classpath is built fine, the -T exec starts fine, but then it calls jibbit.core, which calls create-basis in tools.build, which calls tools.deps to make the basis, which is when the message happens#2022-03-1022:11Alex Miller (Clojure team)I don't actually know anything about jibbit so my mental model is a bit weak when it gets there :)#2022-03-1022:12Alex Miller (Clojure team)do you have a JIB_CONFIG env var or a jib.edn file?#2022-03-1022:13Alex Miller (Clojure team)I guess I don't so probably irrelevant#2022-03-1022:13mruzekwNope, neither. Planned on using just the CLI for now#2022-03-1022:18mruzekwI guess the best way to troubleshoot from here would be to pull down jibbit and update its deps?#2022-03-1022:21Alex Miller (Clojure team)oh, it's in your :deps - io.github.atomisthq/jibbit {:git/tag "v0.1.12"}#2022-03-1022:21Alex Miller (Clojure team)is there some reason that's there?#2022-03-1022:22mruzekwOh! No. Damn lol#2022-03-1022:22Alex Miller (Clojure team)I looked everywhere but there :)#2022-03-1022:23mruzekwThat unblocked that#2022-03-1022:23mruzekwAnd now it's working. Thank you! 🙇#2022-03-1022:23Alex Miller (Clojure team):thumbsup:#2022-03-1022:25mruzekwWould it be fair to say that installing a tool as an alias follows?:
:<tool name> {:replace-deps {io.github.<org>/<tool name> {:git/tag "<tag>" :git/sha "<sha>"}}
:ns-default <same value as tool deps.edn :ns-default>
:ns-aliases <same value as tool deps.edn :ns-aliases>}#2022-03-1022:26Alex Miller (Clojure team)yes#2022-03-1022:26mruzekwNice! Thanks. I'll keep that in mind if I want to write/augment a tool to add that to deps.edn#2022-03-1117:34mruzekwHey there, I'm trying to output a string of edn to stdout via clj -T... and for some reason I can't get the namespace to load. I feel like there's something simple I'm missing. Can anyone spot it?
deps.edn aliases
:aliases {:jib-config {:replace-paths ["dev/tools"]
:exec-fn tools.skaffold/get-skaff-config}}
<project-root>/dev/tools/skaffold.clj
(ns tools.skaffold)
(defn get-skaff-config [_]
(with-out-str (clojure.pprint/pprint {:x 1 :y -1})))
Output
❯ clj -T:jib-config
Namespace could not be loaded: tools.skaffold
#2022-03-1117:39mruzekw-X yields the same results
❯ clj -X:jib-config
Namespace could not be loaded: tools.skaffold
#2022-03-1117:44Alex Miller (Clojure team)you want :replace-paths ["dev"]#2022-03-1117:44Alex Miller (Clojure team)the namespace (tools.skaffold) is relative to the path (dev) - those get put together#2022-03-1117:49mruzekwSo paths can only be root folders?#2022-03-1117:53mruzekwI also have dev/user.clj which takes sometime to load and has some extra stdout, which is why I was trying to avoid it.#2022-03-1117:53Alex Miller (Clojure team)yes, that's how the JVM classpath works#2022-03-1117:53Alex Miller (Clojure team)you could put one of these things under another root of course#2022-03-1117:55mruzekwGotcha, thanks. Yeah still pretty new to the JVM#2022-03-1117:55Alex Miller (Clojure team)namespace is munged into a resource path, then that resource path is checked relative to each classpath location#2022-03-1117:56Alex Miller (Clojure team)tools.skaffold => tools/skaffold.clj => look for that in dev/ (and other locations)#2022-03-1117:57mruzekwGotcha, thanks, makes sense#2022-03-1316:18borkdudeI just hit an edge case with :deps/prep-lib. For JVM usage of a library that makes sense, but when using a library for other targets (e.g. CLJS or babashka) there might be reader conditionals that do not rely on JVM compilation.
An example: the specter library uses one compiled Java class, but not for CLJS or babashka. To support running this lib in a clj REPL, I added :deps/prep-lib to it, but this made the library incompatible with babashka:
$ bb /tmp/specter.clj
Checking out: at a7190b1b74f6410c4efb0b5e67ceb5d3180bb757
Error building classpath. The following libs must be prepared before use: [com.rpl/specter]
https://github.com/borkdude/specter/blob/babashka/deps.edn#2022-03-1316:26Alex Miller (Clojure team)incompatible because ... sorry not getting it#2022-03-1316:27Alex Miller (Clojure team)I read not as incompatible but as "requires prep" ?#2022-03-1317:39borkdudeCalculating the classpath for this dep fails with requires prep, yes#2022-03-1318:12Alex Miller (Clojure team)Yeah, we don't have a concept of “conditionally requires prep”#2022-03-1318:29borkdudeMaybe there could be an option of "calculate classpath, skip prep"?#2022-03-1318:37borkdudeNot urgent, as I was just experimenting with this, but maybe something to consider over time#2022-03-1319:56borkdudeOne consideration could also be that tools like clojure-lsp etc need to calculate the classpath but don't need the compiled Java classes. Having Clojure-lsp crash in your editor because of unprepared libs might be a reason to disable it optionally too. #2022-03-1400:21Alex Miller (Clojure team)I do have an option like that buried in the code, don't remember if there's a path to get it there from the cli (or if I want that to even be available)#2022-03-1415:20borkdudeAnother option could be to pass :prep via the CLI, e.g. -Sprep so you will always have the prepped classpath. Is there anything against doing this?#2022-03-1416:02Alex Miller (Clojure team)yes, we really want prep to be an explicit independent action, rather than an implicit thing#2022-03-1416:06borkdudebecause missiles can be launched during prep?#2022-03-1416:07Alex Miller (Clojure team)yes, it's running arbitrary third party code on your machine#2022-03-1416:07borkdudeand without prep, you have a potentially incomplete classpath, right?#2022-03-1416:07Alex Miller (Clojure team)right#2022-03-1416:08borkdudeas in, missing target/classes in git libs, for example#2022-03-1416:08Alex Miller (Clojure team)for libraries, requiring prep should often drive you to artifacts instead#2022-03-1416:08Alex Miller (Clojure team)but it can be useful for local dev, or other situations#2022-03-1416:09borkdudeyeah, preview / dev versions etc, and then publish as artifact when ready to release#2022-03-1416:10Alex Miller (Clojure team)like we were able to do with transit-java recently - made it available as a git lib on a branch, which required prep, and then released as an artifact once tested#2022-03-1416:11borkdudemakes sense#2022-03-1419:05borkdudeThere is no way to use a non-deps.edn project as a deps.edn project when it has a non-standard source folder src/clj right?#2022-03-1419:05borkdudeElse I could have used :deps/manifest :deps#2022-03-1419:18Alex Miller (Clojure team)yeah, there's no way to supply that externally#2022-03-1419:18Alex Miller (Clojure team)is it lein project?#2022-03-1419:20borkdudeyeah#2022-03-1419:44dpsuttonopen an issue for them to keep a lein pom reasonably up to date?#2022-03-1419:45borkdudeit's not so urgent, I'm keeping a fork around with a deps.edn in it#2022-03-1419:45borkdudeuntil the next release#2022-03-1515:39Adam HelinsThe company I currently work for has deployed great efforts trying to make clojure work in the context of Nix. Hours and days spent on it by talented ops folks and it turned out to be a real rabbit hole met with failure.
I believe @ericdallo has encountered unresolved difficulties as well.
The core problem is that we have a really hard time reaching a situation where clojure would work without any access to the internet, for the purpose of doing a "pure" build. That is, even when "everything" seems to be available locally (libs in local maven repo, gitlibs, etc), it still needs the network for some unknown reasons.
Quoting @taeer.bar-yam :
We have actually gotten it to work without internet access
This is part of what's so confusing
We can get it to work after running `unshare -n`
So there is clearly some collection of files that when present will convince it not to go to the internet
We just can't figure out what set of files that is
#2022-03-1517:20ericdalloMaybe @UFDRD93RR may be interested on this as he knows a lot of Nix#2022-03-1517:28kokadaAFAIR when I tried to compile something using tools.deps from scratch in Nix#2022-03-1517:29kokadaAt least looking at all available documentation, I didn't found a way to do a completely offline build of it#2022-03-1517:30kokadaIn leiningen I remember that I got something to work (I don't remember if it did actually compile and run successfully, but I think it did)#2022-03-1517:30kokadaBut tools.deps seems to be lacking some way to do a completely offline build#2022-03-1517:32kokadahttps://github.com/NixOS/nixpkgs/commit/2c3ad4ef9c5715fd5b74b4a664b54f5ab8ac7a2c
Yeah, I am right. I got clojure-lsp to compile successfully from source with leiningen#2022-03-1517:33kokadaSo yeah, I may be wrong but I think the problem is with tools.deps#2022-03-1517:33kokadaIt would be nice to have a flag on it to do a completely offline build (and also another flag to use a custom Maven location instead the default one)#2022-03-1517:37ericdalloAgree with that ☝️#2022-03-1518:44Wanja HentzeHere's some things we did try:
• As mentioned, pre-filling .m2/repository and .gitlibs - both via fixed-output derivations and also just taking them from my home directory (after building normally, to warm the cache) as-is. (doesn't work, it still wants to fetch package manifests)
• Taking .cpcache as-is in addition to the above (same problem)
• Patching everything in .cpcache that refers to local directories, pointing it at our prefetched deps instead (still no luck)
• Passing a really long generated -Sdeps parameter to add an alias that sets every deps to a :local/root pointing at prefetched deps, then putting that one last in the list of aliases passed to clojure (actually seems to work for clojure-source deps, but not in the general case for jar deps; some of them have pom files demanding they be placed in a specific location relative to their parents, it seems)
• Creating a settings.xml telling maven to run offline (doesn't help, it just fails slightly earlier when it realises it would have to talk to the internet)#2022-03-1518:45Wanja Hentze• probably more things that only @taeer.bar-yam knows, I wasn't there for all of it#2022-03-1518:47Wanja Hentze@UFDRD93RR fwiw, giving a different maven location is relatively easy, you can pass -Sdeps '{:mvn/local-repo ".deps/m2"}'
The issue is that whenever it doesn't find what it needs in there, it will still fetch from remote repos, and we have been unable to debug why exactly it makes this decision#2022-03-1519:03Taeer Bar-YamI also pre-populated .config/clojure/deps.edn and .config/clojure/tools/tools.edn (as well as the .clojure version of those files.
Importantly, I can build, and then run unshare -n (cutting off internet access) and then build again, and it's fine. When I build that second time with strace, I see that the only files it access are files that the hermetic nix build should have accessible to it (once we add in all the files we've added in).#2022-03-1519:03Taeer Bar-Yamand when I run strace inside the hermetic nix build, it's not trying to access any files that it can normally but can't in the nix build#2022-03-1520:17kokada@U033F53LU0M yeah, I know. This is exactly what I tried, and this resulted in the same problem you're describing.#2022-03-1520:46kokadaAh, you're talking about the custom Maven flag. Yeah, I am pretty sure that I saw this too and I was just misremembering that tools.deps didn't had way to do#2022-03-1520:47kokadaBTW, I remember doing pretty much all the things that @taeer.bar-yam said too. Except for the strace part, I pretty much simply gave up eventually#2022-03-1610:18Adam Helins@U064X3EF3 Regarding a repro case, we have a private codebase but we'll try to figure out a way to share the relevant bits.
In light of all the above, what has been tried, do you maybe have some suggestions on what would be most useful to you? Reducing the scope of the repro in a meaningful way?#2022-03-1612:17Alex Miller (Clojure team)In regards to a “pure” build, what are you actually trying to do with clojure? Run a repl, start a program with -M or -X or -T? What features are you using in the build? Maven deps? S3 maven deps? Git deps? Local deps? The latter two with deps.edn? pom.xml?#2022-03-1612:21Alex Miller (Clojure team)Ideally I just want a smallish case that illustrates this issue in your env - the deps.edn and the commands you are running#2022-03-1613:05Adam HelinsA very good starter would be to build an uberjar using tools.build . All attempts were trying to prefetch and prepare everything needed in advance (`tools.build` + deps for the uberjar itself) so that we could clojure -T completely offline (which is precisely what fails).
Deps consist of Maven deps mostly, a couple Git ones, and 1 :local/root with a deps.edn.#2022-03-1613:21Alex Miller (Clojure team)can you repro this with like a simple example from the tools.build guide? filing an ask.clojure question would help me keep track of this. I'll be out most of the rest of the week.#2022-03-1613:26Adam HelinsYep, we'll get something out and I'll follow up on ask.clojure
Thanks 🙂#2022-03-1614:20Alex Miller (Clojure team)and in case it's not clear, I do want to help you succeed at this - there should be a way to avoid hitting the net if we have what we need. I assume the issue is in canonicalization, either in resolving version ranges for maven or checking git sha/tags or something like that in git libs#2022-03-1614:22ericdalloReally excited to see that working 🤞#2022-03-1617:00kokadaBTW, since we are already talking about doing reproducible Clojure builds in Nix, it is important to remember that we don't build Clojure/Leiningen itself from source, instead opting for packaging the release jars#2022-03-1617:02kokadaIt would be very interesting that we could build it from source instead, so we could trust the result binary being compiled from source instead of trusting the release binaries from upstream#2022-03-1617:06Alex Miller (Clojure team)well, you can, although I don't know why you'd trust that more than a signed binary#2022-03-1617:06Alex Miller (Clojure team)debian team does this already#2022-03-1621:53kokadaFor one we don't check the binary signature (only the checksum). Probably we should, but not sure how/if this is possible on nixpkgs#2022-03-1621:54kokadaBut I still think we should build from source if possible. This is the only way to really guarantee that the whole system is reproducible#2022-03-1713:24Adam HelinsLet's say that would be a second step, if we ought to prioritize 🙂
But it does feel like a direction since there is a trend evolving towards fully reproducible builds.#2022-03-1716:21Taeer Bar-YamOkay, I made [the reproduction](https://github.com/radvendii/clojure-tools.build-test)
In doing so, I discovered that the problem doesn't happen with old versions of clojure. In particular the version upgrade from 1.10.3.855 -> 1.10.3.933#2022-03-1716:23Taeer Bar-YamI looked at [the changelog](https://clojure.org/releases/tools), and I'm wondering if the problem is tools, and in particular them getting fetched in a different way. Is there a way to disable that feature, to test that out?#2022-03-1716:25Taeer Bar-YamAlternatively, if we know where the tools and tool information gets cached, we could try providing those files to the nix build#2022-03-1515:42Alex Miller (Clojure team)can you file a reproducible case at https://ask.clojure.org ?#2022-03-1803:22Taeer Bar-YamAsked. Here's the link: https://ask.clojure.org/index.php/11644/re-building-without-internet-access-with-nix#2022-03-1818:02imrehttps://github.com/polyfy/polylith/blob/5292e8ab1bea3ff84b08c7f2142dbe7f1349a3f3/deps.edn#L78-L81 that uses the keys :deps and :paths in one of its aliases as opposed to their (extra/replace) counterparts. I failed to find any documentation wrt the usage of these keys in an alias. Does anyone know where I could get more info on why someone would want to use them there as opposed to :extra-*?#2022-03-1818:07Alex Miller (Clojure team)They are synonyms for :replace-deps and :replace-paths - notably they replace the project paths and deps rather than add to#2022-03-1818:52seancorfield@U08BJGV6E see https://clojure.org/guides/tools.build for examples where you have :deps in aliases.#2022-03-1819:09imreThank you both, I see I was looking in the wrong place.
However, I do have a question wrt synonyms @U064X3EF3 - I don't see the phrases replace or synonym at https://clojure.org/guides/tools_build - so my question is, are these synonyms (as in all contexts) or only appear to behave like synonyms due how https://clojure.org/reference/deps_and_cli#_aliased_tool_execution works?#2022-03-1819:10imreWhat is the intended behavior of these keys when the alias in question is used not in a tool execution context but to start a repl together with other aliases, like clj -A:build:dev:test?#2022-03-1819:32Alex Miller (Clojure team)They are synonyms in all alias contexts#2022-03-1819:32Alex Miller (Clojure team)No difference in tool / non-tool#2022-03-1819:34Alex Miller (Clojure team)Well I guess the big difference is that -T removes the project paths and deps regardless#2022-03-1820:22imreOh I see, thank you.
So is my understanding correct that if I want to use an alias both as a tool with only the deps/paths specified in the alias: clojure -T:my-alias and as an additional alias (with the additonal paths/deps) for a repl: clojure -A:dev:test:my-alias then I should just continue using :extra-paths/deps?
And a side question just so I understand properly: if deps/replace-deps and paths/extra-paths are synonyms in every context, then why do both exist? Is using one recommended over using the other in some contexts and why?#2022-03-1820:54seancorfieldHistorical reasons and, now, backward compatibility.#2022-03-1820:56seancorfieldI'm afk right now (on vacation in England) but I thought deps/paths behaved in a context-sensitive manner. If have to double check the source 😁#2022-03-1821:00imreI guess I should be able to throw something together next week to test my assumptions in case there isn't some official-like documentation that explains this. Enjoy your holiday!#2022-03-2010:43practicalli-johnI found the synonyms confusing myself. I prefer using the specific keys so it's very clear what the alias does#2022-03-2112:05imreEnded up with a small demo https://github.com/imrekoszo/try-deps-tool-replace#2022-03-2114:42Alex Miller (Clojure team)in the context of running a tool with -T, the project :deps and :paths are removed, so it may feel more natural in that case to use :deps and :paths rather than :replace-deps and :replace-paths. But in the context of -X where :deps and :paths are not removed, it can be helpful to use :replace-deps and :replace-paths to make that clear. so, that's why both synonyms exist#2022-03-2114:51imreThank you Alex. Is my understanding correct that there in the -T:tool context one could also use :extra-* without any observable difference at runtime (due to the removal you mentioned)?#2022-03-2114:53imreIf the answer to that is yes, then would it be safe to assume that if an alias is to be used both as a -T tool and an extra alias for my repl, it would be better to stick with :extra-* if I don't want it to affect project deps/paths when in the repl?#2022-03-2115:33Alex Miller (Clojure team)yes#2022-03-2115:34imreBrilliant, thank you!#2022-03-1823:44CarloWhen I follow this guide (the uberjar case) https://clojure.org/guides/tools_build, after writing the suggested content in build.clj and deps.edn, I get this interaction:
$ clj -X:build clean
Namespace could not be loaded: build
What am I doing wrong?#2022-03-1900:03Alex Miller (Clojure team)Should be -T not -X ?#2022-03-1901:20CarloThank you 🙈#2022-03-1916:17mruzekwIs there a clojure.spec for deps.edn? I'm working if you can compose aliases in the deps.edn file#2022-03-1917:00borkdude@mruzekw you can't compose them as in make one alias that consists of the merge of two other aliases#2022-03-1917:03mruzekwThanks. The idea was to be able to compose up simple commands for common workflows without having to think about what's needed upfront.
So for instance, have a :env/dev specifying all dev deps/paths, and :dev using that definition in addition to a :main-opts
Allows for a simple clj -M:dev entry point, while also allowing reuse of :env/dev separately.#2022-03-1917:04borkdudeyeah, you can right now only compose that on the command line, like clj -M:env/dev:dev#2022-03-1917:05mruzekwGotcha :thumbsup: Thanks#2022-03-1918:16mruzekwLooks like I could achieve something similar with tools.build#2022-03-2112:07dergutemoritzFTR, there is a ticket about this feature request: https://clojure.atlassian.net/browse/TDEPS-220#2022-03-2112:08dergutemoritzSimilarly https://clojure.atlassian.net/browse/TDEPS-170#2022-03-2116:15mruzekwNice! Having this feature would be awesome.
For now I'm using tools.build and https://github.com/seancorfield/build-clj to create a simple singular interface for all tasks. My goal is for it to appeal to beginners.
clj -T:build run
clj -T:build run-dev
clj -T:build test
...
I'm building that out here for now: https://github.com/mruzekw/clojure-starters/tree/master/skaffold-dev#2022-03-2010:51promesantehi guys, good morning, I'm trying to startup a Clojurescript / re-frame project with clj-new, https://github.com/seancorfield/clj-new, the way suggested there:
clojure -Tclj-new create :template re-frame :name yourname/spa :output front-end :args '[+garden "+10x" +routes]'
and get the following error reported:
Error building classpath. Bad coordinate for library , expected map: nil
what should I do, or might be doing wrong? thanks !#2022-03-2011:28practicalli-johnMissing double quotes around the :args values, the values should be individual strings in a vector
So wrap +garden and +route in double quotes#2022-03-2014:15promesantethie way?
$> clojure -Tclj-new create :template re-frame :name yourname/spa :output front-end :args '["+garden" "+10x" "+routes"]'
Error building classpath. Bad coordinate for library , expected map: nil
if so, I still get the same error
just in case, I tried the following ways as well, always getting the same error:
$> clojure -Tclj-new create :template re-frame :name yourname/spa :output front-end :args "[+garden "+10x" +routes]"
Error building classpath. Bad coordinate for library , expected map: nil
$> clojure -Tclj-new create :template re-frame :name yourname/spa :output front-end :args "["+garden" "+10x" "+routes"]"
Error building classpath. Bad coordinate for library , expected map: nil
thanks !#2022-03-2019:02practicalli-johnclojure -T:project/new :template re-frame :name practicalli/reframe-example :output reframe-example :args '["+garden" "+10x" "+routes"]'
This works with the alias I have in https://github.com/practicalli/clojure-deps-edn/
For your example, it seems the quotes around the vector were changed to double quotes, when they should be single.#2022-03-2019:09practicalli-johnMy only guess is that something went wrong with installing the clj-new project as a tool. Check the .clojure config directory for tools/clj-new configuration, or just try install the clj-new as a tool again
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.381"}' :as clj-new
You can also use clj-new as an alias, in which case -T:alias-name is used instead of -Tclj-new create#2022-03-2300:05promesantehi John, thank you very much for your replies !
You were absolutely right. I had clj-new installed via deps.edn, I reviewed ~/.clojure/deps.edn and it seemed ok.
Nevertheless, I commented all the :new element there, and then reinstalled clj-new "as a tool". And then the error I reported above got fixed: project with re-frame template was created right with the command shown in the README.
Again: thank you very much !#2022-03-2012:10orestisPerhaps unrelated to this channel, but given yet another JS packaging brouhaha (search for node-ipc), I was wondering how much safer the Clojure/JVM ecosystem is, if at all? My understanding is that Maven packages are immutable but it's not clear if my deps.edn file will always generate the same jars etc, seeing that there's no hash of the downloaded JAR anywhere. Can someone with more info shed some light? Thanks!#2022-03-2012:40Alex Miller (Clojure team)Maven artifacts are immutable#2022-03-2012:41Alex Miller (Clojure team)I'm not sure what scenario you're imagining, can you be more specific?#2022-03-2012:50orestisLet's say I host a library on a maven-like server (e.g. http://jitpack.io, maven-restlet, to take example of my current deps.edn). I upload my library foo/nicelib which consists of nicelib-1.2.3.jar and nicelib-1.2.3.pom. I assume if a specify {foo/nicelib {:mvn/version "1.2.3"}} this is what I'm getting.
Now say that a hacker breaks in my maven-like server, and replaces nicelib-1.2.3.jar with a hacked version - is there anyway for the toolchain to detect that and complain about a mismatch in the digest?#2022-03-2012:53orestisTo some extent, git coordinates in deps.edn don't have this issue because tools-deps will clone the repository and can verify that the sha1 requested is the one on the disk. But with versioned artifacts, I'm not sure how you could pin a jar file to its sha digest.#2022-03-2013:45Alex Miller (Clojure team)if the maven server is compromised, then I think you're sunk regardless. maven does support authenticated artifacts (gpg signed) and things on Maven central are required to be signed, but by default there is no way to automatically verify that the key used for the signed artifact was "correct". you really would have to manually check every lib. (The info to do this for the clojure and clojure contrib jars is at https://clojure.org/community/download_key for example). but the signature file is stored in the repo - if the maven server is compromised, the signature file is also vulnerable.#2022-03-2013:46Alex Miller (Clojure team)maven now requires https and have made other changes to minimize MITM attacks (which were previously pretty easy)#2022-03-2013:47orestisRight, thanks. What about the scenario of a malicious library author? Can they upload whatever they want under their account (e.g. delete the previous artifact and upload a new one)?#2022-03-2013:47orestisI mean, are the artifacts immutable only in practice, or also in principle? 🙂 Thanks for taking the time to explain this, BTW.#2022-03-2013:47Alex Miller (Clojure team)generally, you can't delete/replace artifacts in most maven repos (but it's an open system and someone could make a repo that allowed this)#2022-03-2013:48Alex Miller (Clojure team)many people use a enterprise-wide server to serve as a proxy for public maven repos, which has the benefits of caching and giving you a point to detect such changes and/or lock things down#2022-03-2013:49Alex Miller (Clojure team)things like Sonatype Nexus#2022-03-2013:50Alex Miller (Clojure team)clj/tools.deps work fine with those setups - that support is built into Maven. they do not yet work with Git proxies (but I hope to add that)#2022-03-2014:04orestisRight, it seems that AWS CodeArtifact is such a proxy I could use. Thanks for the guidance!#2022-03-2112:12dergutemoritzWhat about shadowing Maven artifacts? AFAIUI tools deps by default checks clojars and maven central in some defined order. Assuming it checks clojars first and falls back to central only when an artifact couldn't be found there (is this correct?), then a malicious user could shadow any artifact from central by putting a homonymous one with the same version on clojars without any dependees noticing, right?#2022-03-2112:13dergutemoritzre this:
> if the maven server is compromised, the signature file is also vulnerable.
Only if the private key of the author is compromised, as well, though, which wouldn't be the case only by breaking into the Maven repo. Or does central store private keys on their servers, too? 😬#2022-03-2113:53Alex Miller (Clojure team)maven does not store private or public keys#2022-03-2113:54Alex Miller (Clojure team)it stores signature files alongside artifacts#2022-03-2113:56Alex Miller (Clojure team)someone with control of the maven contents could provide correctly signed artifacts using a different key. the catch here is that there is no way for a Maven consumer to know what key is the "correct" one to look for other than trusting this key as a manual step per key, which no one does.#2022-03-2113:58Alex Miller (Clojure team)Re "What about shadowing Maven artifacts?" - this is a potential problem but note that for deps.edn specifically the only maven repos used are ones that you define at the top-level (not transitive repos), which differs from Maven's behavior. so you are explicitly in control of the set of maven repos being used. I'm not sure what controls clojars has in place over this wrt Maven Central - I know there have been discussions about it in the past. you could ask about that at #clojars#2022-03-2114:52dergutemoritzSounds reasonable, thanks for the info! Is there some way to make tools deps check signatures when provided with a set of trusted public keys?#2022-03-2115:34Alex Miller (Clojure team)not currently. there is support for that in Maven but it's a huge pain from a user pov (and I've never met anyone that actually did it)#2022-03-2115:36Alex Miller (Clojure team)if you want to put a request on ask for it, happy to gauge interest via that#2022-03-2113:52Maxime DevosI noticed there were some questions on why Guix packages Clojure libraries
instead of just using the Clojure tools (<https://clojurians-log.clojureverse.org/tools-deps/2022-01-15/1642212004.067400> and later messages). Can I try to answer them here, and can I use @ to ping the people that were asking the questions?#2022-03-2114:00pavlosmelissinosBy the way there's a #guix channel but it's really young and tiny so if there are people interested in it please join!
(I'm definitely interested in this discussion)#2022-03-2114:03Alex Miller (Clojure team)I don't think this thread should be here, so would prefer that happen elsewhere#2022-03-2114:36Maxime DevosI've started answering on #guix.#2022-03-2115:26Maxime DevosThe questions about Guix have now been answered in #guix#2022-03-2122:26Alex Miller (Clojure team)there's a new prerelease of the Clojure CLI targeted at https://ask.clojure.org/index.php/10703/error-building-classpath / https://clojure.atlassian.net/browse/TDEPS-153 - errors that happen during concurrent Maven requests, would be great if anyone can try it out, especially if you have experienced that problem#2022-03-2321:57Carlowhat's the right way of writing the jvm option in a deps.edn file?
{:jvm-opts ["-Dsnoop.enabled"]
:deps {}}
I can write -J-Dsnoop.enabled=true on the command line, and I would like the deps.edn equivalent#2022-03-2321:58Alex Miller (Clojure team)there is no way to provide a global jvm opt#2022-03-2321:59Alex Miller (Clojure team)you can provide them in an alias with :jvm-opts as you have above and then use the alias#2022-03-2322:00Alex Miller (Clojure team){:aliases {:snoop {:jvm-opts ["-Dsnoop.enabled"]}}}
clj -A:snoop
#2022-03-2322:04Carlothanks! I'm sure there has to be a reason for that, but I don't see it at first glance#2022-03-2322:05Carlodo you happen to have any insight on why that has to be the case?#2022-03-2322:12Alex Miller (Clojure team)most jvm options don't make sense for the open world of all possible programs you might run in a project#2022-03-2322:12Alex Miller (Clojure team)so I'm not terribly convinced yet that this is a good feature to have :)#2022-03-2322:18CarloI see, thank you! 🙂#2022-03-2400:28practicalli-johnI've just added some pages on JVM options, showing how to use them with Clojure CLI.
https://practical.li/clojure/reference/clojure-cli/jvm-options.html
There are also a number of examples and https://practical.li/clojure/reference/jvm/java-17-flags.html.
There are a lot of options to choose from 🙂
It would be very interesting to know which options are commonly used. I've put together https://practical.li/clojure/reference/jvm/common-options.html.
Appreciate any corrections if something is not quite right.#2022-03-2401:17Alex Miller (Clojure team)I would definitely add the OmitStackTrace one#2022-03-2401:18Alex Miller (Clojure team)-XX:-OmitStackTraceInFastThrow#2022-03-2401:19Alex Miller (Clojure team)That is the most common jvm option I use#2022-03-2401:20Alex Miller (Clojure team)And probably next most common are the —illegal-access flags - debug and deny#2022-03-2401:51practicalli-johnInteresting. It seems the --illegal-access flag was marked deprecated in Java 16, with https://openjdk.java.net/jeps/403. So maybe one day it will be less common :)#2022-03-2402:08Alex Miller (Clojure team)Well I missed that!#2022-03-2402:12Alex Miller (Clojure team)That's actually great, I'm glad they switched the default to deny as that means the reflection canAccess methods will actually work#2022-03-2402:12Alex Miller (Clojure team)The Clojure reflector uses that feedback to look further up the inheritance chain for a public method to call#2022-03-2706:16Ben SlessAren't server and tiered compilation on by default?#2022-03-2407:00miikkaCould there be a concurrency issue with git dependencies? We're occasionally seeing that deps fails to clone a git repository with an error message like this. The dependency it is trying to download occurs in the dependency tree more than once.
fatal: destination path '...' already exists and is not an empty directory.
#2022-03-2407:08miikkaOh, and before that there's log entries saying that it's trying to clone the repo three times. Hints towards a concurrency issue.
Cloning: ssh://.../our-repo.git
Cloning: ssh://.../our-repo.git
Cloning: ssh://.../our-repo.git
#2022-03-2407:23miikkaPosted an ask about this https://ask.clojure.org/index.php/11663/possible-concurrency-issue-with-git-deps#2022-03-2516:03jjttjjI remember seeing something about this recently but can't find it now. Is there a way to use clj without an internet connection?#2022-03-2516:04dergutemoritzhttps://ask.clojure.org/index.php/11644/re-building-without-internet-access-with-nix#2022-03-2516:05jjttjjthat was it, thanks!#2022-03-2516:05dergutemoritzFYI: We're still on it, so watch that place for updates#2022-03-2516:16Alex Miller (Clojure team)I was not able to reproduce that case above, so would welcome more info there#2022-03-2516:17dergutemoritzAye, should arrive any day 🙂 Thanks for your support so far!#2022-03-2516:22Adam Helins@U064X3EF3 Are you on MacOs by any chance? Because it turns out I couldn't reproduce either while my Linux colleagues could, very odd
We did some intensive debugging today. I still have a few things to try and I'll report on ask.clojure#2022-03-2516:23Alex Miller (Clojure team)That is where I tested, but not clear to me why it would be different#2022-03-2518:45favilaI think this may be a platform difference. The different clojure platform installers scripts put their files in different places, but the nix script doesn’t account for this: https://github.com/NixOS/nixpkgs/blob/nixos-21.11/pkgs/development/interpreters/clojure/default.nix#L19-L48#2022-03-2518:45favilaThis is an issue I ran into with our own “bin wrapper” tool wrapping clojure for download#2022-03-2518:46favilaso on some platforms, the clojure bash script doesn’t find the jar and redownloads it#2022-03-2518:52favilayeah, this is macos:
.
├── BOOTSTRAPPED
├── bin
│ ├── clj
│ └── clojure
├── deps.edn
├── example-deps.edn
├── libexec
│ ├── clojure-tools-1.10.3.1087.jar
│ └── exec.jar
├── share
│ └── man
│ └── man1
│ ├── clj.1
│ └── clojure.1
└── tools.edn
#2022-03-2518:53favilathis is linux:#2022-03-2518:53favila.
├── bin
│ ├── clj
│ └── clojure
├── BOOTSTRAPPED
├── lib
│ └── clojure
│ ├── deps.edn
│ ├── example-deps.edn
│ ├── libexec
│ │ ├── clojure-tools-1.10.3.1087.jar
│ │ └── exec.jar
│ └── tools.edn
└── share
└── man
└── man1
├── clj.1
└── clojure.1#2022-03-2718:24kennyDoes a Tool released to Maven support :tools/usage? If yes, how so? Do I need to include my deps.edn in the built jar?#2022-03-2719:10seancorfieldNo, :tools/usage only works with git (and local) source-based deps. Basically deps.edn-only stuff.#2022-03-2719:11seancorfieldArtifacts released to Maven/Clojars have a pom.xml files as the "system of record" inside the JAR and that contains a list of other Maven/Clojars dependencies (that are all "standard" :mvn/version style libraries).#2022-03-2719:12seancorfield@kenny ☝️:skin-tone-2:#2022-03-2719:14kennyGot it. Thank you. It might be nice to add that :tools/usage is not supported for maven to the website docs in https://clojure.org/reference/deps_and_cli#_tool_usage.#2022-03-2719:31seancorfieldhttps://github.com/clojure/clojure-site/issues/584#2022-03-2721:41Alex Miller (Clojure team)I have thought about this quite a bit and I wouldn't close the door on ever supporting artifact based tools, but it raised more issues than I was willing to solve before the initial tool release#2022-03-2721:42Alex Miller (Clojure team)One workaround would be to make git dep wrapper with a tools/usage around a maven artifact#2022-03-2822:22chancerussellIs there some way to “cache” the result of the The CLI will verify that both the tag and prefix sha point to the same full sha commit, and error if not step of the gitlib fetching process? It threw me off a little that clojure -P didn’t cover that (although it does make sense now that I realize that’s what’s going on :))#2022-03-2822:25chancerussellAnd really I’m only asking to make sure I’m not thinking about this completely wrong—it was fairly easy to work around this once I understood what was happening#2022-03-2908:43Wanja HentzeHow did you end up working around it? We're stripping the :git/tag from the deps map now, which feels a little clumsy.#2022-03-2914:02chancerussellThat’s what I did in this case#2022-03-2914:04chancerussellI was using the goofy docker buildkit ssh socket mechanism to hit private repos from the build container and was trying to not pass the ssh socket to a subsequent step after the -P runs#2022-03-3108:35Wanja HentzeThere is now an issue for this: https://clojure.atlassian.net/browse/TDEPS-223#2022-03-3013:34Ben Slessis there a way to set mvn/local-repos in an alias?#2022-03-3013:38Alex Miller (Clojure team)no#2022-03-3013:39Alex Miller (Clojure team)I often set it dynamically at the command line with -Sdeps '{:mvn/local-repo "foo"}'#2022-03-3013:40Alex Miller (Clojure team)as always, feel free to make a request at ask clojure#2022-03-3013:42Ben SlessI think I will, this will help clean up some messy ci scripts#2022-03-3108:42dergutemoritzcan confirm - we currently use a babashka task to patch it into our deps.edn in CI 🙈#2022-03-3108:44dergutemoritzPassing via -Sdeps is a bit fragile IME. We had set up a shell alias for clojure before for passing it in by default but that doesn't work when it's invoked indirectly e.g. from another shellscript.#2022-03-3110:11Ben SlessHere we go
https://ask.clojure.org/index.php/11711/set-mvn-local-repo-via-alias#2022-03-3108:42dergutemoritzcan confirm - we currently use a babashka task to patch it into our deps.edn in CI 🙈#2022-03-3108:44dergutemoritzPassing via -Sdeps is a bit fragile IME. We had set up a shell alias for clojure before for passing it in by default but that doesn't work when it's invoked indirectly e.g. from another shellscript.#2022-03-3110:11Ben SlessHere we go
https://ask.clojure.org/index.php/11711/set-mvn-local-repo-via-alias#2022-03-3102:13Cora (she/her)how might I add resource paths for a given alias?#2022-03-3102:16Cora (she/her)is it just added to extra-paths?#2022-03-3102:17ghadiyeah. "resources" are a maven fiction#2022-03-3102:17dpsuttonjust :extra-paths. My understanding is there’s no such thing as a resource path. Things are found on the classpath or not (clojure source files or traditional “resources”)#2022-03-3102:17Cora (she/her)ok, cool, makes sense#2022-03-3102:17Cora (she/her)thanks!#2022-04-0416:57dpsuttonanyone recognize errors like the following?
Error building classpath. Could not acquire write lock for 'artifact:org.clojure:tools.namespace:1.2.0'
275
java.lang.IllegalStateException: Could not acquire write lock for 'artifact:org.clojure:tools.namespace:1.2.0'
276
at org.eclipse.aether.internal.impl.synccontext.named.NamedLockFactoryAdapter$AdaptedLockSyncContext.acquire(NamedLockFactoryAdapter.java:146)
Apparently we are on '1.10.3.933' which is ancient. I’m looking through the release notes to see if this issue has been addressed. I see some stuff that looks like it might be similar but not exactly the same#2022-04-0417:12dpsuttonI’ve seen that this has been reported before and saw that there is an issue tracking it but don’t see that issue. Also not clear if we should expect this to be fixed or need to run single threaded#2022-04-0417:31Alex Miller (Clojure team)much work has been done on this and I don't believe it is an issue on latest#2022-04-0417:35dpsutton🙏 thanks#2022-04-0611:52grzmI’ve been doing some monorepo spelunking and am calling clojure.tools.cli.api/tree programmatically to get the dependency hierarchy. To get the result as data:
(-> (deps/tree {:project deps-map,
:format :edn})
(with-out-str)
(edn/read-string))
Works a treat. That said, the round-tripping from pprint within cli/tree and re-parsing makes me suspect I’m doing it wrong.#2022-04-0611:55grzmI could call clojure.tools.deps.alpha.tree/trace->tree but there’s some nice setup within cli/tree that I’d rather not duplicate
https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/cli/api.clj#L151-L156#2022-04-0611:58grzmI suspect this information is in the basis as well. My likely-too-brief and naïve inspection of the basis thinks shows that it’s already flattened and inverted from what’s returned by cli/tree (listing parents rather than children).#2022-04-0611:59Alex Miller (Clojure team)There is not sufficient information in the rest of the basis to construct the “tree” (it's not really a tree)#2022-04-0612:00Alex Miller (Clojure team)Something could probably be factored better for programmatic use, feel free to add a request to https://ask.clojure.org#2022-04-0612:01grzmWill do! Thanks, @U064X3EF3 🙂#2022-04-0612:03grzmIs that preferred over opening a TDEPS issue and sending a patch?#2022-04-0612:06Alex Miller (Clojure team)Ask is preferred first, then I make a jira from that#2022-04-0612:06Alex Miller (Clojure team)If you want to provide a patch on that, that would be fine#2022-04-0612:07grzmWilco.#2022-04-0612:47grzmhttps://ask.clojure.org/index.php/11738/how-can-i-return-data-from-clojure-tools-cli-api-tree?show=11739#a11739#2022-04-0812:10borkdudeI guess the answer will be no, but does deps.edn support global exclusions?
E.g. an app has dependencies that depend on org.foo/lib but wants to override it globally with another dep which brings in the same namespaces, but is published under a different organization org.bar/lib.#2022-04-0812:11Alex Miller (Clojure team)No, but I'm pretty sure there is an ask question if you want to upvote#2022-04-0812:14borkdudeI may have bad searching skills, but I don't see one#2022-04-0812:29borkdudeI guess an ugly workaround could be to make a local/root dep indirection and then make an exclusion on that#2022-04-0813:34rickmoynihan@borkdude I’m curious were you forking a transitive dep? I had a similar case recently#2022-04-0813:35borkdude@rickmoynihan I'm asking for something we're considering called tools.misc :
https://github.com/clj-easy/tools.misc
The idea is that you can choose one tool (JSON or http client, etc) but can swap the implementation on the application level to reduce the number of dependencies. A few implementations will be centrally maintained, but it should be possible to override those with your own or other third party implementations.#2022-04-0813:38rickmoynihanok so essentially like putting all your java interfaces in a dep separate from the implementations; but for clojure obviously. Interesting :thinking_face:#2022-04-0813:40borkdudekind of yes#2022-04-1107:15olafHey, where clojure tools is caching the dependencies list/classpath?
I have a project that is using rewrite-clj and in a babashka script in the same project (for error) I imported rewrite-cljs. Since then I’m unable to run a REPL since is requiring the cljs version of the library that is causing conflicts in the code. rewrite-cljs is not present at all in the project, I removed .cpcache , .shadow-cljs and ~/.m2/repository/rewrite-cljs without success. If I run something with clj -M:dev foo and run ps -aef | grep rewrite-cljs in a separate terminal, I see
/usr/bin/java -Dconf=dev-config.edn -Dclojure.basis=.cpcache/1039881647.basis -classpath [...]:/Users/olaf/.m2/repository/rewrite-cljs/rewrite-cljs/0.4.5/rewrite-cljs-0.4.5.jar:[...]
Don’t know how to remove it.#2022-04-1107:34borkdude@eliascotto94 you should not use rewrite-cljs anymore, that library was merged into rewrite-clj#2022-04-1107:34olafYep, I know. Just I cannot remove it from the project#2022-04-1107:35borkdudeYou can do clojure -Stree to see how it's pulled in and then add an :exclusion#2022-04-1107:38olafIt shows only rewrite-clj so is not a dependency at all#2022-04-1107:40borkdude@eliascotto94 Remove ~/.clojure/.cpcache as well#2022-04-1107:43olafTried but nothing changed#2022-04-1107:44borkdudeCan you post the output of clojure -Stree -A:dev in a reply on this message (thread)?#2022-04-1107:45olaf#2022-04-1107:46borkdudeCan you try:
clojure -Sforce -M:dev foo
#2022-04-1107:47borkdudeand do you know for sure that the ps -aef output matches the last java process you spawned and not an old one?#2022-04-1107:47borkdudetry killall -9 java first#2022-04-1107:55olafKilled all node and java. run with -Sforce, opened another window and typed ps -aef | grep rewrite-cljs. Only one process with rewrite-cljs in the classpath#2022-04-1107:55olafSo nothing changed#2022-04-1108:20borkdudeIs the project public?#2022-04-1108:33olafNo, but I can give you access to the repo. If I search inside the project for rewrite-cljs nothing comes out#2022-04-1112:15Alex Miller (Clojure team)Did you check -/.clojure/deps.edn ?#2022-04-1114:21lread@eliascotto94 I see your same question in #rewrite-clj and there you mentioned you are using shadow-cljs.
Try npx shadow-cljs info … I think that is how you ask shadow-cljs to show dependencies…?#2022-04-1114:25lread(Maybe you are using a shadow-cljs.edn?)#2022-04-1118:11borkdude@eliascotto94 I checked out your repo now. And when I ded clojure -Stree -A:dev I see this:
day8.re-frame/re-frame-10x 1.2.5
X binaryage/devtools 1.0.5 :use-top
. com.yahoo.platform.yui/yuicompressor 2.4.8
. zprint/zprint 1.0.1
. rewrite-cljs/rewrite-cljs 0.4.5
X org.clojure/tools.reader 1.3.2 :older-version
X borkdude/edamame 0.0.11-alpha.12 :use-top
X rewrite-clj/rewrite-clj 0.6.1 :use-top
. superstructor/re-highlight 1.1.0
. cljsjs/highlight 10.3.1-0
So it's re-frame10x which pulls this in via zprint.#2022-04-1118:12borkdudebut if you would use a newer zprint, then this problem would be solved since that no longer depends on rewrite-cljs#2022-04-1118:17borkdudeAnother way to do this is put an :exclusion of rewrite-cljs on re-frame 10x.#2022-04-1121:13olafYes, :exclusions worked perfectly. Thanks everyone for the support, it’s the first time I encounter this kind of bug. 🙏 🙏#2022-04-1211:09DenisMcHi, I’m having a weird problem when using tools.deps that is confounding me. I have a build process using deps that compiles my code, has been working fine. I also have a docker build that also used to work fine, but now hangs when attempting to compile. But, it only hangs in the docker build - it builds successfully locally. (local clojure version 1.11.1.1105, docker version from the clojure-latest image, 1.10.3.1087. I thought the smoking gun was that I recently introduced some futures into the codebase to handle some external services, however when I run kill -3 on the process in the docker image I can’t see any references to the futures I created (and the futures are not in top level functions anyway).
When I run lots of kill -3's, the system does seem to be doing work - the main stack trace changes - but it’s crawling along at a snail’s pace.
I realise that this could be caused by any amount of errors I’ve introduced myself, but maybe someone on here might have seen this category of problem before and could point me where it may be most fruitful to look and might save me a few hours of my life! Thanks in advance.#2022-04-1212:36Alex Miller (Clojure team)Is the “build process” a tools.build program or something else? Not sure how to help without more info#2022-04-1214:25DenisMcApologies, should have included that info, it's using clojure.tools.build.api/compile-clj to compile. I'm building an Uber jar after compiling sources but it's not getting that far #2022-04-1214:27Alex Miller (Clojure team)one common reason for compile to take suspiciously long is that you have code at the top level of a namespace that is doing something (compile is a side effect of loading)#2022-04-1214:28Alex Miller (Clojure team)it may be doing something can succeed locally but not in the docker environment, for example#2022-04-1214:30DenisMcWhen top level, you mean (for example) having a (future) call directly in the namespace, rather than within a (defn)? I have introduced a few futures but from what I can see they are all within other functions. #2022-04-1214:31DenisMcNow that you say it though, the stuff I introduced would fit the bill in terms of working locally and not in docker, so that's definitely a smoking gun.#2022-04-1214:31Alex Miller (Clojure team)yes, any code at the top level of the namespace (def's ) will be loaded and evaluated as part of compilation#2022-04-1214:32DenisMcBut defn’s won't?#2022-04-1214:33Alex Miller (Clojure team)no real difference here - each top level form is evaluated. for def , that means evaluating the initialization value. for defn it means evaluating the function definition (but that's not going to invoke the function, unless you cause that to occur)#2022-04-1214:35DenisMcAh ok, that's clearer now. Thanks for the steer.#2022-04-1315:57didibusI often use delay or promise for top level def that aren't constant to avoid that problem.#2022-04-1316:33borkdudeThat is also good practice for graalvm-native compiled Clojure#2022-04-1316:35athomasoriginalIs there a way to see which deps you don’t currently have in your local .m2 . My goal is just to see out of all the deps I need, which would need to be downloaded.#2022-04-1316:37Alex Miller (Clojure team)No, that's not a feature right now #2022-04-1316:38Alex Miller (Clojure team)To get that answer you would at least need to download all the poms #2022-04-1814:34Ben SlessIs there a way to align a dependency's version across multiple deps files?#2022-04-1814:42borkdudeI've made bb scripts for this in the past. There's also a exoscale lib for this called deps-module or something (also works with bb). Forgot the exact name and the phone is a terrible UX to find this.#2022-04-1815:23borkdudefound it: https://github.com/exoscale/deps-modules#2022-04-1818:27emccueMake a synthetic local dependency that just contains the single dependency locked at a version#2022-04-1906:10Ben SlessOne thing I'm still missing is interdependencies between modules#2022-04-1915:30emccueCan you elaborate more?#2022-04-1918:08Ben SlessIf I have module A which depends on B, how do I model that dependency? The git sha is a moving target#2022-04-1918:08emccuefor a monorepo
{:local/root "../component/B"}#2022-04-1918:08Ben SlessBut I'm building artifacts from them#2022-04-1918:09emccueah... no clue - templating?#2022-04-1918:10Ben SlessDoesn't feel right. At least not with strings. Should be able to template things with data#2022-04-1918:11Ben SlessREPL is started at the root of the project, so when would I need to merge all this data? At build time. Think I need to dig into tools.build#2022-04-1922:18favilaFYI, I’m still seeing the “HashMap$Node cannot be cast to HashMap$TreeNode” issue with latest Clojure CLI tools. https://ask.clojure.org/index.php/10703/error-building-classpath?show=11784#c11784#2022-04-2000:15Alex Miller (Clojure team)Yeah, I had another report about it. Haven't had a chance to look at it. Doesn't make any sense to me as everything should be thread local, but obviously something unexpected
Is happening#2022-04-2012:32emccuemy current hypothesis is that the DefaultModelValidator is truly a global singleton in some fashion.#2022-04-2012:36favilaDefaultRepositorySystem is what makes them if you want to chase that down. Or I guess just run a tools deps system, then inspect all the DefaultModelValidator instances#2022-04-2013:46Alex Miller (Clojure team)I assume it's something like that, because Maven is awful#2022-04-2013:47Alex Miller (Clojure team)it is a giant tower of mutable maps, disguised in Java classes, glued together with 3 kinds of dependency injection. there's no reason for any of it to be mutable. why would a validator be mutable - that is a function. bleh.#2022-04-2013:49emccuehttps://get-coursier.io/ if scala's bytecode compatibility wasn't as odious as it is I would almost say this would be an option#2022-04-2013:54Alex Miller (Clojure team)I have to talk myself down from rewriting the whole thing every time I work on it#2022-04-2013:55favilaThe mutability in the validator is for memoization (perf) afaikt#2022-04-2013:59Alex Miller (Clojure team)I know, but you know what I mean#2022-04-2217:27Alex Miller (Clojure team)@U09R86PA4 just released prerelease 1.11.1.1113 - please test#2022-04-2116:41Jem McElwaindoes anyone have an example or suggestions for how to structure a "multi-module" project using tools.build? ideally, i would like to have a single build.clj from a root directory that's able to build jars of sub-projects in a consistent way, but i'm running into lots of problems trying to get this to work. wondering if anyone has advice on best practices here.#2022-04-2117:22seancorfieldHappy to try to help if you can provide more info about what you tried and what doesn't work. We have a monorepo and build over a dozen uberjars from it using a single build.clj at the root. Here's an excerpt from our uberjars task:
(doseq [p projects]
(println "\nRunning: uber on" p)
(let [project-dir (str "projects/" p)
aliases (with-dir (io/file project-dir) (get-project-aliases))]
(binding [b/*project-root* project-dir]
(bb/clean {})
(bb/uber (assoc (:uberjar aliases)
:compile-opts {:direct-linking true})))))
b is tools.build and bb is my org.corfield.build library but it's only a very thin wrapper around tools.build providing "sane" defaults.#2022-04-2117:24seancorfieldEach projects/<foo> folder has its own deps.edn to specify the deps needed for the uberjar (mostly just :local/root to other parts of the monorepo) and there's an :uberjar alias containing data -- options -- for the uber task.#2022-04-2117:24seancorfield(defn- get-project-aliases []
(let [edn-fn (juxt :root-edn :project-edn)]
(-> (t/find-edn-maps)
(edn-fn)
(t/merge-edns)
:aliases)))
is the t.d.a-based helper for getting aliases out of the project's deps.edn file.#2022-04-2117:26seancorfield(our monorepo is slowly migrating from an ad hoc approach to using Polylith which has helped a lot with separating out "build" concerns from dev/runtime concerns)#2022-04-2207:45Adam Helins@U02AEH4M8GY I wrote a little helper for exactly that kind of setup and used it successfully in several projects. Just have to properly document it though 😅
I'm curious, what have you tried?#2022-04-2423:55Jem McElwainMajor problem was just that we were using 7.5.0, most things were fixed just by just upgrading to the latest version and using some of the provided build.clj as reference. Also had some issues with b/write pom skipping local deps, but realized that could be solved by providing extra deps.edn when creating basis.#2022-04-2116:51Alex Miller (Clojure team)should be fine, just need to be careful with calling b/set-project-root! in each per-project context#2022-04-2116:52Alex Miller (Clojure team)there's been at least one thread about this in #tools-build#2022-04-2116:53Alex Miller (Clojure team)see https://clojurians.slack.com/archives/C02B5GHQWP4/p1645694632297419?thread_ts=1645635678.336119&cid=C02B5GHQWP4#2022-04-2210:26lepistaneI am trying to create uberjar for my deps.edn project.
i must be doing something wrong but
when i run
clj -T:build uber :aot true
i get
-T is no longer supported, use -A with repl, -M for main, or -X for exec
Am i using outdated cli or something?
I am using this lib and i see the -T being used https://github.com/seancorfield/build-clj#standalone-cli-usage#2022-04-2210:41pavlosmelissinosI think you might have an older version of the CLI tools. What do you get if you run clj --version?#2022-04-2210:52lepistaneWARNING: When invoking clojure.main, use -M
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
--version (No such file or directory)
Full report at:
/tmp/clojure-10539041283103714829.edn
#2022-04-2210:56lepistanealright you are right. i updated using https://clojure.org/guides/getting_started#_installation_on_linux
Thanks a lot!#2022-04-2210:58pavlosmelissinosI fail to recall the old command for checking the version of your Clojure CLI but after the upgrade clojure --version should give you something like Clojure CLI version 1.11.1.1105
edit: I think it was clojure -Sversion but that doesn't seem to work anymore#2022-04-2210:59lepistaneyeah now it does!
I managed to use -T#2022-04-2214:30practicalli-johnclojure -Sdescribe will show the version of the Clojure CLI tool along with the paths it uses, so quite a useful tool
clojure -Sverbose will show the same info, but also run a REPL or clojure code as well, so useful for debugging classpaths and other issues#2022-04-2214:45pavlosmelissinosOh right, I was thinking of -Sdescribe, thanks#2022-04-2215:09mpenetis it bad to want to have a way to chain multiple -X call in the same process (a bit like lein do ...?#2022-04-2215:10mpenetclj -X:foo bar -X:baz quuq ...#2022-04-2215:10mpenetI guess it becomes a bit ambiguous (-X arg vs -X invocation)#2022-04-2215:11mpenetand classpath isolation would come up I suppose#2022-04-2215:15seancorfieldFor a while it was possible to chain calls but that was removed because of exactly those sorts of issues. Write functions to do the chaining and then you have control of that. See build.clj for example.#2022-04-2215:21mpenetit's easy to emulate other ways, it's not so easy when it wants to be dynamic (in the user controlled sense)#2022-04-2215:21mpenetanyway, I am fine without it, I was just curious#2022-04-2215:23Alex Miller (Clojure team)if you want a program, write a program, and call that :)#2022-04-2215:24Alex Miller (Clojure team)if you want a dynamic program, there's this cool thing called a REPL :)#2022-04-2218:20seancorfield☝️:skin-tone-2: That is why I mostly run our build.clj as a REPL instead of a set of tasks to invoke from the CLI:
(! 543)-> clj -M:build -i build.clj -r
Clojure 1.11.1
user=> (-> {} (build/cold-start) (build/all-tests-ci))
...#2022-04-2218:21seancorfield(I can also do clojure -T:build some-task :the '"args"' when I need something running standalone elsewhere)#2022-04-2218:37mpenetMost build tasks are quite static for us anyway, you never run them manually (ci/cd does the work).#2022-04-2218:38mpenetFor the rest we re trying to standardize profile names for common aliases across repos to lower the learning curve#2022-04-2219:06seancorfieldWe have all sorts of useful stuff in our build.clj script for local dev/test work -- as well as several for CI/CD automation.#2022-04-2220:00borkdude@seancorfield I recently added a :dev alias to a project where I added "." to the classpath, so I could do:
clojure -A:build:dev
and just develop tools build from the REPL in addition to the rest#2022-04-2220:02seancorfieldWe already have a :dev alias for other reasons (Polylith) but I do often add :build when starting a REPL so I can work on build.clj as well as everything else 🙂#2022-04-2220:06borkdudeanother workaround (hack) I sometimes do is clj -Scp $(clojure -Spath -A:build):$(clojure -Spath -A:whatever-else):. , works as well ;)#2022-04-2304:03didibusCan you -A:alias an alias that specified a main-opts and use another -M as the main? I just want the extra-deps from that alias as well#2022-04-2304:04seancorfieldYes, "last one wins" when multiple aliases specify :main-opts. That's in the docs I think?#2022-04-2304:05didibusHum, ok, my issue must be somewhere else then#2022-04-2304:08seancorfieldOh, there might be an issue mixing -M and -A -- don't do that. Just use one or the other and specify multiple aliases.#2022-04-2304:10didibusThat was it, mixing -A and -M don't work, just have to list the alias in the -M itself#2022-04-2304:11seancorfieldThat's just a quirk of the clojure shell script really.#2022-04-2304:12didibusHum, that shell script is more loc than half my clojure programs 😛 , and its bash, so I'm not going to try to figure that one out haha#2022-04-2304:13seancorfieldPah, we have 125k lines of Clojure!#2022-04-2304:20didibusI'm exaggerating a bit, but it's a surprisingly long shell script lol. Ok, so I guess bash wins, the shell script is 487 loc, and borkdude's pure clojure rewrite is 708 loc hehe, I thought it might be smaller, but bash takes the cake today#2022-04-2310:22borkdudeI'd take 10 lines of Clojure over 5 lines of bash any time of the day#2022-04-2310:22borkdudesearch for deps and add them to deps.edn in emacs :)
https://twitter.com/borkdude/status/1517810425689219072 😎#2022-04-2808:47baptiste-from-parishello friends, I struggle for a while now making my private s3 maven repo work on ec2.
Basically
• private maven s3 repo with our jar
• clj with desp & s3 works locally
• mvn with s3 deps work on ec2
• clj with s3 does not work on ec2
• I can aws s3 ls in ec2
• clj cli is 1.11.1.1113
I can’t find why, it’s not an authorisation issue as I made it work with mvn. It should not be a clj issue as it works locally..#2022-04-2808:47baptiste-from-parisif anyone has an idea…#2022-04-2809:40dharriganCan you expand upon what doesn't work for clj with s3? (let's take this to a thread)#2022-04-2810:22baptiste-from-parisOk, finally find it and it’s very tricky because the error message is misleading
I tried my infra on an ec2 t3.nano with only 512mb of memory.
The jar file is around ~100mb so because file is kept in memory contrary to mvn, it could not download the jar.
And the error message was not the good one (could not find the jar on maven…)#2022-04-2811:39Alex Miller (Clojure team)Well if there is something I can improve, would be happy to look at it, but there's no info to go on here#2022-04-2812:23baptiste-from-parisThe underlying error when I tried to do a
(aws/invoke s3-client {:op :GetObject
:request {:Bucket "bucketname"
:Key "/path/to/jar"}})#2022-04-2812:24baptiste-from-parisit looks like it’s part of cognitect.http-client#2022-04-2812:24baptiste-from-paristhe user error was Error building classpath. Could not find artifact name/of/artifact#2022-04-2812:24baptiste-from-paristhe exception must be caught somewhere#2022-04-2812:27Alex Miller (Clojure team)Yeah might be worth an issue on aws-api #2022-04-2812:28Alex Miller (Clojure team)Or tools.deps, not sure where exactly something is being overcaught#2022-04-2812:29Alex Miller (Clojure team)I assume this is using cli to get a jar from s3 repo which uses aws-api which uses http-client and that's where the oome happens#2022-04-2812:30Alex Miller (Clojure team)If you can drop something about this at https://ask.clojure.org I will take a look when I get a chance, I'm out of office today#2022-04-2812:31baptiste-from-paris👌 will do at the end of the day#2022-04-2906:54baptiste-from-parishttps://ask.clojure.org/index.php/11840/tools-deps-not-working-when-downloading-big-jar-small-machine#2022-04-2907:18Ben SlessCan several projects in the same repository, each in their own directory, be requited as git deps?#2022-04-2907:27pavlosmelissinosYou could use :local/root but that will always use the current version of the project in the local filesystem (this is what #polylith uses/suggests).
If you want to pin a project to a specific :git/sha/`:git/tag` I'm pretty sure you can use it as usual and combine it with the :deps/root attribute (haven't tried it myself but I don't see why it wouldn't work)
Reference: https://clojure.org/reference/deps_and_cli#_dependencies#2022-04-2907:35Adam HelinsHow does it play out when two projects within the same repo are requested at the same time?#2022-04-2908:47pavlosmelissinosWell you have to use a different name for each dependency (so you can't use the reverse domain notation) but other than that there shouldn't be any issues, at least in the case of :local/root.
I'm not sure though how well that would work in the case of :deps/root, sorry.#2022-04-2909:51Ben SlessSo using deps/root will clone the entire repository but recognize the correct deps.edn?#2022-04-2910:40pavlosmelissinosI'm not sure if it clones the entire repository (or if it uses something like git's partial clone) but it seems there's https://github.com/clojure/tools.deps.alpha/blob/c815c440e0521647a590a700cf08d6eb9ad8c1f8/src/main/clojure/clojure/tools/deps/alpha/extensions/deps.clj#L32-L35 regardless#2022-04-2912:42borkdudeYes, :deps/root is the thing you can use for that#2022-04-2912:43borkdudee.g. here we have multiple projects within one repo:
https://github.com/babashka/nbb-features/tree/main/features#2022-04-2913:29Alex Miller (Clojure team)a git dep with :deps/root will clone the entire repo and then point into a sub path for the dep#2022-04-2913:29Alex Miller (Clojure team)if you have multiple, they share the same cloned dir (for a given sha)#2022-04-2913:31Alex Miller (Clojure team)I guess more precisely, they share the same git object dir (for all shas) and the same working tree (for a given sha)#2022-04-2914:25Ben SlessCool, just what I needed.
I think that covers everything I need to make a multi module project + build#2022-04-2914:26Ben SlessWhen using this scheme, if one sub project depends on another can it always refer to it by local root?#2022-04-2914:29Alex Miller (Clojure team)yes BUT local deps lose the "git source" tracking and can't be version compared in that way#2022-04-2914:30Alex Miller (Clojure team)we have an open issue about this and it's definitely something to be further explored#2022-04-2914:33Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-132#2022-04-2914:34Alex Miller (Clojure team)Maybe that mostly affects external users of the monorepo (which isn't a thing for you)#2022-04-2915:39Ben Slessbut can they still pin a specific version by requiring it as a top level?#2022-04-2915:40Alex Miller (Clojure team)yes#2022-04-2917:31Ben SlessNice#2022-04-3018:44Ben SlessQuestion regarding collisions, let's say I have a git repo acme/foo, with sub modules bar and bazz, how can I require them both? How meaningful is the key when I specify a git url?
{com.acme/foo.bar {:git/tag "" :git/sha ""
:git/url ""
:deps/root "./module-bar"}}#2022-04-3020:05Alex Miller (Clojure team)They're keys in a map - they have to be unique#2022-05-0102:49Ben SlessWhat I mean is, if I specify a git repo, does the key matter in any way besides the requirement it be unique?#2022-05-0102:53Alex Miller (Clojure team)No#2022-04-2918:09dharriganIs 1.11.1.1113 an official release (or pre-release?)#2022-04-2918:09dharriganAh, yes it is.#2022-04-2918:34Alex Miller (Clojure team)Official - I accidentally called it a pre release on the announcement originally#2022-04-3010:41hkjelsHave any of you worked on a hybrid of monolithic and split repo?
And by that, I mean a monolith that include child-libraries via git submodules, so that others can make use of these child-libraries via deps.edn , while still working with it as if it was a monolith. I’m starting a project where I think it would make sense, but I would like to hear if others have some gotchas and bad experiences with it.#2022-04-3010:43hkjelsOr good experiences for that matter 🙂#2022-04-3010:45lispycloudswe use this approach in #babashka https://github.com/babashka/babashka worked out fairly well I think, cc @U04V15CAJ#2022-04-3010:46lispycloudsfor me forgetting to do a submodule update was the worst of the issues 😅#2022-04-3011:06pavlosmelissinosI don't get it! Why do you need git submodules if you have deps.edn?#2022-04-3011:06borkdudebecause those git submodules are projects of their own. The alternative is to use them as git deps, but this allows me to quicker iterate#2022-04-3011:07borkdudeIt sounds I've been doing exactly what @U0B1SDL67 is asking#2022-04-3011:07borkdudeYou can use local/root in your parent project while the git submodules are still their own independent project#2022-04-3011:10pavlosmelissinos> independent project
>
You mean like a separate repo?
Oh I'm sure there's a reason but I'm wondering how this is better than having everything on the same git repo.#2022-04-3011:12borkdudeYes, a separate repo. E.g.:
https://github.com/babashka/sci
https://github.com/babashka/fs
are projects on their own but also used in babashka. I don't want to bump a git sha every time I make a small change to SCI or fs, so I use them as git submodules.
Putting all those projects inside the babashka repo doesn't make sense since those projects can be independently used from babashka.#2022-04-3011:14pavlosmelissinosOh, I get it now, thanks for the explanation. I was going to recommend #polylith but if you need your projects to be separate repos it wouldn't work, obviously.#2022-04-3011:18borkdudeThat and maybe some problems can be solved using lighter weight ways#2022-04-3011:18hkjelsGreat! This put me at ease about the idea. Thanks#2022-04-3011:20hkjelsI’ve looked briefly at polylith, but it seems a bit cumbersome and overkill#2022-04-3011:20hkjelsmight be that I just haven’t quite understood it though#2022-04-3011:23lispycloudsThe way i think about it is if i have bunch of code deploying as a single purpose set of microservices, polylith seems to be a good fit. ie, one repo and multiple deployable artifacts/services. If its a single deployable artifact, use submodules and/or git/maven deps. Both approaches try to address the code resuse problem but for me it depends on what am I trying to build.#2022-04-3011:26lispycloudsgit/maven deps vs submodules comes down to your dev practices/iteration needs etc like borkdude mentioned#2022-04-3011:27borkdudewell in theory I could also make a small (bb) script that quickly updates the git/shas of the libraries that I'm developing#2022-04-3011:28borkdudebabashka is the only repo where I'm working like this. in other projects I usually switch between local/root and then back to git shas#2022-04-3011:29borkdudein the babashka case, the build is also still based on leiningen which doesn't support git deps, which is why I chose git submodules initially. I needed uberjar, etc to work which wasn't there a few years ago in the deps.edn world#2022-04-3011:29borkdudefor development I use deps.edn there, but the deployment is done with lein#2022-04-3011:29borkdudeI will probably switch over in the future, but it wasn't a huge priority so far#2022-04-3011:31borkdudeit's occurring to me that you can switch between local/root and git shas also via an -Sdeps argument perhaps, for development#2022-04-3012:29Alex Miller (Clojure team)Just fyi, Clojure CLI does not work with git deps using sub modules #2022-04-3012:30borkdudein bb's case that's not a problem, but good point in general#2022-05-0107:26steveb8nI'd like to do this as well. Ideally with different access control on submodules. Sadly I don't think cloud hosts support access control on modules#2022-05-0114:41emccueLets say I have a dependency tree like so
A (v1) -> B (v2)
C (v1) -> B (v1)
And I directly depend on A and C. I should get
A (v1), B (v2), C (v1)
Is there a way I can get the tree of "actual versions used"?
I want to have one file where I declare all the dependencies for the mono repo and every subproject to always get consistent versions of everything.
So if something in the codebase only depended on C (v1) I would still want it to get B (v2) for consistency#2022-05-0114:53Alex Miller (Clojure team)You can see all of this with the tree output of the various commands#2022-05-0114:53Alex Miller (Clojure team)You might find clj -X:deps list to be useful too#2022-05-0114:55Alex Miller (Clojure team)More options with https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/list including edn output#2022-05-0115:25emccueah the :edn part is the key i think#2022-05-0211:46EugenIs there any way I can exclude files ?! I have an issue with what might be a broken shaded jar.
The issue is because the jar manifest has signatures that are not valid ?!
At least that is what I understand.
deps.edn:
{:paths ["src" "test" "resources"]
:deps {org.clojure/clojure {:mvn/version "1.11.1"}
org.apache.directory.server/apacheds-all {:mvn/version "2.0.0.AM26"}}}
the file
(ns user
(:import (org.apache.directory.server.core
DefaultDirectoryService)))
the error
; Evaluating file: core.clj
; Execution error (SecurityException) at sun.security.util.SignatureFileVerifier/processImpl (SignatureFileVerifier.java:316).
; Invalid signature file digest for Manifest main attributes
; Evaluation of file core.clj failed: class clojure.lang.Compiler$CompilerException
clojure.lang.Compiler/load (Compiler.java:7665)
ieugen.ldap-auth-provider.core/eval7557 (NO_SOURCE_FILE:1)
clojure.lang.Compiler/eval (Compiler.java:7194)
clojure.core/eval (core.clj:3215)
clojure.core/eval (core.clj:3211)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.core/apply (core.clj:667)
clojure.core/with-bindings* (core.clj:1990)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:87)
clojure.main/repl (main.clj:437)
clojure.main/repl (main.clj:458)
clojure.main/repl (main.clj:368)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:84)
nrepl.middleware.interruptible-eval/evaluate (interruptible_eval.clj:56)
nrepl.middleware.interruptible-eval/interruptible-eval (interruptible_eval.clj:152)
nrepl.middleware.session/session-exec (session.clj:218)
nrepl.middleware.session/session-exec (session.clj:217)
java.lang.Thread/run (Thread.java:829)
#2022-05-0211:58EugenI managed to get around this by including all the jars instead of the shaded -all jar .#2022-05-0211:58Eugenbut would be nice to know if there are some knobs in deps#2022-05-0215:49seancorfield:exclusions tells it to omit a specific (transitive) dependency.#2022-05-0216:29Eugeni think the issue is with the jar manifest and the fact that those files exist there with different checksums#2022-05-0216:29EugenI am not familiar with the details#2022-05-0216:29Eugenso exclusions might not work in this case#2022-05-0221:13dominicmUnfortunately you can't exclude individual files from jars.#2022-05-0214:43grzmHello! I’ve just recently started using clojure.tools.deps as a library (as opposed to via the clj command line tool), and am seeing WARNING: Use of :paths external to the project has been deprecated, please remove: for the first time. We have a monorepo. Each module in the monorepo has its own deps.edn, expressing dependencies on other modules using :local/root. I can imagine motivations for wanting to flag external paths, and I can also imagine scenarios where this is safe. We always want to depend on HEAD: we never want to refer to a specific commit for these dependencies, so using [:git/url :git/sha] and bumping those each time is something we’d rather avoid.#2022-05-0214:44grzmThis warning has been in the clojure.tools.deps for a long time (over a year), so I imagine there’s been discussion around this in the past. If people have references at hand to point me to, that’d be appreciated: I don’t mean to rehash all of this if it’s already been addressed.#2022-05-0214:55grzmIt looks like https://clojure.atlassian.net/browse/TDEPS-132 is set up to address this.#2022-05-0215:51seancorfieldWe have a large monorepo and use :local/root without that warning -- so you must have :paths that point outside the local deps.edn project scope as well? What are you using :paths for that isn't a "local" directory?#2022-05-0217:33grzmI think I’m seeing when using the clojure.tools.deps library directly (e.g., referring to the clojure.tools.cli.api namespace for monorepo operations). We’re not seeing it when running daily clj commands that rely on the same deps.edn files.#2022-05-0217:34grzmAnd it’s pointing to all of our local/roots, not some exception.#2022-05-0217:36seancorfieldThat warning comes from :paths, not :local/root -- the latter can point outside a project. And if you're not seeing it from the CLI, maybe your CLI is out of date and using a much older t.d.a?#2022-05-0217:56grzmOh, good call. Thank you, rubber ducky 🙂 (seancorfield rubber ducky is the best rubber ducky)#2022-05-1516:43grzmSo, I got to the root of the issue: when supplying a deps-map to clojure.tools.deps.alpha/create-basis via :project. Supplying a path to the source deps.edn file doesn’t have this issue.#2022-05-1517:00grzmSo, what I think is happening is that the :local/root in deps is pulling in :paths from those project’s deps.edn files, and those paths are (unsurprisingly) relative to those projects, and external to wherever create-basis thinks the project root is (probably the user.dir, but maybe it knows it doesn’t have one set) clojure.tools.deps.alpha/validate-paths calls out these paths as external.#2022-05-1517:12grzmClose, but wrong again. The deps-map I was passing as {:project deps-map} contained absolute paths to paths external to user.dir , which I think is assumed to be project root. Thank you all for your unwilling patience and witnessing me think this through in public.#2022-05-0309:56imreIs there any way to make clojure -A:aliases not execute clojure.main with :main-opts in case they are defined in one of the aliases?#2022-05-0309:56imre; clojure -Sdeps '{:aliases {:foo {:main-opts ["-e" ":hello"]}}}' -A:foo
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
:hello#2022-05-0309:57imreI'd like to force this command to bring up a repl instead, are there any switches that can do that for me?#2022-05-0309:59imreI'd like to do this without having to alter the alias. Say, there's an alias in a project that's usually used with -M to execute something, but I want to repl into the environment set up by that alias instead#2022-05-0310:07imresomething akin to a --strict that doesn't support deprecated usages#2022-05-0310:08borkdudeI think this will be the behavior in the future, as -A with main opts has been long deprecated. @U064X3EF3 mentioned that this might happen with the switch to 1.11 but it didn't#2022-05-0310:08borkdudeas a workaround I sometimes use a subshell + -Scp#2022-05-0310:09borkdudeclj -Scp $(clojure -Spath -A:...)
#2022-05-0310:10imreAh, thank you, that's useful!#2022-05-0310:11imreI'm guessing this didn't happen due to not wanting to break users, there might still be many of them around#2022-05-0310:11borkdudeThe warning has been in there for a long while now#2022-05-0310:12imreyeah, I keep purging those from just about any project I encounter in the wild#2022-05-0310:51practicalli-johnAlternatively, avoid using -A execution flag at all.
I use Rebel Readline as my terminal REPL, which uses a :main-opts to start the REPL process and terminal UI, so I can chain any aliases before the rebel alias and only the :main-opts from rebel is used.
I get a much richer REPL UI experience too..
For example I can run rebel with kaocha to include the library and skip its :main-opts and just use the main-opts from :repl/rebel
clojure -M:env/dev:test/kaocha:repl/rebel
Aliases are defined in https://github.com/practicalli/clojure-deps-edn#2022-05-0311:10imreInteresting, thanks for sharing#2022-05-0312:14Alex Miller (Clojure team)I have been prepping a list of deprecations to remove, and that will happen in stages, but haven't started yet. The above will be one of the last things to change.#2022-05-0312:16imreWould you consider an option to add something along the lines of a strict mode cli switch beforehand so as to provide a way to test whether stuff will continue working after deprecations?#2022-05-0313:08Alex Miller (Clojure team)I'll think about it, but I am loathe to add options#2022-05-0313:09imreTotally understandable#2022-05-0317:51seancorfield@U08BJGV6E I've mostly taken to separating :main-opts into additional aliases, so I have clojure -M:test:runner for example but clj -A:test to start a REPL.#2022-05-0317:52borkdudeThis is exactly what I'm using the -Scp workaround for#2022-05-0317:52seancorfield(although, at this point, I've moved nearly all my actual test running into build.clj so clojure -T:build test tends to be how I run a test suite now)#2022-05-0317:53seancorfieldAliases are designed to be composable so having a separation of concerns and the ability to combine them as needed is more idiomatic (you could argue this is a "feature").#2022-05-0318:13imreThanks for the tip, Sean. I also started doing that and perhaps this will nudge me to do more. It would help though if one could create an alias which is the merging of other aliases for brevity at the command line and to try and not break others' workflows#2022-05-0318:33Alex Miller (Clojure team)I've been thinking about this area a lot recently#2022-05-0318:33Alex Miller (Clojure team)so maybe an area of enhancement soon#2022-05-0318:35imresounds intriguing!#2022-05-0319:11seancorfield@U08BJGV6E Go vote for https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases if you haven't already 🙂#2022-05-0409:32simongrayHow do I import this library in my deps.edn? https://mvnrepository.com/artifact/io.github.mightguy/symspell/6.6.154#2022-05-0409:33simongrayThere is a Leningen example but it says [io.github.mightguy/symspell "6.6.154" :extension "pom"] and I have no idea what to do about the :extension "pom" part. Converting it to io.github.mightguy/symspell {:mvn/version "6.6.154"} doesn’t seem to work.#2022-05-0410:39dergutemoritzThis might be an instance of https://clojure.atlassian.net/browse/TDEPS-202#2022-05-0412:02simongrayAha, it would seem so!#2022-05-0413:04Alex Miller (Clojure team)Yes, it is. I've done some of the internal work for this but still more to do#2022-05-0409:45simongray(I wrote io.github.mightguy/symspell-lib {:mvn/version "6.6.154"} as that was referenced elsewhere and this seemed to work)#2022-05-0410:23OlicalHey everyone! Has anyone noticed with Clojure 1.11+ that their deps aren't fetching when they try and run things from the Clojure CLI now? I have to run the exact command with -P then run it again without which is proving REALLY awkward with wrapped tools like Kaocha. This is happening across all projects of mine, so it's in the Clojure CLI install rather than the projects themselves. The only thing that changed (I think) was the Clojure CLI version updating from 1.10 to 1.11+. Has something changed in the CLI contract and now we need to remember to fetch these deps? Is this a bug? Misconfiguration on my part somehow?#2022-05-0413:01Alex Miller (Clojure team)No intentional change that I can think of#2022-05-0413:02Alex Miller (Clojure team)If you have something reproducible, happy to look at it#2022-05-0413:03Alex Miller (Clojure team)What do you actually see when you run? #2022-05-0413:05OlicalHmm, well it seemed like if I ran clj or used a jack-in it would try to start my Clojure projects without fetching the dependencies they required (when it definitely used to!).
I did delete my .m2 recently, so maybe the .cpcache also needs clearing per dir? :thinking_face:
I just tried it in a blank directory and it fetched the dependency! So I'd put it down to .m2/.cpcache interplay (if that's possible?) or the arguments we pass to clj now have it skip fetching deps since we have some combination of -A -M...#2022-05-0413:05OlicalI'll see if I can reproduce it and check if clearing .cpcache etc helps.#2022-05-0413:08Alex Miller (Clojure team)If you have a cpcache, and you have the same command, it won't fetch anything#2022-05-0413:08Alex Miller (Clojure team)If you alter the m2, you'll need to either erase the cpcache or use -Sforce to force a redownload of the deps#2022-05-0413:10OlicalGot it! I feel like I haven't noticed this before and it used to still fetch everything even if I wiped my .m2, but maybe I just got lucky or accidentally fixed it. Thanks for the insight / prompting me to think about it more!#2022-05-0413:10OlicalMakes sense though.#2022-05-0413:10Alex Miller (Clojure team)It has always worked like this#2022-05-0413:16OlicalThen somehow I accidentally avoided it for years and I have no idea how 😅#2022-05-0414:19Alex Miller (Clojure team)by not deleting your m2?#2022-05-0414:20OlicalOh no, I've cleared that many many times, mostly around where I mounted it into docker containers which ends up creating the files under root which complicates other things. (I should work around that by keeping a separate root .m2 from my user really)
I guess I remembered to remove my .cpcache in the past and it slipped from my memory.#2022-05-0507:09brendnzIs it good practice to periodically delete our m2?#2022-05-0509:37OlicalI don't think so if you don't need to. As I mentioned higher up in the thread, I did it because I'd been mounting my .m2 into docker containers which poisons it with root owned files. I could chown them too, but I'm being lazy and silly. The only reason I'd recommend it is if you were REALLY tight on disk space, your .m2 was huge and you know you're only working with a few JVM deps now so you won't need to redownload them all.#2022-05-0509:38OlicalThat way you clear out old versions of deps you no longer use and redownload the ones you need. But as I've found, you must delete your .cpcache directories in your projects to ensure it'll re-fetch dependencies.#2022-05-0513:10Alex Miller (Clojure team)You can also use -Sforce to force recompute of the classpath (and download if needed)#2022-05-0411:39practicalli-johnAre there command line tools (or REPL based tools) to help find the full name of a library to add as a dependency?
I can find versions of libraries using the built-in clojure -X:deps find-versions :lib fully-qualified/library-name , which works very nicely assuming I know the correct library name, but that is the bit I'd like a tool for.
I have been using https://github.com/hagmonk/find-deps although that seemed to stop working. I forked it and got it working and fixed some of the deprecation warnings (which I'll send a PR for, although I think some of the dependency versions in that project also need updating, so may need to hack a bit more)
Any alternative CLI tool suggestions are welcome (otherwise I'll try find some time to improve find-deps).#2022-05-0411:41practicalli-johnRunning find-deps I was getting a very strange warning
Error building classpath. Manifest type not detected when finding deps for swisseph/swisseph in coordinate #:loca
01/swisseph-2.01.00-01.jar"}
Changing to a different directory fixed the issue, so I assume it was some kind of cache issue. I couldnt find that library in the dependency tree for find-deps or online. Very strange.#2022-05-0413:07Alex Miller (Clojure team)There's some speculative work on dep searching in the add-libs3 branch of tools.deps, but it's pretty fiddly, not sure it will ever be integrated#2022-05-0417:21seancorfieldGiven that the Maven coordinates can be arbitrarily different to the namespaces within a library, I don't see how that could work, short of indexing the whole of Clojars (and a lot of Maven Central)?#2022-05-0417:32Alex Miller (Clojure team)search apps exist#2022-05-0417:33Alex Miller (Clojure team)the code in that branch uses them (https://mvnrepository.com, https://grep.app, etc)#2022-05-0417:42seancorfieldOK, and they're essentially brute force searches? (I know "go read the code" 🙂 )#2022-05-0417:43seancorfieldhttps://github.com/clojure/tools.deps.alpha/blob/add-lib3/src/main/clojure/clojure/tools/deps/alpha/repl.clj#L109 onward -- @U05254DQM#2022-05-0419:46agI need something akin to :prep-tasks for lein's uberjar. Basically, some scripts (esp. front-end related - shadow-cljs, etc.) generate a bunch of throwaway files and I need to clean them up before packaging.
What's the recommended way of dealing with that? A custom alias that calls a function to clean them up?
I'm using {uberdeps/uberdeps {:mvn/version "1.1.4"}} for packaging; I don't know if that's relevant#2022-05-0419:51seancorfield@ag Do you have a build.clj file?#2022-05-0419:52seancorfieldI do pretty much all my testing/packaging via tools.build these days and then tasks are "just code" so they can do whatever you need.#2022-05-0419:52seancorfieldhttps://clojure.github.io/tools.build/clojure.tools.build.api.html#var-delete 🙂#2022-05-0419:55agOh, that's interesting. I did not know about this. Still new to deps.edn stuff, been stuck with Leiningen until very recently. I'll try to learn about this. Thank you!#2022-05-0420:00ghadi@ag https://clojure.org/guides/tools_build
the guide shows some examples of builds as ordinary programs#2022-05-0420:20agit's not possible to re-use keys from one alias in another, right? e.g, I have alias :dev that adds extra-paths and deps, if I want to use the same stuff in other alias, e.g. :test, I'd have to repeatedly declare those, correct?#2022-05-0420:21seancorfieldclojure -A:dev:test ?#2022-05-0420:21seancorfieldOr if it's a subset of :dev that you need for :test, split :dev in two.#2022-05-0420:23seancorfield@ag This is typically how I start my main dev REPL for work: SOCKET_REPL_PORT=5000 clojure -J-Dlog4j2.configurationFile=log4j2-sean.properties -M:rebel:portal:everything:dev:test:runner:build:dev/repl but sometimes I add even more aliases (such as :reflect:jedi-time:add-libs 🙂 )#2022-05-0420:48agYes, I know I can chain aliases, but that means some of them won't work without others. If that's accepted, I'm okay with that, I guess.#2022-05-0420:49dergutemoritz@ag https://clojure.atlassian.net/browse/TDEPS-170 and https://clojure.atlassian.net/browse/TDEPS-220 are relevant here#2022-05-0420:53dergutemoritzAnd https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases (linked from the second jira issue) if you'd like to leave an upvote 🙂#2022-05-0421:35Cora (she/her)hmm aliases that depend on aliases might be neat#2022-05-0422:03Alex Miller (Clojure team)well don't get too attached to this particular solution, the problem is about composition and interacts with other problems#2022-05-0506:45practicalli-johnAliases that depend on other aliases (who may also depend on aliases that depend on aliases....) seem a great way to add complexity to an otherwise simple approach configuration.
Excludes and overrides may be needed to manage conflicts and control what is being brought in from other aliases, which would then need tooling to show what each alias is pulling in...
Or aliases could be kept simple, declarative and composable on the command line (it's for a CLI after all). I hope this is the path that is kept.#2022-05-0515:28dvingoSince writing that ticket I started using babashka to create my ~/.clojure/deps.edn file and this solves the developer experience issue for me. It's another layer to deal with but it works. Keeping complexity out of the lower layers is probably a good idea.#2022-05-0611:24socksywhat does :default-deps actually mean? My understanding was you could use it to define some deps that could be overridden from e.g. ~/.clojure/deps.edn, but it doesn't seem to do that. Is it actually just meaning if no deps at all are defined in then it will define some deps? The documentation is not helping my mental model so much#2022-05-0612:09favilaDefault deps means any deps which don’t have a version declared will get that version#2022-05-0612:10favilaThink of it as roughly “managedDependencies” of maven or Lein#2022-05-0612:12favilaUse it by saying :deps {coord nil} in your base deps, then adding default deps with :default-deps {coord some-version} and applying via alias#2022-05-0616:00seancorfield:override-deps is how you override versions of deps -- it says "if this dependency is brought in, by anything, use this version regardless of how the dependency is declared"#2022-05-0616:01socksyand that's something you would put in your ~/.clojure/deps.edn ?#2022-05-0616:02seancorfield:default-deps is kind of weird. Like, you might think you could use it to provide a default version of Clojure if nothing else specifies it, but there are tooling contexts that do supply a specific version of Clojure and your default won't take effect.#2022-05-0616:03seancorfieldI have :override-deps in my user deps.edn file so I can force a particular version of Clojure when running stuff: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L69-L87#2022-05-0616:03seancorfieldThat also allows me to do multi-version testing:
for v in 7 8 9 10 11; do clojure -M:test:runner:1.$v; done
#2022-05-0616:04seancorfield(although these days I mostly run my tests via build.clj since that gives me more control over test setup and context)#2022-05-0616:04seancorfieldI have never found :default-deps useful.#2022-05-0616:04favilaI use default deps to share and centrally manage directly-depended-on versions across modules. It’s great for this use case, but weird because deps doesn’t allow composing deps files (you have to generate them), and only lets this thing come in via an alias, and Alex said he didn’t have this in mind.#2022-05-0616:05favilaand override-deps only for fixing “bad” versions that our deps.edns don’t depend on directly#2022-05-0616:06seancorfield@U09R86PA4 Unless your defaulted dependency is top-level, you understand that if a more recent transitive version of your :default-depsis in the tree, it will take precedence?#2022-05-0616:06seancorfield(although I expect you're using :local/root deps for your modules?)#2022-05-0616:07favila> Unless your defaulted dependency is top-level
You mean unless I do :deps {lib nil}? Yes that’s exactly the intention and how we use it#2022-05-0616:07socksyto be clear, this is about a long standing back-and-forth issue we've had with setting cider-nrepl in a repo with many engineers. It's a struggle to support both people who have no idea about tooling and just want Calva to work out of the bat, and also to support people running e.g. a particular version of cider in emacs which might require a different version of nrepl. We recently switched to putting the cider-nrepl dep in a :default-deps map, but it turned out this wasn't tested properly and never worked — which made me wonder what the heck the point of them is. But it looks like you need to explicitly declare the version to be nil in your :deps map — which is a bit weird, but I guess it makes sense if it's been used historically for deps like org.clojure/clojure which will be there implicitly anyway#2022-05-0616:08favilasome specific built artifact may bring in a different dep transitively and :default-deps will have no affect, but that’s ok.#2022-05-0616:08favilaif it’s not ok (e.g. a CVE), use :override-deps#2022-05-0616:08seancorfieldSo you have it as a top-level dep and you have to use an alias to bring in the :default-deps? Where is your alias coming from if your top-level deps.edn file has the nil version?#2022-05-0616:10seancorfieldi.e., how are you centrally managing a version via an alias across multiple top-level deps.edn files?#2022-05-0616:10favilathere’s a top-level “shared-deps.edn” which just has aliases in it, including the one with default-deps and override-deps. Each module then has a “module-deps.edn” which only has :deps and has {coord nil} for things it needs. We generate the concated deps.edn for each module (just adds the alias keys) and run each module with the required aliases#2022-05-0616:14favilathe concat is done with babashka.deps/merge-deps, which is a workalike to the tools.deps function that merges your user-deps and project deps together.#2022-05-0616:16favilaso it’s functionally like having (merge user-deps <shared-deps in the middle> project-deps), and I really wish that the cli could just do that so I don’t have to generate composed deps.edn. But we’ve strung enough makefile-foo over it and babashka makes it fast enough that it’s tolerable.#2022-05-0616:51seancorfieldWe gave up on that approach and switched to something very close to what Polylith does (and then, ultimately, to Polylith) -- based on discussions with Alex and his recommendations for a monorepo with multiple modules.#2022-05-0616:52seancorfieldi.e., a master project deps.edn in the repo root and the modules as :local/root deps. The extra thing Polylith brought to the table was specific project deps.edn files in separate folders for the artifacts to build (and then depending on modules via :local/root deps).#2022-05-0616:53favilathat’s what we do, we just call those modules “services” (they’re applications not libraries), and they have only local-root dependencies on other modules.#2022-05-0616:53favilaand it is in one of these that we had the clojure nil problem#2022-05-0616:53seancorfieldThen there should be no need to merge deps.edn files.#2022-05-0616:53favilathe merge is so they pick up the aliases#2022-05-0616:54favilathat have the default-deps and override-deps in them#2022-05-0616:54seancorfieldWhen that discussion came up the other day, I didn't realize you were generating your deps.edn files on top of everything else... we tried that and it had horrible edge cases so we stopped doing it.#2022-05-0616:54favilaThe thing that’s maybe different from what you describe is we don’t build one artifact, we build several#2022-05-0616:55seancorfieldWe build about 20 different artifacts from our monorepo.#2022-05-0616:56favilathen I’m not sure what’s different 🙂#2022-05-0616:57faviladid you alter the un-aliased :deps {…} map of in each module’s generated deps.edn?#2022-05-0616:57favilabecause we don’t do that, we just require an alias. The cli literally cannot compute a classpath (blows up) without the alias#2022-05-0617:08seancorfieldWe moved away from aliases because it was often so hard to get tooling at large to respect them. Things have improved a bit but tooling still isn't very consistent.#2022-05-0617:10seancorfieldWe decided that the "complexity" of duplicating a certain number of dependencies between the :dev profile for Polylith and the various projects/*/deps.edn files was less than the complexity of trying to manage that with aliases and fighting with tooling... ¯\(ツ)/¯#2022-05-0823:20danielcomptonIs there a way to run a tool without installing it first? In CI environments, it’d be nice to be able to just specify a tool (and optionally a version) and use it without having to install it first. Prompted by https://github.com/rm-hull/nvd-clojure/issues/142#issuecomment-1120490553, I had a similar issue#2022-05-0908:39practicalli-johnYes. I use an aliases with -T execution option to run all the tools I use.
The alias allows me to specify the version and any default options via :exec-opts when running the tool
It also means I don't have to specify install the tool before using it
For example, I use clojure -T:search/outdated alias to find libraries with newer versions, excluding some commercial libraries https://github.com/practicalli/clojure-deps-edn/blob/live/deps.edn#L360-L385#2022-05-0823:57seancorfield@danielcompton Pretty much all tools can be run via git deps and specifying the fully-qualified name of the functions that are entry points for the tools.#2022-05-0823:59seancorfieldI thought I had an example for deps-new but it's not in the README so I think I only posted it in #calva -- just a sec...#2022-05-0900:00seancorfieldhttps://clojurians.slack.com/archives/CBE668G4R/p1651366338100909?thread_ts=1651355489.468969&cid=CBE668G4R shows how you can invoke deps-new purely via the command-line, without even an alias...#2022-05-0900:01seancorfield(because -T and -X are similar, under the hood, except for whether the project deps are included)#2022-05-1521:19janezjAre there plans to extend tools-deps to support modules? I would really like to build library of modules (like kit-clj) with reusable code that I can include in my inhouse applications. But include only some modules not all, because I want small app jars. Polylith is not a solution because I would like mylib to be someday public and is not suitable for libs. But I want one big lib of prebuild services, to be sure that all parts are working together, and app,deps is easily maintainable. I know exoscale/deps-modules but is not perfect fit.
In fact mylib will be small in LOC just planty of very small modules with glue code, but complete dependency tree would be huge.
:deps {cblus/services {:git/sha "..." :git/url "
#2022-05-1522:22Alex Miller (Clojure team)In short, no I do not expect to make a concrete notion of modules in tools.deps /deps.edn#2022-05-1522:23Alex Miller (Clojure team)In long, I think there are a variety of possible enhancements I am considering, some of which may help you#2022-05-1522:24Alex Miller (Clojure team)Specifying an alias in a dep is one#2022-05-1617:04Ben SlessI have something sketched out, remind me here to upload to github in an hour or two in case I forget#2022-05-1617:33mpenetWe have another lib that adds common project tasks (and includes deps-modules), it also includes a fancy task runner that can understand multi modules repo (or not). I suppose that might get open sourced eventually#2022-05-1617:34mpenetBut that's strictly for us to have "one way to do it" across the org. All this stuff is easy with tools.deps&build#2022-05-1617:55Ben SlessHere @UQ5QFFX54 It's mostly an experiment, but it works https://github.com/bsless/build.modulo#2022-05-1713:51janezj@UK0810AQ2 I understand each module is referenced as separate dependency with all gitrepo coordinates. Alex mentioned new alias features that should reduce coordinate duplications.
I am constantly commenting in and out, so i would like to avoid multiple references of the same repo
{cblus/cblus {:git/url "<https://c>....cblus.git"
:sha "66a3cb06a6ec7096f0ac2a3337971d"}
#_{:local/root "../../gdm/cblues"
I just don't understand what is the output of spit-project (you are referencing ::project :project).#2022-05-2306:09Ben Sless@UQ5QFFX54 relative coordinates for modules should "just work" because they live in the same directory so if you depend on one directly it should be able to find others transitively.
Spit project exists to pin dependencies for git deps, not for transitive does but external#2022-05-1821:33dpsuttonI seem to be having an issue when a project uses :mvn/repos. From the project it declares
:mvn/repos
{"Exasol" {:url ""}}
and i can start a repl and use the dependency: (com.exasol.jdbc.EXADriver/getVersionInfo) and the jar is in (System/getProperty "java.class.path"). However, if i have a local root to this project:
{:deps {metabase/exasol {:local/root "/Users/dan/projects/work/metabase-driver"}}}
and try to start a repl, i get:
❯ clj
Error building classpath. Could not find artifact com.exasol:exasol-jdbc:jar:7.1.7 in central ()
It’s as if it does not consult this extra repo when depending on the project as a local/root#2022-05-1821:36dpsuttona workaround seems to be to manually hoist the transitive :mvn/repos to the top level and clj -Sdeps '{:mvn/repos {"Exasol" {:url ""}}}' but this seems like the tooling should do this#2022-05-1821:36seancorfieldThis has come up before as a "known issue" as I recall...#2022-05-1821:36dpsuttonah i see. i just found it (and with a response from you) on http://ask.clojure.org#2022-05-1821:36dpsuttonshould have checked first#2022-05-1821:38seancorfieldAh, I thought it was familiar -- couldn't remember why it wasn't done but "security" is a good reason.#2022-05-1821:39dpsuttonyeah. it would be nice if you could list them to make it easier to include top level#2022-05-1821:39dpsuttonclj -X:deps list-mvn-repos or something so i could
have tooling audit which are attempted to be included so i can easily include them if necessary#2022-05-1821:40seancorfieldMaybe add that suggestion to the ask post about it?#2022-05-1821:40dpsuttonwill do now#2022-05-1821:45Alex Miller (Clojure team)Yes, please do#2022-05-1914:28Drew Verleeis -Jopt a valid clj-opt argument?
When you read the list it seems like should be, but i get the impression it's just a parameter name. Would this be more accurate if slightly more cumbersome?
Start a REPL clj [clj-opt|Jopt] [-Aaliases] [init-opt*]
This isn't something i'm really suffering over, i'm just trying to solidify my understanding.#2022-05-1914:38Alex Miller (Clojure team)yes, it's a valid clj-opt, not sure I understand the question#2022-05-1914:49Drew VerleeI just mean that this fails
$ clj -Jopt
Error: Could not find or load main class opt
Caused by: java.lang.ClassNotFoundException: opt
As where this does not:
$ clj -Spath
src:/home/drewverlee/.m2/repository/clojure/java-time/clojure.java-time/0.3.2/clojure.java-
#2022-05-1914:50Alex Miller (Clojure team)"opt" is something you have to provide that gets passed through to the jvm#2022-05-1914:51Alex Miller (Clojure team)so clj -J-Xmx256m for example#2022-05-1914:52Drew Verleeyes, that makes sense. That's what i'm learning, and in my mind i'm going to think of it as
-J<opt> Pass <opt> through e.g -Xmx512m#2022-05-1914:53Drew VerleeAs where the "path" in Spath isn't a parameter itself.#2022-05-1914:55Alex Miller (Clojure team)well, I understand the confusion and think about a way to make that clearer#2022-05-2008:59mpenet@alexmiller did you consider a extra-exec-args option?#2022-05-2009:00mpenetI got a use case where it would be nice to share an alias with a pretty big exec-args and just add stuff to it with other aliases#2022-05-2013:00Alex Miller (Clojure team)exec-args are merged if you use multiple aliases so I think you already have that?#2022-05-2013:03mpenetoh they are, didn't think they would be#2022-05-2013:03mpenetI ll check, thanks#2022-05-2013:04Alex Miller (Clojure team)And then command line args are merged at the end of that#2022-05-2222:00Joshua SuskaloI'd like to confirm, the :deps key on aliases is deprecated in favor of :replace-deps, right?#2022-05-2312:24practicalli-johnI consider :deps to be an short (and vague) version of :replace-deps key, so try avoid using it.
I change any :deps to :replace-deps so I have one less thing to remember 😉#2022-05-2222:25Alex Miller (Clojure team)not deprecated, they are both valid alternatives that do the same thing#2022-05-2222:28Joshua Suskaloalright, I was curious because :replace-deps is mentioned in the reference on the site, but just :deps on aliases is not. Is the behavior defined if both are present?#2022-05-2300:38seancorfieldThey are aliases for each other. If have to look in the source code to see which one "wins" if you specified both.#2022-05-2416:35Sam Ritchiehey all — I am looking to run clj -P in my github action, but I want to prepare the deps for one of my aliases. Following this advice: https://gist.github.com/realgenekim/97cd8da28b00761560bef06b8072eb9f
I ran clj -A:test -P
which works! but gives me this warning:
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
is there a non-warning way to write this?#2022-05-2416:35Sam Ritchie[
it’s not that#2022-05-2416:37Alex Miller (Clojure team)-M and -X capture everything after as args#2022-05-2416:37Alex Miller (Clojure team)so clj -P -M:test#2022-05-2416:43Sam Ritchiethank you!#2022-05-2417:10seancorfield@U017QJZ9M7W And remember to add in any other aliases you might need for build tooling etc so those don't get downloaded every time (I assume you're doing this to set up caching?). Our CI pipeline has:
clojure -P -M:build:poly:everything:dev:test:runner
#2022-05-2417:54Sam Ritchieah nice!#2022-05-2417:55Sam RitchieI have separate github actions steps now for lint, test and coverage, so I was doing only the ones I needed to cache for each step, but that’s an excellent tip.
@U04V70XH6 I suspect I’ll have more questions… I’m doing a full switch from leiningen now to get ready for modularizing #sicmutils. next step is to get a clojars publish step going,#2022-05-2418:03seancorfield@U017QJZ9M7W Sure. Feel free to DM me if you have Qs (or ask them publicly).#2022-05-2418:18Sam Ritchie@U04V70XH6 actually here is one. is there a way to stack aliases? for example I am setting up cloverage now and realizing that I need all test deps and namespaces (of course). I’m asking because I THINK i remember seeing that this is not a thing…#2022-05-2418:18Sam Ritchietrying this clj -M:test:coverage --codecov#2022-05-2418:19Sam Ritchiethat seems to work; but semantically seems wrong since coverage won’t work without test , ie folks will definitely forget this#2022-05-2418:26seancorfieldCurrently, there's no way to have an "alias of aliases" but it is a common ask -- and there's at least one http://ask.clojure.org Q about that and/or similar things.#2022-05-2418:29seancorfieldWe have split our :test alias and our :runner alias so that :test is easier to combine with other aliases -- and we have :coverage as an alias, in fact 🙂 Also, since we are migrating to Polylith, we have a :dev alias which is "our production Polylith code" and :everything which is "our legacy production code" so clojure -M:dev:everything is kind of our baseline for any tooling to make any sense of our code...#2022-05-2418:30seancorfieldFWIW, here's how I start my development REPL as an example of combining aliases:
SOCKET_REPL_PORT=none clojure -J-Dlog4j2.configurationFile=log4j2-sean.properties -M:build:dev:test:everything:runner:add-libs:jedi-time:nrepl:portal:reflect:dev/repl
#2022-05-2418:32Sam Ritchieawesome, I found your project here too, this is a huge help https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#2022-05-2514:07seancorfield@U017QJZ9M7W just a caution: although that's up on GH and I use it as an example for what can be done with deps.edn, it's my personal config and subject to change without notice -- and using "RELEASE" for tools is considered poor practice (and that symbolic version is deprecated by Maven).#2022-05-2514:33Sam RitchieIt's a great cookbook and really helpful to see patterns. The linter has already scolded me about RELEASE and I am a version pinner anyway so hear you there too #2022-05-2514:45seancorfieldEverything inside my projects is always pinned. And then I run antq periodically to let me know about potential updates (it's not in an alias -- it's via a bash script that fancies up the report quite a bit).#2022-05-2516:29lread@U017QJZ9M7W everyone is different, but I find babashka tasks an excellent front end for a project’s dev tasks.
For example I recently whipped up a https://github.com/clj-commons/etaoin/blob/e1b7df647740536cec7b8528fe4c84ae21f94ab7/bb.edn#L14 to https://github.com/clj-commons/etaoin/blob/e1b7df647740536cec7b8528fe4c84ae21f94ab7/script/download_deps.clj#L1-L22.#2022-05-2516:35Sam RitchieThat's great, I am going to need some custom goodies to get these multiple modules published, but locally linked for development.#2022-05-2418:49Sam Ritchieis there a pattern for starting a repl a namespace other than user? This seems not to work, I stay in user:
clj -M:test:dev/repl -r -e "(require 'sicmutils.env) (in-ns 'sicmutils.env)"
#2022-05-2923:04Drew VerleeThis question confuses me a bit. I tend to start a nRepl server and client (connected to my editor) and i send expressions e.g (inc 1) from the client to the nRepl server which reads, evals, then prints it. Well it doesn't print it really, it makes message and sends it back to the client so it can print it.
what does the alias :dev/repl do?
nRepl server is used by a couple editor programs like cider w/emacs. But the idea is the same for all the editor's (vscode, intelliji, vim, what-ever, etc...)#2022-05-2923:08Drew VerleeWhat i'm saying is, it looks like your trying to eval things from the command line using -e, but you would be better served using your editor to eval expressions.#2022-05-2923:10Sam RitchieI want to provide an experience for new users that will allow them to write a command that drops them into a REPL with all batteries included, but configured so they understand what namespace they’re in , ie, where all those batteries live #2022-05-2923:17Drew VerleeIf your going to use the command line, your going to have to print things then right? Like you could print a message when they run that command that introduces things.#2022-05-2923:20Sam RitchieYeah that's a good idea too!#2022-05-2923:21Sam RitchieI had wanted to avoid simply pulling everything into the user namespace as it feels too magical, and it doesn't give someone a hint of “ah, I can go read this source file or cljdoc page to see everything configured”#2022-05-2923:52Drew Verlee@U017QJZ9M7W this is awesome btw https://github.com/sicmutils/sicmutils#2022-05-2923:54Sam RitchieThanks!! That's where I'm doing these tools.deps shenanigans https://github.com/sicmutils/sicmutils/pull/517#2022-05-2923:56Sam Ritchie@U0DJ4T5U1 the project is this huge foundation… the next phase is a bunch of visualizations and a pair of executable textbooks built on this code. I'm building up a new head of steam to get that work out, and I hope it will be more broadly compelling (think Mathematica-lite that can run on server or browser etc)#2022-05-2923:56Sam Ritchie@U0DJ4T5U1 I'm in new dad mode with a pair of twins so that comment gave me a much needed smile :) #2022-05-3000:05Drew Verleeno worries. The other day i was remarking that we (humanity/me) need a way to bridge math symbols and programming.#2022-05-2418:50Sam Ritchieah! putting -r at the END works.#2022-05-2418:53Sam Ritchieclj -M:test:dev/repl -e "(do (require 'sicmutils.env) (in-ns 'sicmutils.env) (str \"Clojure \" (clojure-version)))" -r
this gives an experience ALMOST like bare clj#2022-05-2419:13Sam Ritchiefor publishing to clojars; is a good pattern here to keep a pom.xml checked into the repo, and modify the deps with clj -Spom?#2022-05-2419:16Sam RitchieI like this, but this feels like this is going to be tough when I go to modularize the build (because each subproject will have its own pom file etc…) so maybe I will do this for now, pause and then figure out how to go modular.#2022-05-2420:17seancorfieldI keep a "template" pom.xml checked into git and generate the real one as part of the build process for deployment. See honeysql or next.jdbc for examples.#2022-05-2420:17seancorfieldhttps://github.com/seancorfield/honeysql/blob/develop/build.clj#L75#2022-05-2420:18seancorfield(I don't run clojure -Spom -- I do everything via tools.build because you have more control)#2022-06-0215:54Sam RitchieI think I’ve got this all set up now#2022-06-0215:55Sam Ritchiethe only improvement would be the ability to consolidate everything into build.clj, so no one has to run combos of -X , -M with stacks of aliases etc#2022-06-0215:55Sam Ritchie@U04V70XH6 it looks like your run-task is the best model for that, yeah?#2022-06-0215:56Sam Ritchiefor example: https://github.com/seancorfield/honeysql/blob/develop/build.clj#L24#2022-06-0215:57seancorfieldThat lets you run any -M style task in a subprocess.#2022-06-0215:57seancorfield(not every task needs to be run in a subprocess -- it does add some overhead -- but it provides isolation)#2022-06-0215:57Sam RitchieI’m a little surprised tools.build is missing this (interaction between -T tasks and clojure’s -M and -X runners, etc#2022-06-0215:58Sam Ritchieis there some built-in way to trigger, say, clojure -M:eastwood from clojure -T:build without run-task?#2022-06-0215:59Sam Ritchiethe only goal being to replace clojure -M:test:coverage and these various tasks that require stacked aliases with, say, clojure -T:build coverage so I can bake in the stacking#2022-06-0216:03seancorfieldWe run tests and cloverage via run-task as separate subprocesses so that we get isolation from the tools.build/tools.deps dependencies. Alex has said that it's sort of expected for folks to just run those directly via the CLI and not use build.clj for it, but I think it's "expected" by users that clojure -T:build ci should run the CI pipeline precisely so you don't have to run multiple CLI commands directly.#2022-06-0216:05seancorfieldFor a while -X let you execute multiple functions in a pipeline but that was removed (there were some unresolvable conflicts in how to parse functions, symbols, and command-line arguments) so "task pipelines" were supported directly in the CLI. I think task composition is a natural thing to want, and isolation is also a natural thing to want for some tasks...#2022-06-0216:05seancorfieldAt work, we have a way to run -X/`-T` tasks in a subprocess but it relies on some pretty awful hacks so I haven't released it.#2022-06-0216:08seancorfieldSee https://clojure.atlassian.net/browse/TBUILD-6#2022-06-0219:12Sam Ritchiethank you!#2022-06-0219:13Sam RitchieI can see now how I’ll do the multimodule setup. Probably hardcoded local dependencies in the git repo between modules, and then I’ll override these in my build script on a local install or remote publish#2022-06-0219:14Sam Ritchieit is nice to have simple pieces and the full language available. At least, I am deep enough into Clojure, build tooling etc that this feels great and non-confusing. There’s a certain level of confusion in the problem domain that’s hard to get past for someone just beginning, I’d imagine#2022-06-0219:23seancorfieldYeah, I think tools.build is the first time we've all been really exposed to the concepts behind the CLI in terms of computing a "basis" from one or more deps.edn files and using that to run other Clojure tasks... Folks who've built CLI tools using t.d.a. have dealt with this to some extent before, but now it's very much a public API for writing build systems.#2022-05-2812:53BenjaminHow do I include non clj files from an external lib. Specifically the cognitect aws lib depends on service.edn files. I made an intermediate library that depends on com.cognitect.aws/redshift-data#2022-05-2812:56Alex Miller (Clojure team)If you depend on a lib, anything in it will be on the classpath and loadable as a resource#2022-05-2812:56Alex Miller (Clojure team)But not sure if that's what you're asking#2022-05-2812:57BenjaminI see so it should work#2022-05-2813:00Benjamin(ns lambdaisland.classpath.watch-deps
"Watch deps.edn for changes"
(:require [clojure.java.classpath :as cp]
[clojure.string :as str]
[clojure.tools.deps.alpha :as deps]
[lambdaisland.classpath :as licp]
[nextjournal.beholder :as beholder]))
(def watcher (atom nil))
(defn- on-event [opts {:keys [type path]}]
(when (and (= :modify type)
;; On Mac the path will be absolute and include the watched dir,
;; e.g. `/Users/x/project/./deps.edn`
;; On other systems it seems to be relative, like `./deps.edn`
(str/ends-with? (str path) "./deps.edn"))
(println "✨ Reloading deps.edn ✨")
(let [new-paths (remove (set (map str (cp/system-classpath)))
(:classpath-roots (deps/create-basis opts)))]
(doseq [path new-paths]
(println "- " path))
(licp/install-priority-loader! new-paths))))
(defn start!
"Start a file system watcher to pick up changes in `deps.edn'
Options are passed on to tools.deps when creating the basis, you probably want
to at least put the `:aliases` you need in there.
(start! {:aliases [:dev :test]})
"
[opts]
(swap! watcher
(fn [w]
(when w
(println "Stopping existing `deps.edn' watcher")
(beholder/stop w))
(beholder/watch (partial on-event opts) "."))))
(defn stop!
"Stop a previously started watcher"
[opts]
(swap! watcher
(fn [w]
(when w
(beholder/stop w))
nil)))
(comment
(start! {:aliases [:dev]}))
there is something about how this tool works that made it so I ended up with the error
"could not find redshift-something.service.edn" on the classpath.
It worked when I started the program.
So I have libA depends on com.cognitect.aws/redshift-data
and programB depends on libA and ran into the error#2022-05-3111:54janezjI am building uberjar. It complains at runtime that clojure.java.jdbc.Connectable is missing, In the jar there is jdbc.clj and Connectable protocol in there, but no Connectable.class#2022-05-3116:55seancorfieldCan you share more details on how exactly you're building the JAR file?#2022-05-3116:57seancorfieldThe Connectable protocol is in the clojure.java.jdbc namespace so as long as your code is requiring that before referencing the protocol, it should work if clojure/java/jdbc.clj is in the JAR @UQ5QFFX54#2022-05-3117:21janezjAlmost unchanged from the docs
:build
{:extra-paths ["src/build"]
:deps {io.github.clojure/tools.build {:git/tag "v0.8.2" :git/sha "ba1a2bf"}}
:ns-default build}
and buld.clj
(ns build
(:require [clojure.java.shell :refer [sh]]
[clojure.string :as string]
[clojure.tools.build.api :as b]))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file "target/app.jar" )
(defn clean [_]
(b/delete {:path "target"}))
(defn uber [_]
(println "Uber")
(clean nil)
(println "Copy.")
(b/copy-dir {:src-dirs ["src/main" "resources"]
:target-dir class-dir})
(b/copy-file {:src "VERSION" :target (str class-dir "/VERSION")})
(println "Compile..")
(b/compile-clj {:basis basis
:src-dirs ["src/main"]
:class-dir class-dir})
(println "Uber..." uber-file)
(b/uber {:class-dir class-dir
:uber-file uber-file
:basis basis
:manifest {"_date" (java.util.Date.)
;:_git_count_revs (b/git-count-revs)
"_hostname" (:out (sh "hostname"))}
:main 'login.integration}))#2022-05-3117:27janezjlogin.integration.main
...
login.integration/-main integration.clj: 74
login.integration/-main integration.clj: 76
login.integration/-main/fn integration.clj: 76
login.integration/start integration.clj: 27
login.db/insert-event! db.clj: 106
login.db.Db2Writer/write-event! db.clj: 92
java.lang.ClassLoader.loadClass ClassLoader.java: 522
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass ClassLoaders.java: 178
jdk.internal.loader.BuiltinClassLoader.loadClass BuiltinClassLoader.java: 581
java.lang.ClassNotFoundException: clojure.java.jdbc.Connectable
java.lang.NoClassDefFoundError: clojure/java/jdbc/Connectable#2022-05-3117:31hiredmanyour db.clj file likely refers to the Connectable class without loading the associated clojure code that generates the class (the defprotocol)#2022-05-3117:31hiredmanare you using import anywhere?#2022-05-3117:33hiredman(specifically import on of Connectable)#2022-05-3117:34janezjno, just jdbc in the require in namespace.#2022-05-3117:35hiredmanif you are aot compiling your code, then you should end up with a Connectable class in the uberjar (since compilation is transitive)#2022-05-3117:35hiredmanif you don't that means something is loading next.jdbc before your code is compiled, like a user.clj#2022-05-3117:37janezj(defn start []
(mount/start #'config/config #'db/db2 #'db/db-writer)
(mount/start)
(banner/footer)
(jdbc/with-db-transaction [conn db/db2]
(jdbc/db-set-rollback-only! conn)
(jdbc/insert! conn "DEV.SIF_01" {:d464sfr "R247"})) ;; <<<< this works
(db/insert-event! {:bs 666666 ... ;; Exception#2022-05-3117:38hiredmanyeah, that can happen due to classloader stuff#2022-05-3117:39hiredmanmy guess is you have a user.clj that loads next.jdbc, so when compilre-clj goes to aot compile your project, next.jdbc gets loaded before the compilation actually happens, and then during compilation the require of next.jdbc is a noop (because it is ready loaded)#2022-05-3117:40hiredmanthis causes next.jdbc to not be aot compiled#2022-05-3117:40hiredmanso no class files for it in the uberjar#2022-05-3117:40janezjprotocols:
(defprotocol Mock)
(defprotocol DbWriter
(write-event! [this params] "Insert event into database"))
(defrecord MockDbWriter [^ConcurrentLinkedQueue queue]
Mock
DbWriter
(write-event! [this params]
(.add queue {:params params})))
(defrecord Db2Writer []
DbWriter
(write-event! [this params]
(jdbc/with-db-connection [conn db2]
(let [rows (jdbc/insert! conn "sevents" params)]
(log/spy rows)))))
(mount/defstate db-writer
:start (if (:mock-publish? config/config)
(->MockDbWriter (ConcurrentLinkedQueue.))
(->Db2Writer ))
:stop nil)
(defn insert-event! [data]
(let [params (->snake data)
_ (log/spy params)]
(write-event! db-writer params)))#2022-05-3117:41hiredmanso simple usages of next.jdbc work because they just use jdbc loaded from source in the jar#2022-05-3117:42janezjBut program works even when started as clj -T:run:dev
so no user.clj involved#2022-05-3117:42hiredmanbut complex usages, that end up with references to next.jdbc specific classes in your compiled bytecode, fail#2022-05-3117:43hiredmansomething is loading next.jdbc so it isn't getting aot compiled#2022-05-3117:44hiredmana simple way to verify it is because your program is aot compiled but next.jdbc is not aot compiled would be to remove the call to compile-clj from your build.clj and then launch your uber clojure using clojure.main#2022-05-3117:45hiredmanbuild the uberjar without aot compilation, then java -jar whatever.jar clojure.main will get you a clojure repl, then require your main namespace and call your main function#2022-05-3117:46janezj@U04V70XH6 said in the first comment that aot (.class file) in not mandatory?#2022-05-3117:47hiredmanthat is correct#2022-05-3117:47janezjhow to disable aot?#2022-05-3117:47hiredmandon't call compile-clj#2022-05-3117:48hiredmanyou may want to use 'clojure.main as the :main when building the uberjar#2022-05-3117:48hiredmandepending on if you are just trying it out, or are disabling aot compilation long term#2022-05-3117:50hiredmanrunning a clojure program that hasn't been aot compiled from an uberjar is a little more involed#2022-05-3117:52hiredmanhttps://github.com/hiredman/clojure-site/blob/patch-4/content/guides/running_a_clojure_program.adoc#2022-05-3117:52janezjit works without aot#2022-05-3117:52seancorfieldCould this be due to the top-level form that defines Mount state in terms of creating Db2Writer -- could that cause a mismatch on the class instance at runtime perhaps, due to an interaction with AOT?#2022-05-3117:53hiredmanthat wouldn't explain the missing next.jdbc class files#2022-05-3117:54hiredmanthe fact that those classfiles are not generated when aot compiling means for some reason next.jdbc is being loaded before the loading that would trigger aot compilation#2022-05-3117:55hiredmanthe most common reason for that would be a user.clj file#2022-05-3117:55hiredmanand a user.clj file that loads next.jdbc wouldn't break running the program when not aot compiling#2022-05-3117:56hiredmanso https://clojurians.slack.com/archives/C6QH853H8/p1654018921566669?thread_ts=1653998041.071529&cid=C6QH853H8 doesn't rule it out#2022-05-3117:56seancorfield@UQ5QFFX54 Do you have a user.clj file on the classpath?#2022-05-3117:58janezjyes#2022-05-3117:58hiredmanthen that is it#2022-05-3117:58seancorfieldWhat's in user.clj?#2022-05-3118:00janezjI just renames user.clj to xuser.clj#2022-05-3118:01janezjand it works clj -M#2022-05-3118:05hiredmanworking without aot doesn't matter#2022-05-3118:07janezjit works with aot. user.clj caused all the pain#2022-05-3118:10janezjThank you guys, I was completely desperate,.#2022-05-3118:26seancorfield@UQ5QFFX54 So I'm curious, what did you actually have in user.clj?#2022-05-3118:45hiredmanmy guess would be some utility functions to start and stop mount, which caused the project code to be loaded, which caused next.jdbc to be loaded#2022-05-3118:46hiredmanthe project code got aot compiled because calling compile on a namespace bypasses the check that require does to avoid reloading already loaded code, but only bypasses it for the direct argument of compile, anything that the namespace being compiled requires will get that check#2022-05-3118:54seancorfielduser.clj seems to trip a lot of people up -- I wonder if Rich et al would have not added it if they had the chance to do-over? I used it once, in a project ages ago, as a hacky workaround to force compilation of a namespace containing a deftype when a REPL was started, but there are much better ways of handling that sort of thing with the tools we have these days (and that project no longer does that). It seems that user.clj gets recommended in a lot of the examples around the "reloaded" workflow as well to define functions for the REPL (my thoughts on that workflow are well-known 🙂 ) so I can understand why folks end up in that situation...#2022-05-3120:07janezj(ns user
(:require
[cblus.server-components.config :as config]
[rabbit.stream :as rs]
[cblus.log-dev :as ld]
[login.db :as db] ;; <----------- jdbc
[login.login-handler :as login-handler]
[clojure.spec.alpha :as s]
[mount.core :as mount]
[taoensso.timbre :as log]))
#2022-05-3120:10janezjuser.clj -- just :require .. all other code in (comment ...)#2022-05-3120:17janezj@U04V70XH6 Do you have a link to a better workflow? I just remember that I read that you don't like mount.#2022-05-3120:19seancorfieldI use Component -- because it's simple, effective, and you can easily have multiple "systems" up and running at the same time in a single JVM which is great for dev/test. I don't find a need for "reloaded" because I eval every change as I make it, and I use REPL-friendly tricks like #' or function wrappers to ensure that function values are not caught up in expressions -- so the indirection ensures the code always picks up the current definition of functions.#2022-05-3120:21seancorfieldMy REPLs stay running for days -- often for weeks (occasionally for months) -- which is possible if you have good "REPL hygiene" without needing tools.namespace "refreshing" anything.#2022-05-3120:24seancorfieldMy main work REPL was restarted on the 27th which might not sound much given there was a weekend in there where I didn't use it. My HoneySQL REPL has been running since the 20th at this point. I tend to just leave REPLs running (started from the terminal) and connect to them as needed in various VS Code windows.#2022-05-3118:54seancorfielduser.clj seems to trip a lot of people up -- I wonder if Rich et al would have not added it if they had the chance to do-over? I used it once, in a project ages ago, as a hacky workaround to force compilation of a namespace containing a deftype when a REPL was started, but there are much better ways of handling that sort of thing with the tools we have these days (and that project no longer does that). It seems that user.clj gets recommended in a lot of the examples around the "reloaded" workflow as well to define functions for the REPL (my thoughts on that workflow are well-known 🙂 ) so I can understand why folks end up in that situation...#2022-05-3120:00pyrHi! We started using github actions for some of our jobs and I'm hitting a bit of a snag when needing to point to private repositories in dependencies with tools.deps: users need to point to SSH dependencies (say: <mailto:/cdn-cgi/l/email-protection|/cdn-cgi/l/email-protection>:exoscale/some-repo) while github actions has trouble with this (and would likely benefit from a HTTPS url + token).
Is there a canonical way to deal with this?#2022-05-3120:09pyrReplying to myself in case it's useful to anyone else, judging from https://github.com/marketplace/actions/tools-deps-builder#why-an-ssh-key the only way is to create a dedicated SSH key for now#2022-05-3120:34Alex Miller (Clojure team)you can use a personal access token too#2022-05-3120:39Alex Miller (Clojure team)or at least, I have used that for some things from github actions, I guess I haven't used it for this.#2022-05-3120:40pyrBut then you force every user to procure one too, r ight?#2022-05-3120:40Alex Miller (Clojure team)yeah#2022-05-3120:40Alex Miller (Clojure team)I guess to answer your original question - no, I do not know of a canonical way to do this :)#2022-05-3120:41pyrThanks Alex, I can live with that, we'll go for deploy keys I suppose#2022-05-3120:42Alex Miller (Clojure team)you can use the github cli from github actions but I suspect you're back into the same auth question#2022-05-3120:43Alex Miller (Clojure team)I guess that wouldn't help you with access via tools.deps#2022-05-3120:48pyrnope it wouldn't. org-wide deploy-keys on github would make this a non issue, in the meantime it means a bit of busywork, thanks for clarifying!#2022-06-0215:25pyrIs there a clj -X:deps prep like function that I can call to prep a lib when working against it directly (instead of going out to dependencies as clj -X:deps prep does)#2022-06-0215:56seancorfieldSince "prep" is "run this function with this alias" you can use clojure -X:some-alias some-fn directly (or clojure -T:... - I can't remember whether prep runs it as a tool or as an exec function).#2022-06-0216:06Alex Miller (Clojure team)it's like -T#2022-06-0216:07Alex Miller (Clojure team)so if you are in that project, clj -T:alias fn should be the equivalent of what prep is doing#2022-06-0216:07Alex Miller (Clojure team)maybe it would be useful to have some shortcut for "self prep"#2022-06-0216:10seancorfieldThanks, @U064X3EF3 saves me having to go digging in the code! 🙂#2022-06-0216:22Alex Miller (Clojure team)it's a little hard to tell even if you do look at the code, but that is the intent :)#2022-06-0306:52pyrI agree that you can use the targeted alias, as you suggest @U04V70XH6. But I was more interested in a generic way to call something akin to self-prep as @U064X3EF3 suggests. Point taken that it's not something that exists at the moment. Would there be interest in a patch that adds support for self-prep?
The context for us is that we have a (soon to be released) tool which adds conventions around building stuff. Our codebase is large and spread across several repositories which are either standalone or multi-module. Since the engineering team is large as well, we are always careful about ensuring there are conventions and as few snowflakes as possible (hence why we built a tool instead of distributing build.clj files around).
The specific need for self-prep is when building jars with the tool, it would be interesting to try launching self-prep. For now, I'll make the tool discover the :deps/prep-lib and mimick prep's behavior in that case.
Oh and by the way if the above didn't make it obvious, we're transitioning wholesale away from leiningen and on to deps over the course of the next two months 🙂#2022-06-0307:18seancorfieldAnd just to be clear clojure -X:deps prep doesn't work for the current project? (I'm a bit surprised about that)#2022-06-0307:22pyrit only runs the :deps/prep-lib task in dependencies, not in the current project (by design)#2022-06-0307:36pyrOK, scratch the suggestion to make a patch for it, I guess it's a bit too specific to the tool we built and it's easy enough to bake the behavior in it#2022-06-0223:05DiegoWhat the best way to run something like this in my codebase?
clojure -Sdeps '{:deps {cljfmt {:mvn/version "0.8.0"}}}' \
-m cljfmt.main fix
Thinking like I could make alias or something.#2022-06-0223:36seancorfield:aliases
{:format {:extra-deps {cljfmt {:mvn/version "0.8.0"}}
:main-opts ["-m" "cljfmt.main" "fix"]}
..}
Then clojure -M:format -- like that?#2022-06-0223:36Diegothanks!#2022-06-0308:03practicalli-johnI have these cljfmt aliases in practicalli/clojure-deps-edn
;; cljfmt-check - check/report formatting issues
:format/cljfmt-check
{:extra-deps {cljfmt/cljfmt {:mvn/version "0.8.0"}}
:main-opts ["-m" "cljfmt.main" "check"]}
;; cljfmt-check - check/report formatting issues
:format/cljfmt-fix
{:extra-deps {cljfmt/cljfmt {:mvn/version "0.8.0"}}
:main-opts ["-m" "cljfmt.main" "fix"]}
https://github.com/practicalli/clojure-deps-edn#2022-06-0314:09Sam RitchieQuestion about git dependencies. if I have a repository with multiple projects in subfolders, is it possible to specify one of the subfolders as a git dep?#2022-06-0314:10Sam Ritchiethis is obviously not a clojure project but I’m looking to organize #sicmutils more like https://github.com/twitter/algebird, with algebird-core , algebird-generic etc as various modules. I know they can depend on each other with :local/root and picking up the root as a git dependency will correctly pull in the submodule. but can a user use git deps to grab a submodule directly?#2022-06-0314:15Alex Miller (Clojure team)Yes, by specifying :deps/root you can pick out a sub folder of a repo#2022-06-0314:16Alex Miller (Clojure team)I'm assuming you don't mean actual git sub modules (those don't work with tools.deps)#2022-06-0314:20Sam RitchieCorrect just subfolders #2022-06-0314:52pyrI mentioned this this morning, but we just changed our tool's repository's visibility: https://github.com/exoscale/tools.project. This tools tries to bring a certain number of convention to handle both standalone projects, and multi-module projects. This solves most of the gripes we had dealing with multi module projects in the leiningen world. The project itself isn't so big, but it serves as an aggregator for the many common things we rely on (antq, clj-kondo, cljfmt, slipset-deploy, deps-modules, and more)#2022-06-0315:05Sam RitchieAwesome I will look closely. So this provides conventions for local multi module development, and proper references between modules when you do a release?#2022-06-0315:06pyrAmongst other things yes#2022-06-0315:11mpenet@U017QJZ9M7W most things are built on a task runner that you can tell what to run tasks on (all modules or a subset) and when a command doesn't need that (when we detect you run that command from inside a module dir already) it will default to the corresponding standalone command#2022-06-0315:12mpenetex clj -T:project install from the root will run install for all modules in a single process, from a module dir it will just install that module#2022-06-0315:15pyrThe task part is documented here: https://github.com/exoscale/tools.project/blob/main/doc/target/task.md#2022-06-0315:28borkdudeI may have found a reasonable solution for shell quoting issues. If anyone ever runs into that again (e.g. on Windows) it'd be interesting if that solves the issue for the user.
https://github.com/babashka/cli#usage-with-the-clojure-cli#2022-06-0315:39Darin Douglassi just noticed that metadata annotation stuff, pretty neat! thoughts about making it :babashka.cli/cli so we can auto-namespace the key from the main babahska.cli namespace?#2022-06-0315:40borkdudeThe metadata is designed to be used without the dependency, so that's not really a concern. I guess you could use :as-alias for this as a solution#2022-06-0315:40borkdudeAlso the key can be used in deps.edn now and it will match the name of the library coordinate which is kind of neat ;)#2022-06-0315:41Darin Douglassah i skimmed over that part, makes sense 👍#2022-06-0315:41borkdudeI might have just added it :)#2022-06-0315:41Darin Douglass😄#2022-06-0315:47borkdudeYeah, I could move the namespace to org/babashka/cli.clj perhaps to fix this. Not sure if that's worth it... Also not consistent with the rest of the babashka libs#2022-06-0315:48Darin Douglasseh, not really worth it imo babashka is a pretty unique initial namespace name#2022-06-0315:50borkdudeSo I could consider :babashka.cli/opts
(require '[babashka.cli :as cli])
::cli/opts {...}
vs
:org.babashka/cli {...}
Hmm, not much difference I'd say#2022-06-0315:51Darin Douglassi was mainly thinking of the use-case where babashka.cli was already required and aliased in the code#2022-06-0315:51Darin Douglassthe only-exec-dep paradigm makes :org.babashka/cli make a lot more sense#2022-06-0316:36seancorfield@U04V15CAJ I haven't looked closely at it yet, but (how) does it handle argument values that can be symbols or keywords?#2022-06-0316:37borkdudeDo you mean {:option 'foo} or {:option :foo} ?#2022-06-0316:38seancorfieldYes. With standard -X, you just say foo or :foo -- because strings are the "odd one out"#2022-06-0316:38borkdudeThere is no automatic coercion until you tell it to, either via metadata or via data in your deps.edn.
:org.babashka/cli {:coerce {:option :keyword}}
And you call it with --option foo which will then be turned into {:option :foo}#2022-06-0316:39seancorfieldI'm thinking of commands like clojure -Ttools install which takes a qualified symbol as a key, whose value is a hashmap (with one or more keyword keys and string values), and then :as as a key, whose value is a symbol.#2022-06-0316:40seancorfieldAh, so you'd have to explicitly specify conversion to keyword, symbol, or even via EDN to a hash map?#2022-06-0316:40borkdudeYes#2022-06-0316:41seancorfieldAnd you would need a specific structure to mimic/wrap the tools install (since it can have any arbitrary symbol as a key). You'd need to specify actual options for the libname and the coordinates. Just trying to get a good mental model of what it really is/does.#2022-06-0316:41borkdudeI haven't added the :edn option yet, but this can be added#2022-06-0316:42borkdude> You'd need to specify actual options for the libname and the coordinates.
Yes, that#2022-06-0316:43borkdudeIt's not intended to focus on tools install itself, but to call exec functions like you would do with -X or -Tantq#2022-06-0316:43borkdudeHere is an example for antq:
https://github.com/babashka/cli#antq
and here is one for clj-new:
https://github.com/babashka/cli#clj-new#2022-06-0316:47borkdudeSo you get:
clj -M:antq --upgrade --skip github-action
vs.
clj -Tantq outdated :upgrade true :skip '["github-action"]'
#2022-06-0316:49seancorfieldUnderstood. I was just thinking about different -X/`-T` argument formats that I've seen.#2022-06-0316:49borkdudeYeah, I think what tools install itself does is kind of an exception to the rule#2022-06-0316:50seancorfield(and I had looked at the clj-new example -- on my phone this morning, before I even got out of bed 🙂 )#2022-06-0316:52seancorfieldAnd how does it read the :org.babashka/cli data from the alias in the deps.edn file?#2022-06-0316:54borkdudejust a moment, dinner, back in a bit :)#2022-06-0316:54seancorfieldJust looked that up. Sneaky 🙂 Relies on reading the clojure.basis and getting the metadata from :resolve-args. I didn't realize the whole alias data ended up in there.#2022-06-0317:56borkdudeYeah, I was happy I didn't have to do anything else. I was contemplating some project level config file but this seems the most natural place#2022-06-0317:57borkdudeBtw, you can use any function in :coerce and :collect metadata, the keywords are just for convenience (and .edn)#2022-06-0410:23borkdude@U04V70XH6 From the next version (0.2.10) on, you can use clj-new like this in lein:
{:clj-1.11 {:dependencies [[org.clojure/clojure "1.11.1"]]}
:clj-new {:dependencies [[org.babashka/cli "0.2.9"]
[com.github.seancorfield/clj-new "1.2.381"]]}
:user {:aliases {"clj-new" ["with-profiles" "+clj-1.11,+clj-new"
"run" "-m" "babashka.cli.exec"
{:exec-args {:env {:description "My project"}}
:coerce {:verbose :long}
:collect {:args []}
:aliases {:f :force}}
"clj-new"]}}}
If clj-new would add that metadata, the :coerce, :collect and :aliases could be left out.#2022-06-0410:23borkdude$ lein clj-new app --name foobar/baz --verbose 3 -f
Generating a project called baz based on the 'app' template.
#2022-06-0410:24borkdudeI have some trouble deploying to clojars now (see #clojars) but this should be up once that is fixed.#2022-06-0413:15borkdudeDeployed#2022-06-0418:43seancorfieldI'm afk but if you want to create an issue on the clj-new repo explaining the change that would make it easier for your lib...#2022-06-0418:43borkdudeWill do, have a nice weekend!#2022-06-0519:03borkdude@U04V70XH6 https://github.com/borkdude/clj-new#babashka-cli#2022-06-0519:03borkdudeIf that looks good, I'll create the PR#2022-06-0519:06borkdudeThe other only change is this extra metadata on the namespace:
https://github.com/borkdude/clj-new/commit/af63fe0c48dbde993fc179717090c0119ced4295#2022-06-0519:27seancorfieldSure, go ahead and send a PR. Do you want to do a similar one for deps-new? I'd much rather encourage folks to use that instead of the older one with all the Leiningen/Boot baggage.#2022-06-0519:40borkdudeSure!#2022-06-0520:17borkdudeOK, both done now#2022-06-0907:31borkdudeThanks for merging! Note that I mentioned a clj-new / deps-new version here:
https://github.com/seancorfield/clj-new#babashka-cli
and here:
https://github.com/seancorfield/deps-new#babashka-cli
but those have to be updated when the babashka CLI stuff is going to start working.#2022-06-0907:49borkdudeI've updated the docs here to use the newest SHAs:
https://github.com/babashka/cli#clj-new
https://github.com/babashka/cli#deps-new
We could also just link there#2022-06-0917:21seancorfieldAh, I didn't catch the version number had changed. I'll need to do new releases (and I'll update those references). I'll try to get to that tomorrow or over the weekend.#2022-06-1005:11seancorfieldNew releases of deps-new and clj-new are available with support for babashka.cli! (v0.4.10 was broken, sorry, so please use v0.4.12)#2022-06-0316:58pyrSo quick one while I'm still looking up a few things wrt tools.project. I ended up adding a :deps/prep-lib key for tools.project itself, which does aot compilation of all namespaces at direct-linking true to speed it up. with this It's quite fast. This does make the tool usage a bit cumbersome though since you have to issue clj -X:deps prep :aliases '[:project]' before launching any command.#2022-06-0316:59pyrDo I have another simple resort to make the tool available in AOT compiled fashion? I suppose I can't publish to a maven repo since there are dependencies to artifacts published as git only?#2022-06-0317:00pyrMaybe building an uberjar and publishing that, which could be OK since this is a tool, but this feels quirky#2022-06-0317:32seancorfieldBy "publishing" you mean putting it somewhere for folks to download, rather than using Clojars, right? Have you considered Graal and making native binaries?#2022-06-0317:34pyrI did but some of what tools.project does doesn't work in graal. As for publishing I meant clojars#2022-06-0317:35pyrI think for now I'll drop the :deps/prep-lib#2022-06-0318:10Alex Miller (Clojure team)I would not (ever) publish an uberjar#2022-06-0318:16borkdudeYou can however distribute an uberjar and let people call that - but this probably not how you intend this to be used. It would be similar to distributing a graal binary though. There are also tools to make jars be wrapped in an executable wrapper#2022-06-0318:19borkdude@U06V097TP What didn't work for you in graal? I have a native version of tools deps here (mostly experimental):
https://github.com/babashka/tools-deps-native#2022-06-0318:22borkdudeWhat I often do with projects that need some Java compilation is publish the Java compiled stuff separately as there's usually not a lot of churn there and then do the rest as git deps / mvn deps more often#2022-06-0319:18pyrThe thing is, I want stuff to work out of the box for git clone + clj -T. We'll likely distribute a graal'd tools.project too but it should work without first having to go fetch stuff left and right which is what I'm focusing on atm#2022-06-0319:31borkdude@U06V097TP kk - let me know if you want some help with the graalvm stuff - it's tricky with tools.build#2022-06-0318:11Alex Miller (Clojure team)This is actually something I've thought about as I have the same problem with tools.build#2022-06-0318:13Alex Miller (Clojure team)With tools.build you could easily make an alias at the root to do that for all subprojects - would that at least reduce the need a bit?#2022-06-0319:21pyrWas this directed at me? If so I'm not sure I get the point you're making#2022-06-0318:16seancorfieldI tend to run tools.build/`build.clj` via a REPL to avoid startup costs: clj -M:build -i build.clj -r and then (build/some-task {:the "args"}) interactively as needed.#2022-06-0318:17seancorfieldIt makes it easier to chain tasks as well: (-> {:projects '[api auth]} (build/run-tests) (build/uberjars))#2022-06-1005:11seancorfieldNew releases of deps-new and clj-new are available with support for babashka.cli! (v0.4.10 was broken, sorry, so please use v0.4.12)#2022-06-0418:35Sam Ritchieno more project.clj! The end of an era… https://github.com/sicmutils/sicmutils#2022-06-0522:34agI need to run some code in build.clj via tools.build and -T:build that requires some dependencies of the project, but
> -T:an-alias remove all project deps and paths
Is there a way to forcefully include project dependencies without repeating them in the :build alias?#2022-06-0522:50seancorfieldCreate a basis in that code, using the aliases that have those dependencies.#2022-06-0523:06agI reconsidered, I think I'm just gonna use -X. Thank you, Sean.#2022-06-0607:39pavlosmelissinosIs there a way to get the dependency tree of a project as data? I'm trying to figure out the paths that lead to a specific transitive dependency and clojure -Stree | grep <dependency name> -B 8 unfortunately includes a lot of garbage from other branches, which I'm not interested in.
It's quite possible that I'm focusing on the wrong thing, so happy to consider a different approach as well!
edit: The real "why" is that there's a known vulnerability for one of my transitive dependencies, so I'm trying to figure out which of my direct dependencies will be affected if I upgrade it (override it with a newer version)! 🙂#2022-06-0613:14Alex Miller (Clojure team)You can get data from clj -X:deps tree :format :edn#2022-06-0615:00pavlosmelissinosOh sweet, thanks! Is this documented? On my phone right now but I swear I looked in the https://clojure.org/reference/deps_and_cli first and couldn't find it. 😅#2022-06-0615:34Alex Miller (Clojure team)-X:deps tree is documented there and the docs for the tree function are at https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/tree#2022-06-0619:34dpsuttonwhat’s the way to invoke -X with an environment variable? clj -X :filename $OUT_FILE style?#2022-06-0619:38hiredmanWith any additional escaping, those args pass through read-string#2022-06-0619:41dpsuttonyeah. but i need to enable expansion in the shell and then ensure it’s readable to clojure and i’ve forgotten the exact sequence of double and single quotes necessary to thread that needle#2022-06-0621:10borkdude@dpsutton
$ bb -e "*command-line-args*" $OUTFILE
("foo")
$ bb -e "*command-line-args*" \"$OUTFILE\"
("\"foo\"")
$ bb -e "(edn/read-string (first *command-line-args*))" \"$OUTFILE\"
"foo"#2022-06-0621:50seancorfieldclojure -X :filename '"'$OUTFILE'"'#2022-06-0621:50seancorfield(I use that in several invocations -- also with '"'$(...)'"')#2022-06-0806:14Ronnie Nissanhey. I am having a problem installing deps-new on Fedora Linux, using both the fish and bash shells.#2022-06-0806:14Ronnie Nissanwhen I run
clojure -Ttools install io.github.seancorfield/deps-new '{:git/tag "v0.4.9"}' :as new
I get
Unreadable arg: "{:git/tag"#2022-06-0806:17hiredmanAre you sure you are using the quotes like you have there verbatim?#2022-06-0806:18hiredmanI would expect an error like that if the single quotes were missing for example#2022-06-0806:20Ronnie NissanI am using the exact same command as the one on the repo's readme#2022-06-0806:28hiredmanAre you running it as part of a shell script or typing it directly in the terminal?#2022-06-0806:30hiredmanThe command from the readme doesn't fail like that using bash for me#2022-06-0806:30Ronnie Nissandirectly in the terminal#2022-06-0806:31hiredman(completely different error about my git trying to launch an x11 program without an x11 display before cloning from GitHub)#2022-06-0806:31hiredmanA regular terminal emulator?#2022-06-0806:34hiredmanAre you sure that clojure is the tools.deps related script? What does clojure -Sdescribe say?#2022-06-0806:37Ronnie Nissan#2022-06-0806:40Ronnie Nissanas for the terminal, I am using the gnome terminal emulator. and I do not think that should matter#2022-06-0806:40Ronnie Nissanmy clojure and java install are all inside a podman container (using distrobox)#2022-06-0806:40hiredmanAh#2022-06-0806:41hiredmanYou may need some extra quotation to pass the args through the extra layer there#2022-06-0806:42hiredman'\' instead of just '#2022-06-0806:43hiredmanOr not, distrobox would just run the shell#2022-06-0806:44hiredmanWhat kind of error do you get if you replace :git/tag with :foo/bar in the command?#2022-06-0806:44Ronnie NissanUnreadable arg: "{:foo/bar"#2022-06-0806:45Ronnie Nissanthe escaping worked, kinda#2022-06-0806:45Ronnie NissanCloning:
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.git/coord-err (git.clj:44).
Library io.github.seancorfield/deps-new has invalid tag: v0.4.9
#2022-06-0806:45hiredmanAre you using the distrobox export feature?#2022-06-0806:45Ronnie NissanI am#2022-06-0806:46hiredmanThat is what is making it require extra quoting#2022-06-0806:46Ronnie Nissanbut running the command inside the container has the same resoult#2022-06-0806:47hiredmanHow are you running it inside the container? Launching a shell in the container and typing it in?#2022-06-0806:47Ronnie Nissanyes#2022-06-0806:48Ronnie Nissandistrobox-enter clojure where clojure is the name of the container#2022-06-0806:51Ronnie Nissanso from inside the container, with the escape, it returns this, Key is missing value: new#2022-06-0806:51Ronnie Nissanmaybe I should nuke the container and start fresh#2022-06-0806:53hiredmanTry removing the export#2022-06-0807:07Ronnie Nissannow it works, weird#2022-06-0807:07Ronnie Nissanmaybe the full path of the clojure bin should be specified when running it#2022-06-0921:55Drew VerleeHas anyone written an experience report covering a switch from boot to deps and tools build? I know it will be quite specific but it still might help.
I read sean corfield's blog post where he mentions the change. It was useful, I'm just looking for more.#2022-06-0922:30seancorfieldHappy to discuss that change in more detail. I don't think many Boot users existed, let alone have made the switch (compared to the larger Clojure community).#2022-06-0922:33seancorfieldI think an important first step for anyone contemplating such a migration is to externalize their dependencies. Long before t.d.a was a thing, we'd decided to put our dependencies into EDN files, which our Boot tasks read in. We found that easier than digging around inside build.boot files, and it also allowed for us to have a single build.boot file and pick up different dependencies from different subprojects, based on the arguments passed in. @U0DJ4T5U1#2022-06-0922:38Drew VerleeThanks sean. The first step i have is to pull the dependencies from boot into a deps.edn file, so it's good to hear that's where other people went as well.#2022-06-1022:03Drew Verlee@U04V70XH6 Like i said yesterday, my first step is to move the dependencies from boot into a deps.edn file. I went ahead and did that. I take it you, like i just did, wrote some clojure functions to convert between the two formats (deps->boot & boot->edn)? Are those lying around anywhere?
Those or any other helpful pieces of functionality you used along the way to convert would be much appreciated.#2022-06-1022:04Drew VerleeThough i'm guessing from there it will be better to just convert to boot file to deps manually... to much past that point will be different and it's not that big 🙂.#2022-06-1022:05seancorfieldThe only code you need in your build.boot file is to read the new deps.edn, parse it as EDN, lift the dependencies out and tell Boot about them.#2022-06-1022:05seancorfieldI can't remember how you do that with Boot (it's been 3 1/2 years) but ISTR it's pretty straightforward to just add deps programmatically?#2022-06-1022:06seancorfieldIt doesn't need to round-trip. You create the deps.edn files once from whatever you had in Boot (this was a one-off, manual process for us), and then your Boot script uses those while you gradually rewrite the rest to deps.edn aliases and build.clj etc.#2022-06-1022:07seancorfieldI guess we might still have this somewhere in our repo's history but there have been thousands of commits since then 🙂#2022-06-1022:09Drew VerleeYeah. I just did the one off boot to deps transfer using clojure. I'm glad to hear there weren't any hiccups when you did it.#2022-06-1022:51seancorfieldIt'll depend on how complex your build script is and whether you do fancy stuff with dependencies and paths etc... Ours was fairly complex, and I think it took me a couple of days, over the year-end holiday, to get it converted and all working properly for my team mates to come back to in early January 🙂#2022-06-1022:51seancorfieldI did our lein to boot migration the same way -- when everyone else was out of the office for a few days.#2022-06-1003:08tianshuCan I depend on a Java project through tools.deps via git support? I think the git repository can be cloned, but only maven pom.xml is resolved. So I got the dependencies of that project, but the project itself is not accessible.#2022-06-1003:17hiredmanNo#2022-06-1003:21tianshuAnd is it possible for a library use deps.edn , but depends on some java file? When it is a dependency in another project, there's no way to get the java file compiled?#2022-06-1003:31hiredmanNot really, there are things you can do using prep tasks, but really the java ecosystem is not made to be consumed as source#2022-06-1003:36tianshuThanks#2022-06-1006:50Christian JohansenHow do I provide private Maven repo username and password on a build server? I have :mvn/repos in my deps.edn, and locally I have a ~/.m2/settings.xml file with settings.servers.server.{id, username, password}. I guess I could do echo ,,, > ~/.m2/settings.xml on the build server, but are there other options? Environment variables or similar?#2022-06-1009:58Alex Miller (Clojure team)You need a settings.xml#2022-06-1011:12Christian JohansenOk, thanks!#2022-06-1107:51pyrI couldn't find whether this was discussed before, but it feels a bit odd that aliases cannot hint at their preferred call structure and that is a peculiarity the caller has to know about (i.e: -M vs. -X). I suppose this stems from the fact that there's only so much the large bash wrapper can figure out, right? Or let me rephrase: if there was a way for tools to hint at their preferred calling convention, would a clojure cli flag honoring that be considered?#2022-06-1113:27Alex Miller (Clojure team)Not sure I understand what the request is at the end#2022-06-1114:19borkdude@U06V097TP The usual solution to this is to use a task manager thing like Makefile, or https://book.babashka.org/#tasks`bb.edn`#2022-06-1120:43seancorfield@U06V097TP An alias can contain both :main-opts and :exec-fn so its behavior depends on whether you use -X or -M. Generally tho', looking at the alias in deps.edn tells you which usage it expects: :main-opts implies -M, :exec-fn or :exec-args or :ns-default implies -X. Given my first point, usage could not be inferred however.#2022-06-1120:44seancorfieldIf an alias contains :extra-deps and :exec-fn, you can still use it with -M and it will bring in those deps but then expect you to provide main opts somewhere else, either via other aliases in combination or on the command-line.#2022-06-1120:45seancorfieldSimilarly, if an alias contains :extra-deps and :main-opts, you can still use it with -X and it will bring in those deps but then expect you to provide exec opts somewhere else, either via other aliases in combination or on the command-line.#2022-06-1120:48seancorfieldFor a long time, my dot-clojure deps.edn file had aliases with both :exec-fn and :main-opts so you could invoke certain tools with either -X or -M (and so you got to choose how to provide command-line arguments). And that's another reason why the tool can't decide for you: clojure -[X or M]:some-alias :arg '{:value 42}' -- those arguments will be read as EDN and put into a hash map if you use -X but they'll be treated as strings if you use -M.#2022-06-1116:22Sam Ritchiequestion - when I use create-basis in tools.build to create a basis and pass it to pom, it looks like that command adds org.clojure/clojure to the pom. i THINK I don’t want to do this, but instead let the user specify their Clojure version — so
1. is that correct? and
2. what is the best way to prevent tools.build from adding this dep?#2022-06-1116:46Alex Miller (Clojure team)Clojure is defined as a dep in the root deps.edn. If you want to not include the root, you can pass :root nil to create-basis#2022-06-1116:47Alex Miller (Clojure team)Whether you want to do that is hard for me to answer :)#2022-06-1116:47Alex Miller (Clojure team)You do need something to declare a dep on Clojure or you can't run Clojure programs #2022-06-1117:04Sam Ritchie@U064X3EF3 I had been browsing and seen many library that didn’t declare the dep at all, which made me think folks have settled on providing clj and cljs scoped as "provided"#2022-06-1117:04Sam Ritchiewhich is eminently reasonable (that is how it should be done ALWAYS for cljs I’d say). but an explicit Clojure dep makes sense too obviously, at the lowest version you can support#2022-06-1117:20Alex Miller (Clojure team)I think not specifying is a reasonable choice for libraries#2022-06-1117:21Alex Miller (Clojure team)but I understand the many reasons libs do specify one#2022-06-1117:21Alex Miller (Clojure team)the important thing is that ultimately some app is going to make the choice anyways#2022-06-1116:23Sam Ritchieetaoin for example https://clojars.org/etaoin does not declare org.clojure/clojure as a dependency, but I don’t see an obvious spot where this is excluded: https://github.com/clj-commons/etaoin/blob/master/build.clj#2022-06-1116:29lreadWarning @sritchie09 that build.clj is pretty new and has not been used for a real release yet!
As far as I understand it, I think the best practice is for a library to explicitly depend on the minimum Clojure version required for the lib.
https://clojars.org/rewrite-clj/versions/1.1.45.#2022-06-1116:29lread(If that’s what you are wondering about?)#2022-06-1116:36lreadAn app writer will typically specify clojure as a dep overriding any library clojure deps.#2022-06-1117:03Sam Ritchie@lee okay that makes sense! and in that case that is easy to manage#2022-06-1117:06lreadI find it especially nice in deps.edn because dep resolution seems simple, clean and intuitive to me when compared to lein deps resolution (which maybe I really just haven’t taken the time to understand).#2022-06-1317:10seancorfield@alexmiller I noticed t.d.a got updated with a fix for the prep lib directory issue -- and it looks like the CLI got updated too, to 1.11.1.1129?#2022-06-1318:36fogus (Clojure Team)I don't know how current this resource is, but https://formulae.brew.sh/formula/clojure#default shows 1.11.1.1124#2022-06-1318:36fogus (Clojure Team)and appears to match https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/clojure.rb#2022-06-1318:40Alex Miller (Clojure team)It's a prerelease, waiting for feedback from the reporter#2022-06-1318:41Alex Miller (Clojure team)Stable is still 1124#2022-06-1318:49seancorfieldCool. I was ready to vendor in 1124 but I've gone ahead and vendored in 1129 so we might catch any regression issues with it.#2022-06-1318:56Drew VerleeIs https://github.com/boot-clj/boot/blob/e6ea562af765ee2b50703ab33a00cf615d0bef43/boot/core/src/boot/task/built_in.clj#L368 with :server option enabled roughly equivalent to https://clojure.github.io/clojure/clojure.core-api.html#clojure.core.server/start-server ?#2022-06-1319:04Drew Verleethe answer appears to be no. For one, boot is launching an Nrepl server.#2022-06-1319:53seancorfieldhttps://clojuredocs.org/clojure.core.server/start-server starts a Socket REPL server.#2022-06-1321:33Drew VerleeRunning the command line tool e.g clj -X:blah results in an exception that compains about missing clojure spec alpha on the classpath. Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/spec/alpha__init.class or clojure/spec/alpha.clj on classpath., compiling:(clojure/run/exec.clj:1:1)
however, that dependency isn't in the clj deps. e.g if i run clj -Stree i don't see it.#2022-06-1321:37hiredmanclojure.run.exec is an internal thing the cli tools use to implement -X#2022-06-1321:38hiredmanI would not be surprised if it failed like that when used with a version of clojure that predates spec being a dep of clojure#2022-06-1321:45Drew Verleeyep, that makes sense, checking...#2022-06-1322:29Alex Miller (Clojure team)Yeah, -X requires at least 1.9#2022-06-1620:15frankitoxAnyone hit an error while trying to use -M flag in Windows? Is it a known issue?#2022-06-1620:23frankitoxI just realize that if I do clj -Mother it fails. But doing clj -M:other works as expected#2022-06-1620:26borkdudeThe latter is the expected syntax#2022-06-1620:30frankitoxOh, so I'm guessing I always should prepend : in Windows. I always skip the first : in Linux#2022-06-1620:31Alex Miller (Clojure team)don't - that's not supported#2022-06-1620:32practicalli-johnall aliases names should start with a : , if they don't then its not the name of an alias - there are many examples here https://github.com/practicalli/clojure-deps-edn/blob/live/deps.edn
Only -T tool names are free from : (although I use aliases for tools to give more configuration options)#2022-06-1620:37frankitoxOh I see, if I use clj -Mother still picks up the :main-opts option, that's what confused me. I'm always prepending : form now on, thanks guys!#2022-06-1620:48practicalli-johnSome versions of the Clojure CLI did still work if it was missed of, but it's definitely a bug
Some projects also included the incorrect command in their docs, so it's an easy trap to fall into... But it's definitely a trap :)#2022-06-1620:34practicalli-johnIs dont suppose there is a way to use an environment variable as part of the path when setting the path value for :mvn/local-repo in a deps.edn file?
At least not without pre-processing the deps.edn file and re-writing with the value of the environment variable
For example :mvn/local-repo "$XDG_CACHE_HOME/maven/repository
I am assuming it needs to be a full path, as the above creates a directory called $XDG_CACHE_HOME in the current directory (of course 🙂 )#2022-06-1620:43Alex Miller (Clojure team)that's not currently supported in a deps.edn file#2022-06-1620:43Alex Miller (Clojure team)you could probably make it work passing on the command line with -Sdeps (but quoting is fun)#2022-06-1620:52borkdude@jr0cket That's where I usually use -Sdeps if it's something dynamic#2022-06-1621:20practicalli-johnOkay, using -Sdeps makes sense as a command line configuration approach.
For the particular :mvn/local-repo , ideally it would live in the user level deps.edn, but using a full path is only a slight inconvenience when sharing that setup as part of practicalli/clojure-deps-edn - I’ve just shared it for now with that configuration key commented (so I just have to remember not to over-ride it when committing new changes 😄 ). Thank you.#2022-06-1621:22borkdudeMaybe it also accepts a relative path? Assuming that in your user deps.edn, the relative path to a user .m2 dir can be relatively(!) well predicted?
EDIT: well, that's not going to work, since it would be relative to the working directory#2022-06-1621:25practicalli-johnYes, :mvn/local-rep will use a relative path, but it will be relative to where the clojure executable command is run. You can have some great fun setting :mvn/local-repl ~/.cache/maven/repository as it will create a directory called ~ in the directory where clojure was run, which under no circumstances should you delete using rm -rf ~ unless you have everything backed up in your home directory and you want a clean start :rolling_on_the_floor_laughing:#2022-06-1621:28borkdudeomg...#2022-06-1622:28seancorfield@alexmiller I just installed 1.11.1.1139 via the Linux installer and tried the install-latest function to update installed tools: it loses the :deps/root key when it upgrades:
diff --git a/tools/poly.edn b/tools/poly.edn
index 46da562..108a51a 100644
--- a/tools/poly.edn
+++ b/tools/poly.edn
@@ -1,5 +1 @@
-{:lib io.github.polyfy/polylith,
- :coord
- {:git/sha "f0d38cb71e9d3f7a620a4baecb491da4aab91edd",
- :deps/root "projects/poly",
- :git/url ""}}
+{:lib io.github.polyfy/polylith, :coord {:git/tag "v0.2.14-alpha", :git/sha "e874e80"}}#2022-06-1622:56seancorfield(never mind that it "updates" the latest master SHA to an earlier release tag and sha)#2022-06-1622:58Alex Miller (Clojure team)In both cases, there's no way for it to know those things#2022-06-1622:58seancorfieldIf the existing, installed EDN file includes :deps/root, it could at least preserve it...#2022-06-1622:58Alex Miller (Clojure team)That seems just as likely to be wrong#2022-06-1622:59Alex Miller (Clojure team)Installing from latest version will always use a tag, so the master sha thing is not a version that's considered#2022-06-1623:00Alex Miller (Clojure team)So this process won't work in all possible cases#2022-06-1623:02seancorfieldSo tools that require :deps/root cannot be safely installed or updated via install-latest? Perhaps that caveat could be added to the docstring and/or docs somewhere.#2022-06-1623:02seancorfieldI'll have to make sure the Polylith docs include a warning not to use install-latest. Issue created in the Polylith repo.#2022-06-1700:25Alex Miller (Clojure team)yeah, I wasn't aware there were any so didn't occur to me#2022-06-1715:36borkdudeI'm trying to expose something as a clj tool, but I don't see why this is failing:
$ clojure -Ttools install io.github.borkdude/jet '{:git/sha "18bfbda4237f295877addafe87e5a5eb8e1b0698"}' :as jet
Execution error (NullPointerException) at clojure.tools.deps.alpha.util.dir/canonicalize (dir.clj:30).
null#2022-06-1715:38Alex Miller (Clojure team)I got
% clojure -Ttools install io.github.borkdude/jet '{:git/sha "18bfbda4237f295877addafe87e5a5eb8e1b0698"}' :as jet
Cloning:
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.git/eval1445$fn (git.clj:113).
Commit not found for io.github.borkdude/jet in repo at 18bfbda4237f295877addafe87e5a5eb8e1b0698#2022-06-1715:39Alex Miller (Clojure team)not sure what version of clojure you're using, it has changed several times in this area lately#2022-06-1715:39Alex Miller (Clojure team)but indeed, I don't see that as a valid commit sha#2022-06-1715:41borkdudeForgot to push facepalm Now it works#2022-06-1715:41Alex Miller (Clojure team)% clojure -Ttools install-latest :lib io.github.borkdude/jet :as jet
Checking out: at 73e0be021a290a95bf1d671102be1b4b25c2b5a4
Installed tool jet as: io.github.borkdude/jet {:git/tag "v0.2.18", :git/sha "73e0be0"}#2022-06-1715:41Alex Miller (Clojure team)using the new install-latest ...#2022-06-1715:42borkdudeCool. But that one didn't have the exec function yet, just added it :)
Is there also a way to invoke a tool with a default function, so you don't have to write:
clj -Tjet exec ...
#2022-06-1717:57practicalli-johnThis is why I use an alias rather than the tool name. With an alias you can specify the function to call#2022-06-1715:42Alex Miller (Clojure team)no, we're kind of on the fence about making that optional, so it's required right now#2022-06-1715:44borkdudeCan you also install a mvn dep as a tool? I guess not right, since it won't have the :ns-default info available?#2022-06-1715:48borkdudeAnyway, #jet is now available for deps.edn too:
https://github.com/borkdude/jet#depsedn
I was able to support both -M and -X/-T style invocations with minimal boilerplate thanks to #babashka-cli :-D#2022-06-1715:50Alex Miller (Clojure team)right, you can install, but you can't publish the usage stuff. I'm not sure yet what a good way to do that is, whether it should be something you added to pom or put it in a resource in the jar etc. just needs a bit more thought#2022-06-1716:41seancorfieldPresumably, there's a potential parsing conflict if the exec function is omitted (made a default)? clojure -Tthing what are '{these args?}' vs clojure -Tthing func what '[are these args?]' since you can't really assume anything based on the number of arguments?#2022-06-1716:43Alex Miller (Clojure team)yes, that's part of it#2022-06-1819:29borkdude#2022-06-1905:16seancorfieldSo this Q doesn't get lost in the higher-traffic channel there.
I can confirm that install-latest gives me v0.4.9 too while https://github.com/seancorfield/deps-new/releases shows three releases since then.#2022-06-1920:08seancorfieldhttps://ask.clojure.org/index.php/11983/why-does-tools-install-latest-not-always-find-latest-version#2022-06-1906:22thhellerI'm guessing version numbers are maybe not compared numerically? if they are sorted that way at all I don't know
(sort ["0.4.1" "0.4.9" "0.4.12"])
=> ("0.4.1" "0.4.12" "0.4.9")#2022-06-1906:32Cora (she/her)that's lexically sorted#2022-06-1906:33Cora (she/her)https://github.com/owainlewis/semver#2022-06-1909:02borkdudeThere's also: https://github.com/xsc/version-clj
user=> (v/version-sort ["0.4.1" "0.4.9" "0.4.12"])
("0.4.1" "0.4.9" "0.4.12")
#2022-06-1920:08seancorfieldhttps://ask.clojure.org/index.php/11983/why-does-tools-install-latest-not-always-find-latest-version#2022-06-2214:43Joshua SuskaloWhere can I post to put in a feature request to make sourcehut git dependencies auto-resolve the url based on groupname the same way that github and gitlab do?#2022-06-2214:43Joshua SuskaloAnd would a PR implementing this be likely to be accepted?#2022-06-2214:44Joshua SuskaloOr patchsets etc#2022-06-2214:48Joshua SuskaloFor those unfamiliar with sourcehut (https://sourcehut.org/, https://sr.ht/), it's a git forge like github or gitlab that's focused on email-driven contribution, as well as being a high-performance, low-overhead forge.#2022-06-2214:50Alex Miller (Clojure team)http://ask.clojure.org#2022-06-2214:50Joshua SuskaloThanks#2022-06-2215:09Joshua Suskalohttps://ask.clojure.org/index.php/12005/tools-deps-integration-with-sourcehut#2022-06-2214:48Joshua SuskaloFor those unfamiliar with sourcehut (https://sourcehut.org/, https://sr.ht/), it's a git forge like github or gitlab that's focused on email-driven contribution, as well as being a high-performance, low-overhead forge.#2022-06-2223:07Drew VerleeSo based on this ticket https://github.com/polyfy/polylith/issues/66 i'm assuming that deps.edn :paths doesn't resolve paths outside the directory e.g "../foo"#2022-06-2223:12seancorfieldI think it still works but you'll get a WARNING that it will go away at some point...#2022-06-2223:13seancorfield...it may have already gone away (I stopped doing :paths outside the directory where deps.edn is a long time ago so I don't know for sure @U0DJ4T5U1)#2022-06-2223:13seancorfieldBut it is why Polylith went through that massive change last year.#2022-06-2223:17Drew Verleeif i do -Spath it it still shows up. I don't want this to be a long term solution.#2022-06-2223:18seancorfieldWhat still shows up? The WARNING? If so, your project needs to stop using :paths outside the "project".#2022-06-2223:18Drew VerleeThe path still shows up in the output#2022-06-2223:19seancorfieldYeah, so it hasn't been made illegal in t.d.a yet.#2022-06-2223:19seancorfieldLike I said, I wasn't sure -- I stopped doing it last year to avoid the WARNING 🙂#2022-06-2223:19seancorfieldWhat exactly is your question? I'm confused at this point...#2022-06-2223:23Drew VerleeTo give context, there is a macro which reads a file and turns it into something the browser client thats being built needs. In the console logs that variable is "nil".
Which means either it's nil in the cljs context but is getting read correctly (i feel like i need a whiteboard explanation). Or that file isn't getting read because the path isn't valid. I think it's the later and i'm about to sanity check it by trying something simplier.#2022-06-2223:25seancorfieldYou have a macro that reads a file at expansion time? That's a bit... dangerous...#2022-06-2223:25seancorfieldBut I'll assume your macro is attempting to read a file from the classpath so it's using io/resource, yes?#2022-06-2223:25Drew Verleeyes#2022-06-2223:26seancorfieldAnd your deps.edn file has :paths (or :extra-paths) that contains a relative or absolute filepath that is outside the project?#2022-06-2223:26seancorfield(because if that file is inside your deps.edn project, all of this is irrelevant 🙂 )#2022-06-2223:28Drew Verleethe project has a shadow-cljs.edn with :deps true and a deps.edn with a :deps {foo {:local/root "../bar"}} and that "bar" has a deps with with :paths "../resources"
And yes that's awkward. The goal was to have Boot and deps co-exist for a bit so if there is a way to have boot do something like :local/root this would probably get easier and cleaner#2022-06-2223:29seancorfieldCan you move that resources folder down into bar and update the Boot file to reflect that it moved?#2022-06-2223:30Drew Verleeidk, that was going to be the next thing i tried 🙂#2022-06-2223:39Drew VerleeThat seems to do what i want, weather or not it's what i need.... will wait for another day! I hope to not be like this for very long 😆#2022-06-2223:39seancorfieldMigrations from one tech to another can be a bit painful 🙂#2022-06-2223:39seancorfieldI'm migrating a lot of legacy code into Polylith right now... it is tedious and frustrating at times.#2022-06-2223:41Drew Verleei can imagine.#2022-06-2223:45Drew VerleeI have looked at polylith but i haven't had to manage an application that seemed complex enough to warrant more thought about organization so i never could wrap my ahead around the problem it was solving.#2022-06-2300:31seancorfieldAs far as I'm concerned, any repo that produces multiple application artifacts is a good candidate for Polylith.#2022-06-2415:45Joshua Suskalo@U04V70XH6 is there somewhere that I can see the basic changes for that alteration of polylith? I've been using polylith for a monorepo for a while and we're using this style of deps at the moment.#2022-06-2415:46Joshua SuskaloOh wait, I think I see. This actually shouldn't be a concern because paths aren't being directed outside the current directory, only deps with :local/root target outside the current directory.#2022-06-2416:26seancorfieldI'd say "read issue #66" and the associated code changes but it was a massive change early last year -- so it depends on when you started using Polylith? 🙂#2022-06-2416:27seancorfieldWe started using Polylith while that work was happening on a branch and so we started with :extra-deps instead of :extra-paths for :dev and used :local/root right from the start.#2022-06-2416:28seancorfieldAnd that all came about because we had been on bleeding edge versions of the Clojure CLI / t.d.a for a while and saw it coming. This is the same reason we always update our vendored-in CLI to the latest prerelease versions, and update our repo to the latest Clojure prerelease versions: to test against upcoming changes so we aren't caught out when they drop.#2022-06-2416:28Joshua SuskaloYeah, looks like we started using it after the issue. The only place we're using extra paths is in the dev alias on the root deps.edn for the development project.#2022-06-2416:29seancorfield(I mention this because there are always people complaining at each new Clojure release about changes that happened in early alphas that they should have tested against and provided feedback on!)#2022-06-2416:29Joshua SuskaloYeah, I always try to stay on the bleeding edge to catch stuff like that.#2022-06-2416:30seancorfield"extra paths .. in the dev alias" -- because you have some Cursive users?#2022-06-2416:31seancorfieldUsing :extra-paths instead of :extra-deps means you also have to duplicate all the :extra-deps libs from bricks up into :dev as well which is a pain.#2022-06-2416:38Joshua SuskaloAfter reading through the issue 66 it looks like another solution to this problem would be to make a #include "some-dir/blah" to allow you to read in arbitrary other edn data into the current file as a value. This would allow multiple edn files, one per project, while one large file at the root enumerates the projects. That way you can have a "working directory" of the root, but with separation of the data.
That said, this adds complexity to the clojure cli that's definitely not necessary, and honestly I like having deps edn files in components as it allows specifying library dependencies of components with the component rather than needing to always include them in every project.#2022-06-2416:41seancorfieldPretty sure the core team wouldn't want to support #include or anything like that since it's meant to be "just EDN" with no custom readers. And it would be a terrible mistake for Polylith to add "proprietary" extensions to deps.edn because then you couldn't use any standard t.d.a tooling with those files.#2022-06-2416:41Joshua SuskaloYeah, agreed on both counts.#2022-06-2416:43seancorfieldIt frustrates me already that Polylith doesn't rely on t.d.a more and ends up having to re-implement stuff that t.d.a already supports (see the recent ticket about aliases in paths -- built into t.d.a but Polylith had to re-implement it b/c it doesn't use create-basis etc)#2022-06-2311:59Christian JohansenCan tools deps check out private github repos with a personal access token? Specifically, deps.edn has <mailto:/cdn-cgi/l/email-protection|/cdn-cgi/l/email-protection>:… dependencies, how can I use a personal access token in github actions for tools deps to find the dependencies?#2022-06-2312:07Alex Miller (Clojure team)If you can use git clone outside the Clojure CLI, then it should work#2022-06-2312:07Alex Miller (Clojure team)So solve that problem first#2022-06-2312:08Alex Miller (Clojure team)The CLI is just shelling out to git#2022-06-2312:08Christian JohansenRight, thanks#2022-06-2312:08Christian JohansenI’ll try to use an SSH key instead, I think that’s easier#2022-06-2312:08Alex Miller (Clojure team)Some of that stuff is set up for you in gh actions#2022-06-2312:09Christian JohansenThe checkout thing works with a token, but I’m not sure how to combine it with the tools deps step#2022-06-2312:09Christian Johansen(I’m rather new at github actions…)#2022-06-2312:38Alex Miller (Clojure team)been a while since i used a token - what does that look like? does it get passed on the command line or is it ambient somehow#2022-06-2312:51Christian JohansenI’m not entirely sure 😅 With the checkout action you just specify the token. I think you can use it as a password with your username. Anyway, I got my action pipeline working with an ssh key 👍#2022-06-2314:20practicalli-johnAre you sure a token is needed?
If an action is within the scope of the current repository, a GITHUB_TOKEN is automatically created for the GitHub action by GitHub.
Or you can set a GITHUB_TOKEN environment variable in the repository or the organisation that repository belongs too, usually if you need to update or otherwise affect another repository/ organisation or user.#2022-09-2312:41jjttjj"Bumping" this because I see it was already discussed but I'm not quite sure I'm clear on an answer, at least for my situation.
I'm trying to use clj from a github action in a private repo that has a dependency on another private repo in the same organization. I'm using an oldish version of https://github.com/DeLaGuardo/setup-clojure (v3.5).
Getting the following error:
Unable to clone /home/runner/.gitlibs/_repos/github.com/my/proj
fatal: could not read Username for '': terminal prompts disabled
Is there something I need to do with the GITHUB_TOKEN (that github actions set)? Do I need to clone a repo using the token as the password, and possibly set git to use caching?
(Just wondering if there's a quick fix I'm missing before I start spamming changes to my workflow yml since I don't currently it setup to run locally)#2022-09-2312:44dharriganWould this be a case of using clojure, not clj as clj uses rlwrap which expects a terminal?#2022-09-2312:49jjttjjNope, I'm using clojure#2022-09-2312:50jjttjjI think I might need to use the configure_git option with https://github.com/marketplace/actions/private-actions-checkout, attempting that now#2022-09-2313:30dharrigan:thumbsup:#2022-09-2313:37jjttjjAh, didn't seem to work#2022-09-2313:41dharriganLet me DM you#2022-09-2314:14jjttjjOk I got it working, this is what worked for me.
After creating a secret for the organization that has a read-only personal access token, I put this in the github hactions yaml:
...
jobs:
build:
...
- name: Use github personal access token to authenticate private repos
run: git config --global --add url.":${{ secrets.ORG_PERSONAL_ACCESS_TOKEN }}@github.com/".insteadOf ""#2022-09-2314:17practicalli-johnThe github_token that GitHub automatically creates can only be used for the current repository, so as you rightly have done an org (or user) token is required to access other repositories (that are not public)#2022-06-2314:23practicalli-johnI assume to access a private repo you can create a token on the private repo and set that as the GITHUB_TOKEN value from the GitHub action that is accessing it (in theory)#2022-09-2312:41jjttjj"Bumping" this because I see it was already discussed but I'm not quite sure I'm clear on an answer, at least for my situation.
I'm trying to use clj from a github action in a private repo that has a dependency on another private repo in the same organization. I'm using an oldish version of https://github.com/DeLaGuardo/setup-clojure (v3.5).
Getting the following error:
Unable to clone /home/runner/.gitlibs/_repos/github.com/my/proj
fatal: could not read Username for '': terminal prompts disabled
Is there something I need to do with the GITHUB_TOKEN (that github actions set)? Do I need to clone a repo using the token as the password, and possibly set git to use caching?
(Just wondering if there's a quick fix I'm missing before I start spamming changes to my workflow yml since I don't currently it setup to run locally)#2022-06-2416:37Alex Miller (Clojure team)@borkdude picking up from other thread...#2022-06-2416:38Alex Miller (Clojure team)so the case is:
root deps - has whatever clojure version
user deps - has :deps {org.clojure/clojure {:mvn/version "1.10.3"}}
tool deps - has :deps {org.clojure/clojure {:mvn/version "1.11.1"}}}#2022-06-2416:38Alex Miller (Clojure team)?#2022-06-2416:38borkdudecorrect#2022-06-2416:40borkdude(afk for dinner, will be back)#2022-06-2416:40Alex Miller (Clojure team)ah, so the tool's deps are not top level#2022-06-2416:41Alex Miller (Clojure team)it will be {:deps {tool-lib tool-coord}}#2022-06-2416:41Alex Miller (Clojure team)so anything defined top level in root/user deps.edn will be used over the tool#2022-06-2416:42borkdudeAre you explaining the status quo or actually how it's indented to work?#2022-06-2416:42Alex Miller (Clojure team)status quo#2022-06-2416:42Alex Miller (Clojure team)don't remember if I've thought about this in the past (usually would only affect clojure version)#2022-06-2416:43Alex Miller (Clojure team)and if you're on latest CLI, you'd be on latest clojure version#2022-06-2416:45seancorfieldDidn't this come up in another thread here (or in #tools-build maybe?) a few weeks back?#2022-06-2416:45Alex Miller (Clojure team)maybe? so many threads :)#2022-06-2416:45Alex Miller (Clojure team)I did definitely consider both options back when this was implemented. don't remember if this aspect was considered or not, may have missed it.#2022-06-2416:46dpsuttonI think it is this thread: https://clojurians.slack.com/archives/C02B5GHQWP4/p1651524280682919#2022-06-2416:46seancorfieldI think that was around getting an "unexpected" version of Clojure because a project didn't specify a dependency (on Clojure) -- and then they got different Clojure versions with tools vs other stuff. So maybe not entirely related.#2022-06-2416:46seancorfieldOh yeah, that was it -- :default-deps not "working" for Clojure itself (due to different Clojure versions in CLI vs tool chain stuff).#2022-06-2416:56Alex Miller (Clojure team)I don't really see this as an important issue - several things have to combine for it to surface as an actual problem for a user.#2022-06-2416:57seancorfieldWhich was the thread that started this discussion BTW? I think I must have missed that...#2022-06-2417:06Alex Miller (Clojure team)In announcements#2022-06-2417:43borkdude> several things have to combine for it to surface as an actual problem for a user.
If a user has clojure 1.10.3 in their user deps, tools depending on 1.11 won't work, isn't that an important problem?#2022-06-2417:43borkdudeIn general I think more specific settings should override global settings#2022-06-2417:44seancorfield-T will ignore user deps other than the specified alias and for an installed, named tool the user deps doesn't come into play.#2022-06-2417:45borkdude@U04V70XH6 Well, they did come into play, which was why this discussion started#2022-06-2417:46borkdude@U04V70XH6 Easy to repro:
Put clojure 1.10.3 in your user deps.edn (top level)
Then try:
clj -Ttools install io.github.babashka/http-server '{:git/tag "v0.1.4"}' :as serve#2022-06-2417:47borkdudeand then: clj -Tserve exec#2022-06-2417:48borkdudeThis will fail. Then update user to 1.11 and it will work.#2022-06-2417:48Alex Miller (Clojure team)"If a user has clojure 1.10.3 in their user deps" is a thing most people don't do, and even if they do, most tools do not require a newer version.#2022-06-2417:48Alex Miller (Clojure team)I'm not saying this is not an issue, it just seems less important than many other things#2022-06-2417:49borkdude¯\_(ツ)_/¯
#2022-06-2417:49seancorfieldWhy would anyone put a version of Clojure itself in their user deps? That seems like a terrible idea.#2022-06-2417:50seancorfieldPut aliases for different Clojure versions in user deps, sure, but not any top-level deps.#2022-06-2417:50borkdude@U04V70XH6 Because the clojure CLI did not have 1.11 and I wanted a clojure 1.11 REPL everywhere? And I didn't expect those deps to override project-specific clojure versions since most tooling doesn't work that way. But I understand now that this doesn't work this way. TIL.#2022-06-2417:51borkdude(and btw, I wasn't the one running into this problem, it was someone else, so apparently I'm not the only person doing terrible things)#2022-06-2417:51seancorfieldclj -A:1.11 -- use an alias#2022-06-2417:52seancorfieldAnd if a tool depends on a minimum version of Clojure, tell folks their CLI needs to "at least x.y.z.*" for it to work.#2022-06-2417:53seancorfieldFolks need to be encouraged to keep their CLI install up-to-date 🙂#2022-06-2417:53borkdude@U04V70XH6 This was before the clojure CLI was updated#2022-06-2417:53seancorfieldCLI 1.11.. has been available for quite a while...#2022-06-2417:54borkdude@U04V70XH6 I'm talking about why I added that thing there a while ago and then I of course forgot about it.#2022-06-2417:54Alex Miller (Clojure team)if you can make an ask.clojure question, we can track votes/experiences#2022-06-2417:54borkdudeBut it was @UK0810AQ2 running into this problem#2022-06-2417:55Alex Miller (Clojure team)I would like to address this, just not at the top of my list (unless 10 people are having the same problem)#2022-06-2417:55seancorfieldClojure 1.11.1 was released on April 5th and the CLI was updated to 1.11.1.1105 that same day 🙂#2022-06-2417:56borkdude@U04V70XH6 If you want all the details, I had org.clojure/clojure {:mvn/version "1.11.0-alpha4"} in there and updated with every alpha release ;)#2022-06-2417:57borkdudeThis is actually a good way to force-test a new Clojure release in all your projects, now I see why it is like this :P#2022-06-2417:58borkdudeAnyway, thanks for clearing it up. Now I know what to tell people if they run into this problem.#2022-06-2417:58Alex Miller (Clojure team)I don't think the fix for this is obvious, and it may interact with other possible dependency management like features#2022-06-2417:58borkdudeIf it happens more often (not sure what my threshold is) I'll create an ask issue#2022-06-2418:03Alex Miller (Clojure team)thx#2022-06-2610:59borkdudeMade tool dependency (org.babashka/cli) compatible with 1.9 now which reduces the likelihood of giving problems.#2022-10-0612:26borkdudeJust FYI, I came across one user who added deps in their user deps.edn here:
https://clojurians.slack.com/archives/CHY97NXE2/p1665059013827909?thread_ts=1665055904.775729&cid=CHY97NXE2#2022-06-2418:16pyr@jr0cket The only catch with using a token is that you then force everyone to have a token handy. Setting up a key that has read-only access to the repos you're interested in is likely the best approach (but will likely require eating a new seat in your org)#2022-06-2418:24practicalli-johnI am confused now. I thought you were talking about GitHub actions. I don't understand the question anymore, sorry.
I only use SSH keys if it's people on teams accessing repositories. Tokens are much more work for people, but useful for systems access.#2022-06-2419:14pyrI agree. I was only saying that with tools.deps you have to make a choice on which to use (or alternatively, go with different deps.edn files for different context, adapting dependency declarations as needed)#2022-06-2419:15pyrAh wait sorry I replied to you but you weren't the one asking the question in the first place! sorry, I should have directed my comment to @U9MKYDN4Q#2022-06-2511:16victorbquick q: Can I somehow hook into tools-deps installation lifecycle? For example, I'd like to prevent downloading dependencies based on some heuristics, so I have a cli that exits with 0/non-zero status code, which I'd like to use to prevent downloads. npm has "scripts" in its package.json, where you can add a preinstall script that gets run before installs, so I guess i'm looking for something similar#2022-06-2511:27Alex Miller (Clojure team)No, don't have anything like that#2022-07-0415:09Benjamin[
I have a dir like this full of jars without a pom. How do I add them all to the classpath?#2022-07-0415:17delaguardoI would generate deps.edn in that repository using babashka with :paths pointing to each jar#2022-07-0415:21Benjaminworks 🙂#2022-07-0415:31Alex Miller (Clojure team)clj -Scp "*"#2022-07-0415:32Alex Miller (Clojure team)jvm classpath has a special case for * as a wildcard#2022-07-0416:54Benjamin:thumbsup: thanks Alex#2022-07-0416:55Alex Miller (Clojure team)and note that this is not a shell expansion thing - the jvm expects to see "*" as part of a path item#2022-07-0416:55Alex Miller (Clojure team)and not everything you think would work at the shell works in the jvm, it's much more limited#2022-07-0416:56BenjaminI see#2022-07-0416:56Benjamin"jdbc/*" works like expected#2022-07-0515:36Ben SlessIs it possible to write a tool that dynamically brings in the dependencies it needs based on some user inputs? Is add-deps the only option? I guess I can always rebuild the deps and get to the tool entry point myself#2022-07-0515:44borkdude@ben.sless Probably not what you want, but slightly related: https://github.com/borkdude/deps-infer
(resulted from a brainstorm in this channel with @hiredman)#2022-07-0515:54Ben SlessI want to load correct dependencies depending on the input format, source, etc, but I dont want to drag in potentially heavy deps for every use case#2022-07-0516:16Ben SlessI can risk it and use the add lib branch#2022-07-0601:36bocajI’d like to use azure-functions’ maven tooling to run and deploy clojure code on their azure-functions framework. Right now I’m copying deps from deps.edn to the pom.xml so that my dependencies are available to the azure-functions environment. What is currently possible for syncing dependencies from deps.edn to pom.xml? Or, perhaps I should use b/write-pom and refer to my clojure code as a module from a parent pom.xml#2022-07-0601:59Alex Miller (Clojure team)you can generate a pom from your deps.edn#2022-07-0602:00Alex Miller (Clojure team)clj -X:deps mvn-pom#2022-07-0602:01Alex Miller (Clojure team)this functionality will use a pom.xml as the source (if it exists) and update only specific sections (<dependencies>, path stuff, repositories) - the rest is left as is so you can put all your pom metadata there#2022-07-0603:05bocajOh! That's convenient.#2022-07-1708:23BenjaminJo I have an error with ion-dev and a git dep:
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:126)
the alias in .clojure/deps.edn
:ion-dev
{:deps {com.datomic/ion-dev {:mvn/version "1.0.304"}}
:main-opts ["-m" "datomic.ion.dev"]}
the dep in the project dep
org.sg.reply-bot/reply-bot {:git/url "#2022-07-1712:29Alex Miller (Clojure team)What was the original command? Is this Clojure cli or a Datomic command?#2022-07-1712:29BenjaminCli with alias#2022-07-1712:30Alex Miller (Clojure team)What was the command and what version of the cli is it?#2022-07-1712:32Alex Miller (Clojure team)The jsch ssh stuff is from a very old version of tools.deps so I suspect the first step is to update your cli#2022-07-1712:41BenjaminChecking#2022-07-1712:49Benjaminturns out I run clojure with bb
Clojure CLI version (deps.clj) 1.11.1.1113
#2022-07-1712:57Benjaminprobably I need to update babashka tasks then. I just tried just shelling out to "clojure" and it worked 🎉#2022-07-1920:13p-himikFeels like I'm blanking on something obvious - with deps.edn and CLI, how can I refer to an alias with :main-opts while ignoring them?
I.e. I want to use that alias' dependencies but instead of respecting its :main-opts I want to just run the CLI REPL.#2022-07-1920:17p-himikA, found a discussion: https://clojurians.slack.com/archives/C6QH853H8/p1651571780896929#2022-07-2023:39bocajI’m trying to uber this dependency and keep running into an error having to do with the License file.
org.apache.hadoop/hadoop-mapreduce-client-core {:mvn/version "3.3.3"
:exclusions [org.slf4j/slf4j-log4j12]}
Execution error (FileNotFoundException) at java.io.FileOutputStream/open0 (FileOutputStream.java:-2).
/var/folders/r8/py0yt76j2gv0l1ywbww_5z080000gn/T/uber15373383667740882819/META-INF/license/LICENSE.base64.txt (Not a directory)
#2022-07-2023:40bocaj(b/uber {:class-dir class-dir
:uber-file "dbloader.jar"
:basis uber-basis
:exclude [#"(?i)^META-INF/license/.*"
#"^license/.*"
#"LICENSE(.)"
]
:main 'dbloader.main})#2022-07-2023:41bocajunzip -l ~/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/3.3.3/hadoop-hdfs-client-3.3.3.jar | grep LICENSE #>> META-INF/LICENSE.txt
#2022-07-2023:49bocajApologies this is fixed in newer tools build#2022-07-2110:20pieterbreedIn my project I'm getting this warning:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/pieter/.m2/repository/com/fzakaria/slf4j-timbre/0.3.20/slf4j-timbre-0.3.20.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pieter/.m2/repository/org/slf4j/slf4j-simple/1.7.29/slf4j-simple-1.7.29.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See for an explanation.
From my understanding; the fix is to find the specific dependency that includes a specific jar and exclude it. In this case if I could somehow exclude slf4j-simple...jar from the classpath that would solve the issue.
When I run clj -Stree I don't see an entry for slf4j-simple.
What is the recommended way for tracking this down and fixing?#2022-07-2110:24p-himikWith -Stree, do you specify the same exact aliases as you do when you get the warning?#2022-07-2110:25pieterbreedhmmmm :thinking_face: let me see#2022-07-2110:26pieterbreedthis is it, thank you thanks3#2022-07-2510:23pinkfrog{:exclusions [foo/lib]} excludes foo/lib, what’s the counterpart that ensures a specific version of foo/lib is used?#2022-07-2510:24p-himikJust add it to your :deps or :extra-deps with that version.#2022-07-2510:30pinkfrogIt is a lib. So other libs conflicts with it. I cannot control the :deps of the project.#2022-07-2510:39p-himikI don't understand what you mean. Can you give a concrete example? With a description of what you can and cannot control.#2022-07-2510:41delaguardoyou control your deps.edn
this should be a place where you declare some specific version of some library. and it will take precedence even if another lib in your dependency tree excludes it#2022-07-2510:44p-himikTo make make the above more concrete with an example.
Your deps.edn:
{:deps {lib/a {:mvn/version "1.0.0"}
lib/b {:mvn/version "0.0.2"}}}
The lib/a's deps.edn:
{:deps {lib/b {:mvn/version "0.0.3"}}}
Your project will end up having lib/b version 0.0.2 even though lib/a depends on a newer version.#2022-07-2511:34pinkfrogRight. But My goals is the contrary, to forcefully obey
{:deps {lib/b {:mvn/version "0.0.3"}}}#2022-07-2514:11p-himikI believe the only reasonable way to do it is to just specify that version manually.
But you can have a test that makes sure that it's the right version at all times.#2022-07-2514:40ChaseI set up a test alias according to the cognitect test runner repo and when invoking it with clj -X:test it runs all tests twice. When I use my editor's built in test functionality it only runs them once as expected. nbd but just curious why that might be happening.#2022-07-2514:46p-himikMaybe it gathered all your tests twice? Not sure why exactly - maybe some bug in the runner, maybe something's wrong in your setup.#2022-07-2514:49Chase:test {:extra-paths ["test"]
:extra-deps {io.github.cognitect-labs/test-runner
{:git/url ""
:sha "9e35c979860c75555adaff7600070c60004a0f44"}}
:main-opts ["-m" "cognitect.test-runner"]
:exec-fn cognitect.test-runner.api/test}}}#2022-07-2514:50Chasethat's in my deps.edn and my minimal tester is just:
ns practice.scratch-test
(:require
[clojure.test :refer :all]
[practice.scratch :refer :all]))
(deftest a-test
(testing "add"
(is (= 0 (add 1 2)))))#2022-07-2514:50Chasedid I do something silly there?#2022-07-2514:50Chasethe results:
λ ~/projects/clojure/practice : clj -X:test
Running tests in #{"test"}
Testing practice.scratch-test
Testing practice.scratch-test
Ran 2 tests containing 2 assertions.
0 failures, 0 errors.#2022-07-2514:52p-himikAnd no "test" in the :paths in deps.edn?#2022-07-2514:53p-himikI just did the same thing (but replaced 0 with 3 and add with +), and my tests ran only once.#2022-07-2514:58ChaseI didn't even catch that it wasn't catching the failing test. That is a problem. Hmmm#2022-07-2514:59ChaseAnd no, I have :paths ["src" "resources"] in the deps.edn#2022-07-2514:59ChaseMy repl connected editor catches that failing test.#2022-07-2515:00ChaseIs my deftest form actually correct? I'm coming back to clojure after a little break so was just going off memory there.#2022-07-2515:01p-himikYour deftest is correct.
But I don't know why it tests the same thing twice and doesn't catch the error. If you can share a minimal reproducible example as a Git repo, I'd take a look.#2022-07-2515:04ChaseWill do, give me 1 minute. Thanks!#2022-07-2515:06Chasehttps://github.com/Chase-Lambert/clojure-practice#2022-07-2515:09p-himikYou have test/scratch_test.clj with (ns practice.scratch-test ...).
And you have test/practice/scratch_test.clj with the same NS.
Remove the first one, change the second one (it won't fail otherwise).#2022-07-2515:09Chaseaccidentily left in a dep for criterium fyi, I just push a commit removing that.#2022-07-2515:10Chaseoh my goodness, I left that file in there huh. I told you I was doing something silly! Sorry for wasting your time.#2022-07-2515:10p-himikNo problem. :)#2022-07-2515:10Chaseoh wait, but it's still not catching the failed test (even though it now only runs once of course)#2022-07-2515:11ChaseNope, still being silly, I had changed the test to be successful. Hahaha!#2022-07-2515:11ChaseI need more coffee#2022-07-2515:12ChaseThanks again for the assistance#2022-07-2720:55seancorfield@alexmiller "This branch is 8 commits ahead, 117 commits behind master." -- can we get t.d.a master merged to add-lib3, or is there some impediment to that now?#2022-07-2720:57seancorfieldI just did a test merge locally and there are no conflicts, but I didn't attempt to run any tests etc.#2022-08-0113:41Alex Miller (Clojure team)pushed#2022-08-0115:40seancorfieldAppreciated!#2022-07-2721:27agWhile trying clj -T:build jar, I'm getting weird error I haven't seen before.
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).\norg.objectweb.asm.Type\n
Anyone familiar? Any pointers are appreciated.#2022-07-2721:29seancorfieldSounds like a version conflict somewhere in the dependencies. What is your :build alias and what is the jar task in build.clj? Also, do you have more of a stacktrace?#2022-07-2721:33ag> What is your :build alias
:build
{:extra-paths ["src"]
:extra-deps {io.github.clojure/tools.build
{:git/tag "v0.8.3" :git/sha "0d20256"}}
:ns-default build}
> Do you have more of a stacktrace?
It's a bit overloaded, but I couldn't decipher it. Let me dig a bit more, maybe I could still figure it out on my own.#2022-07-2721:38agjar task:
(defn jar [_]
(b/write-pom {:class-dir class-dir
:lib lib
:version nil
:basis basis
:src-dirs ["src"]})
(build-cljs+css nil) ;; this can be ignored (the problem is not here)
(b/copy-dir {:src-dirs ["src" "resources"]
:target-dir class-dir})
(println "BUILD: Producing jar:" jar-file "...")
(b/compile-clj {:basis basis
:src-dirs ["src"]
:class-dir class-dir})
(b/uber {:class-dir class-dir
:uber-file jar-file
:basis basis
:manifest {"Main-Class" "iroh_front_end.server.system"}})
(println "BUID: Done building jar."))#2022-07-2721:39ag#2022-07-2721:42seancorfieldSo it's coming from compilation of some core.async stuff that is referenced from iroh_front_end.shadow_cljs_helpers whatever that is...?#2022-07-2721:44hiredmanyou have some bogus dependency information somehow, core.async uses tools.analyzer.jvm, which has a dependency on the library where rg.objectweb.asm.Type comes from#2022-07-2721:44hiredmanso if org.objectweb.asm.Type is missing, somehow the jvm isn't being started with asm on the classpath#2022-07-2721:45seancorfieldYup, I was just about to ask how you compute basis and whether you are missing some aliases there?#2022-07-2721:47hiredmanit would be very surprising to get tools.analyzer.jvm on the classpath, but not org.objectweb.asm.Type with any basis#2022-07-2721:47hiredmanmy guess would be some kind of uberjar as a dependency shenanigans#2022-07-2721:48thhellerit probably shouldn't even be compiling that file in the first place? I guess you use it for development stuff? I hope you are not using shadow-cljs in any production setup?#2022-07-2721:50hiredmancljs recently (maybe months ago?) switched to a vendored version of some of the core libs because of clashes, I wonder if it is the cljs stuff in your build#2022-07-2721:52agOh, so the issue it seems that I need a dependency (shadow-cljs) included, but it's in :extra-deps key in an alias.
how do I include deps in an alias with clj -T?
doing clj -T:shadow-cljs:buld jar, didn't work.
I tried clj -A:shadow-cljs -T:build jar - still didn't work#2022-07-2721:53thhellerI strongly recommend to never run shadow-cljs as part of a tools.build task#2022-07-2721:54thheller(since CLJS compilation needs access to all your CLJS dependencies, so you'll just end up including your whole project, which build tasks aren't designed to do I guess)#2022-07-2721:55agoh#2022-07-2721:56seancorfieldIt's somewhat confusing that your task is called jar but it's actually building an uber JAR 🙂#2022-07-2721:56agah, right... gotta rename it. good point#2022-07-2721:56seancorfieldSee https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-create-basis for how to specify aliases when creating the basis.#2022-07-2721:57seancorfieldThat's where you would specify :shadow-cljs rather than on the command-line.#2022-07-2721:58agAh... I see#2022-07-2721:58seancorfieldBut as Thomas says, you probably need to run a shadow build to generate JS and then copy that into your uberjar, rather than trying to run Clojure's compile-clj with all the Shadow stuff included.#2022-07-2722:00seancorfieldI would have expected that to be done via your build-cljs+css function?#2022-07-2722:01thhellerI mean if thats running a new JVM process thats fine, but if its running as part of the build task JVM thats not great#2022-07-2722:02ag> I would have expected that to be done via your build-cljs+css function?
that's what it does, but I also have shadow-cljs-helpers.clj, they need shadow-cljs dep included#2022-07-2722:02thhellersimilar as to how compile-clj is launching a new JVM I guess#2022-07-2722:02seancorfieldbuild.clj can easily spawn separate processes to run this stuff in isolation. That's what my build-clj wrapper does for running tests, for example, and it provides for running arbitrary tasks in isolation too.#2022-07-2722:03agSean, what's the format of :aliases of create-basis? Should it be string or vector?#2022-07-2722:04aglooks like vector of keywods... I think#2022-07-2722:06agThat did it. Thank you everyone. I think that fixed the issue.
Really appreciate your help. Without you, I'd be stuck for a very long time, for sure.#2022-08-0208:32thhelleris there something in tools deps nowadays that can run a CLJ function from the command line, without it doing any argument parsing itself? -X presumes the only argument is a EDN map and parses it as such. -M -m only runs -main? I want clj -? some.ns/some-fn foo?#2022-08-0208:34thhellerjust remembered I opened a ticket for that for clojure.main itself which would work I guess https://clojure.atlassian.net/browse/CLJ-2316#2022-08-0210:28borkdudeMy opinion: introducing a new -f flag seems too much overkill to me for such a simple tweak (and babashka uses -f to indicate --file...), but I think it would be good to have this. lein and bb do support -m foo/bar
@thheller What about doing this in user space (for now)?
(ns run-main)
(let [[main-fn & args] *command-line-args*]
(apply (requiring-resolve (symbol main-fn)) args))
$ clj -M /tmp/run_main.clj clojure.core/prn 1 2 3
"1" "2" "3"
#2022-08-0210:31borkdudeWhen moving this to its own -main function, you could hide this behind an alias, so you can do:
clj -M:run clojure.core/prn 1 2 3
#2022-08-0210:39thhellerjust checking if there is something built-in nowadays that I maybe missed#2022-08-0210:40thhellerI'd prefer if -m just did this by default, similar to how lein run -m foo/bar or lein run -m foo does (ie. accepting both)#2022-08-0309:06Benjaminprobably this question was asked before; I am guessing there is a good reason why git deps are not resolving lein project "manifest type" (aka parse project.el)#2022-08-0312:36Alex Miller (Clojure team)Hasn't been implemented. Doing full support for project.clj would require pulling in a lot of lein as a dependency and don't want to do that. More limited support for simple project.clj is something we have a patch for but I haven't had a chance to look at it yet#2022-08-0313:06Benjamin:thumbsup:#2022-08-0421:46borkdudeIf I were to store some information about a project in deps.edn, what's the best place? A top level org-name fully qualified key or a similarly keyed alias?#2022-08-0421:55Alex Miller (Clojure team)aliases are places to store arbitrary data under a name#2022-08-0604:31didibusAs a follow up, was deps.edn always envisioned to be open keys to outside tooling? Is it part of the philosophy that other tooling could use it to add more key/values to it for their own use?#2022-08-0605:41Alex Miller (Clojure team)top-level, no#2022-08-0605:42Alex Miller (Clojure team)but we thought it might have have procurer config extensions there. I regret putting the :mvn/local-repo and :mvn/repos there though. they may ultimately move under a :procurers key or something.#2022-08-0618:27didibusI think my line of thinking is, for example, with lein, the config file was called project.clj, with deps its called deps.clj. The names themselves reflect a bit of a different mindset. project.clj made it sound like this was the config for your project. deps.edn makes it sound like it's the config only for your deps.
So a question that arise is for example, where do you store your project name, version, description, license, and url. In the official examples, they're stored inside build.clj as top level vars.
As more user friendly, all-in-one, higher level tooling is built and meant to be used on top of tools.deps and build.tools. That tooling might need some config. Would it be in the spirit of tools.deps to put it all under an alias inside deps.edn, say have a :project alias with it? Or its more expected that another config file is made like project.edn ?#2022-08-0618:28Alex Miller (Clojure team)> So a question that arise is for example, where do you store your project name, version, description, license, and url. In the official examples, they're stored inside build.clj as top level vars.
This is only one option of course - you can load these from an .edn file, load them from aliases in your deps.edn, whatever.#2022-08-0618:31Alex Miller (Clojure team)that is, tools.deps and tools.build do not answer this question. in general, if you want to make data accessible to both people and programs, .edn files are a good choice (or other formats more accessible to other toolchains if needed)#2022-08-0618:32Alex Miller (Clojure team)if you do put it in an alias in your deps.edn, one benefit is that it is also accessible in the basis you create in your build.clj#2022-08-0618:35didibusOk make sense. And if say you were to build extensions on top of tools.deps. Say I need to start an alias and inject deps into a particular alias. Cider does this for example to inject its middleware.
But for some reason, as part of that injection process, there might be some config needed, would it make sense to allow for new keys under the alias, maybe :cider/completion-engine ... for example. So when cider launches the alias it would inspect the alias for its own config as to what to inject into it before launching it.#2022-08-0618:37Alex Miller (Clojure team)"inject deps into a particular alias" - I don't know what that means#2022-08-0618:38Alex Miller (Clojure team)aliases are names for data. aliases provided on the command line via -A etc are data understood by tools.deps and the Clojure CLI itself.#2022-08-0618:39didibusI mean say you want to launch a repl under the user :dev alias. But you want it to have nRepl and the cider middleware for nRepl.
You tell cider to run clojure -M:dev basically. It'll probably add extra-deps as it runs the command. But what if the extra-deps it adds are configurable, could it make it that the config is directly read from the :dev alias?#2022-08-0618:39Alex Miller (Clojure team)if you need to add stuff you can always add an additional alias - merging that data is handled for you by the CLI#2022-08-0618:40Alex Miller (Clojure team)and that can be provided ad-hoc via -Sdeps (or in basis calls via :extra)#2022-08-0618:40didibusRight, but the config is per-alias. So as a separate alias you'd need a reference back to each alias.#2022-08-0618:41Alex Miller (Clojure team)is this config that the CLI understands or just other stuff?#2022-08-0618:43didibusFor example,
Cider will run:
clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.9.0"} cider/cider-nrepl {:mvn/version "0.28.3"} refactor-nrepl/refactor-nrepl {:mvn/version "3.5.2"}} :aliases {:cider/nrepl {:main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]}}}' -M:dev:cider/nrepl
So it can inject an alias and deps form the command. But if say I wanted to tell Cider, for my dev alias, use an older version of the middleware, or use a specific completion engine for example. It be nice to do in my deps.edn:
{:aliases {:dev {:cider/completion-engine :foo, ...}}}#2022-08-0618:43didibus> is this config that the CLI understands or just other stuff?
Other stuff. It would be for tools that make use of the cli or tools.deps.#2022-08-0618:43Alex Miller (Clojure team)why do you need to put it under :dev?#2022-08-0618:44Alex Miller (Clojure team){:aliases {:my-cool-config {:dev {:cider/completion-engine :foo ...}}}}#2022-08-0618:44Alex Miller (Clojure team)your code can use the :dev alias as a lookup key in your own config#2022-08-0618:45Alex Miller (Clojure team)(I'm not convinced that this is actually per-alias data)#2022-08-0618:46didibusYou don't per-say. Was asking like what you saw as best-practice for this stuff going forward. Personally I think config like that is more brittle, because like say I decide to rename my alias to :devo, its more remembering from my part that there might be other aliases that refer to this alias I also need to update, but it does work.#2022-08-0618:47Alex Miller (Clojure team)I think it's brittle to manipulate data controlled by other people :)#2022-08-0618:47didibusIsn't that kind of the point of namespaced keys though? The key is for some other namespace.#2022-08-0618:47Alex Miller (Clojure team)the more it can be additive, the less brittle it is#2022-08-0618:48didibusI agree. I just always thought that's what's beautiful about open maps and namespaced keys. You can allow them to be extended externally by people who don't control it.#2022-08-0618:52Alex Miller (Clojure team)I'm fine with that happening in the config file (and the code is written to be tolerant of that as much as possible). I'm objecting to the idea of "injecting", but maybe I'm just reading too much into that#2022-08-0618:58didibusI think I got the gist of the mental model:
1. deps.edn can be used for project/tool-level config, but that's just one option for how this could be done
2. If using deps.edn for project/tool-level config, prefer having a dedicated alias for them
3. If a dedicated alias isn't ideal for "good reasons", use namespaced keys
4. Don't add keys (even namespaced) to the top-level map#2022-08-0619:06Alex Miller (Clojure team)Sounds right#2022-08-0615:59radsNow that :main-opts is deprecated with -A, are there any explicit use cases to call out for -A other than starting a REPL?
clj [clj-opts] [-Aaliases] [init-opts]
-Aaliases
Concatenated aliases for REPL execution, ex: -A:dev:mem
Just want to clarify this for myself and others as I go through and update docs which previously used clj -A:foo to invoke main functions#2022-08-0617:14Alex Miller (Clojure team)-A will continue to work for all modes but, yes it is expected that primary use is supplying aliases with repl execution#2022-08-0617:15radsThanks!#2022-08-0814:49Drew VerleeI want to use a different java version per project/directory. Emacs calls clojure/clj which i assume is using the system set version of java. Is there a way to have deps manage the version of java? Given it's clj that seems to pick the java (is that right?) it would seem up to deps to configure the version of java.#2022-08-0814:51Drew Verleeor maybe if emacs was calling clj from inside the project context something else would set the version of java?#2022-08-0814:52p-himikJAVA_CMD environment variable + something like dotenv.#2022-08-0814:54Alex Miller (Clojure team)this is considered to be external to clj#2022-08-0814:54Alex Miller (Clojure team)better to set JAVA_HOME (JAVA_CMD is kind of last resort)#2022-08-0814:55Alex Miller (Clojure team)clj just uses whatever java is on the command line OR java as set at JAVA_HOME OR whatever is in JAVA_CMD#2022-08-0814:56p-himikAh, of course.
Actually, where did JAVA_CMD come from? Compatibility with some existing tools? Not easy to find online.#2022-08-0814:57Alex Miller (Clojure team)didn't come from anywhere#2022-08-0814:58Alex Miller (Clojure team)I think maybe lein uses that too ?#2022-08-0814:58Alex Miller (Clojure team)it's not a standard thing or anything (like JAVA_HOME is)#2022-08-0815:05lreadIf you are using sdkman you might want to have a look at its https://sdkman.io/usage#env#2022-08-0815:23practicalli-johnWe use https://www.jenv.be/ to manage versions of the Java runtime.
I would not expect Clojure CLI or Leiningen to care about and certainly not manage the version of it's underlying host.
The only reason I've needed to use a tool to manage Java versions is for bugs in driver libraries (usually the more niche libraries) or really old projects with less maintained libraries that are not worth investing time to make better#2022-08-0815:40Drew VerleeI think i understand. I'll need to set my version of java when ever i enter the project, possibly using JAVA_HOME. @U05254DQM i used jenv to set the local java version in the project, but when emacs starts a repl via cider-jack-in, it seems to sue the global version of java.
Maybe i just need to give up on having emacs start the Nrepl, it was nice because then it could keep the nrepl client and server in sync, but i guess i could just use the RELEASE version in my project deps?#2022-08-0815:45Drew Verleethough maybe i'm not setting JAVA_HOME... let me check 👀. it seems like i am setting it with something like java_home.#2022-08-0815:48Drew Verleeactually maybe the practical solution is to just update java 🚀 java#2022-08-0816:23practicalli-johnUpdating to the latest LTS version of Java (17) means you can forget about Java versions for a few years for the majority of Clojure projects.
I wouldn't set JAVA_HOME anywhere, it probably interferes with the jenv tool (and similar tools). There aren't many guarantees that environment variable would be used.
Unless an OS path or environment variable related to the Java excecutable location is specifically set in Emacs e.g .spacemacs.env then Cider jack-in should use the same Java version as running Clojure or lein on the command line. Cider jack-in creates the Clojure or lein command and runs it in the context of the current project.#2022-08-0816:31practicalli-johnTo use Cider connect, I run a repl process using one of the alias from my https://github.com/practicalli/clojure-deps-edn user level config, which I can then use for all projects and minimises any version maintenance.
I have a few variations, e.g :repl/rebel to run a rich ui in the terminal along with the repl process and nrepl or :repl/headless to run a a repl process and nrepl without a cli
I also have aliases to do hotloading and reloading namespaces that I can add to the mix when needed.#2022-08-0817:56Drew VerleeThat's a good point, i didn't check spacemacs.env. i could have Java home set there, which could be leading to my confusion.
Thanks a lot for the help.#2022-08-0818:34dergutemoritz@U0DJ4T5U1 If you're looking for a more general solution, I recommend stopping by in #nix. With it, you gain a way of reliably declaring external dependencies like Java or the Clojure CLI (basically like jenv but for any package). Combined with direnv and emacs-direnv this even works transparently with cider-jack-in.#2022-08-0818:34dergutemoritzHowever, it comes with a bit of a steep learning curve 🙂#2022-08-0818:35dergutemoritzAt least once you want to move beyond these basic features#2022-08-0823:02Cora (she/her)(or sdkman jabba asdf-java) w/ direnv is magical#2022-08-0902:10Drew VerleeNow, how do i switch between my package managers?#2022-08-0910:19dergutemoritzIf you invest in learning Nix, you pretty much won't have to anymore 😄 As mentioned above, it's not specific to any one language ecosystem but a general solution for declaratively managing packages both on the system and on the project level.#2022-08-0907:32mpenetis there a way the tagged litterals support in tools.deps deps.edn reading could be opened to customization somehow? It's a bit a trick question as it would require to allow to express where to find the functions behind these#2022-08-0907:33mpenetwas it ever considered even#2022-08-0907:35mpenetI know I asked this before but I never really got an answer. I am dreaming of having something similar to what aero provides for deps.edn files (literals for including other edn files in locations, pointing to other key values in the file, basically aero #include #ref and #merge).#2022-08-0907:35mpenetshort of forking tools.deps to add support for this directly this is not currently possible#2022-08-0907:51mpenetit's worth noting aero has 0 dependencies, so the price would be small (vendoring these few litterals is also quite easy)#2022-08-0907:59nivekuilwhen I tried aero I didn't really get the benefits of having this custom clojure-like DSL and it was annoying how it behaved (iirc with #merge and #ref combined). I would rather just write clojure that generates edn#2022-08-0908:00nivekuilnot sure what you need it for but could you generate what you need from tools.build?#2022-08-0908:14mpenetI currently do that, write clojure that generates edn, but it's cumbersome. We also wrote libraries to make this easier for mono-repos/multi-modules setups.#2022-08-0908:17mpenetcomposing edn files and enabling more dynamism is powerful, we use that a lot at work. That's one of the strengths of edn to allow these kind of things from the user#2022-08-0908:18mpenetwith stuff like #ref #include & #merge you'd basically open the door to complex monorepo like setups (dependencies version sharing, alias sharing etc) with 0 file generation and no alias juggling at invocation time. And on top of being quite lightweight, that'd purely be opt-in, if you don't care about it you don't have to use it.#2022-08-1213:06grzmIs there a way to reference an alias that includes a :main-opts but does not actually call the -main when referenced? Similar to how you can use -X with an alias that includes an exec-fn but include it with -A and it won't call the function. Example in thread 🧵#2022-08-1213:08grzm% cat deps.edn
{:aliases
{:main {:extra-paths ["src"]
:main-opts ["-m" "com.grzm.ex.main"]
:exec-fn com.grzm.ex.main/exec}
:exec {:extra-paths ["src"]
:exec-fn com.grzm.ex.main/exec}}}
% cat src/com/grzm/ex/main.clj
(ns com.grzm.ex.main)
(defn exec [_]
(println "Hello, exec!"))
(defn -main [& _args]
(println "Hello, main!"))
% clj -X:exec
Hello, exec!
Called exec-fn
% clj -A:exec
Clojure 1.10.3
user=>
Opened a repl with the appropriate classpath. Didn’t call exec-fn.
% clj -M:main
Hello, main!
Called -main.
% clj -A:main
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
Hello, main!
I’d like this last to open a repl, rather than call the -main Is there an alternative incantation that would do this that I’m missing?#2022-08-1213:16Alex Miller (Clojure team)No, separate into two aliases#2022-08-1213:29grzmCool. Thanks, @U064X3EF3!#2022-08-1213:32grzm(And thanks for calc-trace 🙂)#2022-08-1511:49arohnerWhat is the rationale for not supporting :exclusions at the top level of deps.edn? Is that a missing feature, or intentional? Sometimes I’ll want to completely replace a dep with internal code, and it’s not easy because I have to :exclude it from each path. It’s also risky because any future dependency could reintroduce a dependency on the library#2022-08-1511:52p-himik> completely replace a dep with internal code
Just adding the same namespaces to one of the dirs in your :paths should do that, no?#2022-08-1511:54arohnerthe dep still gets resolved and downloaded, and might appear ahead of src on the classpath#2022-08-1512:02p-himik> might appear ahead of src on the classpath
Oh? How?#2022-08-1512:02arohnerhttps://github.com/griffinbank/rules_clojure#2022-08-1512:05p-himikNot sure what I'm looking at and why. If you're using tools.deps.alpha, the content of :paths should appear before the content of :deps, as far as I know.#2022-08-1512:16Alex Miller (Clojure team):paths will appear before deps on the classpath#2022-08-1512:20Alex Miller (Clojure team)but to the original question, just haven't gotten around to thinking about it properly. I actually don't think we have a ticket for it if you wanted to add a request at https://ask.clojure.org#2022-08-1512:23arohnerThe other reason to want to exclude globally is some java deps do magic if they’re on the classpath at all, e.g. slf4j injection#2022-08-1512:24arohnerrules_clojure compiles very differently: each library is compiled separately, with their minimal classpath. In the case of 3rd party deps, that might not even include src at all#2022-08-1516:38zaneCross-posting this here in case anyone has ideas. https://clojurians.slack.com/archives/C03S1L9DN/p1660343169880939#2022-08-1517:03Alex Miller (Clojure team)prep-lib is a way for a git dep to prepare a lib to be on the classpath, but I think what you're after is not about classpaths#2022-08-1916:00DerekEncountering one of those nasty CRC errors when creating an uberjar with build-clj. Is there anyway to quickly identify which JAR is malformed? Details in thread:#2022-08-1916:01DerekExecution error (ZipException) at java.util.zip.ZipInputStream/readEnd (ZipInputStream.java:409).
invalid entry CRC (expected 0x843c9fb1 but got 0xee027fb2)
#2022-08-1916:01Derek{:clojure.main/message
"Execution error (ZipException) at java.util.zip.ZipInputStream/readEnd (ZipInputStream.java:409).\ninvalid entry CRC (expected 0x843c9fb1 but got 0xee027fb2)\n",
:clojure.main/triage
{:clojure.error/class java.util.zip.ZipException,
:clojure.error/line 409,
:clojure.error/cause
"invalid entry CRC (expected 0x843c9fb1 but got 0xee027fb2)",
:clojure.error/symbol java.util.zip.ZipInputStream/readEnd,
:clojure.error/source "ZipInputStream.java",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.util.zip.ZipException,
:message
"invalid entry CRC (expected 0x843c9fb1 but got 0xee027fb2)",
:at
[java.util.zip.ZipInputStream readEnd "ZipInputStream.java" 409]}],
:trace
[[java.util.zip.ZipInputStream readEnd "ZipInputStream.java" 409]
[java.util.zip.ZipInputStream read "ZipInputStream.java" 198]
[java.util.jar.JarInputStream read "JarInputStream.java" 194]
[java.util.zip.ZipInputStream closeEntry "ZipInputStream.java" 142]
[java.util.zip.ZipInputStream
getNextEntry
"ZipInputStream.java"
120]
[java.util.jar.JarInputStream
getNextEntry
"JarInputStream.java"
129]
[java.util.jar.JarInputStream
getNextJarEntry
"JarInputStream.java"
166]
[clojure.tools.build.tasks.uber$explode invokeStatic "uber.clj" 150]
[clojure.tools.build.tasks.uber$explode invoke "uber.clj" 141]
[clojure.tools.build.tasks.uber$uber$fn__3529$fn__3533
invoke
"uber.clj"
257]
[clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
[clojure.core$reduce invokeStatic "core.clj" 6885]
[clojure.core$reduce invoke "core.clj" 6868]
[clojure.tools.build.tasks.uber$uber$fn__3529 invoke "uber.clj" 256]
[clojure.core.protocols$iter_reduce invokeStatic "protocols.clj" 49]
[clojure.core.protocols$fn__8230 invokeStatic "protocols.clj" 75]
[clojure.core.protocols$fn__8230 invoke "protocols.clj" 75]
[clojure.core.protocols$fn__8178$G__8173__8191
invoke
"protocols.clj"
13]
[clojure.core$reduce invokeStatic "core.clj" 6886]
[clojure.core$reduce invoke "core.clj" 6868]
[clojure.tools.build.tasks.uber$uber invokeStatic "uber.clj" 254]
[clojure.tools.build.tasks.uber$uber invoke "uber.clj" 241]
[clojure.lang.Var invoke "Var.java" 384]
[clojure.tools.build.api$uber invokeStatic "api.clj" 480]
[clojure.tools.build.api$uber invoke "api.clj" 408]
[org.corfield.build$uber invokeStatic "build.clj" 250]
[org.corfield.build$uber invoke "build.clj" 208]
[clojure.lang.AFn applyToHelper "AFn.java" 154]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$apply invoke "core.clj" 662]
[clojure.run.exec$exec invokeStatic "exec.clj" 48]
[clojure.run.exec$exec doInvoke "exec.clj" 39]
[clojure.lang.RestFn invoke "RestFn.java" 423]
[clojure.run.exec$_main$fn__205 invoke "exec.clj" 180]
[clojure.run.exec$_main invokeStatic "exec.clj" 176]
[clojure.run.exec$_main doInvoke "exec.clj" 139]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.main$main_opt invokeStatic "main.clj" 514]
[clojure.main$main_opt invoke "main.clj" 510]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause "invalid entry CRC (expected 0x843c9fb1 but got 0xee027fb2)"}}#2022-08-1916:18DerekI copied the project’s deps.edn file into a pretty bare repository and binary searched it, but it would be nice if the error had context? Maybe there’s a verbose setting I’m missing#2022-08-1916:19Alex Miller (Clojure team)I don't think there is a way to quickly do so right now#2022-08-1916:19DerekThanks Alex#2022-08-1916:20Alex Miller (Clojure team)but you could clone tools.build, hack whatever debug you want in and point to it instead using org.clojure/tools.build {:local/root "../mybuild"} etc#2022-08-1916:20DerekTrue#2022-08-1916:21DerekI was able to identify the jar using a bare repo approach (with the problematic deps.edn file) and luckily there’s a more recent version of the package I’m exploring#2022-08-1916:02Drew VerleeIs there a way to get a list of all installed tools from the top level clj? or search from there for existing tools?#2022-08-1916:17Alex Miller (Clojure team)clj -Ttools list
#2022-08-2003:45Drew VerleeTy alex!.#2022-08-1923:14hlship:default-deps provides a set of default coordinate versions to use if no coordinate is specified. The default deps can be used across a set of shared projects to act as a dependency management system:
#2022-08-1923:14hlshipWhat does that mean in practice? How do you not specify a coordinate, and how would this apply across a set of shared projects?#2022-08-1923:18Alex Miller (Clojure team)It is possible to specify a nil coordinate that will be filled by a default dep in an alias, but generally I consider this an unfinished part of tools.deps. The intent was in the vein of Maven/lein managed deps, but I've been thinking off and on about a different direction to go with this which might overlap with other things (like bill of materials poms)#2022-08-2214:51Darin Douglassooo the rare BOM POM#2022-08-1923:16hlshipI guess you can specify nil for the coordinates, but I still don't see how that would apply across shared projects.#2022-08-2021:23grzmI'm trying to use clojure.tools.deps to include a maven dependency with a definition like:
<dependency>
<groupId>io.envoyproxy.controlplane</groupId>
<artifactId>java-control-plane</artifactId>
<version>0.1.32</version>
<type>pom</type>
</dependency>
That <type>pom</type> is tripping me up. Is there a way to reference this type of dependency in deps.edn ? (https://search.maven.org/artifact/io.envoyproxy.controlplane/java-control-plane/0.1.32/pom for a direct link)#2022-08-2021:40grzmLooks like the answer is :extension "pom"
clj -Sdeps '{:deps {io.envoyproxy.controlplane/java-control-plane {:mvn/version "0.1.32", :extension "pom"}}}'#2022-08-2021:42grzm(Not sure yet if that's gonna do what I hope it does…, but we'll see)#2022-08-2021:57grzmLooks like what I really wanted was one of the modules in that pom: io.envoyproxy.controlplane/api {:mvn/version "0.1.32"}#2022-08-2022:16seancorfieldI think the pom type is what's called a BOM -- Bill of Materials -- and that's not supported by t.d.a. yet so you have to explicitly include the specific dependencies from the BOM directly.#2022-08-2022:20grzmMakes sense. I really only want that module anyway. Circuitous way to find it, but I did.#2022-08-2022:48Alex Miller (Clojure team)confirming what Sean said - BOM / pom deps not supported#2022-08-2223:44hlshipTIL that :jvm-opts can be specified in multiple aliases, and the results are concatinated. Let's see if that's in the docs ... yep: If multiple maps with these keys are activated, :jvm-opts concatenate.#2022-08-2407:56octahedrionwhen using a library which has :deps/prep-lib in its deps.edn along with a build.clj to compile that lib's Java classes, if those classes aren't compiled then I get the expected error Error building classpath. The following libs must be prepared before use: [that/library] . However, when I then do clj -X:deps prep I don't get Prepping io.github.puredanger/cool-lib in /Users/me/demo/needs-prep , I get Error building classpath. The following libs must be prepared before use:... again. The library with the :deps/prep-lib is being used by the other project via :local/root . There must be an error in my code but I can't see it. Here's my deps.edn code:
:deps/prep-lib
{:ensure "target/classes"
:alias :build
:fn compile}
:aliases
{:build {:deps {io.github.clojure/tools.build {:git/tag "v0.8.3" :git/sha "0d20256"}}
:ns-default build}
and in my build.clj also at the root level I have:
(defn compile [_]
(b/javac {:src-dirs ["java"]
:class-dir class-dir
:basis basis
:javac-opts ["-source" "8" "-target" "8"]}))
clj -T:build compile works#2022-08-2411:57Alex Miller (Clojure team)At a glance, that seems like a valid setup. The prep function does have a debug mode if you pass :log :debug at the end of the command line#2022-08-2411:57Alex Miller (Clojure team)Also what's your clj -version#2022-08-2412:02octahedrionClojure CLI version 1.11.1.1155#2022-08-2414:40seancorfield@U0CKDHF4L you say your build.clj but what about the project you're preparing? That is where the :build compile will be run, not in your project.#2022-08-2414:54octahedrionthe build.clj is in the project A that needs preparing not in project B that has A as a dependency#2022-08-2415:04seancorfieldOk just checking since you were talking about "my" build file at the root level...#2022-08-2415:05octahedrionI'm developing both projects#2022-08-2415:05octahedrionI think there's something very simple and obvious I'm missing#2022-08-2415:10octahedrionlet me see if I understand the process:
:deps/prep-lib is used by the prep task invoked by clj -X:deps prep, its :alias is then looked up in deps.edn 's :aliasesmap, so a :build key in there having a :ns-default build would mean it will look for a namespace build which means it will load build.clj from the root and call the :deps/prep-lib's :fn key compile in there#2022-08-2415:11Alex Miller (Clojure team)yes, that setup you had all looked correct. what's the output with the debug log setting I posted above#2022-08-2415:11octahedrionah - you mean like clj -X:deps prep :log :debug ?#2022-08-2415:11Alex Miller (Clojure team)yes#2022-08-2415:12octahedrionsame thing#2022-08-2415:13Alex Miller (Clojure team)can you actually paste the full output#2022-08-2415:13octahedrionError building classpath. The following libs must be prepared before use: [that/library]#2022-08-2415:14Alex Miller (Clojure team)can you try adding :force true to that command too#2022-08-2415:14octahedrionclj -X:deps prep :log :debug :force true
Error building classpath. The following libs must be prepared before use: [that/library]
#2022-08-2415:21Alex Miller (Clojure team)hrm#2022-08-2415:24Alex Miller (Clojure team)so you can change to the that/library directory and do clj -T:build compile successfully?#2022-08-2415:26octahedrionyes#2022-08-2415:26octahedrionthat's how I've been using it as a dependency while I figure this one out#2022-08-2415:27Alex Miller (Clojure team)ok, let me think about this for a few#2022-08-2415:31octahedrionif I rename the :deps/prep-lib key to something else then I get
clj -X:deps prep :log :debug :force true
Unqualified function can't be resolved: prep
which confirms it's using that key in the dependency's deps.edn#2022-08-2415:32Alex Miller (Clojure team)that's not what that error means#2022-08-2415:33Alex Miller (Clojure team)I'm wondering if you have something weird with your :deps alias#2022-08-2415:33Alex Miller (Clojure team)can you do clj -Spath -A:deps#2022-08-2416:32seancorfieldI tried to repro and it worked as expected for me (once I spelled the :alias value correctly -- ahem!) so I'm also thinking there might be something odd with your :deps alias -- do you have :deps in your ~/.clojure/deps.edn file perhaps? @U0CKDHF4L I mistakenly did that in the dot-clojure repo I published and it broke the root (system) :deps alias behavior so I removed it -- but a few people who based their dot-clojure file(s) on mine may stay have that in there.#2022-08-2507:58octahedrionI tried creating two projects A and B where B is the project A depends on and is its only dependency and B has no dependencies and a single java class to compile along with the same :deps/prep-liband build.clj and I get the same error.
@U04V70XH6 I checked my ~/.clojure/deps.edn which I hadn't edited as I've recently moved to a different machine - well, it was the default example one I think, but odd - datawise it was {{}}but with comments ;; The deps.edn file describes the information needed... . When I changed it to {} , clj -X:deps prep worked! Which makes sense.
Anyway all's well that ends well and thank you @U064X3EF3 and @U04V70XH6 for all your help! facepalm#2022-08-2514:38seancorfieldGlad you got it working!#2022-08-2522:06twashingi. I have a tools.deps project
ii. that has a git dependency.
iii. tools.deps has previously downloaded the dependency foo/bar.
iv. And it’s in my local /Users/foo/.gitlibs/_repos/ssh/foo.github.com/foo/bar
But trying to run a repl now, gives me the error. What am I missing?
Error building classpath. Unable to fetch /Users/foo/.gitlibs/_repos/ssh/foo.github.com/foo/bar
ssh: Could not resolve hostname : nodename nor servname provided, or not known
fatal: Could not read from remote repository.
#2022-08-2522:18hiredmandns error resolution is failing for the name#2022-08-2522:50twashingI was previously able to run the same command, without access to that domain name.
What’s strange is that the message says... Unable to fetch /Users/foo/.gitlibs/_repos/ssh/foo.github.com/foo/bar.
But that directory is there.
Looking at the source file... https://github.com/clojure/tools.gitlibs/blob/master/src/main/clojure/clojure/tools/gitlibs/impl.clj#L91-L92
it is shelling out to git --git-dir...
What stumps me is, if it worked previously (fetch from local git repo), what could cause it to break :thinking_face:#2022-08-2523:07hiredmanare you saying you never had access to https://foo.github.com or whatever the real domain name is?#2022-08-2523:09hiredmantools.deps works by building a cache of deps -> classpath, which means the first time you run clj it launches a java process that does a lot of stuff, then it figures out a classpath, and stores that in a file named by a hash of the deps.edn file (plus some other stuff)#2022-08-2523:09hiredmanthen it launches your java process with your classpath#2022-08-2523:10hiredmanthen on subsequent runs of clj if your deps.edn hasn't changed, it only launches your java process with your classpath#2022-08-2523:12hiredmanif you had access to https://foo.github.com sometime in the past, got on the code, then lost access, but nothing invalidated the cache, then you would never know#2022-08-2523:12twashingOk then I’ll ask the question in a different way.
• Can tools.deps download git deps in machine1:~/.gitlibs
• Then can I directly rsync that to machine2:~/.gitlibs?#2022-08-2523:13twashing@U0NCTKEV8 From your breakdown, ~/.gitlibs only works on the machine it was downloaded on.#2022-08-2523:14hiredmanyes, but sometimes tools.deps needs to access the origin of the git repo#2022-08-2523:14hiredmanspecifically, I believe, if you are using tags instead of full shas#2022-08-2523:15hiredmanbecause a tag can change, so it tries to fetch tags#2022-08-2523:15twashingHmm… :thinking_face: So bottom line, ~/.gitlibs is not portable in the way ~/.m2/repository is.#2022-08-2523:15hiredman(it might unconditionally fetch even if you aren't using tags if it gets to that point, I forget)#2022-08-2523:17twashingDo you know of any tools that can give me that portability? From the advertisement, git --git-dir... should point to a local directory.#2022-08-2523:18hiredmanhttps://clojure.atlassian.net/browse/TDEPS-223 is maybe this, or at least related to it#2022-08-2523:19hiredmansure, the git call does point to local directory, but what is failing is the call does a "fetch" to pull the tags from the remote repo, which you apparently don't have access to from machine2#2022-08-2717:33armedHello, I’m a bit confused. I’ve a ~/.clojure/deps.edn with a few aliases. If I cd to ~/.clojure and run clj -A:portal -Stree it finds :portal alias and prints all dependencies. However in all other directories clj does not find :portal alias and prints WARNING: Specified aliases are undeclared and are not being used: [:portal]. What I’m missing here? Using latest clojure version 1.11.1.1155 on macos.#2022-08-2717:48armedOk, clojure cli lookups for $CLJ_CONFIG first, then $XDG_CONFIG_HOME and fallbacks to $HOME/.clojure. Recently I defined XDG_CONFIG_HOME for another tool to work. So that broke clojure’s search path. Fixed by defining $CLJ_CONFIG#2022-08-2722:05seancorfieldOr move that to ~/.config/clojure/deps.edn so it's XDG-compliant?#2022-08-2910:17practicalli-johnI agree with Sean that .config/clojure is a more discoverable place
I also moved all my Clojure dev tooling to XDG specification
https://practical.li/blog/posts/adopt-FreeDesktop.org-XDG-standard-for-configuration-files/#2022-08-2717:53ChaseIn my -main function I am using print and was seeing the expected output in the terminal when running clj -M -m practice.core but not when building and running the uberjar with java -jar ... . In looking at the print docs I figured I needed to add a call to flush and sure enough that fixed it. I had read that using the clj -M -m method of running a program is equivalent to using java -jar so I'm curious if there are other small gotchas like this.#2022-08-2718:14hiredmanIt is just not true that they are equivalent#2022-08-2718:18ChaseThat's what I figured. Tbf, I think that conversation was more in the line of starting a backend server. Maybe they meant in that specific instance you could use either or.#2022-08-2720:45Alex Miller (Clojure team)You would need to share more of your code and uberjar to really say more#2022-08-2722:08ChaseEh, it's just my little practice repo. Nothing really going on there. More a curiosity than a problem. https://github.com/Chase-Lambert/clojure-practice/blob/main/src/practice/core.clj#2022-08-2723:22Alex Miller (Clojure team)when build the uberjar, are you setting practice.core as the Main-Class ?#2022-08-2800:33ChaseYep. The code works. I was just curious about the difference between clj -M -m ... and java -jar ... as the former worked without calling (flush) and the latter did not. I am wondering what other differences there are because they are running the same function.#2022-08-2800:37hiredmanhttps://github.com/clojure/clojure/blob/master/src/clj/clojure/main.clj#L616#2022-08-2800:37hiredman-M goes through there, which has a call to flush in a finally#2022-08-2800:38ChaseAhh, yep, that explains it. Cool. Thanks for sharing that#2022-08-2800:38hiredmanIt also sets up a bunch of bindings which aot'ing your own entry point won't have#2022-08-2800:47ChaseIt's cool how readable the clojure source is. I imagined a ton of wild optimizations but even though some of this is over my head it seems quite approachable to me.#2022-08-2804:23Alex Miller (Clojure team)The repl is really just calls to read eval print in a loop. So not much magic#2022-08-2801:52zaneThere’s no way to set the .m2 location via an environment variable, is there? Just :mvn/local-repo?#2022-08-2804:23Alex Miller (Clojure team)Correct#2022-08-2807:06Ben SlessIt's still not possible to set via alias, right?#2022-08-2910:25practicalli-johnBen, unless something has changed recently then :mvn/local-repo is a top-level key and not usable in an alias.
The value of this key could be set on the command line when running a REPL or other tasks
clojure -Sdeps '{:mvn/local-repo ".cache/temp-deps"}'#2022-08-2910:26Ben SlessThat's what I've been doing until now, I just recall some time ago discussing it with Alex. Wondered if things changed since#2022-08-3015:57Drew Verleecan you tell deps to use a different git commit of a local git repo/project then the one the project is currently set to?#2022-08-3015:58Alex Miller (Clojure team)if you're using it as :local/root then no#2022-08-3015:58Alex Miller (Clojure team)if you're using it as a git dep, then you can pick any commit you like#2022-08-3015:59Alex Miller (Clojure team)recent versions of the CLI do also support using local file-based git urls#2022-08-3016:06Alex Miller (Clojure team)for example, something like this works:
{:deps
{foo/bar {:git/url "../../code/tools.deps.alpha/"
:git/sha "b2096587b8c9ead9f47daa6258d3a4fddb18ebde"}}}
even if the local path is at some newer sha#2022-08-3016:07Drew Verleeawesome! i should have looked closer at the docs. I'm considering that it might be worth cloneing more projects so it's easier to work with them.#2022-08-3016:07Alex Miller (Clojure team)it will "check out" that file based project into the gitlibs and create a working tree at the requested sha#2022-08-3117:16Damian KoncewiczHello.
I have some problem with
clojure -Ttools install com.github.seancorfield/clj-new '{:git/tag "v1.2.399"}' :as clj-new
it showing me this error:
Library com.github.seancorfield/clj-new has invalid tag: v1.2.399
How can I fix that?
Thank you for your help#2022-08-3117:18seancorfieldYou're on Windows? Using Powershell or cmd?#2022-08-3117:19Damian KoncewiczI am on windows using Powershel#2022-08-3117:19seancorfieldSee https://clojure.org/reference/deps_and_cli#quoting#2022-08-3117:20Damian Koncewiczthank You i try right away 🙂#2022-08-3117:20seancorfieldO/S usage is about 90% Mac/Linux, 5% Windows/WSL2 (Linux), and 5% Windows/Powershell-or-cmd so pretty much all the tutorials and readmes out there are written for Linux since that covers 95% of usage.#2022-08-3117:23Damian Koncewicz@U04V70XH6 it is working. Thank You very much!! 🙂#2022-09-0114:29practicalli-johnIs there much benefit (or disadvantage) of including the .cpcache directory when copying over the project to build inside a docker container?
.cpcache is not currently included in the Git repository (and it doesnt feel like it should be), so its not used in our CI, so this would only be local (if its of use at all). Any thoughts or experiences welcome#2022-09-0114:35dpsuttoninside of there are lots of references to /Users/dan/.m2/… I suspect all of these paths are either (a) thrown away inside of the docker container so they don’t help at all, or (b) it will be an explicit error.#2022-09-0116:29Alex Miller (Clojure team)you'll want to rebuild your .cpcache in the container to make the paths right, so better not to include. (can use clojure -P whatever to "prepare" the classpath - will include grabbing maven libs, making the classpath cache etc#2022-09-0117:17practicalli-johnAh, that makes lots of sense. thank you.#2022-09-0304:47jasonjcknThis questions probably been asked a hundred times, I did read through the chat history a bit, IF you have java sources as part of your Clojure project, is the recommended setup is to move them to a different folder and use :deps/prep-lib ? This still seems to leave a lot to be desired, like if I change a java source it won’t automatically recompile the affected classes, any other solutions here that behave like lein?#2022-09-0305:03seancorfieldprep-lib is intended to be a one-off step for libraries so that consumers can use them. It is not intended to be the normal way for you to compile Java code in a project you work on. Write a build.clj with a task that does the Java compilation -- see the tools.build documentation for guidance. HTH @U0J3J79FE#2022-09-0305:06jasonjcknTo get the java source files compiled before a cider-jack-in, is there a lein-like prep-tasks, or I should change the command cider uses to start the repl?#2022-09-0305:06seancorfieldYou can then call that build.clj function from any other functions in build.clj that might need to leverage the compiled Java classes -- how exactly you do that, is up to you.
At work, we have a prep... function in our build.clj that can be invoked manually as needed and is automatically called from a couple of other functions to ensure that the "compile" step always happens in our CI pipeline, for example.#2022-09-0422:06didibusHow do people publish both a readme with the correct git deps which needs the sha and do the commit? Do you just make a release and push a README update separately after?#2022-09-0422:11lreadYa it’s a bit of a catch-22. You want your docs that describe your release to be included in your release but they really can’t.#2022-09-0422:16didibusYa, I guess I'll leave the sha empty for the release, and then push one more commit which just adds the sha back to the doc#2022-09-0422:27seancorfieldYup, that's what I do: update everything -- code, docs, build, etc -- and then cut the release, and then update the README (and CHANGELOG) to have the sha of that release. With major.minor.commits, at least you can figure out what the commits part is going to be in advance.#2022-09-0422:29didibusHow do you figure out commit in advance?#2022-09-0422:30seancorfieldgit rev-list --count HEAD tells you what it is now: so you know it's going to be one more when you make that final commit to update the version number immediately prior to release.#2022-09-0422:30seancorfieldtools.build has a function for getting this number so you can do a bunch of stuff programmatically in build.clj as well if you want.#2022-09-0422:35didibusCool, ah ya I think I remember seing it: (b/git-count-revs nil) I think#2022-09-0423:51lreadBut you are kinda outta luck for the :git/sha in your docs.
I mean like: io.github.cognitect-labs/test-runner {:git/tag "v0.5.1" :git/sha "dfb30dd"}
(related: gonna get around to that cljdoc/<version> git tag sometime soon @U04V70XH6!)#2022-09-0423:53seancorfieldYeah, I put :git/tag "v1.2.3" :git/sha "..." in when I make a release and then update it with the short sha immediately afterward.#2022-09-0423:54seancorfieldSee https://github.com/seancorfield/build-clj/commit/7ac1f8dee188894cee4199aaf2fbe7c6708a8680 and then https://github.com/seancorfield/build-clj/commit/7081997c3a21f50e96ee98a20f3bd97be9064a6d -- but it's kind of annoying.#2022-09-0501:56skylizeWhile git still uses SHA1 instead of SHA256 it should be viable to build a tool that predicts the next hash. Some ideas on it here https://stackoverflow.com/questions/16951299/is-it-possible-to-precompute-a-git-commit-hash-such-that-it-can-be-placed-in-the#2022-09-0501:57skylize... or more specifically, viable to predict the short hash, which is all we care about here.#2022-09-0502:01seancorfieldInteresting. Thank you @U90R0EPHA!#2022-09-0500:08didibusWhen using git libs, what should my lib name be? With Maven it was kind of obvious it was group/name, but now say my namespaces start with com.foo but I publish git deps on github, so is the git lib com.github.foo ?
If I put that as the git lib, it will figure out the URL, but what happens if someone else also depends on com.foo {:git/url "..." ...} ?
Will it now pull the same lib twice?
So basically, is the lib name in a dependency declaration just whatever someone puts? So they could even by accident typo it and it would work if a git/url is provided? And suddenly they pulled in the same lib twice with potentially different shas ?#2022-09-0500:10seancorfieldio.github.<username>/<repo-name> is the coordinate.#2022-09-0500:11seancorfieldThe nses are unrelated and can be anything.#2022-09-0500:11seancorfieldNow, you could tell people to use a different libname -- but then they'd have to specify the :git/url in full.#2022-09-0500:14seancorfieldLike:
didibus/my-lib {:git/url "" :git/sha "..."}
vs
io.github.whoever/whatever {:git/sha "..."}
Does that help @U0K064KQV?#2022-09-0500:14didibusThe thing is, it's not the coordinate, its whatever you want. I tried it, so I can pull multiple different version of the same lib:
com.github.xadecimal/riddley {:git/tag "0.2.2" :git/sha "d1ac17e"}
com.wtv/riddley {:git/url ""
:git/tag "0.2.2"
:git/sha "d1ac17e"}
com.lololol/riddley {:git/url ""
:git/tag "0.2.1"
:git/sha "c905720"}
com.xadecimal/riddley {:git/url ""
:git/tag "0.2.1"
:git/sha "c905720"}#2022-09-0500:15seancorfieldIt's convention to use io.github.<username>/<repo-name> so you avoid conflicts 🙂#2022-09-0500:15didibusBut I own com.xadecimal and that's my namespace prefix#2022-09-0500:15seancorfieldWith :local/root, the libname can also be absolutely anything.#2022-09-0500:15didibusSo now it seems at risk of conflicts#2022-09-0500:16seancorfieldI don't understand.#2022-09-0500:16didibusI don't know, this seems like a potential disaster slowly in the making 😛#2022-09-0500:16seancorfieldWhat's your GH username?#2022-09-0500:17didibus(ns com.xadecimal.riddley)
;; my code
http://www.xadecimal.com <-- my website
I publish to git and clojars
com.xadecimal <-- My clojar groupId#2022-09-0500:19didibusSo now in github, where I publish as a git deps, ideally it should be:
com.xadecimal/riddley {:git/url ""}
But you can also write it like:
com.github.xadecimal/riddley {...}
Except now this is treated as two separate lib which will just get random order for which one it uses at runtime#2022-09-0500:19seancorfieldIf I was using your lib from GH, I'd use:
io.github.xadecimal/riddley {:git/sha "d1ac17e667d0159d65a7b6d3fc5893bc0402ff7d"}
so that it deduces the URL.#2022-09-0500:19didibusIsn't that just a 3rd way to import the lib?#2022-09-0500:20didibusSo what will happen is in a chain of transitive deps, if a lib depends on a different version of my lib but just happen to put a different name in their dependencies it won't conflict#2022-09-0500:20didibusSo we're back at tools.deps not detecting and resolving conflicts between libs#2022-09-0500:20seancorfieldWell, that's the danger if you publish to Clojars and to GH.#2022-09-0500:21didibusIt's the danger if I publish only to GH as well#2022-09-0500:21seancorfieldBut, yeah, for :local/root and git deps, the lib name can be "anything" so you're responsible for conflicts. But bear in mind that something published to Clojars cannot depend on a git dep#2022-09-0500:23didibusRight, which is where I'd expect tools.deps to manage conflicts across. Without it, I feel it's pretty precarious, like I'm in the JS ecosystem 😛
I think it could introduce a mandatory lib name inside deps.edn or something and rely on that for conflicts. Now it just relies or whatever name people want to give the lib in their deps#2022-09-0500:23seancorfieldIf you only "publish" via GH deps and encourage folks to use the "standard" (convention) then conflicts are less likely (and t.d.a is more likely to be able to detect them). If you go off-script then...#2022-09-0500:24didibusAs far as I can tell, t.d.a mentions no standard, the doc just mentions the multiple ways to do it#2022-09-0500:25didibusAnd even then, I'm very hesitant to adopt languages with unreliable package managers that can create conflicts, random ordering of deps at runtime are the most costly bugs#2022-09-0500:25didibusAnd solving them are tons of headache#2022-09-0500:25seancorfieldIf you're publishing to Clojars, be consistent and document the git deps using the same coordinate 🙂 So far, the reality has been that this isn't a problem with Clojure.#2022-09-0500:26didibusBecause no one uses git deps yet#2022-09-0500:26didibusI don't know, this seems like a huge disaster in the making#2022-09-0500:27seancorfieldtools.build is GH only so that's clearly not "no one".#2022-09-0500:27didibusIts a build tool that can't conflict not included in your prod app#2022-09-0500:28seancorfieldI think you're imagining a problem that won't come to pass. ¯\(ツ)/¯#2022-09-0500:28didibusNow all it takes is one typo in a popular lib pulling in a common package, and suddenly tons of people are secretly getting two versions of that package, it goes for a while without issue, until they diverge in version enough#2022-09-0500:28seancorfield"popular lib" and "no one" seem mutually exclusive?#2022-09-0500:29didibusNo, eventually more people will move to git deps, so it's just an issue that will creep on the community#2022-09-0500:30seancorfieldThe deps/CLI docs say:
> repository lib - the Clojure CLI uses a convention where the URL does not need to be specified if you use a library name like io.github.yourname/time-lib for the GitHub url#2022-09-0500:30seancorfieldWhere are you seeing it suggest other git approaches?#2022-09-0500:31didibushttps://clojure.org/reference/deps_and_cli#_coord_attributes#2022-09-0500:32seancorfieldJust because it shows both io.* and com.* mapping to the same thing?#2022-09-0500:32didibusI mean, it doesn't mention any preferences, it just says here's many ways to specify the URL#2022-09-0500:33seancorfieldI quoted from the guide above where it mentions the convention. The reference just goes into more detail. I think you're making a mountain out of a molehill.#2022-09-0500:33didibusThe issue is, at least if it used the URL for conflicts... Then at least within git deps it would detect conflicts, though I still wish it also detected them across maven#2022-09-0500:34didibusMaybe I read that different then you, but that just tells me that if I want I can use a different lib name and it avoids me having to specify the URL.
Which I find is even more risky, because now it recommends people to use a different name that can conflict#2022-09-0500:35didibusWhy does T.D.A bother checking the sha? I mean, at this point, its all pretending to detect conflicts anyways#2022-09-0500:36seancorfieldI think you'll find that libraries will be clear and consistent about how to refer to them and folks will use whatever libname is documented in the README...#2022-09-0500:36didibusI'm making a lib myself, and even I had no idea what name to use#2022-09-0500:36seancorfieldYou can't compare a git dep and a Maven dep -- neither is obviously newer than another.#2022-09-0500:36didibusIt just feels weird that the group name in my deps is not the namespace prefix#2022-09-0500:36seancorfieldYou are publishing to Clojars already. You have a group/artifact name already yes?#2022-09-0500:37didibusNo I was planning going only GH for now, but now I'm thinking no GH since its unreliable and only Clojars probably#2022-09-0500:39seancorfieldClojars has group name restrictions so if you are publishing there, you have less leeway. And by now you should be used to Clojure not telling you how to build and distribute projects! 🙂#2022-09-0500:40didibusIt's not that, I cannot trust any dependencies now. Like almost need to have a build step to validate we don't have git deps#2022-09-0500:41seancorfieldI don't know whether libraries will ever really move to GH only since they can't be used in another lib that wants to publish to Maven/Clojars -- and some companies simply don't allow "random" GH libraries to be used right?#2022-09-0500:43seancorfieldIf you want other libraries to use your lib, you pretty have to publish to Clojars. That's been my deciding factor: is it tooling? GH only; is it a reusable library? Clojars only.#2022-09-0500:44didibusI mean, if t.d.a doesn't take reliability and reproducibility of GH deps seriously, probably people won't adopt it. But I think if it did, having internal mirrors of git deps is much easier than Maven, and the publishing story for them is much more convenient. I could see it becoming popular if it adds features to properly detect conflicts and maybe also to detect conflicts against Clojars/Maven, which it very much could if it just introduced a more formal lib name and version as part of deps.edn#2022-09-0500:44didibus> is it tooling? GH only; is it a reusable library? Clojars only.
I guess until t.d.a matures aroud Git deps, I'll have to follow that as well.#2022-09-0500:49seancorfieldThis isn't just a git deps issue. Suppose I fork a library that's published to Clojars and I publish it under different coordinates (but with the same namespaces) -- that's not a detectable conflict.#2022-09-0500:49seancorfieldAnd that's true of Java libraries published to Maven today. The group/artifact is independent of the Java packages.#2022-09-0500:49didibusSome languages actually validate that the namespace prefix all match, I think Maven Central does that as well#2022-09-0500:50seancorfieldNo, it doesn't. I use a bunch of Java libraries from Maven that are totally inconsistent in package names.#2022-09-0500:50didibusHum, ok maybe Maven Central doesn't. I wasn't sure.#2022-09-0500:51didibusBut still, I think that's more acceptable.#2022-09-0500:52seancorfieldMy point is that this isn't a unique problem to t.d.a and git deps and if your concern was a real world problem we'd be running into it already with Java libs and stuff on Clojars.#2022-09-0500:52didibusThe scenario now is just prone to typos, or even people just randomly using a diferent convention, io.github or com.github or groupID with url, etc.#2022-09-0500:53didibusI disagree, it's very different problem. One requires a fork that did not change namespace names, and for libs to depend on the fork and the original#2022-09-0500:53didibusThe other just requires people to declare the deps with a different name#2022-09-0500:53didibusIt's the same lib, not a fork#2022-09-0500:53seancorfieldIf you don't use :git/url then you can't typo the libname -- else t.d.a won't find it.#2022-09-0500:54seancorfieldio.gitflub.xadecimal/riddley isn't going to resolve.#2022-09-0500:56seancorfieldIf you use :git/url then, yeah, you need to be more careful. But I think that's really only likely for regularly-published libs (to Maven/Clojars) that folks want to test against the latest source version -- and they already have a group/artifact so they'll just comment out :mvn/version and add :git/url / :git/sha.#2022-09-0500:59seancorfieldI'm basing this on my experience as an early adopter (and fairly extensive user) of the CLI, t.d.a, and git deps with several projects out there using :git/tag / :git/sha and, at work, relying on a number of tools (and some libraries) via git deps.
Once you've had real world experience doing this, we can compare notes. Otherwise you're just imagining problems that I don't think will happen in the real world.#2022-09-0500:59seancorfield(and this isn't the first time we've had this kind of discussion -- LOL 😆 )#2022-09-0501:00didibusio.github.xadecimal/riddley
com.github.xadecimal/riddley
com.xadecimal/riddley {:git/url "https://github.com/xadecimal/riddley.git"}
are all going to resolve and won't conflict though#2022-09-0501:01seancorfieldWell, choose your poison, document it in your readme and live with it. Users of your library will generally do what you tell them in the README.#2022-09-0501:02seancorfieldIf you think you're going to publish to Clojars as well you need to pick coordinates you can use for both.#2022-09-0501:03seancorfieldClojars lets you use a domain you own or you can also verify a github-style domain (by logging in via GH if your username matches as I recall). So those are your Clojars choices.#2022-09-0501:04seancorfieldIf you put two different group/artifact values in your README, you're just asking for trouble.#2022-09-0501:05didibusHopefully the problem doesn't manifest. But I've seen all possible version issues happen in prod in my short career. They're my most dreaded issues, so I'm sensitive to it 😛#2022-09-0501:05didibusYa, so I think I will go with explicit git url:
com.xadecimal/riddley {:git/url ""
:git/tag "0.2.1"
:git/sha "c905720"}#2022-09-0501:06didibusThis one allows me to change my git hosting in the future without breaking people and still detecting conflicts once I change to say gitlab or some other git hosting
And it can match Clojars or Maven Central coordinates#2022-09-0501:07seancorfieldThere ya go! You looked at the tradeoffs, made a decision, and now you're going to stick with it 🙂#2022-09-0501:07didibusIt should also detect conflicts between git deps and clojars correct? Since they'll have the same coordinate?#2022-09-0501:08seancorfieldIt'll detect that they can't be compared, yes.#2022-09-0501:08didibuslol, well the decision I have not made yet, is if I want to ban git deps from work 😛#2022-09-0501:09seancorfieldThat will ban several tools -- including tools.build and build-clj and deps-new#2022-09-0501:09didibusThis behavior makes me more worried as a user of git deps, as in, someone who might pull in a git deps, what kind of hidden conflicts will I suffer from if I do 😛#2022-09-0501:09didibusYa, might need to accept-list specific build tooling#2022-09-0501:10seancorfieldSo much easier to go to a given GH repo and SHA and see exactly what it depends on tho'...#2022-09-0501:10seancorfieldWith some random JAR on Maven, you can't do that.#2022-09-0501:10seancorfieldYou have to trust that contents are built from something you can trust.#2022-09-0501:11didibusBut I don't understand why equal coordinates can't be treated equal by t.d.a and just conflict?#2022-09-0501:12seancorfieldEqual coordinates => choose the newest version (and throw an error if you can't determine the newest one). That's exactly what it does.#2022-09-0501:13didibusHum... ok so wait, maybe there is no problem then.
You mean that:
io.github.xadecimal/riddley
com.github.xadecimal/riddley
com.xadecimal/riddley {:git/url ""}
garbage/whatever {:git/url ""}
All these actually are treated equal and will be auto-resolved to only pull in the newest version?#2022-09-0501:14seancorfieldNo, coords = group/artifact#2022-09-0501:14didibusI forgot that, when I tested, I thought it won't throw a conflict error, but I forgot t.d.a auto resolve to newest. So I may be worried for nothing#2022-09-0501:14seancorfieldNot url#2022-09-0501:14didibusOh 😞#2022-09-0501:22didibusGenerally, I trust the domain validation done by Maven Central.#2022-09-0501:22didibusAnd the groupId prevents obvious name squatting issues#2022-09-0501:26seancorfieldBut you have no guarantee of what's in a given Maven JAR: just compiled code. You can't guarantee what it was built from. Do you decompile and verify every Maven JAR? With GH deps you know exactly what source code is being used...#2022-09-0501:27seancorfieldAnd with all the git deps, the source is downloaded locally to your CLI process for inspection#2022-09-0501:30didibusYes I agree, there's a lot to like about Git deps, that's why I'm more so disappointed that it introduces other issues. I don't feel it needs too, seems a bit self-inflicted. Like deps.edn could add a :lib key and a :version key, and then t.d.a could use that for conflicts.
Or it could use the actual coordinate, in the sense of where it pulls from, everything pulled from the same place would conflict. Not as good since it can't detect conflicts across sources, but still better.#2022-09-0501:32seancorfieldCan't be in deps.edn in the repo - that would allow people to hijack other libs#2022-09-0501:37didibusI was thinking the combination, like if inside deps.edn you declare a deps on com.foo/bar then in the deps.edn of com.foo/bar it has to say :lib com.foo/bar#2022-09-0501:39didibusThat way, it's not just my README that enforces the coordinate name of my lib, but if I say the lib is meant to have coordinate com.foo/bar that's how people have to declare it.#2022-09-0501:43seancorfieldInteresting idea. You should put that on http://ask.clojure.org! I'd vote for that optional check.#2022-09-0501:45seancorfieldJust to mess with you further, a git dep could have pom.xml and no deps.edn I believe:wink:#2022-09-0502:10didibuslol, I need the Jackie Chan emoji that goes like Why? for that one.
But, I think that would be fine, you can validate that the coordinate used to declare the dependency on it is equal to its groupId/name#2022-09-0502:11seancorfieldHow? I'm talking about a random git repo with a pom and no deps.edn#2022-09-0502:12seancorfield🥋 Jackie Chan?#2022-09-0502:13didibusSo a package with a deps.edn depends on a git does that only has a pom file correct?
And pom files include groupId and name no?
So t.d.a can compare the coordinate I used in my :deps map again what's inside the pom no?#2022-09-0502:14seancorfieldIs that required in a pom file? Does any tooling check that?#2022-09-0502:15didibusThis emoji :rolling_on_the_floor_laughing:#2022-09-0502:16seancorfieldI don't know if aether exposes that when analyzing a pom file but it's an interesting option. Again, add that to http://ask.clojure.org item!👍#2022-09-0502:17didibusI'm not sure, I assumed it was required in a POM, but maybe it's not.
That said, I think what I'm asking could be optional, might have to be anyways for backwards compatibility.
So maybe it can be if pom.xml or deps.edn declare a lib group/name then assert it matches otherwise fallback to current behavior.#2022-09-0503:05didibushttps://ask.clojure.org/index.php/12147/add-validation-for-groupid-name-coordinate-deps-tools-deps#2022-09-0501:17didibusSo to recap my concern:
{io.github.xadecimal/riddley {...}}
{com.github.xadecimal/riddley {...}}
{com.xadecimal/riddley {:git/url "" ...}}
{com.xadecimal/riddley {:mvn/version ...}}
{garbage/whatever {:git/url ""}}
Will all resolve and will not conflict with one another, and be treated as separate libs if I understand, even though they are all the same lib, thus allowing to bring multiple conflicting versions on the classpath and having random order of which one will actually be used in production.
And this is true even in the transitive dependency closure.
Is this just an accepted issue, or are there plans to address this in the roadmap?#2022-09-0501:28hiredmanSame issue exists with maven alone, just takes more effort to trigger, because you have to publish the same artifact under different group and artifact ids#2022-09-0501:29hiredmanWhich definitely happens on clojars (people have a habit of publishing 3rd party jars to clojars under their own group id, so the same library is there under different maven coords)#2022-09-0501:34didibusYes, but at least here, in theory, it is two different libs, albeit with people not taking care to properly namespace their code so it won't conflict.
I guess we don't know the statistical likelihood of this scenario, and shall see if it happens a lot or not. But I also feel t.d.a could address it before we even have to find out, and was curious if it's already planned, or if it's predicted to not be a major issue and won't get addressed.#2022-09-0501:53hiredmanNah, it is the same lib, just published under different names#2022-09-0501:56hiredmanAlso git is a distributed vcs, and tools.deps supports local repos too, so you can go absolutely wild, no reason the artifact id has to be anything like the url of the repo#2022-09-0501:58hiredmanThis is fundamentally the issue in maven, that the maven coordinate names have no relationship to the names of the code in the artifact#2022-09-0501:59hiredmanWhich can express itself in different ways (badly packaged jars that include some dependency in them), the same code packaged slightly different ways and pushed to different maven coords, etc#2022-09-0502:00hiredmanThe difference between this and git deps is who gets to assign the name#2022-09-0502:00didibusIt's a fork no? If it's published by someone else? I mean yes it might be the same code, but in theory it's a fork.
I guess it's philosophical if a fork is the same lib or not.
Anyway, I feel the forking scenario logically would be much less common. Also, for some forks you'd want them to be treated differently, so you can have both the fork and the original on your classpath (for good forks that remembered to rename their packages/namespaces)
But here it's the same lib in every possible way. Same code, same git repo, same author, same publisher, etc.#2022-09-0502:01hiredmanIn maven it is whoever publishes an artifact (which doesn't have to be the person who wrote or packaged it)#2022-09-0502:01hiredmanFor git deps the consumer chooses the name#2022-09-0502:02hiredmanYou can absolutely find the exact same code published on clojars under different artifact names#2022-09-0502:04didibusYes, same code, I said same code and author and publisher and git repo, not or.#2022-09-0502:05didibusAnyways, consumer choosing the name seems at odds with conflict resolution.#2022-09-0502:08hiredmanI think the best thing would be to use the git url as the artifact name directly, instead of having the url in the version map, but the ship has likely sailed on that#2022-09-0502:09hiredmanAgain, that doesn't solve all possible clashes, but puts things in a similar place to maven#2022-09-0502:10hiredmanBut, depending on your view of the capabilities of the consumer, putting the choice in their hands could be seen as a way to give them an escape hatch to rectify poorly chosen names#2022-09-0502:11hiredmanSo maybe instead of making the git deps situation more closely match maven, there should be a way to override the artifact id for maven deps#2022-09-0502:11hiredmanFor the discerning build engineer#2022-09-0502:21didibusI think overriding might be a good feature in addition to also being better at properly identifying conflicts between equivalent coordinates.#2022-09-0502:24didibusRight now I'll create an http://ask.clojure.org that suggests adding a validation that if a lib defines a :lib com.foo/bar inside it's deps.edn, then people that take a dependency on it have to use that same lib name otherwise t.d.a errors.
That would solve the issue, in that the owner can now again decide what the coordinate should be and make it standard.
And possibly a second feature could be to override that when needed, if say someone else happened to use the same lib-name.#2022-09-0502:29hiredmanThat would lock things down beyond what maven does#2022-09-0502:37didibusHow so? Seems more to me it would bring parity. You publish a lib on git, you get to choose the unique name for it that will also be used for conflict resolution.#2022-09-0502:43seancorfieldNope. Any old garbage can be published to Maven/Clojars once you have the group verified.#2022-09-0502:43seancorfieldYou can't currently enforce much on the Maven side. But I think this would be a great enhancement on the git deps side.#2022-09-0502:57didibushttps://ask.clojure.org/index.php/12147/add-validation-for-groupid-name-coordinate-deps-tools-deps#2022-09-0503:30Alex Miller (Clojure team)I didn't read all of this, but basically you, as a library maintainer, should tell people what coords to use. To date, this has not been an issue. I'm open to adding more validation around this - at the moment it's low priority. Also, I've scoped out how to validate maven deps against deps (maven deps usually have git coords and that's sufficient to determine “newer”. Not implemented yet, but there's a hole for it.#2022-09-0503:32didibus@U064X3EF3 The request on http://ask.clojure.org summarizes it, you don't have to read all this. But basically, I'm asking for a way for tools.deps to assert that users use the coordinate that the lib maintainer want them to use.
That will also let the lib author make sure if they double publish to maven central or clojars as well as git deps, that the coordinate used by users is same between those.#2022-09-0617:07socksyis there an easy way to list the local/roots (and only the local/roots) recursively? Would be great for CI scripting in a monorepo#2022-09-0617:09Alex Miller (Clojure team)no, but you can get the data with which to compute this either programatically or via clj -X:deps tree :format :edn#2022-09-0712:58souenzzoI had this problem, did this solution.
But in my case, this solution is too slow, and requires to download all the deps.
End up parsing deps.edn manually and searching for any map with :local/root inside it, in a recur way.
cc @U066F8MGF#2022-09-0713:19socksybut there the issue is that you want to have the resolution of the :local/roots of those deps, which you get for "free" with clj -X:deps list . Unfortunately it does not seem so straightforward to filter out the :local/roots there — I guess a tree walker is needed with clj -X:deps tree#2022-09-0617:10Alex Miller (Clojure team)or clj -X:deps list :format :edn if the structure doesn't matter as much as the end list#2022-09-0617:10Alex Miller (Clojure team)throw some bb on that and you're in business :)#2022-09-0704:29CrispinHi! I want to add a path to the classpath, but only in a deps.edn :aliases.
I want to use codox to generate documentation, but it's for a babashka
pod. And that code, with the docstrings, is not in the classpath because
it can't be compiled by the normal compilation process (it is embedded
in the code that is compiled with macros so it can be injected into
babashka via it's interface). Any ideas how to extend the classpath in
an alias? I have tried a bunch of things with no success.#2022-09-0704:31seancorfield@Crispin use :extra-paths inside your alias to specify the extra directors#2022-09-0704:32seancorfieldAnd perhaps show us exactly what you have tried and exactly how it failed?#2022-09-0704:32CrispinI tried :paths, and :clj-paths, and :main-opts passing extra flags#2022-09-0704:33Crispinit failed with
Could not generate clojure documentation for pod.epiccastle.bbssh.utils - root cause: java.io.FileNotFoundException Could not locate pod/epiccastle/bbssh/utils__init.class, pod/epiccastle/bbssh/utils.clj or pod/epiccastle/bbssh/utils.cljc on classpath.
java.io.FileNotFoundException: Could not locate pod/epiccastle/bbssh/utils__init.class, pod/epiccastle/bbssh/utils.clj or pod/epiccastle/bbssh/utils.cljc on classpath.
#2022-09-0704:34Crispinah bugger, that adds the classpath allright, but now it fails because it cant compile (which is why it's not in the classpath to begin with)#2022-09-0704:35seancorfieldIf Codox loads the code to generate docs, it is a compiling it.#2022-09-0704:36Crispinit now doesnt load (even though the classpath is added) because the code cant compile:
Caused by: java.lang.ClassNotFoundException: babashka.pods
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:69)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at clojure.lang.DynamicClassLoader.loadClass(DynamicClassLoader.java:77)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at clojure.lang.RT.classForName(RT.java:2209)
at clojure.lang.RT.classForNameNonLoading(RT.java:2222)
at clojure.lang.Compiler$HostExpr.maybeClass(Compiler.java:1041)
at clojure.lang.Compiler.macroexpand1(Compiler.java:7062)
at clojure.lang.Compiler.analyzeSeq(Compiler.java:7110)
... 156 more
#2022-09-0704:37seancorfieldThat says Babashka pods isn't on the classpath#2022-09-0704:37seancorfieldSo I expect you need to add that as :extra-deps in that alias#2022-09-0704:38seancorfield(sorry, on phone)#2022-09-0704:38Crispinbabashka.pods only exists inside sci context over in babashka#2022-09-0704:38Crispinand this code that I want codox to run over, is text files that get injected into the babashka context#2022-09-0704:39seancorfieldThen you can't use it with Codox#2022-09-0704:39Crispinlooks like codox cant do it... as long as its searching for code on the classpath#2022-09-0704:39Crispinthanks anyway#2022-09-0704:39seancorfieldAsk in #babashka?#2022-09-0704:41Crispinsure#2022-09-0704:41CrispinI also might be able to do something hacky, like preprocess those source files to remove offending vars...#2022-09-0704:42seancorfieldOr get Codox to run under bb?😁#2022-09-0706:49borkdudeOr use #quickdoc #2022-09-0720:03matthavenerthis feels like a very simple problem but I'm having trouble googling it. when I run clj -A:myalias it's failing to compile some change I've made, but it doesn't produce an error, it just exits with a 1 error code. Is there some trick to getting the compiler to print the eval error?#2022-09-0720:07hiredmandepends, clojure itself (the clojure.main entry point) always produces some kind of error message, possibly just pointing you to a file that contains the full error message#2022-09-0720:08hiredmanif you aren't getting any error message at all then you may need to look at whatever that alias is actually launching (assuming it isn't just clojure.main), that might be swallowing any errors#2022-09-0720:10hiredmanadditionally, when an issue report is vague, it is often good to revisit and verify the underlying assumptions in the report: how do you know it is failing to compile if it just exits without an error message?#2022-09-0720:13matthavenerif I remove the namespace I require from the gen-class'd namespace with -main it works fine#2022-09-0720:13matthavenerthat lets me start a repl and then manually require that same ns#2022-09-0720:14matthavenermaybe I'll try to make an actually minimal testcase#2022-09-0720:14hiredmanwhat is the the alias doing?#2022-09-0720:15hiredmanwhat are the main opts?#2022-09-0720:15matthavenerit's actually two, both adding :extra-pathsand :extra-deps#2022-09-0720:15hiredmanbut no :main-opts?#2022-09-0720:16matthavenernope#2022-09-0720:16hiredmanare you aot compiling your gen-class somehow? (gen-class is a no op without aot)#2022-09-0720:17matthavenerI assume I'm doing something weird. I have an alias named nrepl that adds nrepl to the deps and then includes a path with a user.clj#2022-09-0720:17matthavenerI don't think I'm doing aot, not for dev#2022-09-0720:17hiredmanoh, try it without the user.clj (rename it to foo.clj or something)#2022-09-0720:18matthavenergood idea#2022-09-0720:19matthaveneryeah, without that user.clj I don't think it pulls in the offending namespace so it works fine#2022-09-0720:19matthavenerwhat's the accepted way of adding nrepl? I put this in my user.clj and it seemed so.. wrong:#2022-09-0720:19matthavener(ns user
(:require
[proj.main :as main]
[nrepl.server :as nrepl]))
(defonce nrepl-server (nrepl/start-server))
(spit "./.nrepl-port" (:port nrepl-server))
(main/-main)
#2022-09-0720:21hiredmanand what happens if you start your repl without the user.clj and load proj.main and call (proj.main/-main)#2022-09-0720:22matthaveneryeah, then I see the compile error when I (require 'proj.main)#2022-09-0720:25hiredmanoh, of course, user.clj is loaded so early it doesn't have all the error reporting stuff around it#2022-09-0720:25matthavenerah I see#2022-09-0720:26hiredmanI don't know how people launch nrepl these days#2022-09-0720:26matthavenerI suppose I should just make my user.clj more simple and have some go function that actually runs my app#2022-09-0720:26matthavenerI was used to lein repl doing everything for me#2022-09-0720:27matthavenerthank you for your help, that error reporting explanation makes sense#2022-09-0720:27matthavenerI just thought I might be missing some glaringly obvious flag on clj or doing something silly#2022-09-0921:28hlshipWhen using :local/root, is it possible to specify aliases to use when extracting classpath and dependencies from the identified directory?
Rationale: I have a multi-module project; I have module C that depends on B that depends on A. I'm adding a :local alias that uses :override-deps to convert from :mvn/version to :local/root for these intra-module dependencies, useful for local development.
However in C's deps.edn, I specify :local/root for B but the transitive to A is the :mvn/version coordinate, not the :local/root coordinate.
My workaround is to recapitualte B's local dependencies in C's :local alias.#2022-09-0921:36Alex Miller (Clojure team)no, not currently#2022-09-0921:39Alex Miller (Clojure team)the ticket for this is https://clojure.atlassian.net/browse/TDEPS-116 - I think it needs some thought about both the semantics and usage controls#2022-09-0921:40Alex Miller (Clojure team)I think your current solution is probably the best option right now#2022-09-1622:51Alex Miller (Clojure team)Prerelease version of the Clojure CLI 1.11.1.1161 is now available:
• Adds new CLJ_JVM_OPTS env var that can be used to set jvm opts on the internal processes that build classpaths
• Adds support for including the not-standard but common JAVA_OPTS for setting jvm opts on user programs, which may be useful in cases where people have previously asked for "global" jvm opts
• This support has been implemented for Linux and Mac, but not yet for Windows (but I will do that before I do an official release)#2022-09-1622:53Alex Miller (Clojure team)(Also, I learned that since Java 9 there is a new standard JDK_JAVA_OPTIONS env var - you can set that to send jvm options to all the Java things)#2022-09-2219:48practicalli-johnThis is really useful, thanks. I’m using JDK_JAVA_OPTIONS to pass options to the java command to set heap size constraints
https://docs.oracle.com/en/java/javase/14/docs/specs/man/java.html#using-the-jdk_java_options-launcher-environment-variable
I’m using it in the Dockerfile for my current project and its working well
# JDK_JAVA_OPTIONS environment variable for setting JVM options
ENV JDK_JAVA_OPTIONS "-XshowSettings:system -XX:+UseContainerSupport -XX:MaxRAMPercentage=90"
# Start service (over-rides default entrypoint command of image)
CMD ["java", "-jar", "/service.jar"]
#2022-09-2219:48practicalli-johnThis is really useful, thanks. I’m using JDK_JAVA_OPTIONS to pass options to the java command to set heap size constraints
https://docs.oracle.com/en/java/javase/14/docs/specs/man/java.html#using-the-jdk_java_options-launcher-environment-variable
I’m using it in the Dockerfile for my current project and its working well
# JDK_JAVA_OPTIONS environment variable for setting JVM options
ENV JDK_JAVA_OPTIONS "-XshowSettings:system -XX:+UseContainerSupport -XX:MaxRAMPercentage=90"
# Start service (over-rides default entrypoint command of image)
CMD ["java", "-jar", "/service.jar"]
#2022-09-1715:23zalkyHi folks: I'd like to roll an OSS library as a jar, and I'm wondering if you need to include the license in the jar file, and if so, does it matter where it goes?#2022-09-1715:24zalkyI've looked at a lot of existing projects and some do and some don't.#2022-09-1715:39Alex Miller (Clojure team)seems like a good idea#2022-09-1715:39Alex Miller (Clojure team)I think it is common to include at the root but there is no explicit place anything looks at#2022-09-1715:41respatializedhttps://docs.oracle.com/javase/tutorial/deployment/jar/manifestindex.html
Is there a way to document the license and its path using the jar's Manifest file?#2022-09-1716:15Alex Miller (Clojure team)no#2022-09-1716:15Alex Miller (Clojure team)the most common way to specify a license is with a Maven pom, which has some attributes for this#2022-09-1716:15zalkyIt seems that lein adds both a copy of the license from the project root into the META-INF folder of the jar, but also adds a license xml section into the pom.xml. Am I correct that tools.build does not provide any such feature?#2022-09-1716:16Alex Miller (Clojure team)it does the same#2022-09-1716:17Alex Miller (Clojure team)or rather it makes a pom, and can sync from a template pom.xml if you provide one, which can include license info#2022-09-1716:18zalkyI see, so that would add the pom xml, but not the actual license.#2022-09-1716:18Alex Miller (Clojure team)and you would need to copy the license file in#2022-09-1909:46darnokCan org.clojure/clojure dependency be overridden by dependency? I wanted to use one version of Clojure in all dependencies. I created an artificial project with dependency org.clojure/clojure and used it as a dependency in other projects (instead of org.clojure/clojure). It looks like the dependency version is not picked up from this artificial project.#2022-09-1910:20borkdudetools deps generally picks the newest version of any dep, unless you use it as a top level dep#2022-09-1910:21borkdudeyou could probably achieve your goal with an alias in which you declare your preferred clojure version as a top level dep#2022-09-1912:03darnokWe use mono repo with lots of similar services. I could have an alias, but I wanted to have the version in one place and not all over the repository in each deps.edn file 😞.#2022-09-1912:05darnokMaybe there should have a hierarchy of deps.edn files, or a edn reader which allows to merge deps.edn file with some other deps.edn file before resolving file/dependencies.#2022-09-1912:08Alex Miller (Clojure team)Note that org.clojure/clojure is always a top level dep via the default merged in from the root deps.edn, so if you don't specify it, you'll get the default from your clj install#2022-09-1912:10Alex Miller (Clojure team)So even in the example you gave at the top, the clj version will take precedence over the transitive version you're pulling in#2022-09-1912:20darnokOk#2022-09-1912:52mpenetwe have a workaround using templating at work to solve this without using custom aliases: https://github.com/exoscale/deps-modules#2022-09-1912:52mpenet@darnok maybe that's useful to you, it's quite opinionated but it works well for us#2022-09-1916:01grzmHave people been playing around with adding things to deps.edn like :deps/prep-lib but for their own project management? Particularly in a monorepo, I’ve been thinking about adding something like :my-ns/clean-lib and mimicking (or leveraging, as much as possible) the tooling around how :deps/prep-lib works. (There’s less of a need for this additional indirection in our monorepo as we control all of the naming within, but it’s been something I’d like to think though)#2022-09-1916:10grzm@darnok We do something similar to exoscale/dep-modules, but without the custom keys: just merging in our set of versioned deps with the template is good enough for our use-case. For example, we have a “base” deps.edn that looks like this:
{:aliases
{:default-deps
{cheshire/cheshire {:mvn/version "5.7.0"},
cider/cider-nrepl {:mvn/version "0.25.2"}}}
And then deps.template.edn files that look like:
{:deps {cheshire/cheshire {}, cider/cider-nrepl {}}}
A script picks up every deps.template.edn file and merges the deps listed there into whatever is in the [:base :default-deps] (omitting deps that aren’t there at all, walking into aliases and whatnot). And we can override anything there just by including what we want in the deps.template.edn itself. It’s been good enough for our use-case. I’m sure others have use-cases where this doesn’t fit. We haven’t yet had a need for adding meta or anything to indicate complete replacement instead of merging, or things like that. What’s nice is that it’s just data, so you easily make something that fits what you need. Clojure is good like that 🙂#2022-09-2001:27cflemingI’m having a problem with the Cursive build, due to the fact that I’m using a fork of Clojure. I’m using :classpath-overrides like this:
{:deps {com.cursive-ide/clojure {:mvn/version "1.10.4"}}
:aliases {:no-clojure {:classpath-overrides {org.clojure/clojure ""}}}
:mvn/repos {"" {:url "my-repo-url"}}}
What I’m finding since the base Clojure version has moved on from the version of my fork is that I’m getting incorrect versions of spec. My Clojure version depends on the spec versions from Clojure 1.10.3, but the versions from 1.11.1 are different. What seems to happen is that deps returns the latest spec version (corresponding to 1.11.1), but the classpath-overrrides is processed later, leaving the spec versions without a valid parent in the deps tree. I can’t even add the right spec versions explicitly to my :deps, since they’re still filtered out earlier in the process than the classpath-overrides is run. Is there any useful workaround to this problem?#2022-09-2001:36hiredmanI may be misunderstanding, but I don't believe any dependency information is conveyed when using classpath overrides#2022-09-2001:37hiredmanvery literally "replace the path this dep would have used with this other path" no search for or usage of dependency information found there#2022-09-2001:38cflemingRight, it looks like not. The problem is that that means there’s no way to remove a dep and its dependencies, and there’s no way I can see to manually force the dependencies I want since they’re overridden earlier in the process.#2022-09-2001:38hiredmanA local dep might be what you want, since that includes dependency information#2022-09-2001:40cflemingA local dep for the fork, you mean? I don’t think the problem is the dependency information from the fork (since my fork correctly specifies the dependencies it needs), it’s that that info is discarded early in the process.#2022-09-2001:40hiredmanNo, it is#2022-09-2001:41hiredmanBecause you are bringing the fork in using the classpath override, the dependency information from the fork (what version of spec it wants) isn't considered#2022-09-2001:41cflemingNo. I bring the fork in as a normal dep, with dependencies as usual. What I’m doing with the classpath override is trying to get rid of the mainline version of Clojure so it doesn’t conflict with the fork.#2022-09-2001:43hiredmanAh, yeah, you need a top level exclusion, I forget if that is a thing#2022-09-2001:44cflemingI don’t think it is, unfortunately.#2022-09-2001:47cflemingI can’t even manually add :exclusions to all my deps coords, since Clojure comes in from the system deps file.#2022-09-2001:48hiredmanIf you are publishing to your own repo, you could squat org.clojure/clojure and use a special version#2022-09-2001:49cflemingSadly:
> tools.deps guarantees that the “central” and “clojars” repositories will be checked first, in that order, for Maven libraries. The lookup order of other repositories is undefined.#2022-09-2001:51hiredmanI guess I don't know mavens algorithm exactly there, if you specify a version that doesn't exist in central, but does in your repo, it won't check your repo because it checks central first?#2022-09-2001:55cflemingI’m not sure, in that case. I guess it probably would, I’d just have to use a version numbering scheme that didn’t ever conflict with Clojure.#2022-09-2001:56cflemingAnd versions which are always seen by Maven as higher than public versions.#2022-09-2002:00hiredmanI believe the version algorithm tools.deps uses isn't just a straight comparison, but also considers where it is specified, and if i am remembering that correctly, then what you directly specify in your deps.edn should always win#2022-09-2002:21cflemingIt starts to sound like relying a lot on implementation details, though. I’ll see what’s involved in just updating my fork, which might be the easiest solution.#2022-09-2002:43cflemingIndeed, updating my fork was the easiest thing to do. But it would be good to have a not-horrible workaround for this.#2022-10-1915:19ambrosebs@U0567Q30W try adding this to the :no-clojure alias:
:deps {org.clojure/clojure {:mvn/version "1.11.1" :exclusions [org.clojure/spec.alpha]}}
#2022-09-2006:35pezWe're running into a strange problem where a dependency on com.datomic/datomic-free messes up the cider/piggieback dependency. The repository is public, here: https://github.com/skydread1/flybot.sg/tree/42-simple-clojure-backend
There's an alias dev-server that brings in datomic-free:
:dev-server
{:extra-paths ["src/clj"]
:extra-deps {com.datomic/datomic-free {:mvn/version "0.9.5697"}}}
If I specify that alias on the clojure command line, together with the injection of piggieback and using piggieback nrepl middleware, I get the error that the namespace cider.piggieback is not found:
$ clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"},cider/piggieback {:mvn/version,"0.5.3"}}}' -M:dev-server -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware cider.piggieback/wrap-cljs-repl]"
Execution error at nrepl.cmdline/require-and-resolve (cmdline.clj:221).
No namespace: cider.piggieback found
Full report at:
/var/folders/t5/gqxhj8pd6p9_tnvy6sbtmy480000gn/T/clojure-13582953176357101871.edn
Whereas if I don't use that alias, the nREPL server starts without complaints:
$ clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"1.0.0"},cider/cider-nrepl {:mvn/version,"0.28.5"},cider/piggieback {:mvn/version,"0.5.3"}}}' -M -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware cider.piggieback/wrap-cljs-repl]"
nREPL server started on port 49890 on host localhost -
What could we further do to investigate/fix this issue? Anyone has an idea?#2022-09-2006:56thhellerwhat does the Full report say? should contain the cause exception#2022-09-2007:10pezThanks. forgot to include the full report:
{:clojure.main/message
"Execution error at nrepl.cmdline/require-and-resolve (cmdline.clj:221).\nNo namespace: cider.piggieback found\n",
:clojure.main/triage
{:clojure.error/class java.lang.Exception,
:clojure.error/line 221,
:clojure.error/cause "No namespace: cider.piggieback found",
:clojure.error/symbol nrepl.cmdline/require-and-resolve,
:clojure.error/source "cmdline.clj",
:clojure.error/phase :execution},
:clojure.main/trace
{:via
[{:type java.lang.Exception,
:message "No namespace: cider.piggieback found",
:at [clojure.core$the_ns invokeStatic "core.clj" 4163]}],
:trace
[[clojure.core$the_ns invokeStatic "core.clj" 4163]
[clojure.core$ns_resolve invokeStatic "core.clj" 4370]
[clojure.core$ns_resolve invokeStatic "core.clj" 4360]
[clojure.core$ns_resolve invoke "core.clj" 4360]
[nrepl.cmdline$require_and_resolve invokeStatic "cmdline.clj" 221]
[nrepl.cmdline$require_and_resolve invoke "cmdline.clj" 211]
[nrepl.cmdline$fn__2105 invokeStatic "cmdline.clj" 225]
[nrepl.cmdline$fn__2105 invoke "cmdline.clj" 225]
[clojure.core$map$fn__5931$fn__5932 invoke "core.clj" 2759]
[clojure.core$map$fn__5931$fn__5932 invoke "core.clj" 2759]
[clojure.lang.PersistentVector reduce "PersistentVector.java" 343]
[clojure.core$transduce invokeStatic "core.clj" 6946]
[clojure.core$into invokeStatic "core.clj" 6962]
[clojure.core$into invoke "core.clj" 6950]
[nrepl.cmdline$__GT_mw_list invokeStatic "cmdline.clj" 252]
[nrepl.cmdline$__GT_mw_list invoke "cmdline.clj" 250]
[nrepl.cmdline$build_handler invokeStatic "cmdline.clj" 261]
[nrepl.cmdline$build_handler invoke "cmdline.clj" 254]
[nrepl.cmdline$options__GT_handler invokeStatic "cmdline.clj" 330]
[nrepl.cmdline$options__GT_handler invoke "cmdline.clj" 321]
[nrepl.cmdline$server_opts invokeStatic "cmdline.clj" 380]
[nrepl.cmdline$server_opts invoke "cmdline.clj" 367]
[nrepl.cmdline$dispatch_commands invokeStatic "cmdline.clj" 480]
[nrepl.cmdline$dispatch_commands invoke "cmdline.clj" 473]
[nrepl.cmdline$_main invokeStatic "cmdline.clj" 496]
[nrepl.cmdline$_main doInvoke "cmdline.clj" 491]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.main$main_opt invokeStatic "main.clj" 514]
[clojure.main$main_opt invoke "main.clj" 510]
[clojure.main$main invokeStatic "main.clj" 664]
[clojure.main$main doInvoke "main.clj" 616]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.lang.Var applyTo "Var.java" 705]
[clojure.main main "main.java" 40]],
:cause "No namespace: cider.piggieback found"}}
#2022-09-2007:24thhellerhmm not sure but should it maybe be -A:dev-server -M in the command line? not sure if -M maybe drops -Sdeps or so if used with alias?#2022-09-2007:34pezDoesn't seem to be the issue. I get the same result. Also, it is the datomic-dependency that makes the difference. If I replace it with some other dependency in the alias, piggieback is found and the REPL starts.#2022-09-2007:36thhellermy guess would be on a dependency conflict. I'm guessing that the presence of piggieback causes clojurescript to be added and loaded as well#2022-09-2007:36thhellerand clojurescript (or rather the closure-compiler) can fail to load with a conflicting guava dependency on the classpath#2022-09-2007:36thhellerwhich in turn comes from datomic#2022-09-2007:37thhellerdunno where that actual exception ends up though. it usually throws a proper exception making this easier to debug#2022-09-2007:38thhellertry adding this dep as well [com.google.guava/guava "31.0.1-jre"] via the alias or -Sdeps#2022-09-2007:38thhellerconfirms that its the dependency conflict if it works 😉#2022-09-2007:44pezConfirmed then. 😃 Because works! Thank you! Check here, @loic ^ That's your workaround.#2022-09-2007:58LoicThank you!#2022-09-2113:49pezDoes tools.deps have something equivalent to Leiningen's :injections? I think what that does is to inject requires in all namespaces, but I am actually not sure. In any case if I am using tool that has instructions to specify :injections for Leiningen, what could some equivalent thing be with tools.deps?#2022-09-2113:49borkdudeI'm pretty sure it hasn't#2022-09-2114:06Alex Miller (Clojure team)it doesn't, but I wrote up an example of how to do modify your repl at https://insideclojure.org/2020/02/11/custom-repl/#2022-09-2114:08Alex Miller (Clojure team)(the commas in the alias there are no longer necessary though, that's been fixed)#2022-09-2116:01pezVery funny result trying that custom repl in Calva, @alexmiller 😃 Is it nREPL that isn't a true REPL enough, maybe?#2022-09-2116:02pez(Calva pops up a REPL input prompt and generally stops working)#2022-09-2116:03Alex Miller (Clojure team)Well yeah, nrepl is not a true repl so will not work with that#2022-09-2206:24Ben SlessI remember reading an in depth breakdown of why this is so. Do you happen to have a link to it at hand?#2022-09-2409:25cfleming@UK0810AQ2 https://groups.google.com/g/clojure-dev/c/Dl3Stw5iRVA/m/IHoVWiJz5UIJ#2022-09-2409:25cflemingNot sure if that’s what you were thinking of, but there’s lots of discussion there.#2022-09-2908:35Ben SlessI think that's the one, thanks.
This got me thinking again about listeners. How would that work in Clojure?#2022-09-3007:43cflemingWhat sort of listeners do you mean?#2022-09-3007:47Ben SlessIf I'm getting it right, a Listener is just a generic REPL interface with arbitrary read/print methods, including graphical, all connected to the same process. It's an interface by which the process sets up REPLs which listen to input#2022-09-3007:48Ben Slesshttps://mobile.twitter.com/RainerJoswig/status/1220457695414771714#2022-10-0103:33cflemingI really wish CL didn’t have such a high bar to entry. I’m sure there’s a lot to learn there, but I don’t have a year to spare.#2022-10-0104:21seancorfield@UK0810AQ2 I think tap> has radically opened up the way multiple processes can "listen" to evaluations happening in a core process now.#2022-10-0104:23seancorfieldFor a long time, I worked with just a Socket REPL -- and Chlorine (Atom) and then Clover (VS Code) -- but the pace of evolution of Calva (and lack of progress on Clover) made me switch over completely a while back to nREPL + CIDER middleware. I'm still not convinced about the latter. Calva "relies on CIDER" for some functionality but, I'll be honest, I haven't figured out what yet and the middleware just makes stacktraces so much worse so I may go back to plain nREPL soon.#2022-10-0104:30seancorfieldBut I just wish I didn't even need nREPL. I wish maintainers of editor plugins were more committed to supporting Socket REPL.
One particular use case where this matters deeply to me is being able to connect your editor to a remote Clojure process (or even a remote non-Clojure process that contains the Clojure runtime as a library), where a Socket REPL is started as a JVM option. No dependencies required at all.
Right now, the heavy dependence on nREPL means I a) need nREPL as a production dependency and b) need some code in all my apps to start an nREPL server based on, say, a JVM property and c) I need to provide that "non-standard" JVM property -- rather than zero dependencies and an official, core, documented JVM property that works with any Clojure-containing process. Sure nREPL has no other dependencies but it's still an extra 67K to drag around.#2022-10-0105:54Ben SlessTaps feel about 25% of the way there. You can listen on outputs, but input source is indiscriminate.
I wish there was more work to close this gap in tools, but most effort is directed elsewhere at the moment#2022-10-0106:04seancorfieldRight, each distinct input needs it's own REPL essentially. But you can use tap> to have multiple "listeners" for the same core evaluation. I think a combination of the two can reach most use cases -- but there are some interactive uses where REPL (or pREPL) alone are claimed not to be sufficient: very large output (including infinite output) and very long-running processes -- where folks say (rightly) that it is important to be able to limit/interrupt the evaluation and/or "printing" the output.#2022-09-2118:12borkdude@pez You could make your own evil middleware with nREPL but you didn't hear this from me#2022-09-2118:47seancorfieldI'm looking at https://github.com/clojure/brew-install/blob/1.11.0/deps.edn and notice that it depends on org.clojure/tools.deps.alpha but clojure.exec.run says that it only depends on Clojure's core https://github.com/clojure/brew-install/blob/1.11.0/src/main/clojure/clojure/run/exec.clj#L11 -- I was thinking of depending on this as a git dep to get access to the exec fn stuff but that would drag in t.d.a. Should that dependency actually be in there?#2022-09-2118:49seancorfieldI guess I could use :exclusions to prevent it dragging in t.d.a (and slf4j-nop) but...#2022-09-2118:52seancorfield(background: we have an exec-command function at work in our build.clj that allows us to set up and run Java processes that pass exec args etc but it currently relies on exec.jar as a local dep because we have it vendored into our repo -- and I'd like to open source this, likely as part of my build-clj wrapper, which means it needs to operate independently of where the CLI is installed...)#2022-09-2120:36Alex Miller (Clojure team)We have intentionally not packaged or committed to any specific api for the exec stuff#2022-09-2120:37Alex Miller (Clojure team)And I have changed it in breaking ways several times as it has evolved#2022-09-2120:38Alex Miller (Clojure team)I do want to make it accessible and it probably will be a separate lib (or inside Clojure) in the future#2022-09-2120:39Alex Miller (Clojure team)So, I do not have an answer for you at this time, other than just copying what you need of it and retaining the copyright and license (with the knowledge it may change)#2022-09-2120:39seancorfieldRoughly translated: do not do this. Yet (maybe). Fair enough. Obviously we could just shell out and run clojure -X... with appropriate arguments but it sure would be nice to have something closer to the java-command/`process` stuff built-in.#2022-09-2120:40seancorfieldAt least if I used it as a git dep, I could pin to a version, and not have to copy any code, and then I could update the git dep as exec evolves over time (until something like it is released).#2022-09-2120:41Alex Miller (Clojure team)I need it in tools.build too (and elsewhere), so I feel your pain#2022-09-2120:41seancorfieldI might try that at work, anyway, to sanity check my idea of no longer depending on the local vendored exec.jar file. Yeah, it's build-clj where I feel the need for this.#2022-09-2120:41Alex Miller (Clojure team)It's intentionally not a git dep as I don't want people to depend on it#2022-09-2216:08Drew Verleehow is the value of :main-opts used? At a high level i know your supposed to pass it a string that matches the namespace that has a main function that you want to invoke. However, the arguments are a vector and nearly every example i see is a tuple where the first arg is '-m' . i take is this '-m' and not the :main-opts is really want makes it at a function that invokes the main function in that namespace.
So a simple question is, why doesn't :main-opts just add the '-m' or is there something else that could be passed intead of '-m' for :main-opts?#2022-09-2216:09Drew Verleelink for reference https://clojure.org/reference/deps_and_cli#_execution#2022-09-2216:12Drew Verleei guess i'm not sure what options clojure.main takes, here is clojure main https://clojure.github.io/clojure/clojure.main-api.html#clojure.main/main#2022-09-2216:12Drew Verleeahh the doc string says what else can be passed#2022-09-2216:12Drew Verleemain-opt:
-m, --main ns-name Call the -main function from namespace w/args
-r, --repl Run a repl
path Run a script from a file or resource
- Run a script from standard input
-h, -?, --help Print this help message and exit
#2022-09-2216:12practicalli-john-M tells the Clojure CLI to use clojure.main to run the code (as opposed to clojure.exec)
-m defines the namespace that clojure.main should look in for the -main function
To me t makes sense to include the -m in the main-opts as the next thing in the list is the namespace#2022-09-2216:14practicalli-johnI see the main-opts as a Clojure version of the command line arguments that are passed after -M (and any aliases following -M). Missing out the -m and just having the namespace would make a difference between the command line and the :main-opts configuration (which I would find confusing)#2022-09-2216:18Drew Verleethanks, yeah. I just think i have done anything but use -m. Here is another thing that can be done:
cat foo
(println (+ 1 1))
clj foo
WARNING: Implicit use of clojure.main with options is deprecated, use -M
2
Though i'm not sure how to avoid getting that warning, i don't really mind as i'm not sure when i would ever need this...#2022-09-2216:21practicalli-johnThe practicalli/clojure-deps-edn aliases have some additive arguments, but I don't recall using something instead of -m
`:repl/cider
{:extra-deps {nrepl/nrepl {:mvn/version "1.0.0"}
cider/cider-nrepl {:mvn/version "0.28.5"}}
:main-opts ["-m" "nrepl.cmdline"
"--middleware" "[cider.nrepl/cider-middleware]"
"--interactive"]}`
#2022-09-2216:26practicalli-johnThe depreciation message I believe was in part to help move people away from the -A flag and use the more appropriate -M flag, so f -M isn't used then I'd expect to see a depreciation message (unless using -X or -T instead, in which case main-opts sent used anyway)#2022-09-2217:03seancorfieldI use clj -M:build -i build.clj -r to start a REPL in "build" mode. -i runs build.clj as an "init script" and then -r signals that a REPL should be started. At that point I'm in the user ns but build.clj has been loaded so I can do stuff like (build/database-setup {}) or (build/run-tests {:projects '{api auth login]})#2022-09-2217:32Drew VerleeOh interesting, thanks sean.#2022-09-2218:59practicalli-johnThats an interesting alternative to having a user namespace do everything like that. Thanks Sean.#2022-09-2222:22seancorfieldGiven the various caveats around user.clj I avoid it completely.#2022-09-2223:10skylize> clj foo
> WARNING: Implicit use of clojure.main with options is deprecated, use -M
> 2
> Though i'm not sure how to avoid getting that warning,
If you use the -M flag, like it asks, then it won't warn you anymore. 😉
$ clj -M foo
The -M flag is separate from -m. Currently if you don't use a conflicting flag (`-X` or -T) then it starts just like -M, but gives you this warning.
You'll notice in @U04V70XH6’s example above, he attaches his aliases to the -M flag. But if you did not want to use an alias, you could still use -m in the command line.
$ cli -M foo -m "foo.cmdline"
#2022-09-2223:59Drew VerleeThanks. I didn't even notice i left the -M off!#2022-09-2308:50practicalli-john@U04V70XH6 what are the caveats of using a user.clj ? (I searched but couldn't find anything)
I always define the custom user namespace as dev/user.clj so the dev path has to be explicitly added via alias, controlling when it lads.
Would be interested to know what else to be aware of. Thanks.#2022-09-2316:08seancorfield@U05254DQM I'm surprised you can't find problems talked about in this Slack, since it's come up quite a few times. I'd have to search Slack to give you examples... Maybe later on today if I have time...#2022-09-2416:34seancorfield@U05254DQM so I had a quick search for user.clj here this morning and found lots of threads with people running into problems with it. It's loaded outside the context of normal code running or a repl: it can swallow errors, many dynamic vars are not bound, it can load code in ways that break regular code in several ways (because of the difference in context). If it accidentally ends up in a library, it can cause all these problems for downstream users (Leiningen briefly had a bug that caused this to happen). Apparently, even the way different editor plugins interact with starting repls can cause user.clj to be problematic (I did not dig into that particular thread).#2022-09-2510:00practicalli-johnThe startup order seems something to investigate for constraints, as some contexts are not created before a (custom) user namespace is loaded.
I am assuming a namespace loaded via an -i flag loads later than the user namespace (hopefully at the end), possibly removing those context constraints (something to investigate at some point).
Other concerns seem due to convoluting the user namespace with the application code, which isn't a problem if kept separate.
FYI: I'd searched the web and didn't find anything, seemed to be a lot f noise in slack, but found a couple of threads. Thanks.#2022-09-2706:52Adam HelinsIs there anything to be wary of when requiring the same gitlib twice in a project using different :deps/root?#2022-09-2707:03Alex Miller (Clojure team)I think the big one is if there are transitive local deps via git deps (as they can’t be compared)#2022-09-2707:07Adam HelinsThat could be mitigated by using the same SHA but essentially those transitive local deps would be imported twice, right?#2022-09-2707:55Adam HelinsAh, I might have misunderstood your statement. Let me provide some context.
I'm looking for an efficient way of sharing libs in a nascent public monorepo. Exposing them as git deps with :deps/root seems to be the most straightforward approach.
Let's say that:
A) Consumers must always use the same SHA when importing more than 1 lib
B) Those libs refer to other local libs via :local/root
Thinking more about it, my concern is that something might go wrong indeed with those local transitive deps.#2022-09-2708:48Adam HelinsBut transitive :local/root s are trickiest when working and changing stuff in such a local repo.
As opposed to a git dep being cloned once and not touched afterwards. So maybe I worry too much?#2022-09-2916:36Adam Helins@alexmiller Okay, I finally understood what you meant by problems arising from git deps having the same local transitive deps. It wasn't so obvious at first.
Turns ou you had the exact same discussion with @seancorfield before: https://clojurians.slack.com/archives/C6QH853H8/p1628884285236600
Did someone find a working alternative? 🙏#2022-09-2916:44Alex Miller (Clojure team)I have not worked on it so nothing has changed since then#2022-09-2917:02Adam HelinsI've spent countless hours on related topics and tooling and I'm convinced this would really help in improving the ecosystem (for reasons I can happily discuss) 🙂
For the time being, I think I have worked out a hack. I'd be grateful if you could indicate me if this can work indeed or if I got somehow lucky trying.
Instead of using :local/root , use :git/sha for transitive local dependencies with the actual production :git/url. Please don't think about development for now, I have a solution. Just about how those deps are consumed externally.
When the monorepo is stable, point them all to the same SHA. Commit. New SHA is what user can use when importing one or several libs from the monorepo. Something like that will happen:
Checking out: https://... at e6fd4e17003af53e434e9d684d0b833af7aad2aa
Checking out: https://... at e6fd4e17003af53e434e9d684d0b833af7aad2aa
Checking out: https://... at e6fd4e17003af53e434e9d684d0b833af7aad2aa
Checking out: https://... at 2504dbfd3fd3c2ad605cec3fcc66ce093d57aaef
(Former identical SHAs are what those libs where using (transitively) ; latter one is what I used externally to get them)
Does it always work?#2022-09-2917:20Alex Miller (Clojure team)it should#2022-09-2918:57Adam HelinsAwesome, I think I got it fully working simply because such git deps can compare (always?)
Thanks very much for the work you've put into git deps#2022-09-2918:58Alex Miller (Clojure team)yep, git repos compare and prefer "most descendant" commit#2022-10-0521:07winsomeI just watched "The Secure Software Supply Chain" talk from Strange Loop this year. What's the story on reproducible builds with Clojure? After a quick experiment with some local projects it doesn't look like we're there out of the box, but is there support for this on the horizon?#2022-10-0521:08Alex Miller (Clojure team)Yes, I’m very interested in working in this direction#2022-10-0521:08Alex Miller (Clojure team)I talked to Kelsey about it a bit after the talk#2022-10-0521:10winsomeI dabble in the NixOS world and my understanding is that reproducability with the JVM is tough.#2022-10-0521:14winsomeDo you know if the CLJS folks are looking at it as well? JS has it's own dependency craziness.#2022-10-0521:40Alex Miller (Clojure team)no, I don't have an understanding of the equivalent issues there#2022-10-0521:42Alex Miller (Clojure team)reproducibility is not really the bit that I'm focused on (re-reading the original question above) - that has never been a strong goal for us from Clojure perspective. what we do want is more traceability and verification#2022-10-0521:44Alex Miller (Clojure team)I do think it's important that we should be able to connect signed commits in git -> signed releases, with verification, and better tracking of both provenance and full dep set when you do an application build, and having those tools be as automatic and included as possible#2022-10-0607:02thhelleras far as CLJS is concerned the only real hurdle is (RT/nextId) not being controllable. (gensym) uses it, so it pops up all over the place and ens up generating slightly different code each time since its an ever increasing integer#2022-10-0607:02thhellerclojure.lang.RT that is#2022-10-0607:03thhelleranother hurdle is the :advanced compilation done by the Closure Compiler, its not fully deterministic either but would be closer with more "stable" JS generated by the cljs compiler#2022-10-0607:03thheller(regarding reproducibility that is)#2022-10-0610:17Wanja HentzeWe have experimented a bit with this, and opened https://ask.clojure.org/index.php/12249/bytecode-not-100-deterministic-given-identical-inputs at some point#2022-10-0610:21Wanja HentzeBy using the patched hashCode and patching a logging library we use, @U06GVE6NR was able to get a sizeable production clojure project to build reproducibly.#2022-10-0610:24Wanja HentzeSo I feel like the way to reproducible builds is probably not super hard#2022-10-0610:31Wanja HentzeDepending on your setup, you may also need to do (some of) the things in https://reproducible-builds.org/docs/jvm/
Nix takes care of most of that if you're actually building via a derivation, though.#2022-10-0722:52lread@U04V5VAUN your signing of the jars might not be for naught after all!#2022-10-0722:54lreadI’d like to take credit for being responsible about my verified github commits, but I only set it it all up because @U04VDQDDY’s planck repo required it!#2022-10-0615:54borkdudeA small script to safely convert project.clj to deps.edn, even when project.clj contains read-eval 🔥
https://twitter.com/borkdude/status/1578050265567076353#2022-10-1115:58borkdudelein2deps now also supports evaluation of project.clj (evaluates ~ when the --eval option is given) and also ... Java compilation, PR-ed by @dergutemoritz 🔥
https://github.com/borkdude/tools#lein2deps#2022-10-1116:53dergutemoritzNot just Java compilation - it also generates a :deps/prep-lib which mimics Leiningen's default Java compilation behavior. That allows generating a deps.edn for libraries which are still primarily on Leiningen which allows using it as a git dependency via tools.deps.#2022-10-1116:53dergutemoritzNot just Java compilation - it also generates a :deps/prep-lib which mimics Leiningen's default Java compilation behavior. That allows generating a deps.edn for libraries which are still primarily on Leiningen which allows using it as a git dependency via tools.deps.#2022-10-1202:41Drew VerleeIs there a way to ~automatically~ programatically add exclusions to deps.edn at the root level? If not, is there a reason why that would be hard to do? Or is there something else i should consider?
I know i can get a tree of deps using clj -Stree, it would seem simple enough to find the dep i wanted to exclude and walk up the tree and add it to the parent.#2022-10-1204:10Alex Miller (Clojure team)If you want programmatic I would use tools.deps rather than the cli#2022-10-1204:10Alex Miller (Clojure team)But it is somewhat tricky in that it’s not really a tree but a graph, and a dep can appear at many points#2022-10-1209:25borkdudeIt seems :deps/prep-lib only "activates" if you use the project as a dependency, vs. when you start the project locally. Is that intentional?
{:tag :a, :attrs {:href "/cdn-cgi/l/email-protection", :class "__cf_email__", :data-cfemail "43212c312827362726032e72"}, :content ("[email protected]")}#2022-10-1209:26borkdudeAnother suggestion: the prep-lib error message could include what to run#2022-10-1209:27borkdudeThis also didn't work:
$ clj -Sdeps '{:deps {current/current {:local/root "."}}}' -X:deps prep
Error building classpath. The following libs must be prepared before use: [current/current]#2022-10-1418:14borkdude@U064X3EF3 An idea?#2022-10-1418:16Alex Miller (Clojure team)Yeah, have thought about that, wasn’t sure if it would be like 20 lines of commands at some point :) #2022-10-1418:17borkdudeWhat about the first issue, shouldn't it also just work locally?#2022-10-1418:19borkdudeI.e. in an unprepared project, run a REPL, get an error about prep and be able to run clojure -X:deps prep#2022-10-1421:47Alex Miller (Clojure team)That probably makes sense but I want to go back too the origins sheets to make sure I’m not forgetting something #2022-10-1317:40Lone RangerDoes anyone here know about boot and cljsjs? I'm asking because I'm wondering if this step in the cljsjs readme is still relevant or this could be accomplished some other way with tools-deps now: https://github.com/cljsjs/packages/wiki/Creating-Packages#5-add-buildboot-and-readmemd-files#2022-10-1317:52respatializedI think you could adapt the Boot task into a deps.edn alias analogous to those used by stuff like tools.build - just a main fn that you can invoke from the alias to accomplish all the packaging tasks. https://clojure.org/guides/tools_build
I've never used cljsjs, but having used Boot a little bit ages ago its idea of "pods" is a little wonky and hard to directly translate into ordinary Clojure code. It looks like that description gives a decent overview of the steps you'd need to define in the function you run under a deps.edn alias, though.#2022-10-1317:53Lone RangerOk. Wasn't sure if there was something "special" to boot, I'm assuming it's just because converting over a zillion packages in the cljsjs repo isn't worth it#2022-10-1317:53Lone RangerI've heard, though, that boot is w00t#2022-10-1317:53Lone Ranger😛#2022-10-1317:58borkdudehttps://www.reddit.com/r/bullcity/comments/xo77l9/does_anybody_know_what_this_little_fella_is_on/#2022-10-1321:42seancorfield@U3BALC2HH FYI, we converted our work setup from Leiningen to Boot in 2015 but started to run into bugs/problems with both Boot's pods and the sort of shadow file system, so we switched to CLI/`deps.edn` in 2018. We've since added build.clj/`tools.build`. Everything you can do in Boot as tasks just maps to functions in build.clj except for the classloader isolation of "pods" -- but, as @UFTRLDZEW hinted above, there are definitely problems with that approach and spawning new Java processes (easy with tools.build) is simpler and safer.#2022-10-1321:43seancorfieldI think at this point almost no one is still using Boot (compared to Leiningen and Clojure CLI).#2022-10-1322:00Lone RangerAwesome, thank you!#2022-10-1517:12ChaseI'm curious how folks run multiple aliases that each have their own :main-opts . It's my understanding that only the last alias's main opts are run. So what would you do if you want to run something like clj -M:repl:server where both repl and server aliases have opts?#2022-10-1517:15ChaseI'm currently just running the aliases individually in a separate terminal but I imagine there might be a more efficient way#2022-10-1517:27Chasehaha, I did a search of the channel and it looks like I asked the same question over a year ago. Time flies! It seems the answer is I just can't do what I was attempting but I'll leave the question here in case things have changed or someone can recommend a better workflow#2022-10-1520:02seancorfieldYou can write your own function that launches both things and have an alias for that.#2022-10-1520:03seancorfieldPersonally, I never start a server from the command-line if I'm developing - I always start everything from the repl#2022-10-1522:51skylizeIf the built in clojure repl is enough, you can -r as one of your main opts to also launch the repl.#2022-10-1619:13ChaseThanks folks. I usually connect my editor through a nrepl so not sure the built in one will work. I have wanted to move to a more repl based workflow for starting servers and such so this is a good time to start. That is one of the things that libraries like Component, Integrant, Mount, etc. help with too right?#2022-10-1619:15seancorfieldI'm not talking about the built-in one. I start nREPL from the command-line, connect my editor to that, then start my web server or applications from within the editor, using that REPL connection.#2022-10-1619:15seancorfieldJust to clarify what I'm suggesting.#2022-10-1619:17ChaseYep, I gotcha. I was responding to both of ya with the same comment#2022-10-1619:56ChaseSo I made a dev alias that loads in a dev/user.clj file that I setup like so:
(ns user
(:require
[ring-app.core :refer [-main]]))
(defonce server (-main))
(defn start []
(.start server))
(defn stop []
(.stop server))
(defn restart []
(stop)
(start))
(comment
(start)
(stop)
(restart)
,)
where (-main) starts my jetty server. The thing I can't figure out is how to refactor this such that the server isn't automatically started when invoking clj -M:dev:repl . What should I do instead of the defonce ?#2022-10-1711:50practicalli-johnI have an alias that loads dev/user.clj via an extra path. In the user namespace I have everything in a comment form, except for the ns form that defines any requires
So namespaces are loaded, but code that initiates anything is in a comment
The only exceptions to that rule are println statements to show useful information on REPL startup and global logging context for mulog.
Here is what I am using on my current project (it uses integrant, but thats abstracted to the system namespace so any component system could be used - or none)
https://gist.github.com/practicalli-john/904b5d9ae398aaec2649b50b98a709eb#2022-10-1716:54seancorfield☝️:skin-tone-2: Yup, that's what I have, essentially. I have a -main so that I can "run" my dev startup via -M but I can safely load the ns without it doing anything.#2022-10-1718:07ChaseSounds good. I just moved the server invocation to the comment block and changed the start, stop, and restart functions to take that server as an argument.#2022-10-1616:36BenjaminJo:
SLF4J: Found binding in [jar:file:/home/benj/.m2/repository/ch/qos/logback/logback-classic/1.2.8/logback-classic-1.2.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/benj/.m2/repository/org/slf4j/slf4j-nop/1.7.36/slf4j-nop-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
how do I find exactly which dep contributes which files on the classpath? -Stree seems useful but prints only the "artifacts" or something right?#2022-10-1616:51Benjaminfor some reason slf4j-nop doesn't show up in clj -Spath#2022-10-1617:09Benjaminok lol I had the issue with logback-classic 1.2.8 and it is fixed with 1.4.4#2022-10-1723:15souenzzosee also: clj -Stree
This may be useful too
(map str (iterator-seq (.asIterator (.getResources (.getContextClassLoader (Thread/currentThread))
"/org/slf4j/impl/StaticLoggerBinder.class
"))))
#2022-10-1814:53dergutemoritz@alexmiller When trying to figure out how to invoke clojure.tools.deps.alpha/prep-libs! directly, I got confused by its config argument. It appears to be unused and oddly, the prep CLI API function passes the basis function as its value: https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/cli/api.clj#L102 -- maybe some leftover from a refactoring?#2022-10-1814:56Alex Miller (Clojure team)that is confusing looking but also right#2022-10-1814:57Alex Miller (Clojure team)inside tools.deps, "config" usually refers to the top level procurer config (currently just :mvn/repos and :mvn/local-repo) which is part of the basis#2022-10-1814:57Alex Miller (Clojure team)it's passed through a lot of layers in case it is needed in some future evolution so may be unused in many cases#2022-10-1814:59Alex Miller (Clojure team)pretty much anywhere you see config in those apis, that's what it means#2022-10-1814:59dergutemoritzI see, makes sense!#2022-10-1814:59dergutemoritzSo passing the function value of basis is a legit bug then?#2022-10-1814:59Alex Miller (Clojure team)no, that's correct - it is the map containing those keys if they exist#2022-10-1815:00Alex Miller (Clojure team)it contains other stuff too#2022-10-1815:00dergutemoritzHm? No it's the value of (defn basis ...)#2022-10-1815:00Alex Miller (Clojure team)oh, then that's bad#2022-10-1815:00dergutemoritzAh I found the culprit: https://github.com/clojure/tools.deps.alpha/commit/b04ac4f25508df1d8e643baa1baec7abe91e2f75#2022-10-1815:01dergutemoritzit used to be shadowed by the destructuring binding of basis which was removed by that commit#2022-10-1815:01Alex Miller (Clojure team)yeah, will fix#2022-10-1815:01dergutemoritz👍#2022-10-1815:02dergutemoritzThanks for the clarifications 🙂#2022-10-1815:03Alex Miller (Clojure team)now I have to remember if I intended to remove it entirely or what I was doing then :)#2022-10-1815:04dergutemoritzJust take a reference of your brain as of that time! 😄#2022-10-1815:19Alex Miller (Clojure team)if only#2022-10-1815:22dergutemoritzWe need a cybernetic Datomic implant#2022-10-1815:41Alex Miller (Clojure team)pushed fix for next release#2022-10-1815:41dergutemoritz🙌#2022-10-1902:28jasonjcknI’d like to do clj -P inside a dockerfile/docker build stage to cache all the maven dependencies, then clj -M at the entrypoint, but i’m concerned clj -Mwill still try to fetch dependencies over the network and error out and for various reasons I can only do network calls in docker build . is my concern warranted (?), if so is there a fix (other than producing uberjars).#2022-10-1902:36seancorfieldYou might be able to use the -Scp option to pass it a pre-built classpath so it just uses what is specified without trying to download anything.#2022-10-1902:37jasonjcknAh yah, so -Spath
and then java -cp with the saved result of -Spath. But clj itself has a -Scp option in case you need to run Clojure tools without recomputing/fetching anything.#2022-10-1902:41seancorfield(you mentioned clj -M so I figured you were trying to do something specific with :main-opts)#2022-10-1903:01hiredmanclj won't even run the process that has access to the code to fetch deps unless it hasn't computed the classpath for your deps.edn yet#2022-10-1903:02hiredmanWhich is what -P is for#2022-10-1903:04jasonjcknif i run -P, but then change the deps.edn it does a network call right#2022-10-1903:04hiredmanRight#2022-10-1903:04jasonjcknso that was the basis of my concern, you’re saying it won’t do a network call if deps.edn has the same content#2022-10-1903:06hiredmanDepends what you mean by same content, if the bytes hash the same (so whitespace and comments, etc count)#2022-10-1903:06hiredmanYou need to make sure you are using the same set of aliases as well#2022-10-1903:06jasonjckngotcha… well excellent, that’s what I was hoping for#2022-10-1916:49Carsten BehringI was playing with something like that in the past. And I think there is no guaranty that clj does not "try" to access the network. The -P is more for "reducing download time" then to guarantee that no network call occur, I believe.
Maybe worth to check with clojure.core team.#2022-10-2009:43Wanja HentzeThere are a few cases in which tools.deps will always hit the network, regardless of what is cached.
The one we've ran into most often is https://clojure.atlassian.net/browse/TDEPS-223.
That particular issue can be worked around by making all the git deps use full hashes and no tags.#2022-10-2009:45Wanja HentzeOffline builds are not really a design goal I think, so you need a few tricks to make them work. Alternatively you can run a pre-populated git + maven caching proxy within your sandboxed build, but that's a bit more involved.#2022-10-2401:18Alex Miller (Clojure team)Clojure CLI pre-release 1.11.1.1177 is now available:
• https://clojure.atlassian.net/browse/TDEPS-70 - Detect missing jar in classpath and recompute
• https://clojure.atlassian.net/browse/TDEPS-232 - Add more generic posix installer
• https://clojure.atlassian.net/browse/TDEPS-200 - Clean up default user deps.edn
• https://clojure.atlassian.net/browse/TDEPS-219 Fixed -X:deps find-versions doesn't find Maven versions that start with a letter
• Add :n option to -X:deps find-versions and default to 8
• During clojure.tools.cli.api/prep, use :exec-args of alias being prepped, if present
• During clojure.tools.cli.api/prep, also prep current project if needed
• Fix bad invocation of deps/prep-libs! in clojure.tools.cli.api/prep
• Update deps to latest#2022-10-2401:22Alex Miller (Clojure team)If you are an active user of -X:deps find-versions or -X:deps prep , would love to have your feedback on those
If you are a user on Linux or BSD (or even Mac - but note that you will nuke your brew setup so you should know what you're doing to recover!) and wanted to try the posix installer:
curl -O
chmod +x posix-install-1.11.1.1177.sh
sudo ./posix-install-1.11.1.1177.sh
#2022-10-2403:34seancorfieldThat says posix in line one and then linux in line 2&3.#2022-10-2403:36seancorfieldAlso, is the linux script still there? There are various tools/installers out there which expect that name I think?#2022-10-2403:56Alex Miller (Clojure team)Thanks, and yes this is an addition#2022-10-2403:57Alex Miller (Clojure team)Depending on feedback I might just make them the same, but they will just be same thing under different names#2022-10-2409:29borkdudeNice work making the linux installer compatible with macOS as well. I've been using my own tweaked script for that in CI for a while, I can finally throw that away#2022-10-2412:04Alex Miller (Clojure team)Well props on that to this guy! https://ask.clojure.org/index.php/12328/linux-install-sh-could-be-more-compatible#2022-10-2410:20borkdudeJust synced deps.clj with the newest Clojure CLI changes
Without changing your local clj installation, for trying out the 1177 updates you can now also do:
curl -sLO
clj -M deps.clj -X:deps find-versions :lib cheshire/cheshire
or replace clj -M with bb if you have babashka installed.#2022-10-2417:18hlshipI'm having an issue related to GitHub actions ... my library defines a :deps/prep (see https://github.com/pedestal/pedestal/blob/hls/08192022-deps/service/deps.edn#L41) but fails when invoked on the CI server:
https://github.com/pedestal/pedestal/actions/runs/3314689356/jobs/5474302457
with Error building classpath. The following libs must be prepared before use: [current project]
I've been unable to reproduce this locally, even after deleting .cpcache and ~/.m2/.repository.#2022-10-2417:19hlshipCLI version 1.11.1.1177, Clojure 1.11.1#2022-10-2417:22Alex Miller (Clojure team)this is new in the 1177 prerelease (not a stable release yet) - the error is finding that the project itself needs prepping (prior this wouldn't require prepping)#2022-10-2417:24hlshipFeel like I'm in a chicken-or-the egg situation here; the project needs prepping but the prep command fails because the project need prepping.#2022-10-2417:27Alex Miller (Clojure team)I'll take a look at it. you can fall back to the last stable version of CLI to get past though#2022-10-2417:27Alex Miller (Clojure team)which is 1.11.1.1165#2022-10-2417:28Alex Miller (Clojure team)kind of curious if you intended to get the latest alpha of the CLI or if you accidentally got that - I just released it last night#2022-10-2417:29Alex Miller (Clojure team)I see you have
- name: Setup Clojure
uses: DeLaGuardo/#2022-10-2417:30Alex Miller (Clojure team)I believe using cli: 1.11.1.1165 there would drop you back#2022-10-2417:30Alex Miller (Clojure team)and I'm guessing you can't repro locally if you're not on latest there#2022-10-2417:33hlshipThanks; when you mentioned 1177 I though about dropping back. I'll do exactly that.#2022-10-2417:58Alex Miller (Clojure team)also, I filed this: https://github.com/DeLaGuardo/setup-clojure/issues/75#2022-10-2614:42Alex Miller (Clojure team)I fixed this prep issue, but also made it not the default behavior, and that's in latest stable. Also, clojure-setup action now only uses stable versions (but this is now in latest stable)#2022-10-2418:09Alex Miller (Clojure team)if you have tooling that needs to know the "latest CLI version", please start shifting to use this file which will be updated automatically to have that info:
https://download.clojure.org/install/stable.properties#2022-10-2418:12Alex Miller (Clojure team)the contents of that file are "version sha" (sha is of the clojure-tools-$version.tar.gz)
For example, current contents are:
1.11.1.1165 517ba45cfe83b750a58f8246bcee569ae149d8724690f2dbc8ff3187153fe9d1
#2022-10-2418:16hiredmanuser=> (doto (java.util.Properties.) (.load (.getContent (java.net.URL. ""))))
{"1.11.1.1165" "517ba45cfe83b750a58f8246bcee569ae149d8724690f2dbc8ff3187153fe9d1"}
user=>
#2022-10-2418:19Alex Miller (Clojure team)in bash...
read stable_version stable_sha < stable.properties
is useful to know#2022-10-2418:44seancorfieldNice. I've updated our vendor script at work to check against the stable version and require an explicit flag to allow other versions to be installed, just as a sanity check.#2022-10-2418:45seancorfieldI also switched to the posix- install script and was a bit surprised to get Access Denied trying to curl the 1165 version of it (which doesn't exist) rather than a 404 (not a big deal but figured I'd mention it).#2022-10-2418:49Alex Miller (Clojure team)maybe it's there and I'm denying you :)#2022-11-0323:41Eugenok, I am also getting this error while tring to add deps for dialog lib.
How do I fix it - it's not clear to me what I need to do.
Error building classpath. The following libs must be prepared before use: [amperity/dialog]
amperity/dialog {:git/url ""
:sha "f00a9c9099a78415ff2188877708f98c1ae57f6b"}
#2022-11-0323:43EugenI did upgrade to latest cli:
clj --version
Clojure CLI version 1.11.1.1182
#2022-11-0400:08seancorfieldI'm on my phone but look in the official docs for prep lib#2022-11-0401:42Alex Miller (Clojure team)clj -X:deps prep#2022-11-0409:34Eugenhi and thanks @U064X3EF3, @U04V70XH6.
I did find the docs and I tired but did not work for me.
I am using polylith structure and I have the lib added as a development dependency (dev alias)
clj -X:deps prep
clj -X:deps:dev prep
Error building classpath. The following libs must be prepared before use: [amperity/dialog]
I can probably add the lib as normal dependency and run prep then - but it feels like something is missing#2022-11-0409:37Eugenp.s. Addin the library as :deps {} works of course .
Is this an issue that should be added and tracked?#2022-11-0412:45Alex Miller (Clojure team)I don’t understand from the history here what didn’t work, so don’t know#2022-11-0412:50Eugenadding a git lib with prep to an alias does not prep the library#2022-11-0412:51EugenIMO I can't supply the alias to prep command so I can prep the lib#2022-11-0412:51Alex Miller (Clojure team)Oh, that does work, just not the way you tried it#2022-11-0412:51Eugen:)#2022-11-0412:52Alex Miller (Clojure team)clj -X:deps prep :aliases ‘[:dev]’#2022-11-0412:54Alex Miller (Clojure team)https://clojure.github.io/tools.deps.alpha/clojure.tools.cli.api-api.html#clojure.tools.cli.api/prep#2022-11-0412:57Eugenanother case of documentation :(#2022-11-0412:57Eugenthanks#2022-11-1520:07phronmophobicoh, I had the same problem yesterday.#2022-10-2513:29timoIs it possible to specify a path to a deps.edn-file? Background is that I want to run a backwards-compatibility-test inside my project with the prior version of my deps each time my CI runs. So I would download from github the deps.edn-file from main-branch and run my tests with it.#2022-10-2513:53Alex Miller (Clojure team)no#2022-10-2516:13borkdudewith deps.clj it is possible ...#2022-10-2516:14borkdudebut you could also just swap the deps.edn file on disk perhaps#2022-10-2513:56Alex Miller (Clojure team)Clojure CLI pre-release 1.11.1.1182 is now available (diffs from 1177 in bold):
• https://clojure.atlassian.net/browse/TDEPS-70 - Detect missing jar in classpath and download automatically
• https://clojure.atlassian.net/browse/TDEPS-232 - Add generic posix installer
• https://clojure.atlassian.net/browse/TDEPS-200 - Clean up default user deps.edn
• https://clojure.atlassian.net/browse/TDEPS-219 Fixed -X:deps find-versions doesn't find Maven versions that start with a letter
• Add :n option to -X:deps find-versions and default to 8
• During -X:deps prep, use :exec-args of alias being prepped, if present
• During -X:deps prep, ~also prep current project if needed~ add
• Fix bad invocation of deps/prep-libs! in clojure.tools.cli.api/prep
• Improve error message when local dep directory is missing
• Update deps to latest#2022-10-2515:03Alex Miller (Clojure team)if anyone has any feedback on the pre-release, let me know today, will probably do a stable release tomorrow#2022-10-2621:09Noah BogartI'm trying to migrate my company from leiningen to deps.edn. I've got it mostly working (step by step), and i'm currently stuck on a repl-related issue. in leiningen's project.clj, i can say :repl-options {:init-ns crossbeam.dev :init (start) :nrepl-middleware [cider.nrepl/cider-middleware]} This will pass the middleware to nrepl, start up the repl, and then enter crossbeam.dev and execute crossbeam.dev/start before passing control to me. In deps.edn, I have :aliases {:repl {:main-opts ["-e" "(require,'crossbeam.dev),(in-ns,'crossbeam.dev),(start)" "-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]" This seems to be the same (except for printing #object[clojure.lang.Namespace 0x70949361 "crossbeam.dev"] as it loads), but I'm unsure that it's doing the same thing. Is this roughly correct?#2022-10-2621:15seancorfieldIf you're on Clojure 1.10 or later, that could be "-e" "((requiring-resolve '" and if you're on a recent CLI version you should be able to omit those , but otherwise, yeah, that seems right.#2022-10-2621:16Noah BogartI forgot about requiring-resolve, but I'd like for the repl to start in .clj file and just do a requiring-resolve invocation on it.#2022-10-2621:18seancorfieldI've never understood why anyone cares what ns the REPL starts in -- it is going to track whatever REPL you're eval'ing code from your editor with anyway?#2022-10-2621:18seancorfield(and I never type into a REPL -- I don't even have the REPL visible in my editor)#2022-10-2621:20seancorfieldAt this point, I've also moved any app start stuff into REPL command snippets in VS Code/Calva, so that I can start a "bare" REPL without it starting any processes -- and then I can control what is started/stopped.#2022-10-2621:21seancorfield(but if you want all your devs to be able to start the app the same way, it makes sense in a .clj file in the repo so everyone can invoke it that way)#2022-10-2621:21seancorfieldBut we have ~20 apps so we tend to start/stop different apps depending on what part of the repo we're working in 🙂#2022-10-2621:29Noah BogartSadly, we have a somewhat complicated docker set up and need a consistent entrypoint for our devs of varying clojure experience. Good point about the repl, you’ve made me realize that I can remove the in-ns stuff because we run the repl headless (in docker). Should have noticed that already.#2022-10-2621:31Noah BogartThanks for the input #2022-10-2715:29Noah BogartThe uberjar output by b/uber is much bigger than the one created by leiningen. is there any way to determine the source of the difference? (If this is something to ask in #C0AB48493 I can move the question over as well.)#2022-10-2716:22seancorfieldWas your uberjar AOT'd with Leiningen? Perhaps that's the difference? Either way, I'll be interested to hear what comparing the contents of the two JARs reveals...#2022-10-2716:22Noah Bogartit was, we have :aot :all in the :uberjar profile.#2022-10-2715:53borkdudeyou can unzip it and see what's in there#2022-10-2715:53borkdudezipinfo ~/.m2/repository/io/github/borkdude/SnakeYAML/1.33/SnakeYAML-1.33.jar
#2022-10-2715:53borkdude(from my bash history)#2022-10-2716:26Noah BogartThanks for the suggestion, I'm looking now with zipinfo and seeing that 1) all of the dependency source files are included (.clj, .java) and 2) certain dependencies are included as well. makes me wonder how leiningen did it. For example, we use fundingcircle/jackdaw, which has a dependency on aleph. the tools uberjar has aleph/flow.clj etc and both aleph/utils/RequestTimeoutException.java and compiled aleph/utils/RequestTimeoutException.class files, whereas the leiningen doesn't have any files related to aleph in it.#2022-10-2716:32seancorfieldHow does jackdaw declare the dependency on aleph? I wonder if it assumes a Maven scope that tools.build/t.d.a don't understand?#2022-10-2716:33seancorfieldHmm, nope, it's a regular dependency... I would expect Leiningen to just add all that in...#2022-10-2716:35seancorfieldIt makes no sense for it to be excluded from an uberjar if it is a dependency. An uberjar needs everything so it can be run.#2022-10-2716:35seancorfieldPerhaps there's something in your project.clj that is telling Leiningen to exclude some of that stuff from the uberjar for some reason?#2022-10-2716:47Alex Miller (Clojure team)and it is possible to do the same by creating a modified basis that excludes a dep in tools.build if you want that. not sure why dependency sources would be included unless you're doing something to pull the source classifier jar#2022-10-2717:08dpsuttonleiningen would traditionally make two jars. are you sure you are comparing the uberjar from lein to the uberjar from tools.build?#2022-10-2717:10dpsuttonI made a new lein project, added aleph to it and ran lein uberjar
❯ ls -l target/uberjar
total 18032
drwxr-xr-x 4 dan wheel 128 Oct 27 12:09 classes
-rw-r--r-- 1 dan wheel 8501590 Oct 27 12:09 hi-0.1.0-SNAPSHOT-standalone.jar
-rw-r--r-- 1 dan wheel 13802 Oct 27 12:09 hi-0.1.0-SNAPSHOT.jar
drwxr-xr-x 3 dan wheel 96 Oct 27 12:09 stale
note there are two jars of vastly different sizes#2022-10-2717:14Noah BogartAha, I was looking at the wrong one. Thanks for the help, everyone#2022-10-2717:16seancorfieldNow I'm curious how much difference in size there is (or not) between the actual uberjar from Leiningen (not the library JAR!) and the uberjar from tools.build??#2022-10-2717:16Noah BogartUsing zipcmp:
$ zipcmp -i muncher-0.1.0-SNAPSHOT-standalone.jar muncher-standalone.jar
--- muncher-0.1.0-SNAPSHOT-standalone.jar
+++ muncher-standalone.jar
...
- 1450 faa8953c buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012$fn__63013$fn__63014.class
- 2704 dda4f5d4 buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012$fn__63013.class
- 841 485e6fe9 buddy/core/bytes$concat$iter__63004__63010$fn__63011$iter__63006__63012.class
- 1882 6785282a buddy/core/bytes$concat$iter__63004__63010$fn__63011.class
- 783 ac02f762 buddy/core/bytes$concat$iter__63004__63010.class
+ 1450 3ed59749 buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079$fn__63080$fn__63081.class
+ 2704 00c515eb buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079$fn__63080.class
+ 841 e78f36c7 buddy/core/bytes$concat$iter__63071__63077$fn__63078$iter__63073__63079.class
+ 1882 fd25a2e6 buddy/core/bytes$concat$iter__63071__63077$fn__63078.class
+ 783 97cf0ea9 buddy/core/bytes$concat$iter__63071__63077.class
+ 984 954a77bf buddy/core/bytes$concat.class
- 984 a46bbf08 buddy/core/bytes$concat.class
- 2067 2450f6ba buddy/core/bytes$equals_QMARK_.class
+ 2067 621de668 buddy/core/bytes$equals_QMARK_.class
- 3564 6024f654 buddy/core/bytes$fill_BANG_.class
+ 3564 d744d4cf buddy/core/bytes$fill_BANG_.class
- 1758 dc20a963 buddy/core/bytes$fn__62989.class
+ 1758 00eb8eeb buddy/core/bytes$fn__63056.class
- 2292 98b7ccfc buddy/core/bytes$loading__6789__auto____62987.class
+ 2292 ecfbc801 buddy/core/bytes$loading__6789__auto____63054.class
+ 6129 7c316f0d buddy/core/bytes__init.class
- 6129 ec4dd1d6 buddy/core/bytes__init.class
#2022-10-2717:18dpsuttonwhat about the size difference? that’s comparing things you wouldn’t expect to be identical. But if that’s the diff (and its just counters in the compiler) you are golden#2022-10-2717:18seancorfieldLooks like the sizes are all identical there.#2022-10-2717:20Noah Bogartthe output .class files look to be the (roughly) same, maybe a byte of difference in the occasional file#2022-10-2717:20seancorfield(Clojure compilation is not a reproducible process, i.e., you don't get identical byte code each time you compile the same code but you get equivalent byte code -- those differences in generated classes are just that)#2022-10-2717:20dpsuttoni’m interested in the size of the two jars. if they are close then no problemo#2022-10-2717:21Noah Bogart$ exa -la
.rw-rw-r-- 160M noah 27 Oct 11:24 muncher-0.1.0-SNAPSHOT-standalone.jar
.rw-rw-r-- 31M noah 27 Oct 11:23 muncher-0.1.0-SNAPSHOT.jar
.rw-rw-r-- 159M noah 27 Oct 11:21 muncher-standalone.jar
#2022-10-2717:22seancorfieldAnd I thought some of our uberjar files were big! 🙂#2022-10-2717:23Noah BogartWe don't do any splitting at all, so this is our entire 63k lines of clojure backend lol#2022-10-2717:25seancorfield(our largest one is about 79M at this point -- but we've been working hard to keep them small... most of ours are around 35M... about 18 uber JARs from 130K lines of Clojure)#2022-10-2817:37Ian Fernandezhttps://clojure.atlassian.net/browse/TDEPS-8#2022-10-3119:51tskardalWhen I run clojure -Sdescribe I get a map with symbols as keys instead of keywords. Any idea why? (macOS, clojure installed with homebrew)
❯ clojure -Sdescribe
{ version "1.11.1.1182"
config-files ["/usr/local/Cellar/clojure/1.11.1.1182/deps.edn" "/Users/thomas/.clojure/deps.edn" ]
config-user "/Users/thomas/.clojure/deps.edn"
config-project "deps.edn"
install-dir "/usr/local/Cellar/clojure/1.11.1.1182"
config-dir "/Users/thomas/.clojure"
cache-dir "/Users/thomas/.clojure/.cpcache"
force false
repro false
main-aliases ""
repl-aliases ""}#2022-10-3119:59hiredmanare you sure the : just isn't printing for some reason? there is a suspicious space in front there#2022-10-3120:07tskardalnot sure at all 😛#2022-10-3120:07tskardalthe only reason it bothers me is that Cursive-IDE is not working because of this#2022-10-3120:08hiredmanif you have hexdump something like clojure -Sdescribe | hexdump -b|head -1 | awk '{print $3}' should print 072 (octal for ':')#2022-10-3120:09hiredmanbut I would expect if it is just a printing issue it wouldn't bother cursive#2022-10-3120:09tskardalthe result of that command is 040#2022-10-3120:09hiredmanuser=> (char 040)
\space
user=>
#2022-10-3120:10hiredmanso it is what it is, weird#2022-10-3120:15hiredmanI would check your clojure script#2022-10-3120:15hiredmanlooks like that map should come pretty directly from here https://github.com/clojure/brew-install/blob/1.11.1/src/main/resources/clojure/install/clojure#L464-L475#2022-10-3120:18tskardallooks the same#2022-10-3120:19Alex Miller (Clojure team)that's .... pretty weird#2022-10-3120:19tskardalyup#2022-10-3120:20tskardaloh well. thanks for helping out. I’ll leave it for a while. Not in a hurry 🙂#2022-10-3120:20Alex Miller (Clojure team)maybe try a different terminal?#2022-10-3120:20Alex Miller (Clojure team)something with encoding?#2022-10-3120:21tskardalgood idea. I tried both Terminal and iTerm now, but the result is the same#2022-10-3120:22hiredmanI would try editing the map in the clojure shell script (add an extra key or something) and see if it prints out when you run -Sdescribe again#2022-10-3120:23tskardaladded a :lol which becomes lol when printed 😬#2022-10-3120:25hiredmanfascinating#2022-10-3120:27Alex Miller (Clojure team)well it is Halloween, probably a ghost#2022-10-3120:31hiredmanis /usr/bin/env bash --version actually bash?#2022-10-3120:31Alex Miller (Clojure team)what shell are you using ( echo $0 )?#2022-10-3120:50tskardalI’m using fish.
❯ /usr/bin/env bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)
Copyright (C) 2007 Free Software Foundation, Inc.
#2022-10-3122:14pithylessFish uses spaces to define lists (e.g. a PATH in fish is separated by spaces, not colons). To be backwards compatible with bash, there are code paths that will automagically swap between colons and spaces. From the docs:
> Variables whose name ends in “PATH” are automatically split on colons to become lists. They are joined using colons when exported to subcommands. This is for compatibility with other tools, which expect $PATH to use colons.
I suspect @tskardal may be hitting some weird corner case. 😅 Note: I am not a fish user, but did try looking this up out of curiosity. :)#2022-11-0105:43athosI ran into the same issue on my Mac
Seems like the culprit is the semicolon at this line? https://github.com/clojure/brew-install/blob/9c515c07314f9e1458c3425128105a73577c17c3/src/main/resources/clojure/install/clojure#L370#2022-11-0105:43athosRemoving that semicolon resolved the issue for me#2022-11-0216:22tskardalFor me too! Thanks 🙂#2022-11-0216:24tskardalShould this be reported in some way?#2022-11-0216:30Alex Miller (Clojure team)is it valid to remove that semicolon in other shells?#2022-11-0218:56tskardalI don’t think the shell is relevant here. At least for me the issue is present in bash and zsh too.#2022-11-0219:08Alex Miller (Clojure team)well what is relevant? that is, this does work now for lots of people, so what determines whether it works or not for different people?#2022-11-0219:16tskardalI thought you meant that this “fixes” fish shell, but maybe breaks bash (or others). Maybe I misunderstood.
I did some reading, and experimenting. If I understand it correctly,
with the semicolon, the IFS variable remains : until printing the map . When removing the semicolon it doesn’t really get changed? And perhaps the classpath check isn’t really working? (not sure what’s happening there)
Maybe it should be like this?
# Are .jar files in classpath missing?
oldIFS="$IFS"
IFS=':'; read -ra cp_entries <<<"$(cat $cp_file)"
for cp_entry in "${cp_entries[@]}"; do
if [[ "$cp_entry" == *.jar && ! -f "$cp_entry" ]]; then
stale=true
break
fi
done
IFS="$oldIFS"
Disclaimer: I don’t know what. I’m doing 😅#2022-11-0219:19Alex Miller (Clojure team)the intent here is that using it in the same line means that it is set only for the subcommand in the read and set back after the read#2022-11-0219:20Alex Miller (Clojure team)I mean you don't have to figure it out if you don't know already, but this is work I'll have to do#2022-11-0219:21Alex Miller (Clojure team)and I'm trying to figure out why it doesn't work on your machine so I can at least verify that I've fixed it after, and I still don't understand that#2022-11-0219:23Alex Miller (Clojure team)seems like maybe this has to do with the scope of that IFS setting and the intent not being satisfied that this only is in effect on that one read call#2022-11-0219:23tskardalI see. If you echo "$IFS" on the line below I expect you to see it is still :. And that it doesn’t really just apply to that statement.#2022-11-0219:24Alex Miller (Clojure team)and then that IFS setting affects printing later on some shells?#2022-11-0219:24tskardalI’m not sure why it does, but seems so, yes.#2022-11-0219:27tskardalI’m on GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21) - could it be related to the bash version? As it is bash script after all, and I guess what shell you run it from doesn’t matter.#2022-11-0219:33tskardalI created a script to reproduce it, and it works fine in GNU bash, version 5.1.4(1)-release (arm-unknown-linux-gnueabihf)#2022-11-0219:34tskardalI mean, the colon doesn’t disappear there, but it does when running it on my mac.#2022-11-0219:35Alex Miller (Clojure team)I'm running a newer bash than that and it works fine for me#2022-11-0219:35Alex Miller (Clojure team)but yes, I suspect it's likely bash version related#2022-11-0219:35tskardalyes, upgrading bash on my mac removes the problem 🎉#2022-11-0219:36Alex Miller (Clojure team)I've captured this here: https://clojure.atlassian.net/browse/TDEPS-233#2022-11-0219:36Alex Miller (Clojure team)I try not to depend too much on having a recent version of bash though so I will look at it eventually#2022-11-0219:40tskardalCool 🙂#2022-11-0300:24lreadI've just experienced this same symptom for v1.11.1.1182 on GitHub Actions on macOS.#2022-11-0300:27lreadRunning clojure -Sdescribe as a run step seems to work fine. But when I spawn the same from a bb script to capture output the colons are missing.#2022-11-0305:18Alex Miller (Clojure team)released that in 1.11.1.1189#2022-11-0313:47lreadThanks @alexmiller! New version works as expected on GitHub Actions macOS.#2022-11-0106:33tskardalInteresting, @athos! I'll try it out as soon as I get back to my mac 🙂
@pithyless that could be, but it works fine with fish on linux 🤷#2022-11-0119:37seancorfield@alexmiller I'm seeing something very odd with the CLI/deps cache -- I run this command to generate a classpath at work:
clojure -Spath -A:build:dev:everything:runner:test
but every now and then, instead of returning the full classpath as expected (with hundreds of entries), I get back just the bare Clojure deps as if I had run clojure -Spath (with no aliases).
Is it possible there's a hash collision for the filenames used in .cpcache? If I blow away .cpcache (or use -Sforce), it produces the full classpath again (of course).#2022-11-0119:40Alex Miller (Clojure team)yes, it's possible#2022-11-0119:46Alex Miller (Clojure team)it's actually basically using a crc checksum of the concat'ed alias names (simplifying a little), certainly could be non-unique#2022-11-0119:47Alex Miller (Clojure team)on that particular example, I get a string value of val=||:build:dev:everything:runner:test||||||/usr/local/Cellar/ which yields cksum of 3992683027#2022-11-0119:52seancorfieldIf deps.edn changes, would it recompute the cached data?#2022-11-0119:52Alex Miller (Clojure team)yes#2022-11-0119:53seancorfieldI thought it would but wanted to check.#2022-11-0119:53Alex Miller (Clojure team)the stale check hits all the deps.edn files, any local manifest files, and will also become stale now on missing .jar files#2022-11-0119:53seancorfieldAh, right.#2022-11-0119:54Alex Miller (Clojure team)if you catch it again, throw a -Sverbose on there to see the cp_file and grab that set of files, then -Sforce and compare#2022-11-0119:55seancorfieldOK, I'll remove the -Sforce I've added to prevent the problem and I'll keep an eye on it...#2022-11-0119:57Alex Miller (Clojure team)seems unlikely you'd hit a collision with no aliases though, so maybe worth thinking about other ways you could see that result#2022-11-0119:58Alex Miller (Clojure team)or restating, if that was the collision, seems like you'd see it all the time#2022-11-0119:59Alex Miller (Clojure team)for example, if you missing a file system fsync and did not see the just written file at all, you'd have the same result?#2022-11-0120:01Alex Miller (Clojure team)mac is well known to cheat fsync for example and not actually flush necessarily#2022-11-0120:01seancorfieldThis is Windows/WSL2.#2022-11-0120:05Alex Miller (Clojure team)if you retry without force, do you still get the bare path?#2022-11-0120:07Alex Miller (Clojure team)if so, I presume the file contents has the shorter path and the problem is in building the classpath (or it's a cksum collision)#2022-11-0120:24seancorfieldAfter a force, it produces the right classpath for quite a while. I need to figure out what action somewhere in the system is causing the overwrite. Will keep you posted. It happens pretty often.#2022-11-0300:37seancorfieldOK, it happened today. Here's the verbose output trying to run the CLI after it created the wrong path etc:
(/Developer/workspace/wsmain)-(!2092)-> (cd clojure/ && clojure -Sverbose -Spath -A:build:dev:everything:runner:test)
version = 1.11.1.1182
install_dir = /var/www/worldsingles/build/clojure
config_dir = /home/sean/.clojure
config_paths = /var/www/worldsingles/build/clojure/deps.edn /home/sean/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/1422105138.cp
src:/home/sean/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.jar:/home/sean/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/home/sean/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar
Wed Nov 02 17:33:45
(/Developer/workspace/wsmain)-(!2093)-> (cd clojure/ && tar cvzf /mnt/c/Users/seanc/Downloads/tda_debug.tgz .cpcache/1422105138.*)
.cpcache/1422105138.basis
.cpcache/1422105138.cp
.cpcache/1422105138.libs
I'll attach that tgz file...#2022-11-0300:38seancorfieldLMK if this helps:#2022-11-0300:40seancorfieldRunning it with force fixes the problem:
(/Developer/workspace/wsmain)-(!2095)-> (cd clojure/ && clojure -Sverbose -Sforce -Spath -A:build:dev:everything:runner:test)
version = 1.11.1.1182
install_dir = /var/www/worldsingles/build/clojure
config_dir = /home/sean/.clojure
config_paths = /var/www/worldsingles/build/clojure/deps.edn /home/sean/.clojure/deps.edn deps.edn
cache_dir = .cpcache
cp_file = .cpcache/1422105138.cp
Refreshing classpath
... giant classpath removed ...
and you can see the sizes of the files now which are expected:
(/Developer/workspace/wsmain)-(!2096)-> ls -l clojure/.cpcache/1422105138.*
-rw-r--r-- 1 sean sean 395258 Nov 2 17:39 clojure/.cpcache/1422105138.basis
-rw-r--r-- 1 sean sean 48481 Nov 2 17:39 clojure/.cpcache/1422105138.cp
-rw-r--r-- 1 sean sean 597 Nov 2 17:39 clojure/.cpcache/1422105138.jvm
-rw-r--r-- 1 sean sean 211646 Nov 2 17:39 clojure/.cpcache/1422105138.libs
-rw-r--r-- 1 sean sean 13604 Nov 2 17:39 clojure/.cpcache/1422105138.manifest
#2022-11-0304:06Alex Miller (Clojure team)did this start happening with the latest release?#2022-11-0304:19seancorfieldI hadn't seen this happening until recently so I think it's a new failure mode, yes.#2022-11-0304:22seancorfieldThis is why I was asking about the hash (crc) thing because the cache file has the "right" number for the aliases etc -- it's always 1422105138 for that invocation (when it works) but sometimes it seems to act as if it ignored the aliases (after calculating the correct hash/crc).#2022-11-0304:23seancorfieldI haven't looked at the shell script, but is it possible that two concurrent runs of the CLI script could interfere with each other? Does it use any fixed temporary file paths, anything like that?#2022-11-0304:24seancorfieldI ask because this seems to coincide with me restarting LSP in VS Code which I have configured to a CLI script to generate the classpath because of the workspace / repo structure I have.#2022-11-0304:27Alex Miller (Clojure team)it may be related to the other issue that's being discussed re IFS and I've just released 1.11.1.1189 if you want to bump to that and see if things change#2022-11-0304:30seancorfieldOK, I'll make a note to try that tomorrow at work.#2022-11-0304:31seancorfieldSince it's happening every day or two, I should have a reasonable level of comfort deciding whether that's a fix or not 🙂#2022-11-0304:31Alex Miller (Clojure team)really, that's the only change that I can think of that would be related to this#2022-11-0119:49Alex Miller (Clojure team)but in a particular project the only thing varying there in this case is the concatenated alias string#2022-11-0216:24tskardalShould this be reported in some way?#2022-11-0223:00jasonjcknWould it be possible to get a published jar release for tools.build, the git hashes don’t play well with our networks at work #2022-11-0223:32Alex Miller (Clojure team)Yeah, it’s on my list, I’ll try to look at it soon#2022-11-0223:33jasonjcknthank you 🙏 #2022-11-0223:34jasonjcknfwiw, i unblocked myself for the time being so no rush https://clojars.org/org.clojars.jasonjckn/tools.build#2022-11-0305:19Alex Miller (Clojure team)released as org.clojure/tools.build {:mvn/version "0.8.4"}#2022-11-1422:40jasonjcknthank you @U064X3EF3#2022-11-1422:40jasonjckngratitude-thank-you#2022-11-0223:01jasonjcknWith a jar, i can more easily get it mirrored onto corp artifactory #2022-11-0713:51borkdudegit deps with submodules still aren't supported right?
https://clojure.atlassian.net/browse/TDEPS-126#2022-11-0713:51borkdudeI might have to change my project setup because of this, maybe I'll just have to copy some source around... or put multiple projects into one git repo.. thinking-face#2022-11-0713:59Alex Miller (Clojure team)Correct#2022-11-0714:00borkdudeI could just work around this by making a mvn build which includes the submodules#2022-11-0714:01borkdudewhich also comes with caveats#2022-11-0714:58Adam HelinsAbout multiple CLJC projects in one repo, you might be interested by this: https://github.com/protosens/monorepo.cljc
It contains a Babashka-friendly tool called Maestro for managing a CLJ monorepo from a single root deps.edn. I did announce an experimental version of this way back but this is the new, stable iteration.
So far it has served my purposes very well: being able to maintain any number of "modules" easily combined + being able to expose any as Git libraries.#2022-11-0714:59borkdudeThat's cool, I've seen it before, but I think I want to keep those projects each in their own repos, which imo works better for managing OSS projects#2022-11-0715:00borkdudeI'll just write some bb scripts to update the git sha based on my local checkout more often#2022-11-0715:04borkdudeand I'll start (ab)using the feature that src directories can be missing for developing locally :)#2022-11-0821:19markaddlemanI”d like to add a Maven *-tests.jar to my deps.edn and have tools.deps transitively bring in dependencies. From what I can tell, the maven group and artifact IDs for the *-tests.jar are the same as the real jar. I can’t find a way to express a dependency maven scope within deps.edn. I’m sure this question has come up before but my google skills are failing me.#2022-11-0821:22seancorfield"scopes" are not supported in t.d.a but "classifiers" are: https://clojure.org/reference/deps_and_cli#_dependencies -- I'm not sure if that helps with this case?#2022-11-0821:23markaddlemanoh jeesh…. yet another maven concept I have to learn 🙂#2022-11-0821:23markaddlemanThanks for the pointer#2022-11-0821:23markaddlemanNow I have yet another rabbit hole 🙂#2022-11-0821:24seancorfieldYou could try $tests in the artifact ID -- $sources should bring in the -sources.jar I think, based on my (limited) understanding of this...#2022-11-0821:25markaddlemanThanks. I’m experimenting with that right now#2022-11-0821:26markaddlemanaha! That worked. Thanks a ton!#2022-11-0821:31Alex Miller (Clojure team)Yes, that’s the correct way to do it#2022-11-1010:19danmHow can I point tools.deps at a different Maven settings.xml? I know it obeys the properties in ~/.m2/settings.xml, but for various reasons I can't use that default path (`${HOME}/.m2/settings.xml`), I need to have my settings file elsewhere. The Internet talks a lot about M2_HOME, but mostly that talk is along the lines of "It's deprecated and causes all sorts of issues, don't use it", and I don't know if tools.deps obeys it anyway#2022-11-1010:26Wanja HentzeYou can do the following, slightly different hack: https://github.com/bevuta/clojure-nix-locker/blob/master/utils.nix#L7-L8#2022-11-1010:27Wanja Hentzeand then have .m2/... in that alternative "home directory"#2022-11-1010:28Wanja HentzeI do not know how deprecated/evil this is, I just know it works 😄 Join me in Hyrum's Law-ing it into part of the API#2022-11-1010:33danmHmm. Can have a go and see if that'll work. Bazel really doesn't like you exposing env vars, but I might be able to set the tool option. Then again, telling it the entirety of it's user.home is somewhere else, rather than just Maven settings, might cause other issues...#2022-11-1010:39Wanja Hentzeah, you're using bazel, neat#2022-11-1010:39danmAhha, digging into the tools.deps and maven source is giving me a potential route here#2022-11-1010:39Wanja Hentzeyes, beating tools.deps into shape for use in hermetic build systems is quite a piece of work 😄#2022-11-1010:40danmI think I can set org.apache.maven.user-settings#2022-11-1010:40danmBecause the default settings builder which tools.deps defers to will use that. I think#2022-11-1010:42Wanja Hentzehttps://imgs.xkcd.com/comics/software_development_2x.png
this is how it feels to use tools.deps from Nix (and probably bazel too) 😄#2022-11-1014:44Alex Miller (Clojure team)If you need to set a jvm property on the process that builds the classpath, you can do that by setting env var CLJ_JVM_OPTS (this is preferable to setting JAVA_TOOL_OPTIONS which applies to the user program as well, and also won't spam your stderr)#2022-11-1010:26Wanja HentzeYou can do the following, slightly different hack: https://github.com/bevuta/clojure-nix-locker/blob/master/utils.nix#L7-L8#2022-11-1013:44Carsten BehringCan I exclude "files" (not dependencies) during an uberjar creation ?
I have a case were I get an issue because, I get to files copied to 'META-INF", namely
mailcap and mailcap.default
Removing one manually from the jar solves it.
It seems hat this option:
:exclude - coll of string patterns (regex) to exclude from deps
is for excludings deps only.#2022-11-1014:37Alex Miller (Clojure team)no, it's for paths#2022-11-1014:38Alex Miller (Clojure team)"Use :exclude to exclude specific paths from the expanded deps."#2022-11-1014:38Alex Miller (Clojure team)(also, #C02B5GHQWP4 exists for tools.build questions)#2022-11-1023:28Carsten BehringThanks. The docu use different wording in 2 places
Maybe the Clojuredoc of ubercould be clarified to make it clear that the exclusion is about paths.#2022-11-1016:17FiVoHey, I am building some uberjar with some dependencies coming from a private artifactory. Building the jar gives me
Execution error (HttpResponseException) at org.eclipse.aether.transport.http.HttpTransporter/handleStatus (HttpTransporter.java:472).
status code: 409, reason phrase: Conflict (409)
and further down in the error report
22 {:type org.eclipse.aether.resolution.ArtifactDescriptorException,
23 :message
24 "Failed to read artifact descriptor for io.confluent:kafka-avro-serializer:jar:5.3.2",
25 :at
26 [org.apache.maven.repository.internal.DefaultArtifactDescriptorReader
27 loadPom
28 "DefaultArtifactDescriptorReader.java"
29 255]}
I build the same jar with lein just fine. Is tools.deps doing some stricter checking regarding snapshots and releases as this seemed to be a common problem while googling this issue?#2022-11-1016:18FiVoSo the same thing happens when try to get a repl via clj and it works fine with lein repl#2022-11-1016:18FiVo$ clj --version
Clojure CLI version 1.11.1.1165
`#2022-11-1018:49Alex Miller (Clojure team)can you provide something more to reproduce? I don't have enough info to comment#2022-11-1018:50Alex Miller (Clojure team)not doing anything special wrt snapshots#2022-11-1018:51Alex Miller (Clojure team)or could you provide more of the exception trace?#2022-11-3016:53uwoI've been having a similar problem -- a tools-deps project can't locate a dependency that my lein projects find fine.
I've even used Datomic's bin/maven-install, and I can see that the dependency is available in my local .m2
Clojure CLI version 1.11.1.1200#2022-11-3017:01Alex Miller (Clojure team)Can you share your deps.edn #2022-11-3017:03uwoSure thing#2022-11-3017:07uwo#2022-11-3017:09uwosorry, let me fix that.. I had reverted to a version of datomic that didn't encounter this issue#2022-11-3017:09uwofixed#2022-11-1023:28Carsten BehringThanks. The docu use different wording in 2 places
Maybe the Clojuredoc of ubercould be clarified to make it clear that the exclusion is about paths.#2022-11-3016:53uwoI've been having a similar problem -- a tools-deps project can't locate a dependency that my lein projects find fine.
I've even used Datomic's bin/maven-install, and I can see that the dependency is available in my local .m2
Clojure CLI version 1.11.1.1200#2022-11-1110:50simongrayI recently made a security report using nvd-clojure and I got some terminal output that looks like this (showing mostly issues with transitive dependencies). Is there a one-liner I can use to quickly figure out which of my explicit dependencies result in one of these transitive dependencies?
When I run clj -X:deps tree the output isn’t that easy to sort through (or grep for ancestors), so I spend a lot of time searching through it to figure things out. I’ve tried making a graph using tools.deps.graph which would be idea, except the output is a bitmap image which can’t be searched…#2022-11-1112:26pavlosmelissinosWith clj -X:deps tree :format :edn you can get the output as data, which might help a bit#2022-11-1112:27pavlosmelissinosBut it doesn't solve your other problem (some dependencies not appearing at all)#2022-11-1112:35simongrayyeah… anyway, I found that using grep -B 20 solves grepping the result pretty well, so that at least works ok#2022-11-1112:06simongrayand some of them can’t be found at all in the clj -X:deps tree output#2022-11-1112:54Alex Miller (Clojure team)You might try clj-watson to get a little better output, not sure if it will address your concerns#2022-11-1114:28simongrayI turns out that it was because aliases are not considered when you use -X:deps , while they work with -SDeps . What’s up with that?#2022-11-1114:28simongray(pointed out to me by the wonderful @U7PBP4UVA)#2022-11-1114:29simongrayI see: https://ask.clojure.org/index.php/10245/clj-x-deps-tree-ignores-sdeps?show=10245#q10245#2022-11-1116:01pavlosmelissinosIs there a way to figure out which :mvn/repos entry in deps.edn provides which dependency?#2022-11-1116:20Alex Miller (Clojure team)not currently (and there's not necessarily only one answer)#2022-11-1116:21Alex Miller (Clojure team)but this would be a useful thing to know#2022-11-1116:22Alex Miller (Clojure team)You can see it during download so doing something like
clj -Sdeps '{:mvn/local-repo "tmp"}' -Stree
will show you in the output (and leave you the full lib set in ./tmp)#2022-11-1116:50pavlosmelissinosOh, nice, that will do, thanks a lot. 🙂
My use case is a custom maven repo that I suspect is no longer useful, but I'd like a confirmation before I remove it.
Your workaround is good enough but if this is something that you'd consider improving let me know if you'd like to have the question on http://ask.clojure.org and I'll post it.#2022-11-1118:33Alex Miller (Clojure team)go for it, have been thinking it would be useful to have something like this for a while#2022-11-1120:44pavlosmelissinosFYI https://ask.clojure.org/index.php/12383/programmatically-connect-dependencies-maven-repos-provide#2022-11-1408:49mkvlrthere's no way to use an alias of a git dep, or is there?#2022-11-1408:54vlaaadthere is an issue to support it#2022-11-1408:57vlaaadhttps://clojure.atlassian.net/browse/TDEPS-116#2022-11-1409:13mkvlr@U47G49KHQ thanks, I'll upvote it#2022-11-1418:45socksyhaving problems with default-deps again. In one frontend project, using shadow-cljs + deps.edn, I successfully have the following:
:cider-nrepl {:extra-deps {cider/cider-nrepl nil
refactor-nrepl/refactor-nrepl nil}
:default-deps {cider/cider-nrepl {:mvn/version "0.27.4"}
refactor-nrepl/refactor-nrepl {:mvn/version "2.5.1"}}}
this is overridden in my ~/.clojure/deps.edn here:
{
:deps {cider/cider-nrepl {:mvn/version "0.28.3"}
nrepl/nrepl {:mvn/version "0.9.0"}
refactor-nrepl/refactor-nrepl {:mvn/version "3.5.2"} }
}
in a new, backend project, I have the following:
:cider-nrepl {:extra-deps
{cider/cider-nrepl nil
refactor-nrepl/refactor-nrepl nil}
;; In theory this makes it possible to override them in your .clojure/deps.edn
:default-deps {cider/cider-nrepl {:mvn/version "0.28.5"}
refactor-nrepl/refactor-nrepl {:mvn/version "3.5.5"}}}
when I run the backend project, it does not let me override the deps, but when I run the frontend project, it does. Any idea of a lead to fixing this?#2022-11-1418:54socksyno idea what is relevant version information, but fwiw clojure versions in the frontend project are:
org.clojure/clojure {:mvn/version "1.11.1"}
org.clojure/clojurescript {:mvn/version "1.11.60"}
shadow-cljs has :deps true, with shadow-cljs version of {:mvn/version "2.19.8"} (link to github tree as of that commit https://github.com/thheller/shadow-cljs/tree/dd7d5fd71398a60aaa204759b6e06fb6aefa8e2a)
in backend project clojure version is the same, and ran via clojure -M:foo:cider-nrepl#2022-11-1418:54seancorfieldPretty sure that's not how default deps are supposed to work. You'll get the newest version of a dep that is specified -- you can't override in deps with an older version.#2022-11-1418:54socksyah crumbs. That makes sense but totally breaks my use case#2022-11-1418:56Alex Miller (Clojure team)can you just use :replace-deps ?#2022-11-1418:56seancorfieldSpecifying :default-deps alongside nil deps is going to be the same as specifying those versions directly anyway. And specifying :deps in your dot-clojure file is going to add those deps to every project, even when you build JAR files etc. Do you really want CIDER/nREPL etc in everything you build?#2022-11-1418:57Alex Miller (Clojure team)yeah, I was a little unclear if that was at the root or in an alias#2022-11-1418:58socksynot an alias, since unfortunately the calls in both cases are wrapped in tooling that doesn't make it easy to pass in a custom alias :( I figured for builds I would just isolate them in docker anyway since who knows what people could have locally#2022-11-1419:17seancorfieldQ about classpath ordering: did we establish that t.d.a's classpath ordering is non-deterministic, with only the :paths guaranteed to be at the "front" of the classpath in some order? i.e., the classpath calculated on one machine can differ in order from the classpath calculated on another machine?#2022-11-1419:35Alex Miller (Clojure team)Should not be#2022-11-1419:35Alex Miller (Clojure team)More clearly: it should be deterministic #2022-11-1419:35seancorfieldHmm, I was sure we'd had this discussion... back when the change was made to ensure paths ended up at the beginning of the classpath and not some "random" place within it...?#2022-11-1419:36Alex Miller (Clojure team)I changed it#2022-11-1419:37seancorfieldRight, but I thought that was only to ensure paths were at the front, not to ensure a specific ordering consistent across machines...#2022-11-1419:38Alex Miller (Clojure team)This was a long time ago now, but I am also sorting deps by tree depth and then by name, which should be stable#2022-11-1419:38seancorfieldFor example, if the path to .m2 differs, isn't that going to create different hashes on the paths if they're put into a hash map?#2022-11-1419:38Alex Miller (Clojure team)Not hash ordered#2022-11-1419:38seancorfieldAh, OK, so t.d.a classpath ordering is stable. Thank you. I'll go beat on the Polylith team about that then 🙂 Their computed classpath is not stable.#2022-11-1419:39Alex Miller (Clojure team)I believe order will be extra-paths (in order), paths (in order), then deps sorted by tree depth then alpha#2022-11-1419:39Alex Miller (Clojure team)But going by memory#2022-11-1419:40Alex Miller (Clojure team)I think I have a site issue from you to doc this better actually :)#2022-11-1419:40seancorfieldI got bit because Jetty, bless its heart, decided to keep the same class names across different group/artifact JARs so if you end up with multiple versions (with different coordinates), you can pick up the "wrong" class version...#2022-11-1419:40seancorfieldThanks!#2022-11-1419:41Alex Miller (Clojure team)Evil#2022-11-1518:44baptiste-from-parisHello all, I have a question on deps Tools, the -T option.
I’ve noticed that args need a key/value pair which does not satisfy my use case. For example clj -Tcljd -main 1 2 3 will generate a Key is missing value: 3 error. Is there a way around this ?#2022-11-1518:52seancorfield-T works like -X and calls a function with a hash map.
You can't call -main as that expects a sequence of strings.#2022-11-1518:54seancorfieldYou would need an "exec" function entrypoint such as:
(defn main-x [{:keys [args]}] (apply -main args))
which you could then invoke as
clojure -Tcljd main-x :args '["1" "2" "3"]'
(you could do (apply -main (map str args)) and then pass :args '[1 2 3]' on the command-line if you wanted to avoid the quoting).#2022-11-1518:55seancorfieldBut the usual approach is to have -main parse its arguments into a hash map and then call an "exec" style function -- parsing string args with something like clojure.tools.cli (which produces a hash map of arguments).#2022-11-1518:57seancorfieldThe Cognitect test-runner is structured like that: https://github.com/cognitect-labs/test-runner
It has an api.clj entrypoint for direct exec that massages the initial hash map something before invoking the same function that -main invokes.#2022-11-1518:57baptiste-from-parisOk, thanks a lot Sean 🙏 #2022-11-1707:38Christian JohansenWhat’s the reason paths can refer to paths above the current directory (“../“) but extra-paths can’t?#2022-11-1712:32borkdudePaths outside the current directory are not support (anymore) @christian767#2022-11-1712:33Christian JohansenHmm, ok, that’s a shame. It works in the version I’m on now 😅 What’s the reason for dropping the support?#2022-11-1712:49mpenetwait what? you mean for local/root for instance?#2022-11-1712:49mpenetoh paths/extra-paths#2022-11-1713:00Christian Johansenpaths#2022-11-1713:15Alex Miller (Clojure team)It doesn’t make sense and security implications if you are outside if your directory (someone using this project as a git dep could be referencing arbitrary parts of your file system). If you are in a sub dir of a git dep, you can refer relatively via a local dep#2022-11-1713:17Christian Johansenaha, sounds like I want the latter then 👍#2022-11-1715:21p-himikIs there the right way to solve this without creating a separate repo/branch for the tool?
https://github.com/liquidz/antq/issues/108#2022-11-1715:29Alex Miller (Clojure team)At the moment, you would need some kind of wrapper for the tool. You could either put that in a separate repo or put it in a subdir and use a git dep with a deps root and a local dep back to the root. I think that should work#2022-11-1715:30Alex Miller (Clojure team)But I can think about it some more#2022-11-1715:37p-himik> put it in a subdir and use a git dep with a deps root and a local dep back to the root
Maybe I'm misunderstanding, but doesn't it require being able to use a Git dep that's in a subdir? Is it even possible?
So e.g. suppose we have
antq
- tool
- deps.edn
- deps.edn
with tool/deps.edn being
{:deps {org.slf4j/slf4j-nop {:mvn/version "RELEASE"}
com.github.liquidz/antq {:local/root ".."}}}
How would one install antq using tool/deps.edn instead of deps.edn?#2022-11-1715:47Alex Miller (Clojure team)You can add :deps/root to point to a subdir of a git dep#2022-11-1716:10p-himikAh, nice, thanks!#2022-11-1716:37seancorfieldFWIW, that's typically how Polylith-based libraries and tools are made available (with :deps/root).#2022-11-1716:38seancorfieldSee https://polylith.gitbook.io/poly/install/install/use-as-dependency (Via GitHub section) and https://polylith.gitbook.io/poly/install/install/install-as-clojure-cli-tool#2022-11-1716:39seancorfield(and https://github.com/seancorfield/polylith-external-test-runner which hasn't been announced yet)#2022-11-1817:49seancorfieldRight now, :paths and :extra-paths are the only things that can contain aliases used as data that gets expanded, yes? I'm cleaning up JVM options in our large deps.edn/`build.clj` setup at work and desperately yearning for that same ability in :jvm-opts... is there a Jira ticket for that, perhaps?#2022-11-1817:58Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-184#2022-11-1817:59seancorfieldHah! It was even my "ask" -- for exactly this situation we have at work, which has just gotten more complex since I asked for that feature!#2022-11-1818:00Alex Miller (Clojure team)realistically, I'm unlikely to have time to work on it soon, so depending on your desire for urgency, patches welcome :)#2022-11-1818:01Alex Miller (Clojure team)even if starting a patch uncovered questions, that would be helpful#2022-11-1818:03seancorfieldOK, I might take a run at that since it sounds like it would be accepted/appreciated.#2022-11-1818:06Alex Miller (Clojure team)I'm not opposed to it, just not sure where the dragons lie (maybe none!)#2022-11-1820:07seancorfieldSince I'm not a Maven user, any idea about this:
(~/oss/tools.deps.alpha)-(!2022)-> mvn -version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 19, vendor: Eclipse Adoptium, runtime: /Developer/jdk-19+36
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.15.68.1-microsoft-standard-wsl2", arch: "amd64", family: "unix"
Fri Nov 18 12:05:55
(~/oss/tools.deps.alpha)-(!2023)-> mvn test
[ERROR] Error executing Maven.
[ERROR] java.lang.IllegalStateException: Unable to load cache item
[ERROR] Caused by: Unable to load cache item
[ERROR] Caused by: Could not initialize class com.google.inject.internal.cglib.core.$MethodWrapper
[ERROR] Caused by: Exception com.google.inject.internal.cglib.core.$CodeGenerationException: java.lang.reflect.InaccessibleObjectException-->Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @2bbaf4f0 [in thread "main"]
#2022-11-1820:07seancorfieldDo I just need to use an older JDK to run it?#2022-11-1820:32Alex Miller (Clojure team)well the ClassFormatError and the "module" suggest to me that there is something either class version or module related#2022-11-1820:33Alex Miller (Clojure team)-e will print more of the error#2022-11-1820:33Alex Miller (Clojure team)and -X is debug mode#2022-11-1820:34Alex Miller (Clojure team)oh, you're trying to run the tools.deps build itself, didn't realize that#2022-11-1820:35Alex Miller (Clojure team)I don't currently run have anything newer than Java 17, and it runs with that#2022-11-1820:41Alex Miller (Clojure team)I'm using this now and mvn test works for me:
% mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/Cellar/maven/3.8.6/libexec
Java version: 19, vendor: Homebrew, runtime: /usr/local/Cellar/openjdk/19/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "12.6.1", arch: "x86_64", family: "mac"
#2022-11-1820:42Alex Miller (Clojure team)maybe mvn clean test to make sure you don't have anything old in target?#2022-11-1820:48seancorfieldAh, 3.6.3 is what's installed by apt on Ubuntu for me. It runs on JDK8 just fine. I ought to rebuild my dev env on a more recent Ubuntu at some point 🙂 Thanks for the sanity check.#2022-11-1820:49seancorfieldSo, for TDEPS-184, how do you feel about expanding it to support :main-opts as well? (I have it all working for :jvm-opts already with some fun tests)#2022-11-1820:52seancorfieldIt would be almost trivial at this point to support aliases in :main-opts with the code I've added for :jvm-opts -- and it might be useful to folks...#2022-11-1820:53Alex Miller (Clojure team)hold off on that for now, it plays into other ideas I have#2022-11-1820:54Alex Miller (Clojure team)semantics are different too due to replacement rather than unioning#2022-11-1820:55seancorfieldThat's why I asked 🙂 But it would now be as simple as changing this:
;; handle jvm and main opts
jvm (seq (flatten-opts :jvm-opts exec-argmap merge-edn))
main (seq (get exec-argmap :main-opts))]
to this
;; handle jvm and main opts
jvm (seq (flatten-opts :jvm-opts exec-argmap merge-edn))
main (seq (flatten-opts :main-opts exec-argmap merge-edn))]#2022-11-1820:57seancorfieldThe addition to make_classpath2.clj is this (a variant of how paths are handled in alpha.clj):
(defn- chase-opts-key
"Given an aliases set and a keyword k, return a flattened vector of
options for that k, resolving recursively if needed, or nil."
[aliases k]
(let [opts-coll (get aliases k)]
(when (seq opts-coll)
(into [] (mapcat #(if (string? %) [%] (chase-opts-key aliases %))) opts-coll))))
(defn- flatten-opts
[opt-key exec-argmap {:keys [aliases] :as merge-edn}]
(let [aliases' (assoc aliases opt-key (get exec-argmap opt-key))]
(into [] (remove nil?) (chase-opts-key aliases' opt-key))))#2022-11-1820:57Alex Miller (Clojure team)just drop me a patch and I'll take a look#2022-11-1820:57seancorfieldWith both or just JVM opts?#2022-11-1820:58seancorfieldI'll do both since it's now so easy. And you can decide what to do with it.#2022-11-1820:58seancorfieldI will add some :main-opts tests for this case as well tho'...#2022-11-1821:04Alex Miller (Clojure team)with just jvm-opts, don't want the main-opts right now#2022-11-1821:07seancorfieldGot it!#2022-11-1821:14seancorfieldPatch attached to the ticket (created from the GH PR against my own repo -- so I hope the format is acceptable).#2022-11-1821:24borkdude@alexmiller Would you mind adding or receive a patch /PR adding a deps.edn to tools.namespace so I can use it as a git dep (for ... reasons, I need the test code on disk which I am abusing git deps for since that clones the whole repo)#2022-11-1821:25Alex Miller (Clojure team)Sure, I can add that#2022-11-1821:25borkdudeI did add a deps.edn at one point:
https://github.com/clojure/tools.namespace/blob/master/deps.edn
but it doesn't seem to pull in the deps from the pom.xml#2022-11-1821:25borkdudewhen used as a git dep#2022-11-1821:30Alex Miller (Clojure team)interesting, I see that#2022-11-1821:35Alex Miller (Clojure team)When I run it out of a tools.deps repl it works, so something weird there#2022-11-1821:37borkdudeout of a tools deps repl?#2022-11-1821:37borkdudeI would be fine with just replicating the pom.xml paths and deps in deps.edn though, the trick I used there is admittedly a bit hacky#2022-11-1821:47Alex Miller (Clojure team)I take it back, it is working for me, I flubbed the cli#2022-11-1821:47Alex Miller (Clojure team)do you have a repro of this not working?#2022-11-1821:48Alex Miller (Clojure team)% clj -Stree -Sdeps '{:deps {io.github.clojure/tools.namespace {:git/sha "bd78b34d9551c99910a24ce939c2137fcf1bede1"}}}'
org.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
io.github.clojure/tools.namespace bd78b34
. org.clojure/tools.namespace /Users/alex.miller/.gitlibs/libs/io.github.clojure/tools.namespace/bd78b34d9551c99910a24ce939c2137fcf1bede1
. org.clojure/java.classpath 1.0.0
. org.clojure/tools.reader 1.3.6#2022-11-1822:05borkdude@alexmiller My situation is like:
clj -Stree -Sdeps '{:deps {org.clojure/tools.namespace {:git/url "" :git/sha "bd78b34d9551c99910a24ce939c2137fcf1bede1"}}}'
which outputs:
org.clojure/clojure 1.11.0
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
org.clojure/tools.namespace bd78b34
X org.clojure/tools.namespace /Users/borkdude/.gitlibs/libs/org.clojure/tools.namespace/bd78b34d9551c99910a24ce939c2137fcf1bede1 :use-top
Since I want to override any other tools.namespace dependency, I use org.clojure/tools.namespace as the libname, I thought that was the recommended practice#2022-11-1822:06borkdudebut maybe this prevents including the pom.xml self-reference#2022-11-1822:07Alex Miller (Clojure team)oh, yeah that won't work as is due to the same name#2022-11-1822:07Alex Miller (Clojure team)in this case, probably easiest to just double-maintain the libs#2022-11-1822:08borkdudeagreed#2022-11-1822:10Alex Miller (Clojure team)pushed#2022-11-1822:10Alex Miller (Clojure team)e08261e4b522e45916bdff24a1ac99b478676f12#2022-11-1822:13borkdudeThanks! Does it also need "src/main/clojure" in :paths?#2022-11-1822:14borkdudeI think it does#2022-11-1822:19borkdudebut I can add that manually in my tests to the classpath, so no hurry#2022-11-1822:25Alex Miller (Clojure team)It doesn’t, as that is always in the deps list via the root#2022-11-1822:26borkdudeThat's only src right?#2022-11-1822:26Alex Miller (Clojure team)No#2022-11-1822:26borkdude$ clj -Sdeps '{:deps {io.github.clojure/tools.namespace {:git/sha "e08261e4b522e45916bdff24a1ac99b478676f12"}}}'
Clojure 1.11.0
user=> (require '[clojure.tools.namespace])
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate clojure/tools/namespace__init.class, clojure/tools/namespace.clj or clojure/tools/namespace.cljc on classpath.#2022-11-1822:27borkdudeI also don't see src/main/clojure in the classpath#2022-11-1822:27borkdude$ clj -Spath -Sdeps '{:deps {io.github.clojure/tools.namespace {:git/sha "e08261e4b522e45916bdff24a1ac99b478676f12"}}}'
src:/Users/borkdude/.gitlibs/libs/io.github.clojure/tools.namespace/e08261e4b522e45916bdff24a1ac99b478676f12/src:/Users/borkdude/.m2/repository/org/clojure/clojure/1.11.0/clojure-1.11.0.jar:/Users/borkdude/.m2/repository/org/clojure/java.classpath/1.0.0/java.classpath-1.0.0.jar:/Users/borkdude/.m2/repository/org/clojure/tools.reader/1.3.6/tools.reader-1.3.6.jar:/Users/borkdude/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/Users/borkdude/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar#2022-11-1822:29Alex Miller (Clojure team)oh, should be src/main/clojure#2022-11-1822:29Alex Miller (Clojure team)but notice clojure is there#2022-11-1822:30Alex Miller (Clojure team)daf82a10e70182aea4c0716a48f3922163441b32#2022-11-1822:31borkdudeworks, thanks a lot!#2022-11-1822:31Alex Miller (Clojure team)root deps.edn is: https://github.com/clojure/tools.deps.alpha/blob/master/src/main/resources/clojure/tools/deps/deps.edn#2022-11-1822:32Alex Miller (Clojure team)it has a :deps that includes clojure by default (and this is why you can just run clojure in any directory and start clojure.main repl#2022-11-1822:33borkdudeyes, I get that, but I've never seen src/main/clojure (not just src) be added to the classpath automatically, which was what I was getting at#2022-11-1822:33borkdudethanks for the fix#2022-11-1822:33Alex Miller (Clojure team)ah, then yes. we actually used to do that in the early days#2022-11-1822:34Alex Miller (Clojure team)but I decided that was 2x more special things than we needed :)#2022-11-1822:35borkdudeI'm using git deps to also pull tests to disk which I'm then running with babashka. Currently I'm making it (bb) work with tools.namespace which then makes it work with the cognitect test runner as is#2022-11-1822:37Alex Miller (Clojure team)nice#2022-11-1822:48borkdudeI guess we could have just used a dummy name in that deps.edn local/root, which was not the name of the lib itself, then it would also have worked#2022-11-1822:48borkdudebut then that name would appear in the deps tree which would be a bit odd#2022-11-1822:53Alex Miller (Clojure team)Yes that would have worked too#2022-11-1904:10Alex Miller (Clojure team)would it be wrong if I just automatically set -XX:-OmitStackTraceInFastThrow for every CLI call? omg, worst default setting ever#2022-11-1904:25seancorfieldHahaha... yeah, I have that in almost every :jvm-opts everywhere! And you could put it in JDK_JAVA_OPTIONS env var (JDK 14+).#2022-11-1904:26Alex Miller (Clojure team)but if you do that it nags at you on every startup:#2022-11-1904:26Alex Miller (Clojure team)NOTE: Picked up JDK_JAVA_OPTIONS: -XX:-OmitStackTraceInFastThrow
#2022-11-1904:27seancorfieldOh, I did not know that -- hey @hiredman did you know the JDK warns you about that env var?#2022-11-1904:27hiredmanI did not#2022-11-1904:29Alex Miller (Clojure team)if we set it first in the command, you could even override it if you needed to I believe#2022-11-1904:31Alex Miller (Clojure team)yes, confirmed#2022-11-2016:07gregHi everyone, two short questions about deps' git/url:
1. is it possible to pull lein project?
2. is it possible to pull a project from a repo's subdirectory (e.g. one git repo holds multiple libraries, each within a separate subdirectory, with its own deps.edn)#2022-11-2017:08Alex Miller (Clojure team)1. currently, only if it has no deps (by setting :deps/manifest :deps)
2. yes, by setting :deps/root to the subdir#2022-11-2110:36dergutemoritzRe 1. see also https://github.com/borkdude/lein2deps#2022-11-2116:07borkdudeI have a project that has a dependency on a git lib with a deps/root. That deps/root project has a local/root dependency on another project in that git repo. That should work right?#2022-11-2116:08borkdudeBut it doesn't unless I'm doing something wrong here:
clj -Sdeps '{:deps {io.github.squint-cljs/cherry {:git/sha "0d7ef527ea983cbe4a0f8ee95e388e78d80d14f4"}}}'#2022-11-2116:12Alex Miller (Clojure team)"doesn't work" == ??#2022-11-2116:14Alex Miller (Clojure team)io.github.squint-cljs/compiler-common has two libs in the tree, one as a local coord, one as a git coord - version selection doesn't know how to compare those#2022-11-2116:14Alex Miller (Clojure team)yes, the local is actually in the context of a git coord, but you've lost that context#2022-11-2116:16Alex Miller (Clojure team)we have a ticket for this general problem at https://clojure.atlassian.net/browse/TDEPS-132#2022-11-2116:24borkdudeupvoted#2022-11-2117:10borkdude@alexmiller
I see the problem, I shouldn't have used the compiler-common name twice. So the top level dep is a deps/root in the git repo which then requires the other project via local/root via a different name. I think that ought to work.
{:paths ["src" "resources"]
:deps {io.github.squint-cljs/compiler-common-cherry
{:git/sha "45a6d93d20869183d922041f109124f5625cad39"
:deps/root "cherry"}}}
I pushed that to :git/sha "2f323066cbc74147991c2f33148075af83088057"
But I'm now getting:
$ clj -Sdeps '{:deps {io.github.squint-cljs/cherry {:git/sha "2f323066cbc74147991c2f33148075af83088057"}}}'
Cloning:
Error building classpath. Unable to clone /Users/borkdude/.gitlibs/_repos/https/github.com/squint-cljs/compiler-common-cherry
fatal: could not read Username for '': terminal prompts disabled#2022-11-2117:16dergutemoritzLooks like you're referring to a private repo there#2022-11-2117:16dergutemoritzi.e. https://github.com/squint-cljs/compiler-common-cherry is not publicly readable#2022-11-2117:18borkdudefacepalm#2022-11-2117:20borkdudeof course, I have to add :git/url ""#2022-11-2117:23borkdudeit works now:
$ clj -Sdeps '{:deps {io.github.squint-cljs/cherry {:git/sha "14e0a95e0c01129f3967b9419fd4799376e23d09"}}}'
Checking out: at 14e0a95e0c01129f3967b9419fd4799376e23d09
Checking out: at 45a6d93d20869183d922041f109124f5625cad39
Clojure 1.11.0
user=> (require '[cherry.compiler :as cherry])#2022-11-2117:24Alex Miller (Clojure team)private repos are fine of course (but you have to auth). useful things to know:
• set GITLIBS_DEBUG=true to see git commands being executed
• set GITLIBS_TERMINAL=true to get terminal prompts#2022-11-2117:25borkdudeThis isn't private, I just forgot to add the git/url, I only changed the io.github.* name to avoid the conflict but of course that wasn't sufficient since it's also used to derive the git/url#2022-11-2117:25Alex Miller (Clojure team)👍#2022-11-2117:26borkdudeso it actually works like expected now, I just shouldn't have used the same libname for different purposes#2022-11-2117:30borkdudeio.github.squint-cljs/compiler-common$cherry
for deps/roots ;)#2022-11-2117:44Alex Miller (Clojure team)I don't love it, but should work#2022-11-2117:52borkdudewhat I (kind of non-seriously) proposed was that $cherry should be ignored for derivation of git/url but would still count as a different lib (cause different name)#2022-11-2117:23borkdudeit works now:
$ clj -Sdeps '{:deps {io.github.squint-cljs/cherry {:git/sha "14e0a95e0c01129f3967b9419fd4799376e23d09"}}}'
Checking out: at 14e0a95e0c01129f3967b9419fd4799376e23d09
Checking out: at 45a6d93d20869183d922041f109124f5625cad39
Clojure 1.11.0
user=> (require '[cherry.compiler :as cherry])#2022-11-2215:33borkdudelein2deps can now also be used as a clojure -Ttool
https://github.com/borkdude/lein2deps#usage#2022-11-2511:18Ian FernandezIs there any tooling to convert an pom.xml to a deps.edn? 🙂#2022-11-2511:44borkdude@d.ian.b Perhaps you can leverage tools deps or tools build itself for this, since I assume it can create a basis from a pom.xml#2022-11-2514:16Alex Miller (Clojure team)There’s not really any tool for that although the hard part of reading and understanding the pom is buried in there#2022-11-2514:20borkdude@alexmiller You can't do something like (b/basis "pom.xml") ?#2022-11-2514:21Alex Miller (Clojure team)Well not that of course, but you could include a local dep to a dir containing a pom#2022-11-2514:22borkduderight. and then the basis will contain all kinds of stuff which would let you write a deps.edn from that (with some manual wrangling of course)#2022-11-2514:22Alex Miller (Clojure team)You’ll get something useful, but not sure it’s exactly a deps.edn#2022-11-2514:23Alex Miller (Clojure team)You also won’t get the mvn/repos out of that as we don’t grab those transitively#2022-11-2514:24Alex Miller (Clojure team)Really, you’d only get deps indirectly #2022-11-2514:25Alex Miller (Clojure team)But the code will build the pom model, from which you could get anything#2022-11-2718:33Ben LiebermanOk, continuing over here. I'm wondering if there's any correspondence between the :mvn/repos key and how you would specify a <repository> in a pom.xml. Bc if not, I'm guessing I am going about this wrong.#2022-11-2718:34Alex Miller (Clojure team)there is a correspondence of course. but seeing the actual error would help#2022-11-2718:34Ben LiebermanError building classpath. Could not find artifact org.onebusaway:onebusaway-gtfs:jar:1.3.95-SNAPSHOT in central ()
#2022-11-2718:34Alex Miller (Clojure team)all repos will be tried, but you'll only see a single error reported, so sometimes this masks what's actually happening#2022-11-2718:36Ben LiebermanThis is the relevant portion of the pom.xml :
<repositories>
<repository>
<id></id>
<url></url>
</repository>
</repositories>
#2022-11-2718:37Ben LiebermanThanks to @hiredman I got this to work#2022-11-2718:38Ben LiebermanBad version specified in the README#2022-11-2718:42Alex Miller (Clojure team)it's actually bad that that works#2022-11-2718:42Alex Miller (Clojure team)maven will prevent http downloads but doesn't seem like that's getting blocked by tools.deps#2022-11-2718:43Ben LiebermanI can open a ticket if that would be helpful at all#2022-11-2718:43Alex Miller (Clojure team)I am opening one#2022-11-2902:49devnIs there any way to do global exclusions of transitive deps without endlessly restating them per dep?#2022-11-2904:52Alex Miller (Clojure team)not currently#2022-11-2916:21jjttjjI have a java library I'm working on and a separate Clojure project that depends on it.
Is there way to recompile the java and use it from the the clojure project's repl without restarting the repl?
I'm fine with putting its class output path in my clojure project's :paths even though I know that's deprecated/not a good idea, but that doesn't seem to work.
Is there a way to "refresh" the classpath?#2022-11-2916:40Alex Miller (Clojure team)easily, no. java agents do have the ability to do stuff like this but that's not something built in#2022-11-2920:25pyrThere is this: https://github.com/ztellman/virgil#2022-11-2920:26pyrbut it would require you to bring the java dependency in the main project#2022-11-3015:26Noah BogartIs it possible to set the GITLIBS directory within the deps.edn file and furthermore within an alias? I expect no, given https://ask.clojure.org/index.php/11711/set-mvn-local-repo-via-alias about setting :mvn/local-repo in an alias#2022-11-3015:26Alex Miller (Clojure team)no#2022-11-3015:27Noah BogartWould you like an Ask ticket for this?#2022-11-3015:30Alex Miller (Clojure team)I'm not sure it's feasible to implement#2022-11-3015:31Noah BogartOh? That's surprising to me. If you don't mind, what makes it hard to implement?#2022-11-3015:32Alex Miller (Clojure team)the deps process is the things that reads and interprets the deps.edn so in general, things that affect its environment have to be not in deps.edn#2022-11-3015:34Alex Miller (Clojure team)I guess gitlibs can be configured via Java system property, and maybe it would be possible to read the deps.edn, and set that before resolving deps. but I'd have to think about whether there is a good place to even do that right now#2022-11-3015:34Alex Miller (Clojure team)I mean, you can always create an Ask question regardless#2022-11-3015:45Noah BogartDone! https://ask.clojure.org/index.php/12414/set-gitlibs-in-deps-edn#2022-12-0121:54souenzzohttps://github.com/clojure/tools.build points to maven 0.8.5
but it does not exists on maven
https://mvnrepository.com/artifact/org.clojure/tools.build#2022-12-0122:08hiredmanhttps://clojurians.slack.com/archives/C02B5GHQWP4/p1669247667680269?thread_ts=1669247667.680269&cid=C02B5GHQWP4#2022-12-0122:16seancorfield0.8.4 was the first (and so far only) release of tools.build to Maven. It is not normally released to Maven and I think it might be a manual process?#2022-12-0122:17seancorfieldNormally, you'd use tools.build as a git dep... what is your use case for using it as a non-git dep @U2J4FRT2T?#2022-12-0122:21souenzzooh tools build has a channel. nice.
@U04V70XH6 no specific use-case. just tried to use, and it fails. so reporting.#2022-12-0122:30Alex Miller (Clojure team)there was a snafu in the release process, I will fix it on 0.8.6, but really 0.8.5 is not much different than 0.8.4 so it has not seemed worth it to fix to me#2022-12-0123:21lread@U04V70XH6 fwiw, personally, I tried switching to the maven dep to avoid the :git/sha part. No other reason.#2022-12-0123:25seancorfield@UE21H2HHD I guess I have so much dev tooling that uses git deps that it hasn't mattered to me...#2022-12-0123:27souenzzoi love git deps and usually use it for everything. local/root is awesome too!
some things like being able to overwrite a dep just by placing its symbol and pointing to a "dumb root" is way better than handle maven logics
but some companies only allow maven deps, because they have infrastructure on it (dep usage/analysis etc)#2022-12-0123:27seancorfield(we're using a core.async fork via git deps in production, plus Polylith, an external test runner, tools.build, build-clj, the Cognitect test-runner, clj-watson, the Datomic dev.datafy stuff, Stu's reflector project, the add-lib3 branch of tools.deps.alpha ... all via git deps 🙂 )#2022-12-0221:22hiredmanI write a number of little programs (sometimes actually shell scripts, sometimes clojure programs) to do little things here and there, and I have written enough of these that I have really come to value limiting the number of files needed for these utilities. To that end I have something I have been playing with that might be of interest here: https://gist.github.com/hiredman/cd1b57894a19dda3a205ca129626a672 this is the first part of a file that is a shell script, clojure program, clojurescript program, and deps.edn all in one file#2022-12-0415:28Carsten BehringI came now several times to the question if and how clj aliases can be used to specify "init" type of code. So which should execute one time after Clojure is started.
Specially if there is any form of support of having this specified in two aliases and it executes then one after the other.
I am aware of "-e" options and the various ways to set the main-args and main-exec but it seem none of this is made for "init" code, which should run once and then give back control to Clojure.#2022-12-0415:31Carsten BehringThere is as well "user.clj", but this does not solve the problem that I want to "centralise" this init code and "switched it on" when needed#2022-12-0416:42jjttjjYou can use -e to call load-file in an alias#2022-12-0416:44Carsten BehringHow would it look like to use "-e" in an alias ? I have never seen this#2022-12-0416:53Carsten BehringMaybe my question is too general.
I am in the context of starting an nrepl, and would like to have an alias
which contributes some code to be run after the nrepl is started...
The nrepl I start in the same whay for every project (with an user wide clj alias), but for some projects I want that additional code is executed after (or before) repl is started.#2022-12-0416:53jjttjjHere's an example https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L48#2022-12-0417:04Carsten BehringI checked it. It does indeed work if I only use one alias, namely the dev:repl the examples referes too.
But I want combine 2 aliases, one starts nrepl and one runs cod, like this:
clj -M:repl/cider-refactor:dev/repl
#2022-12-0417:05Carsten BehringIn this case, only the second is executed. The "-e" overrides whatever the first says to start.
Switching order does not work neither, as JVM terminates after the "-e" code finishes.#2022-12-0417:08Carsten BehringWhat would work, but I want to avoid, is a specific user.clj, which does both.#2022-12-0417:12jjttjjOh yeah only one main opts will get used. You could have the loaded file determine what to do based on some cli args it accepts#2022-12-0417:42seancorfield@U7CAHM72M the repo that @U064UGEUQ linked to has code that starts nREPL/cider if it is in the classpath and runs other code which is what you'd have to do.#2022-12-0510:05dergutemoritz@U7CAHM72M One way I've solved this in the past is to have a central user.clj which loads different namespaces based on JVM properties. You can then have aliases with different :jvm-opts setting these properties as these will be concatenated across all aliases. Maybe that's what you meant by your last comment anyway, tho 🙂#2022-12-0500:36skylizeDiverting to new thread here, as sharp tangent from the Calva thread this comment came from.
> @seancorfield it's a great service to civilization, that someone keeps up with Clojure Deps. Its direct-to-git dependencies are "obviously" the long-term Right Thing To Do. Hopefully someone will ring a bell over my grave when Deps is finally finished and stable and has a command line as simple as Lein's. -- @phill
What about the CLI tools do you feel is unfinished? What about its interface do you feel is less "simple" than Lein's?#2022-12-0501:12skylize🧵#2022-12-0501:17seancorfieldThanks for creating this thread @U90R0EPHA - I'll be very interested to hear what folks say (although we've had several threads complaining about the exec-style command-line arguments so that won't be anything new).#2022-12-0501:19Alex Miller (Clojure team)btw I finalized a path with Rich last week to move tdeps out of alpha and I’ll start working through that over the next couple of weeks#2022-12-0607:15dumratLack of templates. I'm aware of clj-new but for beginners looking to just get started with a template, it's painful.#2022-12-0617:42seancorfield@UC1DTFY1G Interesting. Can you elaborate? What sort of templates? Is this a discovery issue or something else? What changes to the docs/ecosystem would help address this?#2022-12-0618:12skylizeI generally use deps-new for bootstrapping. I assume I can make a custom template for it, and keep intending to do so. But I haven't gotten around to it because last I checked the readme offers no meaningful guidance for such customization. Knowing I need to reverse engineer one of the built in templates (and maybe also the library?), I fail to find the motivation to get going.
I also don't know of any other templates besides the ones included. Do they exist somewhere? Discovery here could be as simple as a wiki page on the deps-new repo where people can add links to published templates.
---
I think all of that's likely tangential to @UC1DTFY1G's concern though. My guess is the real issue there is likely that neither clj-new nor deps-new get any mention in the official docs, because they are not part of Clojure's official suite of tools.
How to get a project up and running properly without a basic skeleton is not obvious as a beginner. Maybe some extra-minimalist reproduction of deps-new could be included the either tools.deps or tools.build?#2022-12-0618:14seancorfieldA PR was recently merged to add a place to the readme for folks to have their deps-new templates listed: https://github.com/seancorfield/deps-new#templates -- and there's an expanded section on how to write templates and make them available https://github.com/seancorfield/deps-new/blob/develop/doc/templates.md -- happy to take PRs to continue to improve this!#2022-12-0618:15seancorfield(even tho' I wrote and maintained boot-new, clj-new, and deps-new, I pretty much never used any of them -- I always created new projects from scratch)#2022-12-0618:34skylizeIt's easy to see not bothering with such tools once you know what you're doing. But there are a surpising number of things you need to know that a skeleton can let you put off for learning later.
A few things that come to mind:
For a real project you need a separate src and test folder, where namespaces parallel directory structure, test namespaces parallel source namespaces, and dashes are replaced with underscores in the file names.
You need paths specified for building the classpath. You need to know that test paths and test dependencies, if any, need to be in an alias so they aren't destined to be included in your production build.
And I haven't even touched on the possibility of trying to build a release.#2022-12-0618:42seancorfieldI would love to see deps-new mentioned on http://clojure.org as a way to set up "complete" new projects and it could even have its tools EDN file installed as part of the Clojure CLI -- but I doubt that would happen. Given that it is "just" a thin wrapper around tools.build and it's already EPL1 and has almost no contributors except me, it probably wouldn't be much work to get it turned into a Contrib project, if that would make it more palatable to the core team, but again I don't really see that happening... @U064X3EF3?#2022-12-0618:51Alex Miller (Clojure team)most of what the new templates do, I don't think is needed, and I think it's telling that many people don't use them. I'm not opposed to building some "new" type functionality into the cli but I would like to think about it more - I think the neil approach of adding slices of stuff is a better approach. on build-clj, I actually disagree with the approach of wrapping here. the whole point of tools.build is to start with something simple and grow it, but build-clj hides that (and papers over actually understanding what you're build is and does) and makes it harder to grow it later. I would actually rather have templates in this scenario that don't use a wrapper. but maybe there are things we can do in tools.build to lessen the desire for the wrapper in the first place.#2022-12-0618:59seancorfieldWe're not talking about build-clj -- just to be clear. deps-new just uses tools.build, not build-clj#2022-12-0619:01seancorfield> I think it's telling that many people don't use them
Isn't this perhaps a Catch-22 since it isn't mentioned on http://clojure.org so folks don't know about it? There seem to be plenty of new Clojurians asking about guidance for this sort of thing and folks usually point them to clj-new or deps-new...#2022-12-0619:02seancorfield(and if changing the generated projects in deps-new to not use build-clj would make it more palatable, I'm happy to do that 🙂 )#2022-12-0619:04seancorfieldBut, yes, overall I'd like to see something official that makes it easy for new Clojure users to create "fully-featured" simple application and library projects, whatever form that takes -- and I'm happy to contribute to that with whatever guidance the core team want to offer. As with depstar, I'd much rather the official offerings meant I could retire clj-new and deps-new 🙂#2022-12-0619:12seancorfieldhttps://github.com/seancorfield/deps-new/issues/42#2022-12-0701:47dumratImagine you don't know Clojure. And you see someone doing some cool things with it and you want to try it out. There's a lot of people who are so used to doing npm hairball or whatever and just write js and not worry about how the build system works. Perhaps this person would later be interested as to how to create his own project from scratch. But with Clojure, expecting new users to learn Clojure which is quite different to the majority of languages and then learn new IDEs, extensions, repl stuff, and on top of all this, figuring out how to create a project from scratch, this is too much. All he might want to do might be to just start up a web server and explore how that feels.
Apologies if I'm wrong, but I think this is one more barrier to entry. I feel there should be some "standard" way to quickly get a common type of project underway without having to think. Experts in the language can choose not to use these stuff, but for beginners there should be an easy and a standard way to do it. deps is the right place for it as it is the official thing.#2022-12-0716:23seancorfield@UC1DTFY1G Maybe something like clojure -Ttools create-app :name mycool/newapp or clojure -Ttools create-lib :name dumrat/cool-lib out of the box? (I suggest -Ttools since that's "built-in") I suspect the sticking point would be what's the bare minimum that should be in such generated projects, since "The Clojure Way" is somewhat minimal and composable.#2022-12-0716:41dumrat@seancorfield I really don't mind it being verbose (Though I think some would take issue even with that). As long as it works out of the box and documented somewhere official, I think it'd be great.
I think I understand the reluctance to make tools do more than they absolutely have to, and perhaps I'm wrong. But I think we live in an age where attention spans are extremely short. One thing Clojure excels at (according to me is) helping the programmer hold his attention - primarily REPL + not being statically typed. Even someone like me can spend hours at the REPL and my attention span is extremely short. But getting someone to a decent REPL is perhaps the hard part.
Just what I think. I don't have any evidence to back any of this up 🙂#2022-12-0716:57seancorfieldI think one of the key aspects of a "good" Clojure experience for new developers is an editor that feels approachable with a really good story for connected REPLs so that folks can get past the "typing-into-a-REPL" stage of learning to actually doing stuff. I think that's much more important for onboarding than "creating a new project" but I understand why folks obsess about the latter. I think Calva is the only good option right now for the former (and then there's the difficulty for the Clojure core team of how to guide people to that without seeming to "favor" one editor over another and thus implicitly criticizing other editors -- this is a really tricky thing to navigate for communities).#2022-12-0717:06dumratok, I don't understand the stuff in parens - why can't we just have getting started guides for all major IDEs? And it just sucks that there's no good web based integrated REPL that can be showcased in official site. I think I checked a few languages sometime back and at least Rust and Elm had an interactive playground just one click away from the main site. Ironic that Clojure with one of the best REPLs doesn't have this. I haven't tried Replit in a long time, but last I tried, it didn't work very good for clojure.
Sorry, I don't mean to be mean or blame anyone. I'm very happy with Clojure. Just that recently I've been wondering why more people don't see the light 😄#2022-12-0717:09Alex Miller (Clojure team)I welcome your getting start guide PRs....#2022-12-0717:11Alex Miller (Clojure team)I know this seems like an obvious answer, but the problem is that an expert needs to write it, and the experts are people that work on it mostly, and when they work on it they inevitably want it in their own docs, not on the main clojure site. I have been asking people to write these guides for 10 years and no one has ever even started one. Inevitably I will probably write some of them, but it's a huge time investment, not just at the start but continuously to maintain it.#2022-12-0717:13dumrat@U064X3EF3 Beginner here, but will try to add something. 🙂#2022-12-0717:13Alex Miller (Clojure team)I don't actually think that having a repl on site is important - you can't get any of the affordances with an editor connected repl or structural editing that you get from a real environment. If you are serious about trying Clojure, and there is a well-oiled path to that, then people should take the extra few minutes to do so. the Calva gitpod stuff is pretty slick and my best suggestion atm.#2022-12-0717:19seancorfieldI agree -- for the same reason that it's important to get people past the typing-into-a-repl stage: that's not how you work with Clojure "for real" and it's a very limited experience. Having community members contribute (and maintain) "Getting Started" guides for the major IDEs to http://clojure.org would be the path forward here. Based on the last State of Clojure Survey, that's Emacs, Cursive, VS Code/Calva, and I believe vim/neovim was next... But those are four very different audiences and likely to be at least four different contributors -- and you don't really want to duplicate each editor's Getting Started guide (if there is one) but mostly just point to it (right @U064X3EF3?) which would make the PR for clojure-site pretty small for Calva 🙂#2022-12-0717:23seancorfield(getting people to contribute is hard: years ago, I got tired of people complaining that they couldn't easily contribute to the clojure.java.jdbc docs because of the Contrib process with Jira and patches so I moved the docs to https://clojure-doc.org/articles/ecosystem/java_jdbc/home/ where folks could easily submit PRs etc... and the result was "crickets"... so all those people claiming the Contrib process was an obstacle were just complaining about the process and not really intending to contribute!)#2022-12-0717:40Alex Miller (Clojure team)I actually don’t think a set of links is the right answer (having also tried this in the past). There is a narrow set of things to cover and a lot of value in having that same set covered for multiple envs in a consistent way. Maybe I should just write bad versions and then people would correct them :)#2022-12-0804:04skylize> I think one of the key aspects ... for new developers is an editor with a really good story for connected REPLs [to] get past the "typing-into-a-REPL" stage ... to actually doing stuff. I think that's much more important for onboarding than "creating a new project"
I would argue that "creating a new project" is even more important than editor integration for getting to "actually doing stuff".
If you want to make any kind of progress on anything more complex than toy examples, you need to be able to save your accomplishments into a file, and then be able to load those files back into a repl to run again later. If you don't even know how to set up a functional main namespace to start working out of, editor evaluation is not really going to help you.
Besides that, good editor tooling is mostly agnostic to the choice of Leiningen or Clojure CLI. And the point of this thread is not to try and solve all aspects Clojure's New User Story.
The point is to discuss the contrast between the Clojure CLI Tools New User Story and the Leinengen New User Story; to hopefully shake out what can/should be done to address the widely held opinion that beginners should be pointed toward Leiningen from the outset because "it's easier".#2022-12-0804:10skylize> so all those people claiming the Contrib process was an obstacle were just complaining about the process and not really intending to contribute!
Reasonable but unsound conclusion. Could be those people really did want to contribute, but by the time you made it easy they had moved on and had other things on their mind.
But, yeah. Getting people to follow through on thankless unpaid labor in their free time is certainly not easy or straightforward.#2022-12-0804:12skylize> Maybe I should just write bad versions and then people would correct them
Funny but true. Sounds like a brilliant plan to me. yoda#2022-12-0804:28seancorfield> Could be those people really did want to contribute, but by the time you made it easy they had moved on and had other things on their mind.
I moved it while folks were complaining about the Contrib process so, no.#2022-12-0804:37seancorfield> I would argue that "creating a new project" is even more important than editor integration for getting to "actually doing stuff".
I meant that we know how to solve the "creating a new project" issue -- we just have to either document it or put something into the core tooling. The editor-connected-repl workflow seems much, much harder for us to teach to beginners, and we don't yet seem to have a good handle on how to address that.
Showing someone once how to create a new project is sufficient. It's a simple step to teach. Teaching an ongoing workflow or process is harder 😞#2022-12-0805:13George> The point is to discuss the contrast between the Clojure CLI Tools New User Story and the Leinengen New User Story; to hopefully shake out what can/should be done to address the widely held opinion that beginners should be pointed toward Leiningen from the outset because "it's easier".
I am about a month into learning Clojure and started with the CLI; the book I'm following uses Lein but I convert everything to the CLI.
The CLI syntax definitely is alien at first, but the guides on http://clojure.org are IMO very good. So much of beginner programming is rote following of examples personally I don't think focusing on the command line syntax is that important.
Though I started out creating projects by hand eventually I found using a template from deps-new a big help as there are many little things that I didn't even think about when structuring a project. To that point though modifying that template for my own use I found to be quite easy.#2022-12-0805:21seancorfieldWe never did get an answer the original question "What about the CLI tools do you feel is unfinished?" 🙂#2022-12-0805:35skylizeThe comments that led me to start this thread were from @phill, who has not decided to join in on further conversation. So we may not get any response that answers that question directly.#2022-12-1103:07bringeI think putting getting started guides directly on the site for different editors would be great. I’d be happy to write one up for Calva. I think it would be good to start from absolutely no dependencies (no VS Code, no Java, etc.) to evaluating code from project files with a REPL connected to Calva.
Maybe following, for each editor guide, some framework to keep the guides simple and minimal would be a good idea - something like:
1. installation of deps
2. creation of a basic project
3. start and connect to a repl
4. evaluate forms and see output
For any other information, the reader can be directed to the docs of the editor, or other guides on http://clojure.org, where appropriate.
Some other possible guidelines:
• Every guide should use the Clojure CLI for managing deps and starting the REPL
• Maybe each guide should avoid OS-specific details or any details about how to install the deps, instead pointing to official docs for that information. Or maybe it’s good to go into those details…#2022-12-1103:08bringeI’d be happy to get feedback on the above and any other guidance you might have, @U064X3EF3.#2022-12-1517:31Ben SlessI think it's worth mentioning neil here as a pretty decent "hit the ground running" tool for beginners which I don't think papers too much over what actually goes on#2022-12-0912:29pezI'm trying to get a Leiningen project to use deps.edn for dependencies and am using the lein-tools-deps plugin. Things crash like so:
clojure.lang.ExceptionInfo: Could not find artifact com.github.kenglxn.qrgen:javase:jar:2.6.0 in central ()
{:lib com.github.kenglxn.qrgen/javase, :coord {:mvn/version "2.6.0", :exclusions #{org.apache.xmlgraphics/batik-dom org.apache.xmlgraphics/batik-svggen com.google.zxing/javase}, :deps/manifest :mvn}}
...
It is a dependency on one-time that causes this and we are using qr-gen from there, so I can't simply exclude it. I have reason to believe things might work with version 3.0.1 of qrgen. Is there a way to override the version of this when specifying the dependency on one-time?#2022-12-0912:37pezOK, so ChatGPT could help me with how to specify my own version (I added a dependency on qrgen and exclude it in the one-time dependency declaration, hope that is the way to do it). However, I get the same boring message when using 3.0.1. The QRGen page says it's available from jitpack, so now I am thinking that Leiningen somehow has this configured, but tools-deps don't.#2022-12-0912:50pezHmmm, specifying
:mvn/repos {"" {:url ""}}
At the root of my deps.edn file doesn't work. But it gives a different error message:
Could not find artifact com.github.kenglxn.qrgen:core:jar:2.6.0 in central ()
Could not find artifact com.github.kenglxn.qrgen:core:jar:2.6.0 in clojars ()
Could not find artifact com.github.kenglxn.qrgen:javase:jar:2.6.0 in central ()
Could not find artifact com.github.kenglxn.qrgen:javase:jar:2.6.0 in clojars ()
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
#2022-12-0912:55pezThe REPL starts fine with clojure. Hmmm....#2022-12-0913:06Alex Miller (Clojure team)lein-tools-deps hasn’t been updated in 4 years so not sure if it’s still compatible or not#2022-12-0913:07Alex Miller (Clojure team)If it works with clojure, then my guess is that it’s something in the plugin, probably with pulling the mvn-repos over to the lein context. Have you tried adding the repo to the project.clj too?#2022-12-0913:43pezNo, will try that next. Thanks!#2022-12-0913:46pezThat worked! 🙏#2022-12-0913:58Alex Miller (Clojure team)I'm guessing that plugin just pulls the deps, not the :mvn/repos#2022-12-1011:33pezI now realize it is a known issue: https://github.com/RickMoynihan/lein-tools-deps/issues/59#2022-12-2012:48rickmoynihan👋 Sorry just seen this…
I don’t maintain lein-tools-deps anymore, as Alex says it is 4 years out of date with regards to changes in tools.deps, and we migrated away from it a long time ago.
I’ve updated the README to make this clear.#2022-12-2012:55pezThanks. We're probably also going to migrate away from this soon. 😃#2022-12-1110:48Martynas MaciulevičiusI was reading this piece of documentation: https://clojure.org/reference/deps_and_cli#_dependencies
What I want is to have something similar to Leiningen's profiles where I could have test dependencies separate from production dependencies.
I think this should be achieved by :aliases as I didn't find more places where I could put additional dependencies in.
So now I can't understand how to merge two :aliases together. For instance in Leiningen I could have my dev profile where I could declare dev dependencies (something the REPL would run against). Then I could include that profile into test profile so that the tests would include both lists of extra dependencies.
This is an excerpt from my project.clj file:
:profiles {:dev [:project/dev :profiles/dev]
:test [:project/dev :project/test :profiles/test]
:project/dev { ;; extra deps, extra JVM opts etc
When I run lein test I get all dependencies from :project/dev.
Is there a way to achieve the same in tools.deps?
Can :aliases be used the same way as :profiles in Leiningen? How could I do it without respecifying the same list of dependencies?#2022-12-1111:05practicalli-johnIn simple terms, the top level :deps and :path keys in a deps.edn file represent production (and every other environment). The are always included by default (unless explicitly removed by an alias or using the -T exec option)
So all aliases could be viewed as very flexible, on the fly leiningen profiles, although its not usually a 1-2-1 mapping.
Aliases are only used when they are included in the clojure command line, e.g. clojure -M:env/dev:repl/rebel which has two aliases chained together. Aliases merge with the :deps and :path top level keys (and the configuration from the clojure install which adds src as a path and inlcudes org.clojure/clojure as a dependency
In the above example I included two aliases
• :env/dev for an extra-path to include dev directory on the class path and any libraries I only use during development
• repl/rebel to include library dependencies to run Rebel Readline rich terminal UI and to define the main-opts that run nrepl and the rebel code (rather than running the built-in repl prompt)
Or something like clojure -M:env/test:test/run could be used to set the test environment and libraries and start a test runner.
There are some example of how I use aliases at https://practical.li/clojure/clojure-cli/install/community-tools-available.html
There are many examples of aliases I use in my user level configuration https://github.com/practicalli/clojure-deps-edn#2022-12-1111:06Martynas MaciulevičiusOh so I don't need to do clojure -A:test:dev -M:nrepl?
And instead I can just do clojure -M:test:dev:nrepl ?
Oh wow, your config is really overwhelming. But yes, I can now try to make something out of it.#2022-12-1111:26Martynas MaciulevičiusThanks, I think I managed to make it work.#2022-12-1111:37practicalli-johnYes, clojure -M:test:dev:nrepl is what I would recommend. Note that if using :main-opts key in any of the aliases, only the :main-opt from the alias in the chain is called, so only one thing is 'run'.
More details about aliases can be found in this article https://practical.li/blog/posts/clojure-cli-tools-understanding-aliases/ which may be simpler than poring though my deps.edn file 🙂#2022-12-1111:43Martynas Maciulevičius> one thing is 'run'.
The last one from the : list?
I made separate configs for dev, test, clj-test, babashka-test and nrepl. And out of those only the last three will have -m options. This way I could combine dev and test in any way.
I also used bb.edn file to run everything using babashka. This way I don't need to remember any of the commands and they can be more complicated.#2022-12-1116:06practicalli-johnYes, aliases all start with : as they are keywords, so the last alias name in the chain of aliases. So your aliases are :dev , :test , etc...
-M is the alias option that runs :main-opts with clojure.main (`-X` and -T use clojure.exec instead of clojure.main)
-m is an option that sets the main namespace (-m can also be used in :main-opts ) and a main namespace is required for clojure.main#2022-12-1112:38Martynas MaciulevičiusIs there a way to see whether a dependency from git sources was downloaded to somewhere? I created a private repository and the REPL says that it downloads it but I can't seem to use it in the code :thinking_face: I want to investigate whether my namespace names are correct but I'm not sure how to do it.
The downloaded source project is based on tools.deps too.
This is the output from my project where I used the source project as a dependency:
Checking out:
It seems that it should've worked but where can I find the files then?
It creates .cpcache directory. Is this it?
I also tried to move my sources to src/... and to src/clojure/... but neither seem to be working :thinking_face:#2022-12-1112:58Martynas MaciulevičiusFound it, it's under ~/.gitlibs/libs.#2022-12-1114:13Alex Miller (Clojure team)Use clj -Spath to see the final classpath #2022-12-1216:55adamfreyAt my work we are using google cloud's https://cloud.google.com/artifact-registry tool for storing maven dependencies. Google cloud provides a https://github.com/GoogleCloudPlatform/artifact-registry-maven-tools for artifact registry. Using leiningen we can use https://github.com/AdamFrey/lein-google-cloud-artifact-registry to register that wagon library and pull our deps, but it's unclear to me how we achieve the outcome using tools.deps.
I see https://github.com/clojure/tools.deps.alpha/blob/master/src/main/clojure/clojure/tools/deps/alpha/util/maven.clj#L187 that registers the supported maven connector types which I see includes the AWS S3 Transporter, but I don't know how I can hook into that to register another type. Has anyone done this before that could point me in right direction. Or if it isn't possible right now, would google cloud artifact registry be something that would have to be supported explicitly by the tools.deps library itself, and would a PR be considered?#2022-12-1217:10Alex Miller (Clojure team)Hi, I’m certainly open to adding support for this. Maven wagons are the “old school” spi, newer are Maven transports. If there is a transport impl (or we could write one), I’d prefer to do that rather than dragging in wagon support. #2022-12-1217:11Alex Miller (Clojure team)Alternately, it would be possible to integrate directly as a procurer, it that needs some assessment#2022-12-1219:08adamfreyI see. Thanks for the context. I'll have to do some research to see if they provide a Maven transport for Gcloud artifact registry#2022-12-1219:21Alex Miller (Clojure team)my first pass search answer was that they do not. and also it's probably easiest to integrate as a maven repo source given that they're already doing that with a known url protocol#2022-12-1219:21Alex Miller (Clojure team)https://clojure.atlassian.net/browse/TDEPS-172 is the existing jira I already have for this btw#2022-12-1219:23adamfreyoh cool. I should have search ask.clojure first. Thanks, Alex.#2022-12-1219:28Alex Miller (Clojure team)looking at the set of deps pulled in by that wagon makes me sad :(#2022-12-1219:29Alex Miller (Clojure team)% clj -Xdeps list :extra '{:deps {com.google.cloud.artifactregistry/artifactregistry-maven-wagon {:mvn/version "2.2.0"}}}'
com.fasterxml.jackson.core/jackson-core 2.12.2 (Apache-2.0)
com.google.auth/google-auth-library-credentials 1.0.0 (BSD New license)
com.google.auth/google-auth-library-oauth2-http 1.0.0 (BSD New license)
com.google.auto.value/auto-value-annotations 1.8.2 (Apache-2.0)
com.google.cloud.artifactregistry/artifactregistry-auth-common 2.2.0 (The Apache License, Version 2.0)
com.google.cloud.artifactregistry/artifactregistry-maven-wagon 2.2.0 (The Apache License, Version 2.0)
com.google.code.findbugs/jsr305 3.0.2 (Apache-2.0)
com.google.code.gson/gson 2.8.6 (Apache-2.0)
com.google.errorprone/error_prone_annotations 2.5.1 (Apache-2.0)
com.google.guava/failureaccess 1.0.1 (Apache-2.0)
com.google.guava/guava 30.1.1-jre (Apache-2.0)
com.google.guava/listenablefuture 9999.0-empty-to-avoid-conflict-with-guava (Apache-2.0)
com.google.http-client/google-http-client 1.39.2 (Apache-2.0)
com.google.http-client/google-http-client-gson 1.39.2 (Apache-2.0)
com.google.http-client/google-http-client-jackson2 1.39.2 (Apache-2.0)
com.google.j2objc/j2objc-annotations 1.3 (Apache-2.0)
commons-codec/commons-codec 1.11 (Apache-2.0)
commons-io/commons-io 2.6 (Apache-2.0)
commons-logging/commons-logging 1.2 (Apache-2.0)
io.grpc/grpc-context 1.27.2 (Apache-2.0)
io.opencensus/opencensus-api 0.28.0 (The Apache License, Version 2.0)
io.opencensus/opencensus-contrib-http-util 0.28.0 (The Apache License, Version 2.0)
javax.annotation/jsr250-api 1.0 (COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0)
javax.enterprise/cdi-api 1.0 (Apache-2.0)
javax.inject/javax.inject 1 (Apache-2.0)
org.apache.commons/commons-lang3 3.8.1 (Apache-2.0)
org.apache.httpcomponents/httpclient 4.5.13 (Apache-2.0)
org.apache.httpcomponents/httpcore 4.4.14 (Apache-2.0)
org.apache.maven/maven-artifact 3.6.1 (Apache-2.0)
org.apache.maven/maven-model 3.6.1 (Apache-2.0)
org.apache.maven/maven-plugin-api 3.6.1 (Apache-2.0)
org.apache.maven.wagon/wagon-http-shared 3.3.3 (Apache-2.0)
org.apache.maven.wagon/wagon-provider-api 3.3.3 (Apache-2.0)
org.checkerframework/checker-qual 3.8.0 (MIT)
org.clojure/clojure 1.11.1 (EPL-1.0)
org.clojure/core.specs.alpha 0.2.62 (EPL-1.0)
org.clojure/spec.alpha 0.3.218 (EPL-1.0)
org.codehaus.plexus/plexus-classworlds 2.6.0 (Apache-2.0)
org.codehaus.plexus/plexus-component-annotations 1.5.5 (Apache-2.0)
org.codehaus.plexus/plexus-utils 3.2.0 (Apache-2.0)
org.eclipse.sisu/org.eclipse.sisu.inject 0.3.3 (EPL-1.0)
org.eclipse.sisu/org.eclipse.sisu.plexus 0.3.3 (EPL-1.0)
org.jsoup/jsoup 1.12.1 (MIT)
org.slf4j/slf4j-api 1.7.30 (MIT)#2022-12-1219:30adamfreyjavax.enterprise/cdi-api 1.0
#2022-12-1219:30adamfreylol#2022-12-1219:30Alex Miller (Clojure team)admittedly, we are pulling in half of those already#2022-12-1219:31Alex Miller (Clojure team)might be able to slim that down a lot with a custom transport impl (similar reasons are why I made a custom mvn s3 transport in tdeps)#2022-12-1219:33adamfreyright, I'm seeing the Google wagon repo and your version of s3 transport as inspirations. (assuming I am able to dedicate any time to contributing this)#2022-12-1219:34Alex Miller (Clojure team)it seems like a lot to what amounts to a couple http calls :)#2022-12-1219:36Alex Miller (Clojure team)but anyways, definitely interested in figuring it out#2022-12-1510:31mkvlris it possible when creating a basis for a jar to exclude a dependency and if so, how? Tried manually fiddling with the generated basis but that was tedious and seemed error-prone.#2022-12-1513:28Alex Miller (Clojure team)What does “for a jar” mean? Is this in tools.build context?#2022-12-1513:29mkvlryep, in tools.build#2022-12-1513:29Alex Miller (Clojure team)Is this a top level dep or a dependency?#2022-12-1513:30mkvlrtop-level#2022-12-1513:32Alex Miller (Clojure team)If top, you could I think add an :extra {the/lib nil} in the basis opts#2022-12-1513:32mkvlrI’ll try this, thanks!#2022-12-1513:33Alex Miller (Clojure team)Or if modifying the basis after, the critical part is the :libs and you could just remove the lib from that map#2022-12-1513:34mkvlrlike this?
(b/create-basis {:project "deps.edn" :extra {'io.github.nextjournal/dejavu nil}})#2022-12-1513:34Alex Miller (Clojure team)Yeah#2022-12-1513:34mkvlrstill seeing a warning#2022-12-1513:35mkvlrclj -T:build jar
Producing jar: target/clerk-0.12.707.jar
Skipping coordinate: {:git/sha 31bdc12d78d3a8f48cbd6c3df5b8c79ca88df4f1, :git/url , :deps/manifest :deps, :deps/root /Users/mk/.gitlibs/libs/io.github.nextjournal/dejavu/31bdc12d78d3a8f48cbd6c3df5b8c79ca88df4f1, :parents #{[]}, :paths [/Users/mk/.gitlibs/libs/io.github.nextjournal/dejavu/31bdc12d78d3a8f48cbd6c3df5b8c79ca88df4f1/src]}#2022-12-1513:36borkdudeShould it be {:extra {:deps ...}}?#2022-12-1513:36Alex Miller (Clojure team)Oh, yes#2022-12-1513:36borkdudeWe tried that and it says: expected map, got nil :)#2022-12-1513:37mkvlryeah getting `Bad coordinate for library io.github.nextjournal/dejavu, expected
map: nil#2022-12-1513:38mkvlr(def basis (-> (b/create-basis {:project "deps.edn"})
(update :libs dissoc 'io.github.nextjournal/dejavu)))#2022-12-1513:38borkdudedissocing from libs seems to work..#2022-12-1513:38mkvlrthat seems to do it#2022-12-1513:39mkvlryep, warning is gone, thanks!#2022-12-1513:39Alex Miller (Clojure team)Deps is really a compositional model so excluding things is a lot harder than including them. I thought I had something for the above in there but I guess not. I assume this isnt a dep that could optionally be included rather than excluded?#2022-12-1513:40mkvlrin that case I want to have it be there when consumed at as git dep but leave it out when I build the jar#2022-12-1513:40borkdudeThis is a dependency that does some stuff at startup when in local dev or as a git dep, but as a mvn jar we would like to not use it since we can pre-compute what this lib is doing and also exclude the deps#2022-12-1513:41Alex Miller (Clojure team)Gotcha#2022-12-1513:41Alex Miller (Clojure team)Well I’ll think about better ways to do that#2022-12-1512:39pezadd-lib3 doesn't work if you are using lein-tools-deps. Just logging that here so that it can be found using a search. 😃 I spent way too much time trying to figure out what was going on. The symptoms were that I got an error complaining that the jar wasn't found. If I download it to my .m2 directory, it works adding it, so it seems to be the download that does not happen.#2022-12-1719:37d._.bMoving from lein and one nagging question for me has been around global :exclusions . For an example of why I am wondering, see https://github.com/stuartsierra/log.dev/blob/master/project.clj.#2022-12-1719:39d._.bI worry that specifying exclusions per-dep is going to get extremely wordy, and am interested in any ways to clean up the mess of ensuing :exclusions .#2022-12-1719:40d._.bMaybe in tools.deps land this looks something like a processing step that programatically adds the exclusions per-dep?#2022-12-1719:42d._.bI'm not really sure one way or the other, but I sure would like some top-level method of excluding logging junk!#2022-12-1719:42Alex Miller (Clojure team)Right now, not much to help you, but has been requested several times. Can you add to https://ask.clojure.org ? I can’t immediately find it there#2022-12-1719:43d._.bSure thing.#2022-12-1719:44seancorfieldYou will need exclusions a lot less, in my experience, because of the different algorithm for selecting versions of dependencies. tools.deps uses a much simpler and more obvious algorithm than lein. I found that that helped eliminate nearly all of the :exclusions that we used to need with Leiningen. But it does depend on why you are trying to exclude dependencies.#2022-12-1719:45d._.bAny thoughts on this particular case (other logging libs and bridges)?#2022-12-1719:46seancorfieldWe have not found we need that. We use log4j2 and clojure.tools.logging and have everything bridged to that, and have never had an issue with dependencies bringing in other bridge libraries.#2022-12-1719:49d._.byeah, taking a quick look at my current deps tree, not so bad, so perhaps my "omg im going to need to exclude so much logging junk" isn't as bad as I expected it might be.#2022-12-1719:49d._.bstill a relatively young project, so time will tell#2022-12-1719:51d._.bclj-http bringing commons logging is currently the only one i spot#2022-12-1719:52seancorfieldI just looked. We have four exclusions across 174 deps.edn files in our 134Kloc codebase.#2022-12-1719:53d._.bquasi-related question for you since you mention 174 files (that's a lot of files): do you do anything similar to lein's managed-dependencies?#2022-12-1719:53d._.bthinking about things like jackson#2022-12-1719:53seancorfieldTwo are logging related: one excludes slf4j-nop from build tooling, the other excludes log4j 1.x from an Authorize,Net dep.#2022-12-1719:54seancorfieldWe have one exclusion of Cheshire in one place (because of Jackson issues).#2022-12-1719:55seancorfieldWe keep our deps up to date by running antq regularly and tweaking things only as needed to suppress problematic deps. We also run clj-watson to check for CVEs and we address those as fast as we can.#2022-12-1719:56d._.bmy managed-dependencies question looks like there's a fair bit of prior discussion, so ill keep that out of this thread for now 🙂#2022-12-1719:57seancorfield:override-deps and an alias should handle all of that 🙂#2022-12-2016:34rickmoynihan~Is there a minimal prep-lib example that compiles java sources? For some reason what I have isn’t compiling my java sources, but I don’t see any error~
Solved.#2022-12-2016:36rickmoynihanhelps if you have a build.clj in the right place#2022-12-2210:14Adam HelinsCI started failing like this after some changes in dependencies and little things. Could it be that we screwed up $HOME or something like that? :thinking_face:
Error building classpath. Can't create directory: /.clojure/.cpcache
#2022-12-2211:24practicalli-johnSuggest checking the value of :config-dir and other settings and see if it got set to /
❯ clojure -Sdescribe
{:version "1.11.1.1208"
:config-files ["/usr/local/lib/clojure/deps.edn" "/home/practicalli/.config/clojure/deps.edn" ]
:config-user "/home/practicalli/.config/clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/practicalli/.config/clojure"
:cache-dir "/home/practicalli/.cache/clojure"
:force false
:repro false
:main-aliases ""
:repl-aliases ""}
If so it could be via HOME , XDG_CONFIG_HOME or CLJ_CONFIG environment variables (although it would be pretty obvious in the terminal if HOME was not correct)#2022-12-2212:57Adam HelinsYes, I kinda forgot about -Sdescribe. I spotted a discrepancy between clojure -Sdescribe and bb clojure -Sdescribe which was really helpful. Thanks!#2022-12-2212:19rickmoynihanI’m trying to create a multi-module build with tools.deps/`tools.build`.
I have some sub modules (deps) which are also individually maven artifacts, and I want to also build a parent artifact that simply expresses dependencies on all of the sub module deps. Does anyone know of any simple examples, or ways to do this?#2022-12-2212:43rickmoynihanMy parent deps.edn currently looks something like this (so I can get a single repl for all sub projects together):
:deps {project/submodule-a {:local/root "submodule-a"}
project/submodule-b {:local/root "submodule-b"}
}
Obviously this isn’t quite what I want, as when I create a basis from this it inlines those projects into the parent dep.
I’m assuming (and currently looking into) creating the basis (and ultimately jar) dynamically in my build.clj like this:
(b/create-basis {:project {:deps {'project/submodule-a {:mvn/version version}
'project/submodule-b {:mvn/version version}}}}})
But then I think I’ll need to have installed those submodules into .m2… is there a way to avoid having to pollute .m2 in order to just build the candidate parent dep? I think it would be nicer to separate building from installation if possible.#2022-12-2213:23rickmoynihanhmm also just realised… some submodules depend on each other; and I’d wired up those submodules to use :local/root which works fine at a repl; but it does mean that they are also individually being inlined locally. So I suppose I need to override them too when I create those basis 😞#2022-12-2213:48Alex Miller (Clojure team)What is the end goal of the build? Do you want individual artifacts for each subdirectory with Maven dependencies between them?#2022-12-2213:51rickmoynihan@alexmiller: End goal is to build a set of libraries where:
1. there are individual library artifacts for each subdirectory with Maven dependencies between them
2. a parent library artifact with maven dependencies on all sub projects#2022-12-2213:56Alex Miller (Clojure team)And presumably 3. Work with them locally without artifacts#2022-12-2213:56rickmoynihanyes… well it depends what you mean by that#2022-12-2213:58Alex Miller (Clojure team)3 is easy with local/root and 1/2 are easy with maven/version but it’s hard to do both. You can use aliases with override-deps to swap out but alias application is shallow#2022-12-2214:21rickmoynihanOk I think that might explain why this:
(b/create-basis {:aliases [:overrides]
:extra {:aliases {:overrides
{:override-deps
{'myns/module-a {:mvn/version version}
'myns/module-b {:mvn/version version} 'myns/module-c {:mvn/version version}}}}}})
isn’t doing what I’d expect (when fed into jar) it’s including src paths it shouldn’t.#2022-12-2214:22rickmoynihanI wonder if I’d have more luck reading the deps.edn as data and replacing the :local/root s myself?#2022-12-2214:37Alex Miller (Clojure team)you can use :local/roots in the whole thing, consume with :local/roots and just do the replacements in a custom build script for deployment#2022-12-2214:45rickmoynihanyeah that’s basically what I’m trying to do, from a build.clj at the project root; but the internal deps are :local/root’s.
First I tried it with with :extra :overrides, but it doesn’t seem to work — presumably because the application is shallow?!
So now I’m looking at basically doing something like this, from the top:
(b/create-basis {:project (-> (edn/read-string (slurp "submodule-a/deps.edn"))
(update-in [:deps 'myns/submodule-a] (constantly {:mvn/version version})))})
,,,#2022-12-2214:46Alex Miller (Clojure team)I'm not sure what :extra :overrides means there - :extra is an extra deps.edn (like -Sdeps on the command line)#2022-12-2214:47Alex Miller (Clojure team):override-deps should apply throughout the tree though#2022-12-2214:49rickmoynihanyes I realised that — I did it because :project is set to the submodules “deps.edn” (which doesn’t have the :override-deps alias in it)…
And it looked like :override-deps only has meaning in an :alias (not at the top of a deps.edn map)
Though not 100% sure it’s right#2022-12-2214:49Alex Miller (Clojure team)as deps.edn becomes more complicated, things like slurp / read-string are going to be less likely to work and it will be better to lean on tools.deps apis (like https://clojure.github.io/tools.deps/#clojure.tools.deps/slurp-deps)#2022-12-2214:50rickmoynihanah ok, thanks for the heads up there 🙇#2022-12-2214:50Alex Miller (Clojure team)> :override-deps only has meaning in an :alias
correct#2022-12-2214:51Alex Miller (Clojure team)if you are "changing contexts" in a deps.edn, you'll want to use the tools.build apis for that#2022-12-2214:51Alex Miller (Clojure team)https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-set-project-root.21#2022-12-2214:52rickmoynihanyes I’m already using set-project-root#2022-12-2214:52Alex Miller (Clojure team)I guess, I don't understand why the override thing "doesn't work"#2022-12-2214:53rickmoynihanthough I wrote a macro to help with it:
(defmacro with-project-root [dir & forms]
`(let [orig-root# @#'b/*project-root*] ;; yes this is correct, my cat didn't run over my keyboard I promise! :-)
(try
(b/set-project-root! ~dir)
(let [ret# (do #2022-12-2214:53Alex Miller (Clojure team)in particular, I don't understand exactly what you tried and what you saw and how it differed from what you expected#2022-12-2214:55Alex Miller (Clojure team)seems like rather than with-project-root, you could more directly just bind *project-root*#2022-12-2214:56Alex Miller (Clojure team)(defmacro with-project-root [dir & forms]
`(binding [b/*project-root* ~dir]
should be basically the same right?#2022-12-2214:58rickmoynihanYes, I did actually have a question about that — I guess I was surprised to you setting the var directly, rather than suggesting users bind it (which was my first instinct); so I just assumed you had good reasons to use set. But thanks for clearing that up for me! 🙇#2022-12-2215:01Alex Miller (Clojure team)I was expecting people to change context as a 1-time kind of thing, not as a per-submodule change-and-reset, which in retrospect is an obvious thing. guess I should just add with-project-root to t.b#2022-12-2215:06rickmoynihan> in particular, I don’t understand exactly what you tried and what you saw and how it differed from what you expected
So far everything I’ve tried has had the same effect, which is that the src paths for the independent module jars are being unioned… e.g. if I have in submodule-c:
{:deps {myns/submodule-c {:local/root "../submodule-a"}, myns/submodule-b {:local/root "../submodule-b"}}}
Then my output jar for submodule-c contains :paths from a and b (along with their pom.xml’s etc).
Of course I’m currently just assuming this is because of :local/root (I guess I could be crossing the streams somewhere else)#2022-12-2215:07rickmoynihanI haven’t yet tried just replacing those myself, by slurping and reading the file and munging the data, but that’s what I’m about to try next.#2022-12-2215:08Alex Miller (Clojure team)being unioned ... where?#2022-12-2215:09rickmoynihanI mean the jar contains the union of :paths from the submodules referenced via :local/root#2022-12-2215:09rickmoynihan(Not an explicit call to set/union or anything like that)#2022-12-2215:10Alex Miller (Clojure team)instead of just trying stuff, let's just be really clear about what you're doing#2022-12-2215:11rickmoynihansure#2022-12-2215:11Alex Miller (Clojure team)so in submodule-c, you have a deps.edn like:
{:deps {myns/submodule-a {:local/root "../submodule-a"}
myns/submodule-b {:local/root "../submodule-b"}}#2022-12-2215:12rickmoynihanyes#2022-12-2215:12Alex Miller (Clojure team)you're trying to create a jar for c that references a and b as maven artifacts, not as locals#2022-12-2215:12rickmoynihanyes, but submodule-c is not the parent (the parent also may shortly contain a submodule-d)#2022-12-2215:13Alex Miller (Clojure team)you control everything that is being used to make the jar#2022-12-2215:13rickmoynihanyes#2022-12-2215:13Alex Miller (Clojure team)you can copy the source from c to your jar making location, so that's fine#2022-12-2215:14Alex Miller (Clojure team)you control the basis used with write-pom which defines which deps the jar is published with in the pom#2022-12-2215:15Alex Miller (Clojure team)so if you provide an :extra deps that replaces the top-level deps, that's all you need to do. you don't care about the transitive deps - they don't affect what's in the jar#2022-12-2215:16Alex Miller (Clojure team)so when you create-basis, use :extra {yourgroup/artifact-a {:mvn/version v}, yourgroup/artifact-b {:mvn/version v}}#2022-12-2215:16rickmoynihan> you can copy the source from c to your jar making location, so that’s fine
I could — but I’m not currently copying src files around myself.#2022-12-2215:16Alex Miller (Clojure team)why#2022-12-2215:17Alex Miller (Clojure team)I mean like (b/copy-dir {:src-dirs ["src" "resources"] :target-dir class-dir})#2022-12-2215:21rickmoynihan> so when you create-basis, use :extra {yourgroup/artifact-a {:mvn/version v}, yourgroup/artifact-b {:mvn/version v}}
This was the first thing I tried…. I think it was actually (b/create-basis {:extra {:deps ,,,}}) before trying the form above with override-deps.
Also another reason I moved to try again with override-deps was that I’m also building module-a and module-b, and I didn’t want to maintain separate overrides for each… i.e. I thought the semantics of override-deps were “if a dep uses this coordinate in its deps.edn override it; but if it doesn’t mention it don’t introduce it”#2022-12-2215:23rickmoynihan> why
> I mean like (b/copy-dir {:src-dirs [“src” “resources”] :target-dir class-dir})
I could do that too; but I was avoiding that by doing it in situ by changing b/*project-root*#2022-12-2215:23Alex Miller (Clojure team)you still have to do something to get the sources#2022-12-2215:24rickmoynihanDoes org.corfield.build/jar with the basis not handle that?#2022-12-2215:24Alex Miller (Clojure team)you have code - why maintain anything? just transform the data you have in the deps.edn#2022-12-2215:25Alex Miller (Clojure team)this is the first you've mentioned using that#2022-12-2215:26rickmoynihan> you have code - why maintain anything? just transform the data you have in the deps.edn
yes that’s the approach I’m going to try next#2022-12-2215:26rickmoynihan> this is the first you’ve mentioned using that
Sorry.#2022-12-2215:26Alex Miller (Clojure team)tbh, I don't like Sean's wrapper b/c it only works until something is different, then you have to fall back to using tools.build. the whole idea of tools.build is to start with something that can grow#2022-12-2215:27Alex Miller (Clojure team)this is a case where a lot of things are different#2022-12-2215:27rickmoynihanyeah I see it does the copy-dir#2022-12-2215:28Alex Miller (Clojure team)you're going to twist yourself in knots to create a context where his thing does the one thing it does, or you can just do it directly with tools.build. I think in the end you'll understand more and be able to do the next thing better with the latter#2022-12-2215:34rickmoynihanYeah, that’s fair… I’ve used tools.build before, but also found seans wrapper useful on simpler builds.
However I do also see that the copy-dir inside seans jar is causing my problem, because it doesn’t clean the :target-dir between runs.#2022-12-2215:34rickmoynihanSo I think you’re right, that this is the root cause#2022-12-2215:36rickmoynihanThanks for the help 🙇#2022-12-2215:37Alex Miller (Clojure team)👍#2022-12-2215:37rickmoynihanAnother case of wanting simple not easy 🙂#2022-12-2217:37seancorfield@U06HHF230 A suggestion for you: look at the structure that Polylith uses. It has a development "project" for REPL usage where all code is on the classpath via :local/root (using a :dev alias) and then it has projects folders -- one for each buildable artifact -- with deps.edn that say how to build that specific artifact.
It's primarily intended to build applications (so inlining :local/root deps is acceptable there) but I think it could also be used to build independent libraries, if you can figure out a model for how you version them all (and therefore build them in reverse dependency order, bottom-up, which should be reasonably straightforward?).#2022-12-2217:39rickmoynihanThanks… I’ve been purposefully avoiding polylith at the minute; simply because it seems larger than I need, and more specialised towards application mono-repos.#2022-12-2217:41seancorfield> However I do also see that the copy-dir inside seans jar is causing my problem, because it doesn’t clean the :target-dir between runs.
That is deliberate, so you can combine jar with other actions. As Alex says, that wrapper is fine for the simple stuff but breaks down in more complex settings -- at work we use a mixture of direct tools.deps, tools.build, and a few build-clj wrapper functions... and I'm planning to wean us off it "soon" as well as updating at least deps-new to use raw tools.build instead of the wrapper.#2022-12-2217:42seancorfield(well, Alex doesn't say the wrapper is "fine" but I hope you get my meaning)#2022-12-2217:42rickmoynihanyeah I realise it’s fine for bb/jar’s intended purpose… just not mine (here) 🙂#2022-12-2217:42seancorfieldAnd, yes, the wrapper was born out of an apparent desire for "easy" over "simple" 🙂#2022-12-2217:43rickmoynihanas I said I’ve used the bb wrapper before, and found it helpful at cutting down boilerplate.
I was also assuming when I set out that there might be an easy solution to this; which was partly why I started with your wrapper.#2022-12-2217:45rickmoynihanI think I’ve said this to you before though…
I think a better way to slice “easy” is vertically; separating the needs of applications from the needs of libraries. Where as bb assumes it can add ease for both applications and libraries together.#2022-12-2217:48seancorfieldIndeed. And it started out as a very "simple" wrapper to just cut down some basic boilerplate... but grew far too many knobs and dials in the pursuit of reducing boilerplate in more cases instead of "encouraging" people to just use tools.build... I'm far too accommodating in that area 🙂#2022-12-2217:54rickmoynihanhaha well you do yourself a disservice there, there’s certainly a need for easier solutions to common problems… even if they do bring some complecting to the table.
I think the problem is that easy solutions naturally grow knobs; and become a bit of a mess and hard to maintain overtime… Simple ones, more purposefully leave problems on the table, but are more maintainable over the long term.
Anyway — I do very much appreciate everything you’ve done 🙇#2022-12-2217:55rickmoynihanI would have still liked an easy solution for this; as it’s taking far longer than I was expecting… and I was expecting it to take a while too! :rolling_on_the_floor_laughing:#2022-12-2217:56Alex Miller (Clojure team)I added with-project-root for the next t.b release btw#2022-12-2217:57seancorfieldHeh, well, you've probably seen my long series of blog posts about our monorepo and the various approaches we've taken over the years, and the various things we've had to solve even with Polylith in the mix...#2022-12-2217:57rickmoynihanYes, I’ve returned to them many times over the years#2022-12-2213:59Alex Miller (Clojure team)So not sure if that’s sufficient (depends on your deps)#2022-12-2213:59rickmoynihan3.1. I’d definitely like to work across the whole library as a repl (this bit currently works via :local/root)#2022-12-2214:00rickmoynihan3.2. (optionally) it would be nice to consume the library and its individual bits without artifacts#2022-12-2214:00Alex Miller (Clojure team)consume as in …#2022-12-2214:02rickmoynihanconsume from an application outside of the libraries mono-repo setup… i.e. through a :override-deps {:local/root ,,,} in an alias in my ~/.clojure/deps.edn#2022-12-2214:04rickmoynihan> 3 is easy with local/root
Are you meaning my 3.1 or 3.2 here?#2022-12-2214:04rickmoynihanI’m guessing 3.1#2022-12-2322:55seancorfieldAre these "latest version" scripts documented somewhere? I looked on the deps & CLI guide/reference and the install page and didn't see them (did I just miss them?)...#2022-12-2403:31jasonjcknI don't think they are#2022-12-2411:03delaguardosome info from Alex
https://github.com/DeLaGuardo/setup-clojure/issues/75#2022-12-2614:42Søren SjørupI’m trying depend on a jar I’ve built in deps.edn. But I’m getting Error building classpath. Manifest type not detected when finding deps for * in coordinate ... can someone point me to where this error message comes from so I can better debug? Thank you#2022-12-2615:55seancorfieldHow are you specifying the dependency on the JAR, and did you check that the JAR contains a pom.xml file?#2022-12-2616:27Søren SjørupI’ve since found out that the directory structure inside my jar is not as intended… so sorry to bother you. Also no, I don’t have a pom.xml. I have this in my deps.edn
dk.zoren/jar {:local/root "libs/BTree.jar"}#2022-12-2616:33Søren SjørupI get Unsupported class file major version 61 after fixing the directory layout inside the jar. Thank you, and again sorry to bother.#2022-12-2616:44seancorfieldThe latter says you have some code compiled on a more recent JDK than the one you're trying to run it on. 61 is JDK 17, so you'll need at least JDK 17 to run this.#2022-12-2617:07Søren SjørupYes I found out. But I actually don’t really understand how it happend. I did compile the jar with java 17 and inside the vscode REPL the system property said "java.runtime.version" "17.0.5+8"#2022-12-2617:08Søren SjørupNow I’ve use asdf to install java and restarted VSCode, that fixed it for me.#2022-12-3008:11Jakub Holý (HolyJak)Why is clj giving me
> Error building classpath. Could not find artifact com.datomic:dev-local:jar:1.0.243 in central (https://repo1.maven.org/maven2/)
when I have it in the local m2 cache?
ls ~/.m2/repository/com/datomic/dev-local/1.0.243/dev-local-1.0.243.jar
/Users/me/.m2/repository/com/datomic/dev-local/1.0.243/dev-local-1.0.243.jar
Do I need to explicitly tell deps to look in there?!#2022-12-3011:09practicalli-johnAs I understand it Datomic dev-local libs (or datomic) are not available from Maven central (as its a commercial project)
https://docs.datomic.com/cloud/dev-local.html has details of installing and using this library, basically add the following to the project deps.edn
{com.datomic/dev-local
{:mvn/version "1.0.243"}}
#2022-12-3011:10Jakub Holý (HolyJak)That is true but I have already got them from where they are and have them cached in .m2/repository/ and would expect clj to use it from there, even if it is not in mvn central#2023-01-0423:53jasonjcknis there an equivalent plugin for tools deps https://github.com/ztellman/virgil/#2023-01-0911:45Jakub Holý (HolyJak)Hi again! How do I troubleshoot what config clj sees and why it cannot find a library?
I have in my :config-user file (`"/home/runner/.config/clojure/deps.edn"`) the following (according to cat):
{:mvn/repos {"cognitect-dev-tools" {:url ""}}}
but when I run
clojure -Sverbose -Sdeps '{:deps {com.datomic/dev-local {:mvn/version "1.0.243"}}}' -e '(println "it works!")'
it fails with
> Error building classpath. Could not find artifact com.datomic:dev-local:jar:1.0.243 in central (https://repo1.maven.org/maven2/)
so it seems to ignore this config?! I have no idea where to look for what is wrong 😭 🙏#2023-01-0915:44Alex Miller (Clojure team)You can use -Sverbose to print your path info to see where it’s looking#2023-01-0915:45Alex Miller (Clojure team)But I think here you’re just seeing a confusing error as all repos are checked but it only reports the last failure#2023-01-0915:47Alex Miller (Clojure team)I would ask in #datomic and they can help diagnose there#2023-01-0915:57Jakub Holý (HolyJak)Thank you! I used Sverbose, that's how I fixed my first error and corrected the user deps location. But obviously Clj still does not see all I expect it to see. Locally if I try to remove settings.xml then I get som 4xx error, which I do not see here. So what else could be wrong?
Good to know at least that Clj might be trying the correct repo.#2023-01-0916:00Jakub Holý (HolyJak)Update: instead of relying on config in the user deps file, when I specify the :mvn/repo on CLI via - Sdeps then it works. So it really looks like somehow it is not seeing the file content...#2023-01-0916:17Alex Miller (Clojure team)What does verbose print when it’s not working?#2023-01-1522:54Jakub Holý (HolyJak)Hi Alex, I am sorry, I did not see your question. As far as I remember, verbose printed the path of the file that I was writing. I suspect there is something weird with the GH Action environment / the particular actions I use for this. Anyway I have a working workaround and ton of other stuff to do 😭 so I am closing the lid on this. Thank you for your help!#2023-01-0914:53twashingIs there a way to specify HEAD in tools.deps’ git coordinates?
If I try A, I’ll get an error like B. And this corresponds with the source code C.
# A
{:deps {org.example/my-library {:git/url ""
:git/branch "master"
:git/sha "HEAD"}}}
# B
Library org.example/my-library has prefix sha, use full sha or add tag
# C
#2023-01-0915:00delaguardoyou could use :git/tag "HEAD"
but before starting the repl execute from the project root directory clj -X:deps resolve-git-tags to add explicit :git/sha for every dependency that declares :git/tag#2023-01-0915:07twashingHmm, getting this.
clj -X:deps resolve-git-tags
Namespace clojure.tools.cli.api loaded but function not found: resolve-git-tags
#2023-01-0915:11twashingAhh, it’s git-resolve-tags
https://clojure.org/reference/deps_and_cli#_other_programs#2023-01-0915:27twashingIs there a way to make a feature request for HEAD in tools.deps’ git coordinates?
We have a use case where a platform test pulls in other services. And it should always run its tests against its dependency’s main HEAD.
I’m sure there’s other places it’s useful. But I don’t see a way to make feature requests on in tools.deps’ github repo.
{:deps {org.example/my-library {:git/url ""
:git/branch "master"
:git/sha "HEAD"}}}#2023-01-0915:36Alex Miller (Clojure team)Requests can be made at https://ask.clojure.org but I can tell you in advance that this is intentionally not supported#2023-01-0915:37twashingOk, thanks for the feedback. Do you know the rationale, off the top of your head?#2023-01-0915:38Alex Miller (Clojure team)If you want to “follow along” to a dep, it is better to check out the repo locally and use a :local/root dep#2023-01-0915:39twashingFair enough. Thanks for the backgrounder 👏:skin-tone-5:#2023-01-0915:39Alex Miller (Clojure team)We want git deps to be fixed references to unambiguous commit points (shas)#2023-01-0922:06Colin P. HillAnyone have much experience using the CLI in Powershell? I have a script that reads directly from *in*, and I'm trying to invoke it with gc .\game-data.json | clojure -M .\scripts\wager_returns.clj, but the input stream doesn't seem to receive the data from the pipe – it waits for interactive input.#2023-01-0922:13hiredmanhow do you know it is waiting for interactive input?#2023-01-0922:13Colin P. HillIt hangs, and then I can type things, and then the JSON parser parses what I type#2023-01-0922:15Colin P. Hill> gc .\game-data.json | clojure -M .\scripts\wager_returns.clj
I waited a good ten seconds and then typed this
Execution error at clojure.data.json/-read (json.clj:364).
JSON error (unexpected character): I
#2023-01-0922:18hiredmansounds like '|' is the issue, I don't use PS, but if I recall, it doesn't pipe things based on raw byte streams, but instead can pass structured data#2023-01-0922:18hiredmanso maybe the output of gc is being passed via some structured data thing using |, but clojure is trying to read from stdin#2023-01-0922:18Colin P. HillYeah, that was my thinking. But afaict gc (short for get-content, roughly equivalent to cat) does indeed pipe text. Might still be structured in some way though.#2023-01-0922:19Colin P. HillThough this SO answer seems to suggest I'm doing it right
https://stackoverflow.com/a/11788475#2023-01-0922:23hiredmanhttps://github.com/PowerShell/PowerShell/issues/9497#issuecomment-1246179450 sounds like what you are seeing, but with python#2023-01-0922:25Colin P. HillI'm not sure about that – looks like they're trying to pipe into a PS script.#2023-01-0922:25hiredmanclojure is maybe a ps script?#2023-01-0922:25Colin P. Hillhm, good point, it might be#2023-01-0922:28Colin P. HillPS> Get-Command clojure
CommandType Name Version Source
----------- ---- ------- ------
Alias clojure -> Invoke-Clojure 1.11.1.11… ClojureTools
PS> get-command invoke-clojure
CommandType Name Version Source
----------- ---- ------- ------
Function Invoke-Clojure 1.11.1.11… ClojureTools
Looks like you may be onto something there. A CommandType of Function sure makes it sound like it's a shell function.#2023-01-0922:28hiredmanhttps://github.com/clojure/brew-install/blob/1.11.1/src/main/resources/clojure/install/ClojureTools.psm1 and https://github.com/clojure/brew-install/blob/1.11.1/src/main/resources/clojure/install/ClojureTools.psd1#2023-01-0923:13Colin P. Hillpsclj-a.ps1:
& $JavaCmd -classpath "$classpath;$InstallDir/exec.jar" clojure.main '-e' '(println "Read:" (slurp *in*))'
PS> .\psclj-a.ps1
It waited for me to type and then ^C
Read: It waited for me to type and then ^C
PS> echo "foo" | .\psclj-a.ps1
Same result!
Read: Same result!
psclj-b.ps1:
$Input | & $JavaCmd -classpath "$classpath;$InstallDir/exec.jar" clojure.main '-e' '(println "Read:" (slurp *in*))'
PS> .\psclj-b.ps1
Read:
PS> echo "foo" | .\psclj-b.ps1
Read: foo
#2023-01-0923:28Colin P. Hillpsclj-c.ps1:
if ($MyInvocation.ExpectingInput) {
$Input | & $JavaCmd -classpath "$classpath;$InstallDir/exec.jar" clojure.main '-e' '(println "Read:" (slurp *in*))'
} else {
& $JavaCmd -classpath "$classpath;$InstallDir/exec.jar" clojure.main '-e' '(println "Read:" (slurp *in*))'
}
PS> echo "foo" | .\psclj-c.ps1
Read: foo
PS> .\psclj-c.ps1
Waited for input
Read: Waited for input
Seems workable to me. @U064X3EF3 Is this a known issue and/or does this seem like a reasonable solution? Not seeing any discussion of it in Slack archives or Ask Clojure, but I may have missed something.#2023-01-0923:53Alex Miller (Clojure team)Can you restate the problem?#2023-01-0923:54Colin P. HillIn a nutshell, the CLI on Powershell can't read piped input#2023-01-0923:54Alex Miller (Clojure team)If there’s a bug, please file it in https://ask.clojure.org#2023-01-1001:14Colin P. HillDone
https://ask.clojure.org/index.php/12533/cli-on-powershell-cannot-read-piped-input#2023-01-1009:28borkdudeclj.exe (deps.clj disguised as clj) doesn't have this problem:
https://ask.clojure.org/index.php/12533/cli-on-powershell-cannot-read-piped-input?show=12534#a12534#2023-01-1800:00Jakub Holý (HolyJak)Hello! When I start repl with clojure -A:some-alias , can I somehow tell you to (pre)load a namespace or possibly switch into a ns? Something like #C0AB48493’s :repl-options {:init-ns user} ? Perhaps my only solution is to name the ns user and put it on a dev-only path, then it will be loaded automatically?#2023-01-1800:04seancorfieldYou can pass -i and/or -e -- clojure.main options to load a file or eval an expression -- and then -r to force a REPL to start.#2023-01-1800:06seancorfield(~/clojure)-(!2003)-> cat deps.edn
{:aliases
{:dev {:extra-paths ["dev"]
:main-opts ["-e" "(do (require 'startup) (in-ns 'startup))" "-r"]}}}
Tue Jan 17 16:05:23
(~/clojure)-(!2004)-> cat dev/startup.clj
(ns startup)
(println "Starting up dev...")
Tue Jan 17 16:05:43
(~/clojure)-(!2005)-> clj -A:dev
WARNING: Use of :main-opts with -A is deprecated. Use -M instead.
Starting up dev...
#object[clojure.lang.Namespace 0x1827a871 "startup"]
startup=>
(should use -M instead of -A there I guess)#2023-01-1800:08seancorfieldOr:
clj -M:build -i build.clj -e "(in-ns 'build)" -r
which is how we start a REPL with our build.clj file in the build ns.#2023-01-1800:15dpsuttonI’ve seen this request come up a bunch. I’ve never understood the importance of it. #2023-01-1800:16seancorfieldPeople seem to want to avoid doing a require and in-ns manually? 🙂#2023-01-1800:18dpsuttonYeah. I don’t mean to say it’s wrong. Just would love to hear the benefit#2023-01-1800:40skylizeI think the common reason is to do all your work out of a single namespace, as you would when typing directly in to a CLI; except with a bunch of useful development tools preloaded along with access to your state management system.
I use Calva, where there is definitely friction against that idea, because it automatically switches the repl to the namespace of whichever file you eval code from. Which namespace would be appropriate to load all these tools into?#2023-01-1800:57seancorfieldI use Calva but I have it's REPL window hidden and don't ever type into (so I don't care what ns it thinks it is in 🙂 ) but with the build.clj command above, I type build/test commands into it from time to time while working (e.g., resetting the dev DB, run Polylith tests since the last stable tag & update the tag if successful)#2023-01-1802:01skylize> so I don't care what ns it thinks it is in
>
Well I think you do "care", just not attentively. The vars available to you (or not) are determined by the namespace being evaluated in, right?#2023-01-1802:10seancorfield@U90R0EPHA If you're eval'ing code from your editor window, Calva/nREPL takes care of switching the REPL to the right ns. It only matters if you're typing directly into the REPL windows.#2023-01-1802:20skylizeBut that was my point to begin with. 🤤
Calva (and presumably other nREPL workflows too?) conflicts with the idea of working out of a dev namespace with dev tools already required, because it switches namespaces out from under you.#2023-01-1802:28seancorfieldAh, right. Then we're in violent agreement, as my partner likes to say.#2023-01-1802:29seancorfieldPart of what I liked about Chlorine/Clover was that the REPL window was readonly -- because Mauricio wanted to discourage "bad" behavior 🙂#2023-01-1802:30seancorfieldI like that I can have Calva setup so the REPL window doesn't open by default. Then I use Portal as my "readonly REPL" window.#2023-01-1802:31skylizeYeah. Putting your repl-bound code into a savable file definitely seems superior in most cases.#2023-01-1808:19Jakub Holý (HolyJak)thanks a lot!#2023-01-1819:30dominicmWhat are people doing to generate deps.edn in ~/.clojure with relative paths?
I must admit I'm disappointed that my config will be generated rather than symlinked. As it breaks my current "setup" of using stow to manage my dotfiles.#2023-01-1819:39seancorfieldCan you elaborate on "relative paths"? What is the use case?#2023-01-1819:40dominicmI need to use the deps.edn on a Mac and a Linux machine, and they'll refer to different paths when I want to, e.g. include a library from a directory.#2023-01-1819:41seancorfieldYou have a library in source form on multiple machines? Is that library in git?#2023-01-1819:43dominicmIt is, but I want it in source form because I'll do local development on it. Although that's not a huge issue I suppose, I do that much less on the mac.#2023-01-1819:46seancorfieldSeems a bit dodgy to just assume you have the "same" library code in the "same" place on both machines -- I assume you're doing it this way to have a common alias that brings in that dependency on each machine? (rather than have the dependency in the projects that actually depend on it -- so I guess this is a tooling library rather than an actual dependency?)#2023-01-1819:48dominicmI organize my code in the same way. I use ghq so my repos are always ~/src/github.com/juxt/clip (for example). Then if I'm working on a project and I'm also developing clip, I would launch the project with -A:user/clip so I could develop them in tandem.#2023-01-1819:51seancorfieldHmm, yeah, fair enough... I have "often" felt the need for env vars in deps.edn for situations like this 😐#2023-01-1819:54dominicmYeah. I understand why it's the way it is, too. But, it would be a solution to this kind of problem 🙂#2023-01-1821:17dharriganThe way I do it, and not ideal, is that I use ansible to push out to my linux boxen and mac boxen a templated deps.edn, replacing values depending upon the architecture.#2023-01-1821:18dharriganIf only deps.edn supported variable substitution, even if just from $env variables, would make that all go away.#2023-01-1821:32dominicmI'd be reasonably happy with the ability to include other deps.edn files too#2023-01-1821:33dharriganA bit like aero, eh? 🙂#2023-01-1821:34seancorfield(which we just switched all our config EDN files over to at work!) ❤️ Aero!#2023-01-1821:41dharriganIndeed, I use aero for my config edn files too! 🙂#2023-01-1821:41dharriganmuch love#2023-01-1821:42dominicmEnv vars are probably problematic for caching. But if you could load aliases from files then you could write a few vars into a platform alias and join paths on that.#2023-01-1914:04dvingoI have a small babashka script that will write the fully expanded pathnames (using fs/expand-home) into ~/.clojure/deps.edn - so my deps.edn file lives in a bb script and I put that in a git repo. (Also a happy aero user after reading https://www.pixelated-noise.com/blog/2022/04/28/integrant-and-aero/ !)#2023-01-2006:23dominicmHaving pondered this a little, I think the main thing I'd want is a portable alias/aliases I could use on all machines. That won't use any absolute paths. But I'd like to be able to add aliases that are specific to the machine alongside those aliases which do.#2023-01-2006:24dominicmPhrased like that, I'm wondering if I could abuse the tools aliases for it. Except it wouldn't be a tool, just a normal -A alias#2023-01-1819:42jeroenvandijkI’m seeing
Error building classpath. Malformed \uxxxx encoding.
with a specific dep:
clj -Sdeps '{:deps {prismatic/plumbing {:mvn/version "0.6.0"}}}'
But with lein there doesn’t seem to be an issue. Any idea?#2023-01-1819:42jeroenvandijkI’ve reported it here https://github.com/plumatic/plumbing/issues/148#2023-01-1819:43jeroenvandijkWith the following no issue
lein try prismatic/plumbing 0.6.0
#2023-01-1819:44seancorfieldJust tried it on WSL2/Ubuntu and it works for me. What platform are you on? (and what version of JDK?)#2023-01-1819:44jeroenvandijkMaybe same as https://clojurians.slack.com/archives/C0617A8PQ/p1619001006345400 I’ll try to remove the artifact#2023-01-1819:44jeroenvandijk> Just tried it on WSL2/Ubuntu and it works for me. What platform are you on? (and what version of JDK?)
On mac osx, and I tried with Java 8 and 11.0.7 and Clojure CLI version 1.11.1.1182#2023-01-1819:46jeroenvandijkremoving the maven artifact didn’t help (at .m2/repository/prismatic/plumbing/0.6.0)#2023-01-1819:47dpsutton❯ cd /tmp
/tmp on ☁️ metabase-query
❯ mkdir deps
/tmp on ☁️ metabase-query
❯ cd deps
/tmp/deps on ☁️ metabase-query
❯ clj -Sdeps '{:deps {prismatic/plumbing {:mvn/version "0.6.0"}}}'
Downloading: prismatic/plumbing/0.6.0/plumbing-0.6.0.pom from clojars
Downloading: de/kotka/lazymap/3.1.0/lazymap-3.1.0.pom from clojars
Downloading: de/kotka/lazymap/3.1.0/lazymap-3.1.0.jar from clojars
Downloading: prismatic/plumbing/0.6.0/plumbing-0.6.0.jar from clojars
Clojure 1.11.1
user=>
/tmp/deps on ☁️ metabase-query took 23s
❯ clojure -version
Clojure CLI version 1.11.1.1189#2023-01-1819:48jeroenvandijkOk weird something specific to my setup maybe. So far I’ve only seen it with this specific dep. I’ll try to isolate it more. I have found a workaround so no rush#2023-01-1819:49dpsuttoni’m on osx as well. very strange#2023-01-1819:49seancorfieldI'm on CLI 1.11.1208 but tried with JDK 8 and 19 on Ubuntu -- no problems. Very weird, indeed. Please let us know if/when you figure it out!?!? Curious minds need to know now...#2023-01-1819:49jeroenvandijkYes I will!#2023-01-1819:51jeroenvandijkThe other issue I found in slack sounds as strange as this one. Maybe just unlucky 😬#2023-01-1820:24borkdude@U0FT7SRLP What does:
(System/getProperty "file.encoding")
and
(java.nio.charset.Charset/defaultCharset)
return for you#2023-01-1821:12jeroenvandijk@U04V15CAJ
➜ ~ clj
Clojure 1.11.1
user=> (System/getProperty "file.encoding")
"UTF-8"
user=> (java.nio.charset.Charset/defaultCharset)
#object[sun.nio.cs.UTF_8 0xd02f8d "UTF-8"]
#2023-01-1821:36borkdudeok#2023-01-1900:44lreadMaybe something wackadoodled in your classpath? I expect you can't even get a result for?:
clj -Sdeps '{:deps {prismatic/plumbing {:mvn/version "0.6.0"}}}' -Spath#2023-01-1900:45lreadDid you rm -rf .cpcache?#2023-01-1911:08jeroenvandijkyeah even -Spath doesn’t work here and I’m doing a lot of rm -rf .cpcache lately for another project when things don’t resolve. Maybe it’s related. I’m also on an older OS X. I have been postponing the updates, but maybe it’s time#2023-01-1913:42lreadIf you are super curious, I suppose you could figure out exactly what is going wrong by debugging tools deps.
The error message seems to be generated https://github.com/clojure/tools.deps/blob/6ae2b6f71773de7549d7f22759e8b09fec27f0d9/src/main/clojure/clojure/tools/deps/script/make_classpath2.clj#L224-L234.#2023-01-1913:48jeroenvandijkThanks for the pointer! Yeah i think I’ll do that after I get the work done that initiated this error 😅. Curious to see where this rabbit hole ends#2023-01-1914:00lreadThis seems very similar: https://stackoverflow.com/questions/68003423/java-lang-illegalargumentexception-malformed-uxxxx-encoding-while-mvn-install#2023-01-1915:06jeroenvandijkhmm interesting
grep -rnw ~/.m2 -e '\u0000'
Binary file /Users/jeroen/.m2/repository/org/clojure/clojure/resolver-status.properties matches
Maybe that’s not good#2023-01-1915:07borkdudeyou can try -Sdeps '{:mvn/local-repo "/tmp/mvn"}'#2023-01-1915:08jeroenvandijkclj -Sdeps '{:deps {prismatic/plumbing {:mvn/version "0.6.0"}} :mvn/local-repo "/tmp/mvn" }' -Spath
src:/tmp/mvn/org/clojure/clojure/1.11.1/clojure-1.11.1.jar:/tmp/mvn/prismatic/plumbing/0.6.0/plumbing-0.6.0.jar:/tmp/mvn/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/tmp/mvn/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar:/tmp/mvn/de/kotka/lazymap/3.1.0/lazymap-3.1.0.jar:/tmp/mvn/prismatic/schema/1.2.0/schema-1.2.0.jar
#2023-01-1915:09jeroenvandijkSo something wrong in my ~/.m2 then#2023-01-1915:09borkdudecorrupt files like the stackoverflow issue also indicated#2023-01-1915:12jeroenvandijkI removed the line containing \u0000 in the .m2/repository/org/clojure/clojure/resolver-status.properties. It was looking like this
#NOTE: This is a Maven Resolver internal implementation file, its format can be changed without prior notice.
#Fri Aug 26 16:26:40 CEST 2022
maven-=
maven-metadata-central.xml.error=Could not transfer metadata org.clojure\:clojure/maven-metadata.xml from/to central (https\://repo1.maven.org/maven2/)\: \: nodename nor servname provided, or not known
maven-metadata-central.xml/@default-central-https\://repo1.maven.org/maven2/.lastUpdated=1661524000327
maven-metadata-clojars.xml.error=Could not transfer metadata org.clojure\:clojure/maven-metadata.xml from/to clojars (https\://repo.clojars.org/)\: \: nodename nor servname provided, or not known
maven-metadata-clojars.xml/@default-clojars-https\://repo.clojars.org/.lastUpdated=1661524000346
maven-metadata-my.datomic.com.xml.error=
maven-metadata-my.datomic.com.xml.lastUpdated=1634040492738
maven-metadata-my.datomic.com.xml/@default-my.datomic.com-https\://\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0
#2023-01-1915:12jeroenvandijkNot sure what this file does, but this was the issue!#2023-01-1915:13jeroenvandijkcc: @U04V70XH6 ^#2023-01-1915:13jeroenvandijkThanks for your help everyone!#2023-01-1915:22jeroenvandijkStill curious why this only happened with prismatic/plumbing so far :thinking_face:#2023-01-1916:48seancorfieldGlad you tracked it down! Weird, indeed. Sometimes just blowing away the whole .m2/repository tree and letting it download "the world" again is a worthy exercise:grin:#2023-01-1921:50respatializedFrom within a running Clojure process, is there a way to detect if a specific alias is “enabled?”#2023-01-1921:57Alex Miller (Clojure team)directly, no#2023-01-1921:57Alex Miller (Clojure team)indirectly, probably. presumably you care about the effect, probably on the classpath, which you can see via (System/getProperty "java.class.path")#2023-01-1922:14dpsutton:jvm-opts ["-Dsome.property=some-flog"] and then check it with that. I think we do that with
:ci
{:jvm-opts ["-Dci=TRUE"]}#2023-01-1922:20respatializedthat's a good way to do it; the use case I'm imagining is something like feature flags and custom java properties is a good way to ensure that capability is portable regardless of how the JVM is launched#2023-01-1922:21respatializedthanks!#2023-01-2318:39didibusclojure script.clj will say:
> WARNING: Implicit use of clojure.main with options is deprecated, use -M
Is that normal? I'm just running a script, I'm not trying to call a -main function? And I don't need to pass any alias.#2023-01-2319:03borkdude@didibus Yes, this was introduced a while ago. clojure.main is the namespace that takes care of executing files, -e, -i etc. So you you always have to type -M and then the usual -m or -e , <file.clj> etc.#2023-01-2321:19didibusI see, so you type -M and provide no other option:
clojure -M script.clj
Meh, another esoteric choice by the CLI 😛#2023-01-2717:05Sam Ritchiehey all, I’m wondering if I’ve hit a bug, or if there is some way to prevent this “unintuitive” behavior…
given deps.edn
{:aliases
{:cake {:exec-fn user/cake}}}
and src/user.clj
(ns user)
(defn cake [m]
(prn "you ran cake with " m))
(defn face [m]
(prn "you ran face with " m))
check out this behavior:
[
#2023-01-2717:05Sam Ritchiedepending on whether I supply an even or odd number of arguments, user/face is interpreted as a function OR as an argument#2023-01-2717:07Sam Ritchieis there some way to cause this to error if I supply the wrong number of args? I can do it inside of a babashka task…#2023-01-2717:07Sam Ritchie[
#2023-01-2717:09seancorfieldThis is "expected" but perhaps surprising if you don't know how the CLI figures out what invocation you're trying to run...#2023-01-2717:10seancorfieldSince CLI arguments are passed as key/value pairs, it uses the number of arguments to differentiate between a specific function and k/v pairs or the default function :exec-fn and k/v pairs.#2023-01-2717:11Sam Ritchieyeah, I get it, just trying to noob-proof a template I’m creating with some babashka tasks that call into clojure via -X#2023-01-2717:12seancorfieldHard to noob-proof -X because you have to understand how it is called 😐#2023-01-2717:13Sam Ritchiegot it! bb.edn:
{:tasks
{:init (defn X [cmd]
(do (if (even? (count *command-line-args*))
(apply shell cmd *command-line-args*)
(do (println "Please supply an even number of arguments!")
(System/exit 1)))))
start-clerk
{:doc "Start a Clerk dev server configured with this project's custom JS."
:task (X "clojure -X:dev:nextjournal/clerk user/start!")}}}#2023-01-2717:15seancorfieldYeah, you basically need a wrapper around the invocation...#2023-01-2717:17borkdudeNo, you can call -X with 1 argument, a single map#2023-01-2717:18borkdude@U017QJZ9M7W What I typically do is:
{:task (apply clojure "-X:whatever" *command-line-args*))
Why wrap this with checks?#2023-01-2717:19Sam Ritchiebecause the alias has a default :exec-fn , and I don’t want the user to accidentally call that#2023-01-2717:19Sam RitchieI can move the exec-fn to a NEW alias of course, since I’m using the clerk alias for both extra deps and the exec-fn#2023-01-2717:19borkdudebut you're already providing the exec-fn in the task, so the user can't call another one?#2023-01-2717:20Sam Ritchiethat’s the problem, see my paste above; if you supply an odd number of args, the first one’s interpreted as a new :exec-fn#2023-01-2717:20Sam Ritchieoh sorry I see your “what I typically do is” note#2023-01-2717:20Sam Ritchieit’s because Clerk’s Garden needs a :nextjournal/clerk alias with an :exec-fn that triggers the static build#2023-01-2717:21Sam Ritchieand I think it’s confusing for consumers of this template I’m writing if I put
• clerk dep in one alias with a different name
• garden :exec-fn in :nextjournal/clerk alias
• yet another alias with a different to supply another :exec-fn to start the local clerk dev server#2023-01-2717:22Sam Ritchiethat’s why I’m using bb.edn in the first place, because constructing these big strings out of simple decomplected parts is not something I want to explain to someone new who’s just trying to graph some functions 🙂#2023-01-2717:22Sam Ritchieas much as I am on board with it!!#2023-01-2717:09Yuhri Graziano BernardesHi everyone. I'm trying to use clojure.tools.deps/resolve-deps in a leiningen project (throught a remote REPL). I could download the lib locally, but seems that I need to update the classpath because the lib I just fetched is not there. Does anyone knows how can I do that?#2023-01-2717:13seancorfieldIf you're using Leiningen, you already have Pomegranate available which lets you load new dependencies at runtime.#2023-01-2717:14seancorfieldOtherwise, with tools.deps.alpha you'd need the add-lib3 branch and the add-libs function (which isn't documented and is subject to change). A variant of this may be built into Clojure 1.12 it seems.#2023-01-2717:15Yuhri Graziano BernardesThanks @U04V70XH6. I'll try out Pomegranate#2023-01-2717:39lreadBTW, I recently created a #C04KJME1UPL channel as part of my clj-commons-maintenance-wip-tlc pass on this lib.#2023-01-2811:25Benjamin:lib/hotload
{:extra-deps {org.clojure/tools.deps.alpha {:git/url ""
:git/sha "e4fb92eef724fa39e29b39cc2b1a850567d490dd"}
org.slf4j/slf4j-nop {:mvn/version "1.7.36"}}}
I have this in .clojure/deps.edn and it is useful. But when I start clj in a project that brings in a real slf4j, I get warnings.
Any way to say "don't add slf4j if I already have one".
I just had the idea that I make a second alias for projects where I don't want slf4j-nop#2023-01-2820:29practicalli-johnI included the slf4jnop to suppress the slf4j warning if there is not an implementation added to the class path.
I haven't used slf4j for a few years now (I use mulog) so don't know if there is a nicer way around this.
If including slf4j in most or all projects, I suggest removing the slf4j-nop library from the alias#2023-01-2908:35Benjamin👍 thanks for the great work and community effort btw 🙂#2023-01-2812:17dominicmCongrats on leaving alpha!
(clojure.tools.deps.util.dir/with-dir (java.io.File. "/tmp")
(create-basis nil))
Just noodling about fixing support for toolsbuild/project-root and decided to go straight to tools.deps.alpha. Unfortunately the above code leaves :paths with relative directories still. It doesn't look like util.dir is documented so I am reluctant to rely on it but in order to relativize the :paths consumption out of the basis I will have to. Not a big deal because I can just detect relative paths and handle them, but I thought it was worth a note.#2023-01-2813:24Alex Miller (Clojure team)Can you back up and tell me what you’re trying to do? Are you in tools.build or using tools.deps directly. If the former, you should be changing the project root instead#2023-01-2813:33dominicmI'm using a library based on tools deps#2023-01-2813:38Alex Miller (Clojure team)I’m happy to help but you’re going to have to give me way more context if you want me to do so #2023-01-2814:34dominicmMy library consumes the basis from deps.edn, and produces docker images, jars, etc.
The problem is that currently nothing like project-root is supported, so my library was reading :paths from the cwd rather than relative to deps.edn#2023-01-2815:14Alex Miller (Clojure team)I’m confused by many things here. You opened by talking about moving out of alpha, then said alpha later. I don’t know if that’s relevant or not. You mentioned tools.build but it’s unclear if you’re using that or using it as reference. You’ve mentioned “my library” but I don’t know what library, why no name or link. You said “consumed the basis” - this can mean many things. Is this the code above or something else? In that code, it’s unclear to me if something has changed or if you are changing something, what results you expect, what results you’re getting, etc. #2023-01-2815:16Alex Miller (Clojure team)Nothing material has changed recently in how any of this works. Im unclear if you are seeing a difference you don’t expect, or not seeing some difference you do expect #2023-01-2815:18Alex Miller (Clojure team)The dir ns has tools to set a directory context (effectively a cwd that can change), and you should use that if you are creating a basis in a context other than the current directory#2023-01-2815:20Alex Miller (Clojure team)The :paths in the basis are just the paths in the deps.edn so are always relative#2023-01-2815:21Alex Miller (Clojure team)If you’re looking for the output classpath paths, those are in the :classpath and :classpath-roots keys and should be absolute#2023-01-2816:45dominicmI'm no longer on the alpha tools deps in the library. Transition was pretty seamless.
The user issue mentioned tools build, but I'm using tools deps in my library. The library is pack.alpha, which builds jars, docker, etc. https://github.com/juxt/pack.alpha/
I think I was seeing relative paths in the :classpath for things specified in the roots deps.edn. Nothing has changed here, but everything else including local deps were converted to absolute paths so I was unsure if it's a bug.#2023-01-2819:52dominicmhttps://github.com/juxt/pack.alpha/issues/114 here's the user issue for reference, and the related commits#2023-01-2920:28dominicmSomething that slightly surprises me: extra-paths specified in ~/.clojure/deps.edn aren't relative to ~/.clojure/deps.edn but the CWD. I suppose that's because it's directly merged rather than considered as its own source?
I'm interested in doing this so that I can load a .clj file from ~/.clojure/... which loads a few common dev helpers for all projects I might work on (e.g. https://github.com/gfredericks/dot-slash-2)#2023-01-2920:28dominicmHere's what I tried:
{:aliases
{:user/dev
{:extra-deps {com.gfredericks/dot-slash-2 {:mvn/version "0.1.5"}
vvvvalvalval/scope-capture {:mvn/version "0.3.3"}}
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]
:extra-paths ["mydev"]
:main-opts ["-i" "@mydev.clj"]}}}#2023-01-2920:48Alex Miller (Clojure team):paths and :extra-paths are relative to the location of the project deps.edn, and yes, all deps sources are merged prior to resolution#2023-01-2920:48Alex Miller (Clojure team)you could instead use an alias that had :extra-deps to a local project root#2023-01-2920:49dominicmThat local root is still relative to the project deps.edn in my tests#2023-01-2920:51Alex Miller (Clojure team)it is, but could also be absolute to anywhere#2023-01-2920:52Alex Miller (Clojure team)there is no provided mechanism for making paths relative to the config directory#2023-01-2920:52Alex Miller (Clojure team)(and I don't expect to add one)#2023-01-2920:52Alex Miller (Clojure team)another approach is to put it in a git repo#2023-01-2922:49skylizeUse the absolute path?#2023-01-2922:51dominicmThat's a shame. I don't want to use an absolute path so the same thing works on all of the machines I work from.#2023-01-2923:25skylizeI was going to test if there might be a solution around the`$XDG_CONFIG_DIR` variable. But it looks like even a plain absolute path is not going to work for you.
{:aliases
{:path-test {:extra-paths ["/home/sky/.config/clojure/src"]}}}
$ clj -M:path-test
WARNING: Use of :paths external to the project has been deprecated, please remove: /home/sky/.config/clojure/src
Clojure 1.11.1
user=>
#2023-01-3000:42dominicmhttps://www.git-scm.com/docs/gitattributes
Filter and smudge might be the way to go#2023-01-3114:03Sam Ritchieis there some way to share, say, a version or entry across multiple tools.deps entries? I am thinking of the git sha here: https://github.com/mentat-collective/Mafs.cljs/blob/main/deps.edn#L15-L18#2023-01-3114:03Sam RitchieThis won’t work because we’re EDN not Clojure:
(let [sha "ec666210f11763fce4fec74072acca1e6525d29f"]
{:aliases
{:dev
{:extra-deps
{ {:git/sha sha}
{:git/sha sha}}}}})
This would work BACK IN THE DAY with Leiningen’s project.clj:
(def sha "ec666210f11763fce4fec74072acca1e6525d29f")
{:aliases
{:dev
{:extra-deps
{ {:git/sha ~sha}
{:git/sha ~sha}}}}}
#2023-01-3114:08borkdudeYou can put the dep in one alias and then compose that alias on the command line.#2023-01-3114:08borkdudeit's been a popular request to have derived aliases, but this doesn't exist yet#2023-01-3114:11borkdudeOf course you can generate the deps.edn with a script
(https://github.com/borkdude/rewrite-edn)#2023-01-3114:14Sam Ritchieah, nice#2023-01-3114:15borkdudeneil also supports adding deps + sha#2023-01-3114:15borkdudehttps://github.com/babashka/neil#2023-01-3115:33mpenetwould be nice to have a few tagged literals in tools.deps for that, for instance aero's #ref#2023-01-3115:34mpenet{:thing-version "1.0.0" :deps {.... {:mvn/version #ref [:thing-version] ...}}#2023-01-3115:35mpenetadd #include too on top of it and all the monorepo hacks go away#2023-01-3115:35mpenetsorry, I am getting carried away heh#2023-02-0103:09practicalli-johnclojure --help output seems to have a duplicated -X:deps find-versions line (at least on my machine) for the latest released version of Clojure CLI, :version "1.11.1.1208" - not sure if it was in earlier versions.#2023-02-0121:26Sam Ritchieis it possible to depend on an alias of a git dependency?
I want to depend on clerk as a git dep, but also pull in the extra-deps from this alias: https://github.com/nextjournal/clerk/blob/main/deps.edn#L36#2023-02-0121:27Alex Miller (Clojure team)Currently, no, but that’s something we’ve been talking about lately#2023-02-0121:28Alex Miller (Clojure team)I think it needs some consideration on both export and import sides#2023-02-0121:29Alex Miller (Clojure team)Like a lib would want to say what aliases it exports, and I think a downstream user might want to choose what it imports too#2023-02-0121:29Sam Ritchieagree… in this particular case I am debating between exporting these two dependencies from my project:
https://github.com/mentat-collective/clerk-utils/blob/main/deps.edn#L15-L20
and telling the user to bring their OWN clerk version, but wincing at the “please include both of these and make sure the SHA always matches”#2023-02-0121:29Alex Miller (Clojure team)The case I was looking at is very similar#2023-02-0121:31Sam Ritchieoh! while I’ve got you, I had a Q - how does dependency resolution work between
1. git deps (if I include one and someone overrides it, presumably that works)
2. a git dep and the clojars version of that dep — for this one I assume there is no dependency resolution?#2023-02-0121:32Sam Ritchiein my case this feature only works with git dependencies (the render dep is not available as a jar), so I dodge the problem. but I was was wondering if there were some way to tell tools.deps “I publish this code as a jar under this org/project, version == tag, please resolve that way” or something like that#2023-02-0121:45Alex Miller (Clojure team)at the moment, there is no comparison logic for git to mvn#2023-02-0121:46Alex Miller (Clojure team)so it will just fail and say you need to be explicit at top level about which one to use#2023-02-0121:48Alex Miller (Clojure team)but, there is a hole with a todo where this logic could go. maven jars have a pom, which has scm info, and it should actually be possible to compare the published git scm info for a jar and the git dep scm info and decide which was "newer", so maybe in the future#2023-02-0815:30respatializedIs there any way to invoke a function that takes positional args under an alias using clojure -X ? It seems like :exec-args is only designed for maps; is there something I missed in the docs and examples?#2023-02-0815:33Alex Miller (Clojure team)-X/-T only support functions that take a map#2023-02-0815:34Alex Miller (Clojure team)depending on your goals, https://github.com/babashka/cli might be of interest#2023-02-0815:36respatializedI’m just writing a wrapper fn for the time being; another dep is probably not necessary for my use case.#2023-02-0816:26borkdudebabashka.cli also only invokes functions that take maps#2023-02-0816:27Alex Miller (Clojure team)ah, sorry - I thought the focus was on the cli calling, not the function to call#2023-02-0816:28borkdudethe focus is on making functions that take maps (similar to -X) invokable as unix-like CLIs (like you're used to from tools.cli) using some light metadata annotation#2023-02-0816:30borkdudeat least, that's one of the things that it does. you can also just parse args yourself (like with tools.cli) and then invoke any function you want#2023-02-0916:36Sam Ritchiedoes anyone know how I can add a “provided” dependency to my generated pom.xml, using b/write-pom? I’m reading code here:
https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/tasks/write_pom.clj#L222
and it looks like there is no way to add my own pom entries… dependencies is overwritten by the basis#2023-02-0916:41seancorfieldI'm curious about the use case -- provided has always seemed a bit odd to me: either the dependency is needed or it isn't... :thinking_face:#2023-02-0916:44borkdudeprovided is usually used for local testing or optional deps but in deps.edn you can do this with aliases#2023-02-0916:48Sam Ritchie@U04V70XH6 cljdoc needs these to do analysis if you have a dependency that you want the user to specify#2023-02-0916:48Alex Miller (Clojure team)you are correct that there is no way to add custom deps to the generated pom.xml#2023-02-0916:48Sam Ritchieso if I want to force the user to manually declare an SCI dependency, for example#2023-02-0916:49Alex Miller (Clojure team)if you want to make a feature request --> https://ask.clojure.org#2023-02-0916:49Sam Ritchie@U064X3EF3 I’m going to add it to the basis, and then do
(xml/alias-uri 'pom "")
(alter-var-root
#'write-pom/to-dep
(fn [old]
(fn [[_ {:keys [mvn/scope]} :as pair]]
(cond-> (old pair)
scope
(conj [::pom/scope scope])))))#2023-02-0916:49Sam Ritchie@U04V70XH6 for example if @U04V15CAJ wanted cljdoc to work with https://github.com/babashka/sci.configs (which I know he doesn’t!), he’d need to add provided deps#2023-02-0916:49seancorfieldThat's a "bug" with cljdoc as far as I'm concerned -- and they know about it but haven't come up with a good way to address it yet.#2023-02-0916:50borkdudebut doesn't tools build merge with an existing pom.xml ? I remember there was something like this...? So you could add some stuff in there which should always left in or does it override dependencies?
I think there was already a request for this on ask, I'll see if I can find it#2023-02-0916:50seancorfield(and I've run into it with some of my libraries on cljdoc)#2023-02-0916:50Sam Ritchie@U04V15CAJ yeah, but it only merges one level deep#2023-02-0916:50Sam Ritchiedependencies from the basis wipe out the dependencies entry in the pom.xml template#2023-02-0916:50seancorfield@U04V15CAJ It completely replaces <dependencies> I believe, not merge.#2023-02-0916:51Sam Ritchie@U04V70XH6 I personally like provided dependencies… “I won’t work without this but you need to be explicit about what you want”#2023-02-0916:54borkdude@U017QJZ9M7W clojure.data.xml + add some deps yourself in tools.build? It's not that hard :) it comes with bb too#2023-02-0916:55Sam Ritchieyup, doing that now!#2023-02-0916:56borkdudebtw I can't find the ask issue, so maybe there wasn't one#2023-02-0917:01Alex Miller (Clojure team)it completely replaces some sections -depedencies, repositories, etc#2023-02-0917:06Sam Ritchie(def basis
(-> (b/create-basis
{:project "deps.edn"
:extra
{:deps
{'org.babashka/sci
{:mvn/version "0.6.37"
:mvn/scope "provided"}}}})))
#2023-02-0917:06Sam Ritchiethat works, given
(ns build
"tools.build declarations for the mafs.cljs library."
(:require [clojure.data.xml :as xml]
[clojure.tools.build.api :as b]
[clojure.tools.build.tasks.write-pom :as write-pom]))
(xml/alias-uri 'pom "")
(alter-var-root
#'write-pom/to-dep
(fn [old]
(fn [[_ {:keys [mvn/scope]} :as pair]]
(cond-> (old pair)
scope
(conj [::pom/scope scope])))))#2023-02-0918:56dominicm@U04V70XH6 use case is for libraries with namespaces that integrate with third party libraries that are optional?#2023-02-0918:58borkdude@U017QJZ9M7W What I meant was rewriting the pom after tools build had written it, not actually monkey-patch tools build but since this is in your private build, I guess do what you want :)#2023-02-0918:58seancorfield@U09LZR36F Right, but you don't need those in the POM -- you can use aliases to develop with.#2023-02-0918:59dominicmAh, but cljdoc was maven oriented. And any other maven oriented tools need that annotation.#2023-02-0919:03seancorfieldRight, but cljdoc is also looking at generating docs from source only (git dep) projects so I'd say its current Maven dependence is historical 🙂#2023-02-0919:04Sam RitchieI’ll do it the right way eventually, you’re of course right @U04V15CAJ … #2023-02-0920:38dominicmI'm a couple years out of date, I think 😅 I've got catching up to do#2023-02-0921:11lreadCljdoc will also eventually support static analysis to glean the public API.
Which should help with many awkwardnesses and failed API analyses.
I think this will be the default, and for libs that generate their API at load time - they will still be able to opt for dynamic analysis.#2023-02-0921:13borkdude(A foretaste of docs using static analysis: https://github.com/borkdude/quickdoc)#2023-02-0921:13lread> Maven dependence is historical
But will likely persist @U04V70XH6, I think, anyway.#2023-02-0921:17seancorfield@UE21H2HHD Well, if cljdoc is able to process git deps, those won't require pom.xml and so cljdoc would be able to analyze libraries without the "Maven dependence" which was what I was leaning into there... 🙂#2023-02-0921:34borkdudeMaybe what @U09LZR36F meant is that your analyzed dependency is a mvn artifact and that you can't update your docs unless you push a new version? I don't know if that's still the case, but this was coupled to a mvn version#2023-02-0921:36lreadGotcha @U04V70XH6, makes sense, somebody with a lib that has optional deps would be more likely to take the analyze from git repo sources route (when that becomes available).#2023-02-0921:39lread@U04V15CAJ, if that was what @U09LZR36F was getting at https://github.com/cljdoc/cljdoc/blob/master/doc/userguide/for-library-authors.adoc#articles-override.#2023-02-0918:50Drew VerleeThe trade off between building calling a function through the java command line against a jar, and calling cli -X and passing it a clojure function, is just that the later will be slower right?#2023-02-0918:51Drew VerleeAs in, for functions where the execution time isn't a factor, it's fine to use either route.#2023-02-0918:52seancorfieldThe former can be used standalone, the latter requires source etc.#2023-02-0918:52Drew VerleeYeah.#2023-02-0918:53dpsuttoncan you not do clj -Sdeps '{:deps {my/lib {:local/root "jar.jar"}}}' -X some/fn regardless if source is present or not?#2023-02-0918:57hiredmanwhat the later does is more context dependent than the former, and what it does will determine if it is slower than the former or not#2023-02-0919:02Drew VerleeIs there a way to have two -main functions?
Asking the question another way, If the .clj files on my deps :path already contain a -main function with a the command to generate classes (:genc-class), can I add another -main function, and specify that one in a separate build (build.clj) so I can end up with two jars each that call different main functions?
Maybe using the build api function https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-set-project-root.21! ? which takes, at a glance, a :main option for main class symbol?
Does this question indicate i have strayed from the path and should trying something else?#2023-02-0919:04seancorfieldYou can have -main in every single namespace but you can only have one default -main that is invoked via java -jar without specifying the class to invoke.#2023-02-0919:05seancorfieldjava -jar ... clojure.main -m some.name will invoke -main functions and we choose which one to invoke via clojure.main)#2023-02-0919:06Drew Verleethat makes sense, for some reason i hadn't considered that you could pass the namespace to the java cli.#2023-02-0919:06Drew Verleeis that what it's called? the java cli?#2023-02-0919:07seancorfieldI suspect people just call it "the java command" but I hadn't really thought about it...#2023-02-0919:08seancorfieldjava -jar ... MyPackage.MyClass has been supported for all of Java's life I think -- and clojure.main is "just" a class in the Clojure runtime.#2023-02-0919:10seancorfield(and you can probably just do java -jar ... some.name if you have (:gen-class) in that ns -- I'm just used to bouncing via clojure.main for a ns that might not be AOT compiled)#2023-02-0919:18seancorfieldOh, not java -jar ... MyPackage.MyClass -- java -cp ... MyPackage.MyClass sorry.#2023-02-0919:19seancorfieldHere we go:
(~/clojure/mains)-(!2016)-> cat src/multi/another.clj
(ns multi.another
(:gen-class))
(defn -main
"I don't do a whole lot ... yet."
[& args]
(println "I'm another main!"))
Thu Feb 09 11:18:09
(~/clojure/mains)-(!2017)-> cat src/multi/mains.clj
(ns multi.mains
(:gen-class))
(defn greet
"Callable entry point to the application."
[data]
(println (str "Hello, " (or (:name data) "World") "!")))
(defn -main
"I don't do a whole lot ... yet."
[& args]
(greet {:name (first args)}))
Thu Feb 09 11:18:12
(~/clojure/mains)-(!2018)-> java -jar target/net.clojars.multi/mains-0.1.0-SNAPSHOT.jar
Hello, World!
Thu Feb 09 11:18:22
(~/clojure/mains)-(!2019)-> java -jar target/net.clojars.multi/mains-0.1.0-SNAPSHOT.jar argument
Hello, argument!
Thu Feb 09 11:18:28
(~/clojure/mains)-(!2020)-> java -cp target/net.clojars.multi/mains-0.1.0-SNAPSHOT.jar multi.mains
Hello, World!
Thu Feb 09 11:18:48
(~/clojure/mains)-(!2021)-> java -cp target/net.clojars.multi/mains-0.1.0-SNAPSHOT.jar multi.mains argument
Hello, argument!
Thu Feb 09 11:18:52
(~/clojure/mains)-(!2022)-> java -cp target/net.clojars.multi/mains-0.1.0-SNAPSHOT.jar multi.another
I'm another main!
Thu Feb 09 11:19:02
(~/clojure/mains)-(!2023)->#2023-02-0919:24hiredmanhttps://github.com/hiredman/clojure-site/blob/patch-4/content/guides/running_a_clojure_program.adoc might have some useful context#2023-02-0921:52lreadAh thanks @U04V70XH6 and @U0NCTKEV8 the -cp variant had not occurred to me!#2023-02-1120:11Jeff EvansHas anyone had the problem that trying to resolve a git coordinate project just seems to hang forever? This is both in Cursive (my current IDE, deps aware) and on the command line via clj
#2023-02-1120:26Jeff EvansUpdate: it turned out that it didn't like the :git/tag for some reason. I removed that and put the full value in :git/sha#2023-02-1120:27Jeff EvansAnd that worked. Maybe there was something weird in this GitHub repo? confluentinc/kafka-streams-examples#2023-02-1120:38Alex Miller (Clojure team)Tags are only used to validate the sha#2023-02-1120:38Alex Miller (Clojure team)So if you had just the tag, that’s not allowed#2023-02-1120:39Jeff EvansSorry, to clarify, I had both a tag and short sha#2023-02-1120:40Alex Miller (Clojure team)If you had the repro coords, happy to take a look#2023-02-1120:41Jeff EvansThis hangs
clojure -Sdeps ‘{:deps {io.github.confluentinc/kafka-streams-examples {:git/tag “v7.1.6-18” :git/sha “d1311bc”}}}’
This works
clojure -Sdeps ‘{:deps {io.github.confluentinc/kafka-streams-examples {:git/sha “a3a1b4a528fb205ec86d2bf309431d45fb6d3e9d”}}}’#2023-02-1120:43Alex Miller (Clojure team)You can export GITLIBS_DEBUG=true to see the git commands being run#2023-02-1120:46Jeff EvansAh it's stuck on the tag --sort=v:refname for whatever reason#2023-02-1120:47Jeff EvansOr after it anyway#2023-02-1120:47Alex Miller (Clojure team)The GitHub page for that repo / tag https://github.com/confluentinc/kafka-streams-examples/tree/v7.1.6-18 says that commit is not in that repo#2023-02-1120:47Alex Miller (Clojure team)That can happen with a fork #2023-02-1120:48Alex Miller (Clojure team)There are a very large number of tags in that repo, might actually be taking a very long time#2023-02-1120:51Jeff EvansI tried running the command it printed on my local terminal and it finished instantly#2023-02-1120:52Alex Miller (Clojure team)Well that’s weird then#2023-02-1120:52Alex Miller (Clojure team)When I get a chance I will look at it#2023-02-1120:53Jeff EvansNot urgent, I can work around, but I appreciate it#2023-02-1120:53Jeff EvansBy the way, do you happen to know if test code should be available when using the git resolution?#2023-02-1120:54Jeff EvansOr is it only main?#2023-02-1121:35Alex Miller (Clojure team)Only :paths#2023-02-1121:36Alex Miller (Clojure team)One thing I’m considering is whether to allow git deps to publish aliases for use externally which would allow this#2023-02-1121:37Jeff EvansHow does that work for a Maven project that's just hanging out on GitHub? I think I'm not quite understanding how :paths factors in there#2023-02-1122:39Alex Miller (Clojure team)It wouldn’t #2023-02-1204:47Alex Miller (Clojure team)@U0183EZCD0D this is a bug in capturing large output from the git process. I've pushed a fix, will be in the next build of the CLI. Thanks!#2023-02-1213:56Jesús GómezHello. What is the current way to require the function add-lib or add-libs these days?
https://insideclojure.org/2018/05/04/add-lib/#2023-02-1214:05practicalli-johnThis is how I hotload for libraries, using a rich comment block. It could also be done in the deps.edn file but I prefer the rich comment
https://practical.li/clojure/clojure-cli/projects/hotload-in-project/#2023-02-1214:06practicalli-johnOr if using a terminal UI repl prompt - https://practical.li/clojure/clojure-cli/repl/libraries/#hotload-libraries#2023-02-1214:15Jesús GómezThank you very much.
Kind of an aside, but I notice that the page of tools.deps.alpha say it is obsolete and tools.deps should be used[1], but I didn't find add-libs in tools.deps (maybe I didn't search enough).
Do you know if the function add-libs is part of tools.deps namespace somewhere, maybe a sub namespace I'm not finding?
[1] https://github.com/clojure/tools.deps.alpha#2023-02-1214:19practicalli-johnAs I understand it, tools.deps should be used for everything, except add-libs which has not yet been finalised (although it hasnt changed much in the last year). So just for add-libs, use tools.deps.alpha until Alex is kind enough to find time to finalise the add-libs feature and include it in tools.deps or there preferred way to package it.#2023-02-1214:25Jesús GómezNow I understand. Thank you very much!#2023-02-1214:28Jesús GómezYou have a broken link in the article https://practical.li/clojure/clojure-cli/projects/hotload-in-project/
The first link is pointing to https://practical.li/clojure/clojure-cli/projects/hotload-in-project/hotload-libraries.md#2023-02-1214:31practicalli-johnThanks, I just noticed that too... the book is being refactored and I'm working on improviing that bit today and expanding on hotloading and other tools for a REPL reloaded page... the left hand navigation should all work though...#2023-02-1215:32skylizeBased on recent comments here from Alex, add-libs should probably be out of alpha pretty soon, I think.#2023-02-1216:08Alex Miller (Clojure team)indeed, there will be an official way to do this soon (but it will not be in tools.deps at all)#2023-02-1215:08Jesús GómezWhat happens when I include many aliases and each one has main-opts? Which of those libraries' main methods will be called and with what main-opts?
clj -M:alias-1:alias-2:...:alias-n
#2023-02-1215:14practicalli-johnIn a chain of aliases, only the last :main-opts definition is used, so in the example only :main-opts from :alias-n#2023-02-1215:15Jesús GómezThank you!#2023-02-1215:20Jesús GómezI found the doc reference of this information:
https://clojure.org/reference/deps_and_cli#_execution
(Regarding :main-opts) "If multiple maps with these keys are activated, only the last one will be used"#2023-02-1312:25Sam RitchieQuestion… I think I’ve seen here that :git/tag for dependencies will NOT work without an associated :git/sha.
however, this command seems to work fine:
clojure -Ttools install io.github.seancorfield/deps-new '{:git/tag "v0.4.13"}' :as new
while this one fails:
clojure -Sdeps '{:deps {io.github.mentat-collective/clerk-utils {:git/tag "v0.3.0"}}}' \
-Tnew create \
:template clerk-utils/custom \
:name myusername/my-notebook-project
with
Error building classpath. Library io.github.mentat-collective/clerk-utils has coord with missing sha
is there something about -Ttools install that allows you to forgo the :git/sha?#2023-02-1312:44borkdudeinstall just is more lenient here. there is also install-latest#2023-02-1312:45Sam Ritchiegot it, makes sense after digging through the code#2023-02-1312:45Sam RitchieI wish there were some easy way to have github inject the current SHA into a README.md#2023-02-1312:45borkdudetools like #C03KCV7TM6F allow you to add a dependency by tag:
neil add dep io.github.mentat-collective/clerk-utils --git/tag v0.3.0
#2023-02-1312:45borkdudeand then it adds the sha for you#2023-02-1312:46borkdudewithout providing the tag it picks the last tagged release#2023-02-1312:47borkdudefor README: I sometimes work with templates that I re-generate, but the downside of this is that I regularly receive PRs that update the generated README instead of the template :)#2023-02-1615:58dpsuttonIs it possible for an alias to remove from :paths? Idea would be to create a jar and then use the jar rather than the source paths during testing. Making test startup faster, actually testing the artifact, etc.#2023-02-1616:22Alex Miller (Clojure team)no#2023-02-1616:23Alex Miller (Clojure team)you can use -Scp to override everything and specify the classpath, not sure if that helps?#2023-02-1616:28dpsuttonthat’s a great point. the classpath becomes quite simple in this scenario. thanks alex#2023-02-1619:21borkdudebut you can :replace-paths in an alias#2023-02-1619:21borkdude(aka just :paths currently, I think?)#2023-02-1619:28Alex Miller (Clojure team)Either is fine, they are treated as synonyms#2023-02-2316:39camdezHey all, I’m trying to tighten up our CI process so all dependencies are fetched ahead of time (with -P) and cached. But even with clojure -P -T:tool-name, dependencies still get downloaded when I invoke the tool (later). I can’t seem to figure out how to solve it. Any tips?
It feels like there may be some dependencies getting added at runtime (though I don’t see where).#2023-02-2316:44camdezI’m using seancorfield/build-clj (deprecated, I know…). This is what I see downloaded at invocation time:
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.pom from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.pom from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.pom from clojars
Downloading: org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar from central
Downloading: org/clojure/data.xml/0.0.8/data.xml-0.0.8.jar from central
Downloading: riddley/riddley/0.1.12/riddley-0.1.12.jar from clojars
#2023-02-2316:46borkdudecan someone remind me what use -T has inside a project compared to -X?#2023-02-2316:47practicalli-johnTry using the -X execution flag, -T might be excluding dependencies#2023-02-2316:47borkdude(Btw, https://github.com/babashka/babashka/blob/master/examples/download-aliases.clj is a small babashka script to download all deps from all aliases)#2023-02-2316:47camdez@U04V15CAJ I believe the distinction is that -T “will use only the :paths and :deps from the :build alias”#2023-02-2316:48camdez@U05254DQM Thanks. Pretty sure I’ve tried that also but I will confirm now.#2023-02-2316:48borkdudeOh right, I guess you can't "prepare" all of those combinations in one go since they are exclusive wrt dependencies and paths#2023-02-2316:49camdezYeah. And I’m ok with multiple prep commands, I just want a combo that gets me there…#2023-02-2316:49practicalli-johnI assume a cache is already added to the workflow to cache the libraries#2023-02-2316:49borkdudeStill I don't get the difference between -T and -X the more I think about it. Doesn't -X + :paths + :deps have the same effect as -T? Since :extra-paths would add, but :paths would replace... right?#2023-02-2316:50camdez@U05254DQM Yes, that is correct. But this is literally on the same machine in the same build run. It’s simply not downloading the libraries the first time. (It doesn’t show the download messages there either).#2023-02-2316:50Alex Miller (Clojure team)note the things being downloaded above are poms, used for dependency analysis (not the jars). Also, -X and -T do almost the same thing (and literally share the same code by the time you get to execution)#2023-02-2316:51camdez@U064X3EF3 I see POMs and JARs in the output. Am I misreading?#2023-02-2316:51borkdudeMaybe -T always wipes the top level deps, unlike -X? that must have been it#2023-02-2316:51Alex Miller (Clojure team)oh sorry, I didn't read all of that, yes#2023-02-2316:51Alex Miller (Clojure team)-T excludes the project deps (unlike -X)#2023-02-2316:52camdezThe code for build-clj is pretty simple…I don’t see any dynamic dependency magic happening, so I think it must be in some transitive dependency, but I can’t figure out how to get the transitive dependency tree for a tool… that would be really helpful.#2023-02-2316:53Alex Miller (Clojure team)you can#2023-02-2316:53Alex Miller (Clojure team)tool by name or by alias?#2023-02-2316:54camdezBy alias#2023-02-2316:55Alex Miller (Clojure team)clj -X:deps tree :project nil :aliases '[:toolalias]'#2023-02-2316:55Alex Miller (Clojure team)should be same I think#2023-02-2316:56camdezI had to amend that to clj -X:deps tree :project nil :aliases '[:uberjar]' (hope that’s right), but that gave me this:
org.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
#2023-02-2316:57camdezSo those dependencies don’t seem to appear. I doubt riddley, e.g., is a transitive dependencies of those.#2023-02-2316:57camdez@U064X3EF3 Thanks. I would not have come up with that incantation myself!#2023-02-2316:57borkdudeAt this point I suspect it would be helpful to have a repro deps.edn#2023-02-2316:57camdezStarting to feel that way.#2023-02-2316:58camdezOk, I’ll circle back with one. Thanks for the help, gents.#2023-02-2317:01Alex Miller (Clojure team)well, not sure that will work - removing the project deps.edn also removes the aliases in it so if the tool is defined in the current project, that may not work#2023-02-2317:01camdezAh, fair point#2023-02-2317:02Alex Miller (Clojure team)if you move it to your ~/.clojure/deps.edn, that would work. or you could re-introduce that alias with :extra '{:aliases {:uberjar {...}}}'#2023-02-2317:05camdezGood thinking. Moved it to my ~/.clojure/deps.edn. More reasonable output now, but I still don’t see (all of) the dependencies that get downloaded at run time.#2023-02-2317:06camdezSooo…something is adding dependencies dynamically when the tool runs?#2023-02-2317:08camdezWorking on a clean repro repo.#2023-02-2317:14Alex Miller (Clojure team)assuming you are running uberjar as a tool, and that is using build-clj, which is using tools.build, which is almost surely creating a basis, which will download the project deps#2023-02-2317:15Alex Miller (Clojure team)so you would need to prepare both:
clojure -P -T:tool-name
clojure -P
#2023-02-2317:15camdezOh… I’m not doing the latter. I wonder if that is the issue.#2023-02-2317:16Alex Miller (Clojure team)I think it's definitely the issue :)#2023-02-2317:22camdezIndeed it was. 🎉
Thanks a ton.#2023-02-2317:23camdezSo the (bare) clojure -P does what, exactly? Preps tools.deps dependencies? I guess I assumed that would happen automatically by virtue of preparing anything else.#2023-02-2317:24Alex Miller (Clojure team)clojure does two things - 1) run a process that computes the dependency for the program to run (optionally, guarded by a cache) and 2) run the program using the computed classpath based on #1. -P does just #1.#2023-02-2317:27camdezI (believe) I knew that much but I’m unsure how to square that with clojure -P -T:uberjar not (to my mind) fully computing the dependencies needed to run clojure -T:uberjar.#2023-02-2317:28Alex Miller (Clojure team)the first runs a program that uses tools.deps to download project deps, but clojure -P -T:uberjar does not actually run the program (#2)#2023-02-2317:29Alex Miller (Clojure team)so the first line downloads the deps to run the uberjar program. the second line downloads the deps that will be used by that program#2023-02-2317:31camdezAhhh. I may need to process that a little to update my mental model. 🙂
But, to restate what I think I’m learning, dependencies like riddley are not used directly by the :uberjar (-aliased) tool (hence my trouble finding them in its code) but are needed for clojure to invoke that tool.#2023-02-2317:33Alex Miller (Clojure team)they could easily be transitive deps, don't know#2023-02-2317:33camdezI had been thinking of -P as “prepare everything we need to run this program”, but it’s more like “prepare everything this program needs to run”. Subtle distinction.#2023-02-2317:45camdez@U064X3EF3 @U04V15CAJ @U05254DQM Thanks a ton. Been pulling my hair out over this for a while. CI times are now back below five minutes and some generous soul is saving some bandwidth. 🙂#2023-02-2317:49borkdude@U0CV48L87 I'm happy you solved your problem but I'm not sure how you solved it? By understanding that -P didn't do what you thought it did?#2023-02-2317:50camdezMore or less.
I didn’t realize it was necessary to run clojure -P in addition to clojure -P -T:uberjar in order to prepare everything needed to run clojure -T:uberjar.
I suppose I’ll always add a clojure -P line to my CI builds from here on.#2023-02-2318:10camdez@U04V15CAJ Does that make sense?
I’m still a little fuzzy on the why. It would be clearer to me if I were just using -A , in which case I can imagine that dependencies needed directly by the alias are not necessarily everything needed to run the program in that configuration (i.e. something more globally-scoped within the project).
With -T I figured we were totally ignoring the surrounding context and letting the tool completely take over the dependency list, but perhaps some things are loaded in the process of bootstrapping the tool? I really don’t know.#2023-02-2318:13camdezWait, maybe I totally misunderstood that.
I’m building an uberjar here. It’s probably simply that the full set of project dependencies needed for the uberjar itself were never fully downloaded.#2023-02-2318:14camdezAnd I was mistaking that output for the uberjar tooling downloading its own dependencies.#2023-02-2318:18borkduderight. for creating an uberjar you need tools.build and its dependencies. but when executing build.clj dependencies are pulled while calculating the basis. to "prepare" that you should just do clojure -P combined with all the aliases that you're using in your basis (if any)#2023-02-2318:30camdezOk, yes, I have now confirmed what I said above.
Yeah… I actually understood all of those things and my mental model was correct, I was just misreading the dependency fetches during uberjarring time, thinking they were the dependencies of the tool itself, which I thought I had somehow failed to prepare.
In actuality they were the handful of (general) project dependencies that hadn’t been pulled in by previous steps in the CI process (including testing, hence most dependencies already needing to be pulled). Had I seen a large number of dependencies get pulled at that time then it would have been obvious to me what was going on.
I think they might specifically be dependencies which happen to get shadowed by test dependencies (I do run clojure -P -M:test early in the build, FWIW).#2023-02-2321:12practicalli-johnNot sure if this helps, but I've used -X along with docker build cache in the build and haven't noticed issues
COPY deps.edn /build/
RUN clojure -P -X:env/test:package/uberjar
https://practical.li/blog/posts/build-and-run-clojure-with-multistage-dockerfile/
I haven't used tools.bould yet, but assume it should be very similar (it's on my to-do list)#2023-02-2323:09camdezThanks. I’ve done it with a multi-stage Docker build as well, and it’s a very cool approach. In this case I was using Circle CI’s native caching, but it’s all working great now. Ultimately just a misread on what problem I actually needed to be solving. Classic debugging story.#2023-02-2511:18GeromeHello everybody! I am running into a very strange issue. I checked out the main branch of https://github.com/babashka/scittle and after some initial error I tried running clj -Stree -A:dev and it returns Error building classpath. Library org.babashka/sci has prefix sha, use full sha or add tag. Is there anything I can do to figure what's wrong here? deps.edn is the thread.#2023-02-2511:19Gerome#2023-02-2511:47delaguardowhat is your clojure -Sdescribe?#2023-02-2511:49Gerome#2023-02-2511:51borkdudeCan you try with:
GITLIBS_DEBUG=true clj -Stree -A:dev
#2023-02-2511:55GeromeStill the same#2023-02-2511:56borkdudeWhat happens when you do:
rm -rf ~/.clojure/.cpcache
rm -rf .cpcache
rm -rf ~/.gitlibs
and then try again?#2023-02-2511:56delaguardowierd, there is no such commit https://github.com/babashka/sci/commit/20f08f189ff4027b682f33f93efd20d9e2d2d7#2023-02-2511:57Gerome@U04V15CAJ Still the same error.#2023-02-2512:01borkdudeOh I see. I pulled the latest scittle main locally now (I thought I was on the latest but wasn't and can now repro).
$ clj
Error building classpath. Library org.babashka/sci has prefix sha, use full sha or add tag#2023-02-2512:01borkdudeI'll fix it, sorry for the trouble#2023-02-2512:01borkdudeNot sure what commit that pointed to...
https://github.com/babashka/scittle/commit/12c59b6f68818e0697e7f02d6812df1ea74c8928
cc @U02CV2P4J6S#2023-02-2512:03borkdudeshould work now with latest on main @UPJP9G4G1#2023-02-2512:04borkdudeMaybe the error message could be improved in #C6QH853H8 about non-existing SHAs rather than "this is a prefix sha".
cc @U064X3EF3#2023-02-2512:04GeromeNo problem, I'm happy to have helped in figuring out an error @U04V15CAJ#2023-02-2512:05borkdudeLet's continue in the other thread again :)#2023-02-2512:56Alex Miller (Clojure team)It looked like the sci git sha was <40 chars, so it was a prefix sha#2023-02-2512:57Alex Miller (Clojure team)You should get a different error for a full length non-existing sha#2023-02-2512:58borkduderight, it seems 2 chars were missing, weird. makes sense#2023-02-2513:20pyrHello! I couldn't find in the documentation how to prep a library which is in an alias, what is the recommended way to go about this?#2023-02-2513:20pyr(any variation of clojure -X:deps prep which adds the alias containing the lib fails to create the classpath)#2023-02-2515:36Alex Miller (Clojure team)Can you share the command, the error, and the deps.edn?#2023-02-2515:37Alex Miller (Clojure team)Should be something like clojure -X:deps prep :aliases ‘[:foo]’#2023-02-2520:22pyrThanks, I confirm it works#2023-02-2712:07pmooserI'm having a bit of trouble using a library with a java compilation step with :deps/prep-lib.
I have a project A with a :local/root dependency on a project B that has a :deps/prep-lib entry that compiles java. I can do a clj -X:deps prep from the project directory of A, and that works - I get a message about prepping B, and the compilation results are in B/target/classes. The problem is that the generated classpath of A doesn't seem to include anything like the target/classes directory of B, so when A launches, we get an error that the java class wasn't found (because it's not on the classpath).
When we prep a lib, I assume it's correct for the lib's prep results to be in its own directory?#2023-02-2712:09borkdudeDoes dependency B have the target directory in its :paths in its deps.edn?#2023-02-2712:09borkdudeprep/libs doesn't magically add directories to the classpath, they should be declared in deps.edn#2023-02-2712:09pmooserOk, that explains it then.#2023-02-2712:09pmooserBy the way, you are everywhere.#2023-02-2712:10pmooserI really appreciate your help but I begin to feel a bit guilty that you're the one who answers all of my questions. 🙂#2023-02-2712:11pmooserI feel like this sort of means that I need to have target/classes in the classpath just for this case, even if it isn't required (I don't think) for the normal case where I've deployed a jar.#2023-02-2712:11borkdudeNo problem at all, happy to help. If you want to do anything in return, feel free to sponsor. I'd rather have people do this out of gratitude rather than guilt though :)#2023-02-2712:12borkdudetrue#2023-02-2712:12borkdudeit doesn't hurt to add target to the classpath if it doesn't exist btw#2023-02-2712:13borkdudeYou can see the docs here:
https://clojure.org/reference/deps_and_cli#prep#2023-02-2712:29pmooserOk yes, adding "target/classes" as a resource path in the library (`B`) resolves the problem, as you suggested it would.#2023-02-2714:32vxehello, has pulling from http maven repos been intentionally dropped in some clojure 1.11 verision? I am getting
clojure -Spom -Srepro
...
Error building classpath. Invalid repo url (http not supported):
on clj version 1.11.1.1224 , the command succeeds on 1.11.1.1200 . if intentional are there are any workounds on the maven side to get around the http check, thanks in advance
% java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
#2023-02-2715:04Alex Miller (Clojure team)Yes, this is no longer allowed in the Maven ecosystem to protect against man-in-the-middle attacks so we've followed their lead, see https://maven.apache.org/docs/3.8.1/release-notes.html#cve-2021-26291#2023-02-2715:05Alex Miller (Clojure team)If there is a need here (seems like there might be), please file a question on http://ask.clojure.org and I will consider a way to allow it with some config#2023-02-2715:11Alex Miller (Clojure team)in the meantime, you'll want to fall back to the older release - you can find instructions on how to do that with brew at https://github.com/clojure/homebrew-tools#version-archive-tool-releases (with Linux you just install the older version)#2023-02-2715:46Alex Miller (Clojure team)it would be helpful to know if you are using Maven mirroring in your settings.xml#2023-02-2717:41practicalli-johnUpgrading Nexus server to use HTTPS is the recommended approach I assume Maven gives (or use a mirror service that supports HTTPS)
I vaguely remember we updated Nexus back in 2018 when Maven introduce this change. Or maybe we migrated to artefactory
Leiningen dropped support for http repos around the same time, with release 2.9.x#2023-02-2717:50Alex Miller (Clojure team)yes, that is the recommendation from Nexus#2023-02-2717:50Alex Miller (Clojure team)but I also don't want to make this unusable in the meantime so I gate the check with a flag or something#2023-02-2801:37vxethanks for looking, I filed the question here: https://ask.clojure.org/index.php/12698/deps-all-pulling-from-http-maven-repos thanks#2023-02-2801:38Alex Miller (Clojure team)I don't know if you saw in other threads, but I released a new version of the Clojure CLI today that allows you to reenable it by setting the environment variable CLOJURE_CLI_ALLOW_HTTP_REPO#2023-02-2801:39Alex Miller (Clojure team)1.11.1.1237#2023-02-2823:42vxeawesome thank you 🙏#2023-02-2723:16Alex Miller (Clojure team)https://clojurians.slack.com/archives/C06MAR553/p1677539761360519#2023-02-2723:32seancorfieldWhen will -A stop running :main-opts? 🙂#2023-02-2800:26Alex Miller (Clojure team)“Later”#2023-03-0209:27gregclj by default loads aliases from $HOME/.clojure/deps.edn. Is there a way to extend the list of deps.edn loaded by the clj tool?#2023-03-0209:31gregE.g. I have project specific aliases that I don't want in my $HOME/.clojure/deps.edn and I don't want it in the project deps.edn#2023-03-0209:39delaguardoyou can set CLJ_CONFIG environment variable to point to custom deps.edn
https://clojure.org/reference/deps_and_cli#deps_sources
there is also relevant question https://ask.clojure.org/index.php/9849/teams-common-dependencies-tooling-across-multiple-projects?show=9849#q9849#2023-03-0211:08borkdudeAdditionally, you can use -Sdeps to provide an extra map with aliases#2023-03-0215:03teodorluHi! I'm relatively new to library development, and want to make a library and an app that uses that library. I thought I'd use deps.edn and :local/root to develop the library and the app that uses the library locally.
When I'm working locally with my REPL, is it possible to have a repl in my app that picks up changes in my library source code? How does that work? Is your workflow similar to this when you are developing a library?
Specifically, I'm using Doom Emacs and CIDER, but I assume the REPL question is somewhat independent of editor. Thanks!#2023-03-0215:06Alex Miller (Clojure team)depends what exactly you mean by "picks up changes". nothing will automatically change your running Clojure to reflect changes in the library code, but you can (via several means) reload those namespaces to have those changes take effect (without restarting your repl)#2023-03-0215:25teodorluRight -- so manually running a require reload like this
(comment
(require '[teodorlu.some-lib] :reload))
could suffice?#2023-03-0215:26Alex Miller (Clojure team)yep#2023-03-0215:27teodorluThanks! Then I'll start with that 🙂#2023-03-0215:34borkdudeEither :reload or just re-evaluate those library sources in your REPL from your favorite editor#2023-03-0417:53borkdudeI added a deps.edn to a project.clj project (not my own) with java sources. As I didn't want to disturb the project too much, I added a :deps/prep-lib step which calls lein javac to compile the sources 😂
See #C01ALH5JGRJ for details#2023-03-0921:56sashtonWhat would be the most straightforward way to run with a custom version of tools.deps? I’ve tried using replace-deps with my local root like so: :aliases {:deps {:replace-deps {org.clojure/tools.deps {:local/root "_____/tools.deps"}}} . I’ve seen other mentions of replacing deps this way elsewhere in this channel (https://clojurians.slack.com/archives/C6QH853H8/p1607026303165100).
However, although I’ve added printlns and file I/O in a number of places in my copy of tools.deps to verify my custom version is working, I don’t observe any of my expected output.
I was hoping that running with one of the following would execute using my custom version:
clj -Stree
clj -A:deps -Stree
Is what I am attempting possible?#2023-03-0922:04borkdude@U08TWB99B Your question can mean several things. Do you want the clojure CLI itself to use your fork, or do you want your build.clj to use your tools deps fork#2023-03-0922:04sashtonI guess it would be clojure CLI#2023-03-0922:05borkdudethe clojure CLI uses a pre-built uberjar containing tools.deps. It doesn't use a version that is described in your deps.edn. But you should see that custom version in clj -Stree#2023-03-0922:06borkdudee.g. if you would do:
clj
(require '[clojure.tools.deps])
it should end up using your fork#2023-03-0922:06seancorfieldTo avoid confusion, I would recommend using a different alias for testing, since :deps is a built-in alias in the "root" deps.edn (which is inside tools.deps).#2023-03-0922:10sashton@U04V15CAJ But would that happen before dependencies are resolved? In particular, what I’m trying to do is add some sort of offline mode for git dependency resolution. My work VPN is causing me serious headaches accessing off-VPN git libs, and they change so infrequently that I would be okay running with cached versions for periods of time.
So, if tools.deps is bundled in the uberjar, I’m thinking I’ll need to build that. I have tried poking around to figure out where that is built, but I haven’t figured that out yet, either.#2023-03-0922:11borkdudeyes#2023-03-0922:12borkdudeI mean, yes, you need to build your own uberjar#2023-03-0922:12sashtonAny idea where that happens?#2023-03-0922:12borkdudeif it helps, here is a re-implementation of the bash portion of the CLI: https://github.com/borkdude/deps.clj
but it still downloads the uberjar from http://clojure.org#2023-03-0922:12Alex Miller (Clojure team)it happens in https://github.com/clojure/brew-install#2023-03-0922:13Alex Miller (Clojure team)with clj -T:build release#2023-03-0922:13sashtonThanks! I’ll start there#2023-03-0922:14Alex Miller (Clojure team)the uber jar ends up in target/clojure-tools-<version>.jar#2023-03-0922:14Alex Miller (Clojure team)but this is not a sustainable path to an environment, really only suitable to testing something#2023-03-0922:15Alex Miller (Clojure team)depending on your scenario, you may be able to solve this with the existing git support#2023-03-0922:15Alex Miller (Clojure team)which supports local file path git repos#2023-03-0922:16Alex Miller (Clojure team)but it depends how much git stuff you are depending on and whether those have transitive git deps too#2023-03-0922:17sashtonYeah, I think that would be complicated with transitive dependencies.#2023-03-0922:17Alex Miller (Clojure team)well you can rewrite git urls in git itself#2023-03-0922:18Alex Miller (Clojure team)look up insteadOf#2023-03-0922:18Alex Miller (Clojure team)you should be able to set git config that rewrites your external git urls to local git urls. because tools.deps (really tools.gitlibs) uses git, it will also use that#2023-03-0922:19sashtonThat would be interesting. I’ll take a look at that option first.#2023-03-0922:20Alex Miller (Clojure team)just as a meta observation - start with the actual problem here :)#2023-03-0922:23Alex Miller (Clojure team)"My work VPN is causing me serious headaches accessing off-VPN git libs, and they change so infrequently that I would be okay running with cached versions for periods of time." is a good problem statement :)#2023-03-0922:24Alex Miller (Clojure team)and just fyi, there are also tools that do this sort of thing at an enterprise level (just like they do with maven repos), and I would very much like to support those well eventually, not just for performance but for enterprise control/threat containment#2023-03-0922:34sashtonNice! insteadOf handles my case. I like that much better. Thanks!#2023-03-1001:48aaelonyI have an uberjar that runs fine on my mac, e.g.:
java -cp target/query.jar clojure.main -m aegp.main --help
works fine on my mac.
Relevant part of the deps.edn file is:
:uberdeps {
:replace-deps {uberdeps/uberdeps {:mvn/version "1.2.0"}}
:replace-paths []
:main-opts ["-m" "uberdeps.uberjar"]
}
all my *.clj source files have (:require [...] (:gen-class)), yet a colleague running windows with only a JRE (not JDK) gets:
Error: Could not find or load main class clojure.main
though when he runs it from windows.
is there a better more robust manner to create the uberjar such that it can run on any of windows, macos, linux?#2023-03-1001:55hiredmanDo you know the command line your colleague is using?#2023-03-1002:01hiredmanThat is exactly the error message you would expect if they tried that with a regular jar of your project, not an uberjar, so make sure they are using an uberjar#2023-03-1002:02Alex Miller (Clojure team)that error indicates the clojure.main class is not in the uber jar. if not, then your first command wouldn't work anywhere so something doesn't add up. you might check jar tf target/query.jar | grep clojure/main#2023-03-1002:03aaelonycool, thanks. I'll check with him what he ran exactly. I'm also converting to a lein project to see if lein uberjar works as well..#2023-03-1002:04hiredmanI would not bother with that#2023-03-1002:04hiredmanIf you can run it, but they cannot, the very likely explanation is they are running something different#2023-03-1002:04aaelonyin anycase would be nice to do it correctly (on my end) if I am missing something#2023-03-1002:06hiredmanIf the uberjar worked for you then it is unlikely there is anything wrong with it#2023-03-1002:06aaelonyyeah - works perfectly for me#2023-03-1002:07aaelonycould the version of java he is using be material? I am using 17, I think he is using java 8#2023-03-1002:08hiredmanAre you building the uberjar and sending it to them or are they building it for themselves?#2023-03-1002:08aaelonyembarassingly, I'm slacking him a jar file#2023-03-1002:08aaelony(that I built)#2023-03-1002:09hiredmanThen double check and make sure it is the uberjar and not a regular jar#2023-03-1002:09hiredmanMake sure it is the most recent one#2023-03-1002:10aaelonyhe left for the evening. I'll check with him tmrw. Thanks for the responses#2023-03-1002:13hiredmanMake sure they aren't mixing -cp and -jar flags. They are mutually exclusive, but I forget if you get an error or the java just silently ignores one or the other#2023-03-1415:30lvhHey! I’m trying to set java.library.path via :jvm-opts, but it seems to have no effect. 🧵#2023-03-1415:30lvhsigncryption-demo on main [!] via :coffee: v19.0.2
❯ cat deps.edn
{:paths ["src" "resources"]
:deps {org.clojure/clojure {:mvn/version "1.11.1"}
caesium/caesium {:mvn/version "0.15.0"}
nrepl/bencode {:mvn/version "1.1.0" :lvh/comment "netstring impl"}}
:jvm-opts ["-Djava.library.path=/opt/homebrew/lib"]
:aliases
{:test
{:extra-paths ["test"]
:extra-deps {org.clojure/test.check {:mvn/version "1.1.1"}
io.github.cognitect-labs/test-runner
{:git/tag "v0.5.1" :git/sha "dfb30dd"}}}
:build {:deps {io.github.clojure/tools.build
{:git/tag "v0.9.2" :git/sha "fe6b140"}
slipset/deps-deploy {:mvn/version "0.2.0"}}
:ns-default build}}}
signcryption-demo on main [!] via :coffee: v19.0.2
❯ clj
Clojure 1.11.1
user=> (System/getProperty "java.library.path")
"/Users/lvh/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:."
user=>#2023-03-1415:31lvhthe same string works via JAVA_OPTS#2023-03-1415:32borkdude:jvm-opts needs to go in an alias#2023-03-1415:33borkdudeit's about time... https://github.com/clj-kondo/clj-kondo/issues/1269#2023-03-1418:27practicalli-johnExamples of using JVM options as aliases (if needed)
https://practical.li/clojure/reference/clojure-cli/jvm-options/#2023-03-1610:50dvingoCan this be marked as a bug/feature request in tools.deps ? Is there a reason it's not allowed? Pretty terrible user experience, especially since it's a silent failure.#2023-03-1610:51borkdudeI think it was intentionally not supported and I'll add a check to clj-kondo for the next release, but always good to double-check with @U064X3EF3#2023-03-1610:59dvingoWould love to hear the why - this pretty clearly breaks the principle of least astonishment#2023-03-1611:02borkdudeMy guess would be that it's similar to why :main-opts belong to an alias: you don't always want the same set of main-opts for every invocation.#2023-03-1611:02borkdudebut I can easily see why global jvm opts are desirable#2023-03-1611:05dvingoWhether or not it should be allowed at the top level, I suggest some error be thrown by tools.deps. This is one of those things that's small effort for the tool but provides lots of benefits for users. Especially for new users, the fact that there's not even a warning is really frustrating. Compared to user-facing tools in other programming languages it's pretty embarrassing.#2023-03-1611:07borkdudewell, there's also the docs ;)#2023-03-1611:18dvingoIf I'm not being clear, this about user experience. I think adding a sanity check to deps.edn performed by the tool itself would be a great way to improve the experience of using clojure. The way that a tool interacts with its users says something about tool itself. If it's not clear I'm saying that the behavior of tools.deps makes me think the tool is hostile to users, not welcoming or friendly. I don't think this is intended by the creators of the tool. As someone who has experienced non-clojure developers attempt to use clojure and fail, this stuff matters in changing how clojure is perceived to people who are not steeped in it.#2023-03-1611:19borkdudetop level jvm-opts has come up before and there's a clj-kondo issue for this reason too, so you're not wrong#2023-03-1611:23dvingoThis is low-hanging fruit, there is a great library waiting to be used just for this purpose
https://github.com/bhauman/spell-spec#2023-03-1612:12Alex Miller (Clojure team)There is a ticket for this, but the reason we don’t have global jvm-opts is because you use the CLI to launch all kinds of tools as well as your application, and it seems hard for me to come up with a jvm opt that you would want applied in all those situations#2023-03-1612:14Alex Miller (Clojure team)What I have been considering is whether there should be a well known alias that applies for the “project classpath” context, or whether there is something even more generally useful that could serve this purpose (and others)#2023-03-1612:15borkdudeisn't that already what the top level is for?#2023-03-1612:16borkdudemaybe :replace-jvm-opts could also be an option in aliases to reset the top level jvm opts#2023-03-1612:17Alex Miller (Clojure team)The top level stuff is also used for -X calls#2023-03-1612:17borkdude-X is made in the context of your project classpath, or am I missing something?#2023-03-1612:17Alex Miller (Clojure team)Yes#2023-03-1612:18Alex Miller (Clojure team)I’m not sure if you mean that as a pro or con :) #2023-03-1612:19borkdudeWhat I mean is, when doing -X calls, why would you not want to have the top level jvm opts (if they were supported)#2023-03-1612:19Alex Miller (Clojure team)Not sure, maybe you would#2023-03-1612:21Alex Miller (Clojure team)I also do have an issue to do more validation of deps.edn against the specs, I just need to update some of those before I enable it#2023-03-1613:15dvingoI brought this up two years ago..
https://clojurians.slack.com/archives/C6QH853H8/p1639018565012400
I don't really know what to say. It's obviously an issue other people keep running into.#2023-03-1613:19borkdude@U051V5LLP I think it's good that you expressed your concern and there are issues in both tools.deps and clj-kondo about it. I'll try to get it into the new clj-kondo release, but I'm sure you are aware that each project has a backlog and it can sometimes take more than two years for an issue to be resolved because there are even more important issues to work on.#2023-03-1613:20lvhI think the reason this one may be “special” is that Java library path is more like a class path than any other opt; in this case I definitely want it on by default for this project #2023-03-1613:21borkdudeWasn't there also some global environment variable for this btw? CLJ_JVM_OPTS#2023-03-1613:22lvhMaybe; I’ve been using JVM_OPTS directly since that also does what I want :D#2023-03-1613:24borkdudeCLJ_JVM_OPTS is only for classpath construction, not for the "app" itself#2023-03-1613:30dvingoIt's pretty obvious there are structural issues with the maintenance of these core projects. There's a long chain of great devs who have attempted to contribute to clojure and related core libs are now permanently absent from the community... It may seem like a digression, but the point is that if only one or two people can (practically, not technically) contribute to these core projects, they're just not going to get the attention they require for being core pieces of the clojure workflow. And the language and its community are suffering because of it. I think it's pretty cliché in the clojure community at this point, but that doesn't make it less frustrating. Again the point is that what a tool does says things about the people and community behind it regardless of the explicit words communicated. I know it's pointless to write these words as many before me have done the same to no avail, but, well, I'm frustrated with the clojure community and I think it can be useful for other people to hear that in the hope that it influences some positive changes.#2023-03-1613:31borkdudethere are pros and cons to how projects are managed#2023-03-1713:13borkdudeComing in the release later today#2023-03-1721:06borkdudereleased#2023-03-2311:35dvingoI just wanted to say that the comment I left the other day was unproductive and wanted to apologize for it.
The core desire is to see clojure grow, and I was frustrated from personal experiences of using clojure on projects with other developers (both new to clojure and experienced ones used to leiningen) who had a hard time dealing with the tooling around the language and I thought that if the tooling was made more user-friendly it could help grow the clojure community by making a positive first experience and nth experience for developers instead of one of frustration.
The lack of basic error/syntax checking and reporting to users in tools.deps was one obvious way (to me) to do that,
and the fact that it is unaddressed years later messages to me that this is not important as compared to other things.
I know there is only so much time to get things done so the thought was that if the core were more welcoming to contributors
then these sort of quality of life issues could be addressed more quickly, thus improving the experience of using clojure for everyone.
However, like borkdude said, there are trade-offs in all things and I know this is the way clojure is managed and should not expect differently, but it is still frustrating when you see things that could be made better but are not, even if you understand why they are not.
The desire is to help out but I know from the high profile exits of attempted clojure contributors that path is fraught.
Hence the frustrated comment. Anyway, thanks for the continued work on the tooling, I'm sure it will improve over time
and I know the best/only way out is through.#2023-03-2313:47Alex Miller (Clojure team)Providing better feedback is something we care about and looking back at the prior thread, I did do work at the time to update the specs but never got to the point of enabling it for validation. Because it was never logged it eventually fell off the plate in the usual deluge. Enhancement requests can be logged at https://ask.clojure.org and I sweep through those to make tickets at least once per week. We prioritize (partially) based on votes there. I do have a very recent ticket (https://clojure.atlassian.net/browse/TDEPS-238) that raised this again and when I next have time to work through tdeps issues it’s high on my list. I don’t think we have a good ask Clojure question related to this, so would be happy for you to write one so it can be voted on. I am happy to look at patches for stuff like this and anything else of course - we apply patches from contributors literally all the time all over the contrib projects. Re the so-called “high profile exits”, let’s just say that that summary is reductive and there’s a lot more nuance in what actually happened.#2023-03-2320:31dvingoThanks for taking the time to reply. Just posted a Q here https://ask.clojure.org/index.php/12803/tools-deps-perform-sanity-checking-on-deps-edn-file
Good to know about contributions. I guess this is a grey area in community development about dealing with specific individual incidents.
I specifically have in mind Tim, Zach, and Chas. All people who for many years gave a lot to the community and whose code and protocols we are still using daily. All of them had some sort of falling out that was so bad that they didn't just cut back contributing, but exited the community completely. As an outside observer it is not clear what happened and even after searching the internet for some clues it is still not clear, so all I can do is evaluate the facts that three high profile contributors all left the community after some unobserved interactions with the core team.
From this vantage point in time it appears that these departures were brushed aside and largely ignored by the core team. (I don't see any public communication about what happened).
As someone who wants clojure to succeed and tries to promote it to other developers it is very awkward when newcomers to the language ask what happened when they come across some old blog post or tweet about these events. I don't have an answer for them, and that led to my thoughts about contributions not being welcome.
I know it can be awkward to talk about individual cases and circumstances but the lack of public communication about what happened is I think even more awkward and leaves a bad spot on clojure's history the way that unacknowledged scandals tend to in any domain. I see indirect mention to these departures in 5.3 of A History of Clojure, which from my reading, really only suggests further that you shouldn't waste your time trying to contribute to the core.
I think those dramatic events of the past have helped to (painfully) align expectations about what clojure is and who it is for, but the indirect response to what happened is quite odd and I think makes the idea of wanting to contribute off-putting, but it seems like that might be by design.
To be concrete, something along the lines of "some prominent contributors in the past had expectations which led them to have incorrect beliefs about the clojure contribution process, so to prevent that from happening again here are the changes that were made: xyz" I think would help provide some closure for remaining members of the community and prevent future reductive/inaccurate summaries of what happened.#2023-03-2320:44borkdudeI don't think everything has to do with trying to contribute to core and not getting patches accepted or so.
Chas wanted to have static types and is now using OCaml. The same happened with Antonio Monteiro, heavy contributor to CLJS and author of lumo.
Zach got into an interesting job at Microsoft (I believe) where he doesn't use any Clojure.
Sometimes people just move on to the next thing. 🤷#2023-03-2320:45borkdudeI'm not saying that frustration with the contribution process doesn't have anything to do with it, but also it's probably not the whole story. But I also am an outsider to these things and don't know any details. FWIW#2023-03-2320:47borkdudeI've also seen people go away for 5 years and then take a peek back inside. The Onyx guy (Drogalis) went off to Confluence and kind of disappeared out of the clojure community for a while. Lately I spoke with him here again. These things happen.#2023-03-2320:49borkdudeSorry, that was Confluent (Kafka)#2023-03-2416:17dvingoThanks for the reply and context borkdude. Having a lot of missing details makes it hard to discern what actually happened, but yea, that makes sense that the only constant is change and sometimes people move on.#2023-03-1613:59practicalli-johnI'd like to thank @alexmiller and other core maintainers who have created a very flexible set of tools (Clojure CLI, tools.deps, tools.build, etc)
I appreciate the methodical approach to creating tools and libraries which parallels the approach taken by all Clojure core maintainers and the development of the Clojure language itself. This has greatly helped with the adoption of Clojure and retention of developers and projects using Clojure
The communication about up coming features has been very useful in understanding when to adopt certain features and setting expectations.
There is a small learning curve for these tools and I really appreciate the support provided on this and many other slack channels by @alexmiller which has been highly valuable to me and hopefully the rest of the Clojure community.
Thank you Alex.#2023-03-1616:01orestisUberjar is a jar that contains everything (dependencies and code). Is there a term for two jars where one contains all the dependencies, and another that contains all the code? Think slow moving vs fast moving layers in Docker.#2023-03-1616:15favilaI’m not sure there’s a term of art for that except for (javaee) container-specific deploys (where the container server provides the deps, and WAR files are the application). I think typically when you start separating, you just make a lib directory of dependency jars instead of dependency + application uberjars, because it’s much simpler and safer.#2023-03-1616:15favilathere’s also jar-of-jars, but that’s just another uberjar-like technique to get one self-contained jar#2023-03-1616:17orestisWhat I'm after is essentially creating a docker layer with all my (compiled) dependencies, that should, in principle, not changing very often, and another docker layer with all my (compiled) source. Aim is to deploy to production and get fast startup time (hence the compiled requirement).#2023-03-1616:19favilaif these dependencies you want to be slower-moving are clojure-aot compiled, I think there may be some trickiness there#2023-03-1616:20favilaclojure compilation is transitive to all required code, there is no stable clojure ABI, and I’ve heard can be hard to predict what the output class filenames can be#2023-03-1616:21favilaIOW if this is clojure code, I don’t think you can easily and safely separate the “my aot-ed code” from the “this lib’s aot-ed” code the same way you can with java lib class files#2023-03-1616:23favilabut you could separate by clojure source or java classfile depencency jars vs application clojure source jars#2023-03-1616:24favilabut if you clojure-aot, you probably shouldn’t split up the result of the compile into multiple jars or allow the same clojure source to have aot code on the classpath.#2023-03-1616:43orestisI thought clojure AOT means just generating a .class file per .clj file?#2023-03-1616:43favilathere’s not a straight 1-1 correspondence between class files and clj files#2023-03-1616:44practicalli-johnI would push all the built dependencies to a repository, as part of the CI workflow of each project.
Then in the project that uses the dependencies the docker image can include those dependencies. The cache would be used unless a newer version of that dependency is pushed to the repository
Something like Artefactory or AWS artefact repository#2023-03-1616:44orestishm, 1-1 between class file and namespace?#2023-03-1616:44favilano#2023-03-1616:44favilatry it 🙂#2023-03-1616:45favilathis is why clojure libs don’t distribute jars with class files#2023-03-1616:45favilahttps://clojure.org/reference/compilation#2023-03-1616:48orestisAh, I was thrown by the usual cannot find foo__init.class for the foo namespace.#2023-03-1616:48orestisI still think that AOT compiling Clojure dependencies today and using them with AOT compiled Clojure source tomorrow should work, as long as we're talking about the same Clojure/JVM version.#2023-03-1616:49favilaIt may work most of the time, but really you’re not meant to combine compilation output from different processes running different classpaths#2023-03-1616:50favilaor, you’re on your own, you may encounter subtle problems#2023-03-1616:50favilait’s not impossible, but I wouldn’t trust myself to do it right#2023-03-1616:55favilaHere’s a discussion where Colin Fleming (who made Cursive) asks about using bits of aot-compiled stuff together (for incremental compilation in his case.) The thread touches on some of the challenges in more detail. https://clojurians.slack.com/archives/C06E3HYPR/p1651097379296839#2023-03-1616:55favilaThis is the key:#2023-03-1616:56favila> clojure basically behaves like some kind of whole program optimizing compiler, despite not doing any of things those do
> [5:19 PM] so like taking apart the output of an optimizing compiler and linking it with something compiled separately is tricky, so is doing that with aot compiled clojure#2023-03-1617:28orestisWell I can certainly compile everything together and split the output files. If Docker hashes them and the output is the same, problem solved #2023-03-1617:28favilacompilation output is not deterministic#2023-03-1617:28favilaprobably never going to hash the same#2023-03-1617:32favilaYou can put the contents of .m2 after a clojure -P into your slower-moving layer and probably get most or all of the savings from not having to download dependencies separately, but that doesn’t include the trickier AOT part#2023-03-1617:33favilaor slightly fancier, a tools-deps program that copies all basis jars into a lib directory#2023-03-1617:34favilathen put the lib in a layer#2023-03-1617:42orestisI’m already doing the Clojure -P and Docker is smart to cache that automatically. Definitely helps to not download the universe. My quest for smaller layers is insignificant compared to that. #2023-03-1716:38dominicm@U7PBP4UVA have you looked at pack.alpha?#2023-03-1716:39dominicm> What I’m after is essentially creating a docker layer with all my (compiled) dependencies, that should, in principle, not changing very often, and another docker layer with all my (compiled) source. Aim is to deploy to production and get fast startup time (hence the compiled requirement).
The split of dependencies & application source is exactly what pack takes care of. It doesn’t get involved with compilation though. Splitting the AOT compiled stuff is really difficult as discussed above.#2023-03-1811:28orestisAOT compilation is a basic requirement for us, for faster startup times, so if we can't split AOT-compiled classes that's a no go.#2023-03-1814:49cch1I’ve recently started consuming an old XML lib that ceased development on a SNAPSHOT version ten years ago. I also have a :mvn/repos entry for datomic-cloud.
Now when Clojure CLI processes my deps.edn the delay and noise from checking for updated SNAPSHOTs from datomic-cloud is substantial (interestingly, I don’t see that noise from another custom repo I have configured). Is there any way in deps.edn to reduce the frequency of checking for SNAPSHOT updates? They’re not checked every time clojure runs, but statistically much too often in my case.
If it’s not possible in deps.edn, can maven be configured to adjust the frequency? I’ve tried hacking around in ~/.m2/settings.xml using https://maven.apache.org/ref/3.9.1/maven-settings/settings.html#settings as a guide, but it seems to be ignoring all my attempts.#2023-03-1815:24Alex Miller (Clojure team)with recent releases you can set the update frequency in the :mvn/repos key#2023-03-1815:26Alex Miller (Clojure team){:mvn/repos
{"datomic-cloud" {:url "whatever"
:snapshots {:enabled false}
:releases {:enabled true
:update :daily}}}}
^^ really just disabling snapshots should help the most#2023-03-1815:26Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_maven for more docs#2023-03-1818:27cch1Awesome. Thanks, @U064X3EF3.#2023-03-1819:37Alex Miller (Clojure team)Not many people have used this yet, so interested in feedback#2023-03-2415:55cch1@U064X3EF3 Unfortunately I have many transitive SNAPSHOT dependencies (due to the otherwise excellent io.xapix/paos lib). I essentially set the value of my :mvn/repos to:
{"datomic-cloud" {:url ""
:snapshots {:enabled false}}}
and the impact to the noise and delay on executing clojure and clj was dramatic. Feedback = ❤️ .#2023-03-2409:57mkvlrgiven :main-opts in my :test alias and none in my :dev alias, is there a way for me to start with clojure.main so I can do clj -M:dev:test -e ,,,? clj -M:test:dev will run :main-opts from :test which I don’t want.#2023-03-2411:01borkdudeThe hack I use for this is:
clj -Scp $(clj -Spath -A:dev:test) -M -e ...
#2023-03-2411:02borkdudeOptimally you should be able to:
clj -A:dev:test -M -e
but currently -A: still runs :main-opts which I hope will be removed sooner than later#2023-03-2412:36seancorfieldUse -r to start a REPL: clj -A:dev:test -r#2023-03-2412:39borkdude@U04V70XH6 -A:dev:test will still kick off the :main-opts of the :test alias, no?#2023-03-2412:39borkdudeand shouldn't this be -M -r?#2023-03-2413:02seancorfieldHmm, I shouldn't offer advice at 6 am without coffee. I use -r on some projects but I think maybe I have an empty :main-opts in an alias and use that to override previous :main-opts#2023-03-2413:03seancorfieldAh, you can hack around it by doing clj -X:dev:test -M -r#2023-03-2413:04seancorfieldI will now either go back to bed for a few hours or make a ☕#2023-03-2413:26borkdudeah nice, the -X is good :)#2023-03-2413:27borkdudethanks for the hack!#2023-03-2812:23borkdudeUnfortunately the -X hack didn't work, I'm getting:
Unqualified function can't be resolved: -M
#2023-03-2812:24borkdude@U064X3EF3 When can we get -A with ignorance of :main-opts? 🙏#2023-03-2812:45Alex Miller (Clojure team)After add-libs :) #2023-03-2812:46Alex Miller (Clojure team)Can’t you just split into two aliases as a workaround? Seems like you’re making this so hard#2023-03-2812:47borkdudeyes, that has been the workaround, but it's a workaround that keeps coming up ;)#2023-03-2812:47borkdudebut good to hear it's coming#2023-07-1210:46borkdudeStill hoping this will happen soon.
See discussion in https://app.slack.com/client/T03RZGPFR/C02FBBU61A9/thread/C02FBBU61A9-1689157272.648009
cc @U5H74UNSF @U3X7174KS#2023-03-2417:13AlejandroWhy are the resources and target dirs included in :paths ["src" "resources" "target"] in the documentation of figwheel? Isn't this option for listing source code directories?
https://figwheel.org/docs/create_a_build.html#2023-03-2417:48seancorfieldDid you mean to ask that in the #CALJ3BFLP channel?#2023-03-2417:52AlejandroOh, yeah. But anyway, it's deps.edn.#2023-03-2417:58seancorfieldSure, but only Figwheel users (or possibly only the Figwheel maintainers) will know the answer to this.#2023-03-2417:59seancorfield:paths just puts folders on the classpath so they're available when running code.#2023-03-2417:59seancorfield(so, no, it's not just source code directories -- it's anything you want on the classpath)#2023-03-2418:00seancorfieldHaving target on the classpath sounds a bit odd to me but I guess it depends what Figwheel puts into that folder.#2023-03-2418:01seancorfieldIf you've been using Leiningen (or Boot) previously, maybe this old blog post of mine will shed some light? https://corfield.org/blog/2018/04/18/all-the-paths/#2023-03-2418:18AlejandroYeah, this explains a lot, thanks.#2023-03-2419:42borkdudeI don't think this is the reason figwheel chose this, but having the target/classes directory in paths is somewhat necessary for libraries that support deps/prep-lib#2023-03-2419:42borkdudeexample: https://github.com/http-kit/http-kit/blob/master/deps.edn#2023-03-2709:17gregI experience error saying:
Error building classpath. Unable to compare versions for com.fulcrologic/fulcro-i18n: {:mvn/version "1.0.0", :deps/manifest :mvn} and {:git/url "", :git/sha "72f8cbba597de574480ae21322180f0590213e06", :deps/manifest :deps, :deps/root "/home/user/.gitlibs/libs/com.fulcrologic/fulcro-i18n/72f8cbba597de574480ae21322180f0590213e06"}
Here is a minimal repo reproducing the problem:
https://github.com/rynkowsg/bb-deps-versions-error
Is there something I could do about it other then sprinking some :exclusions within the imported repo or overriding these deps in top repo?#2023-03-2712:53Alex Miller (Clojure team)Maven and git deps can't be compared as versions to determine the newer one to select, so you either need to add exclusions so it's unambiguous, or specify an explicit version as a top dep to avoid the selection#2023-03-2804:55Drew VerleeWhat is the difference between including a Clojure namespaces by adding them through deps.edn :deps as a :local/root vs adding them as a deps.edn :paths?
I have some logic that's not reloaded as expected, and I'm guessing it's because the :local/root logic isn't part of the Clojure tools namespace and component reload flow.#2023-03-2805:05Alex Miller (Clojure team)Depends exactly what you mean, but either way they end up on your classpath, so no effective difference from that perspective. tools.namespace defaults to the dirs in your classpath, which should pull in local dep dirs too, but maybe there's something filtering those out#2023-03-2816:44Drew Verlee@U064X3EF3 thanks alex.
do i need to tell clojure.tools.namespace.repl/set-refresh-dirs about the dirs that I'm pulling in through a local/root? Maybe i should revisit if i need to call the set-refresh-dirs fn at all.#2023-03-2816:48Alex Miller (Clojure team)I think you do need to tell it if you want it to refresh those :)#2023-03-2816:48Alex Miller (Clojure team)but maybe not calling it at all would be the same effect. I don't work in this mode so I'm not an expert in this#2023-03-2816:57Drew Verlee@U064X3EF3
What mode do you mean? The one where i set which dirs to refresh? (as opposed to just all of them?)
I'm using clojure tools namespace along side Component to overcome the "reloading woes" outlined https://lambdaisland.com/blog/2018-02-09-reloading-woes as well as those particles outlined by stuart sierra in his https://www.cognitect.com/blog/2013/06/04/clojure-workflow-reloaded" post.#2023-03-2817:05Alex Miller (Clojure team)I mean all of that :)#2023-04-0409:56pithylessWhat happens if a maven dependency adds <type>pom</type> ?
❯ clj -Srepro -Sdeps '{:deps {com.kohlschutter.junixsocket/junixsocket-core {:mvn/version "2.6.2"}}}'
Error building classpath. Could not find artifact com.kohlschutter.junixsocket:junixsocket-core:jar:2.6.2 in central ()
#2023-04-0409:56pithylessPrevious version that still works:
❯ clj -Srepro -Sdeps '{:deps {com.kohlschutter.junixsocket/junixsocket-core {:mvn/version "2.3.2"}}}'
Clojure 1.11.1
user=>#2023-04-0409:57pithylessOnly difference I can easily spot is the difference in type between https://mvnrepository.com/artifact/com.kohlschutter.junixsocket/junixsocket-core/2.3.2 and https://mvnrepository.com/artifact/com.kohlschutter.junixsocket/junixsocket-core/2.6.2#2023-04-0410:29delaguardoI believe this is a known issue - https://clojure.atlassian.net/browse/TDEPS-202
for now the workaround is to look at pom.xml for junixsocket-core lib and get the same dependencies as inside of pom.xml into your deps.edn#2023-04-0410:34pithylessah ok, I see. Thanks for the clarification!#2023-04-0410:52delaguardoUPD: Please don't use this workaround, this is not supported
btw
I just tried this
~`clj -Srepro -Sdeps '{:deps {com.kohlschutter.junixsocket/junixsocket-core {:mvn/version "2.6.2" :extension "pom"}}}'`~
and it works.#2023-04-0412:05Alex Miller (Clojure team)I would not rely on that continuing to work reliably if it happens to now, this is currently not supported#2023-04-0412:06pithylessOK, thanks for the heads up!#2023-04-0809:34janezjI want to bring an old java project into clojure. The project depends on a lot of jars in ./lib folder.
So I created script to write all :local:root entries, because is close to impossible to convert local deps to maven central deps. But I suspect that even for :local/root transitive dependencies are processed. Is there a way to ignore pom.xml in local jar?
For example
axiom-impl/axiom-impl {:local/root "./lib/axiom-impl-1.2.10.jar"}
clojure -Stree
axiom-impl/axiom-impl /home/jj/workspace/rpt/lib/axiom-impl-1.2.10.jar
. http://org.apache.ws.commons.axiom/axiom-api 1.2.10
. org.apache.geronimo.specs/geronimo-activation_1.1_spec 1.0.2
. org.apache.geronimo.specs/geronimo-javamail_1.4_spec 1.6
. commons-logging/commons-logging 1.1.1
. jaxen/jaxen 1.1.1
. org.apache.geronimo.specs/geronimo-stax-api_1.0_spec 1.0.1
. org.apache.geronimo.specs/geronimo-activation_1.1_spec 1.0.2
. org.apache.geronimo.specs/geronimo-javamail_1.4_spec 1.6
. org.codehaus.woodstox/wstx-asl 3.2.9
. commons-logging/commons-logging 1.1.1
. org.apache.geronimo.specs/geronimo-stax-api_1.0_spec 1.0.1
axis-adb/axis-adb {:local/root "./lib/axis2-adb-1.5.4.jar"}
clojure -Stree
Error building classpath. Could not find artifact wsdl4j:wsdl4j:jar:1.6.2 in central (https://mirror.example.com/nexus/content/repositories/public)
there is nothing interesing in above jar in MANFEST.MF but there is META-INF/maven/org.apache.axis2/axis2-adb/pom.xml
with following deps
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>axis2-kernel</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.geronimo.specs</groupId>
<artifactId>geronimo-activation_1.1_spec</artifactId>
</dependency>
<dependency>
<groupId>xmlunit</groupId>
<artifactId>xmlunit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
#2023-04-0816:39Alex Miller (Clojure team)why do you want to ignore the transitive deps?#2023-04-0816:40Alex Miller (Clojure team)and yes, it is using the pom.xml in the to find those transitive deps#2023-04-0817:15janezjIt is an old project, and part of it was delivered years ago as a folder of jars, and an example how to send some reports to a government agency. All required jars are in the folder. But some poms are for example pointing to sourceforge repos too. I don't know how to setup. And will probably create jars hell because of the duplicates. One way it would be to pack the old junk into docker, jetty and servlet webservice, and second app will be in clojure using this webservices. But I would like to do all the stuff in a single app.#2023-04-0818:04janezjCurrently it is looking for Non-resolvable parent POM for com.sun.codemodel:codemodel:2.2-SNAPSHOT#2023-04-0818:08janezjCurrent build.gradle just collects jars, without transitive resolving and it works
dependencies {
compile fileTree(dir: 'lib', include: ['**/*.jar'])
testCompile 'junit:junit:4.+'
runtime 'com.ibm.db2.jcc:runtime_v1:11.1'
compile group: 'org.eclipse.jetty', name: 'jetty-server', version: '9.4.8.v20171121'
compile group: 'org.eclipse.jetty', name: 'jetty-servlet', version: '9.4.8.v20171121'
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
}#2023-04-0911:48janezjThe best solution would be to use :local/dir. Many Clojurists might not be aware that :local/root is not offline and attempts to load transitive dependencies. This behavior can come as a surprise.#2023-04-1013:29Alex Miller (Clojure team)if you use the proper groupId/artifactId, there will be no jar hell - the best version will be selected. since presumably all of the transitive deps are in your set of local root jars, really only they should get used, but you might see metadata resolution happen#2023-04-1013:29Alex Miller (Clojure team)you can add additional maven repos with :mvn/repos {"repoId" {:url "...repo-url..."}} in the top of the deps.edn#2023-04-1013:30Alex Miller (Clojure team)if you could make a question at https://ask.clojure.org, that's a good way to register a request for a new feature on this#2023-04-1209:19janezjWith a little bit of luck and several hours I found that only one local jar is required: wsdl4j/wsdl4j.
All the others are available from http://dist.wso2.org/maven2 I admit that I am not super happy, because I don't know the ws02. Thanks Alex.#2023-04-1220:27oliverIs there a deps tool similar to https://github.com/weavejester/lein-auto? I'm considering adapting it, but maybe someone has done so already. Also, I was wondering if there is a list of available deps tools somewhere… no luck googling so far.#2023-04-1221:34Alex Miller (Clojure team)https://github.com/clojure/tools.deps.alpha/wiki/Tools is an older list#2023-04-1221:35Alex Miller (Clojure team)I had to lock it down because spam :(#2023-04-1221:36Alex Miller (Clojure team)https://www.clojure-toolbox.com/ has some things under Filesystem Watchers, not sure if any of them suits your needs#2023-04-1308:07oliverThanks a lot… only read this now due to time shift. I will definitely check these out!#2023-04-1421:07seancorfieldFor anyone using my dot-clojure deps.edn or :dev/repl code in some form, I have updated that repo (to :git/tag "v1.0.1" :git/sha "e932f96"):
• the :1.12 alias now references Clojure 1.12.0 Alpha 2
• the :add-libs alias has been removed since you can (should!) use the new clojure.repl.deps stuff in 1.12 Alpha 2 if you really want to dynamically load deps into your running REPL!
(it also no longer forces a DynamicClassLoader either -- I think that was only required for Socket REPLs but I've switched away from those and the previous v1.0.0 version of dot-clojure no longer started one by default)#2023-04-1421:19Alex Miller (Clojure team)I don't think you need it for what people usually do with socket repls either tbh (but it is needed for prepls, which is something that we may address still)#2023-04-1421:42seancorfieldSorry, I should have been clearer that it was only there to support the old add-libs t.d.a branch workflow and isn't relevant with the 1.12 Alpha 2 REPL workflow.#2023-04-1623:24seancorfieldNot sure where best to post this but it's tools.deps-adjacent...
In Clojure 1.12.0 Alpha 2, https://clojure.github.io/clojure/branch-master/clojure.tools.deps.interop-api.html#clojure.tools.deps.interop/invoke-tool provides a way to invoke tools in a subprocess. It lets you specify either a tool name (as a string "mytool", for -Tmytool) or as a single alias (as a keyword :mytoolalias, for -T:mytoolalias) -- but I have a situation where I have a tool that is essentially spread across two aliases: it has a "swappable" portion that needs to be provided by one of two other aliases. Specifically, it's a tool that expects an HTTP client to be provided in order to run, and using the CLI, you can do that as follows:
> clojure -T:the-tool:http-client-a do-some/stuff
# or:
> clojure -T:the-tool:http-client-b do-some/stuff
It would be really nice to be able to do this programmatically via the new invoke-tool function (in build.clj in my case).
Now, invoke-tool doesn't currently check the type of either the :tool-alias or the :tool-name parameter, nor does it currently do anything different based on which one is supplied (except that :tool-alias is preferred: (str "-T" (or tool-alias tool-name))), so I could pass :tool-alias ":dev:+default" in this case and it would "just work" but that seems like a hack...
...would a :tool-aliases option be a useful addition here (a vector of keywords that was str/join'd to make the string passed to -T)?#2023-04-1623:32Hugh PowellHi folks, I've been looking at using the http://javamoney.github.io/ APIs. All of the tutorials/examples that I can find (e.g. https://www.baeldung.com/java-money-and-currency) require the use of the org.javamoney/moneta dependency. As far as I can tell this is just a reference to a pom.xml file, rather than a jar. When I add org.javamoney/moneta {:mvn/version "1.4.2"} to deps.edn and run clojure -Sforce I get the following error
> Error building classpath. Could not find artifact org.javamoney:moneta:jar:1.4.2 in central (https://repo1.maven.org/maven2/)
Is there a way to use this dependency in deps.edn?#2023-04-1623:40seancorfieldYou will need to specify the component subprojects from that pom.xml explicitly for deps.edn#2023-04-1623:43Hugh PowellAha, cool. Thanks 🙇 . So there's no way to reference those non-jar dependencies in deps.edn?#2023-04-1623:46seancorfieldNot currently, no. You'll need the following, explcitly:
org.javamoney.moneta/moneta-core {:mvn/version "1.4.2"}
org.javamoney.moneta/moneta-convert {:mvn/version "1.4.2"}
org.javamoney.moneta/moneta-convert-imf {:mvn/version "1.4.2"}
org.javamoney.moneta/moneta-convert-ecb {:mvn/version "1.4.2"}#2023-04-1623:47Hugh PowellAwesome, thanks 👍#2023-04-1623:50seancorfieldGo vote for https://ask.clojure.org/index.php/10892/add-support-for-bom-bill-of-materials-dependencies#2023-04-1623:59Hugh PowellDone 🙂#2023-04-2616:05Alex WhittHi, did clojure.tools.deps.alpha.libmap get purged from your git history? I was using it for a long time, but I can no longer find it anywhere. We found what looks like the same namespace here:
https://github.com/just-sultanov/clj-deps#2023-04-2616:19hiredmanthat was part of the add-lib branch (not master) of tools.deps.alpha, since then the add-lib branch was superseded by add-lib2 and then add-lib3 branches (add-lib3 is the only one of those that is still around, and it doesn't have libmap) and tools.deps.alpha came out of alpha and goes by tools.deps now#2023-04-2616:20hiredmanand recent alpha's of clojure 1.12 now have add-lib, so the add-lib3 branch is likely not long for this world#2023-04-2616:35Alex WhittThank you. Where is the right place to get the lib-map namespace nowadays?#2023-04-2616:56pavlosmelissinosThat particular namespace doesn't seem to exist anymore.
Which functionality do you need? Perhaps you can still find it under a different name or slightly different API.#2023-04-2617:10Alex Miller (Clojure team)The lib map files have not been created or injected for a while, but the effective replacement for this is new clojure.java.basis api that was added in 1.12.0-alpha2 - the lib map is the :libs key of the basis#2023-04-2617:31Alex WhittI see! Thank you.#2023-04-2820:12imreShould the following work?
{:deps {my.monorepo/foo {:local/root "../monorepo" :deps/root "subprojects/foo"}}}
It doesn't look like it does but reading https://clojure.org/reference/deps_and_cli I'm not sure whether it should.#2023-04-2821:16imreAnd by work, I mean the deps.edn that my.monorepo/foo is going to point to will be at "../monorepo/subprojects/foo/deps.edn"#2023-04-2905:06seancorfieldDoes it work?
What in the documentation makes you think it shouldn't work?#2023-04-2905:08seancorfieldI've never used :deps/root with :local/root, only with git deps so those are honest questions.#2023-04-2914:13imreIt doesn't look like it does.
I'm mucking about with a project referencing libs from a monorepo and temporarily changed the references in the project to a local copy of the monorepo and hoped it would work.#2023-04-2914:14imreFrom the doc:
Common coordinate attributes (all optional):
...
:deps/root - relative directory path within a file based dep to search for the manifest file#2023-04-2914:15imreAnd:
Local project attributes:
:local/root (required) - directory path (will auto detect whether a deps or pom project, also see :deps/manifest)#2023-04-2914:16imreReading the above, local/root can qualify as a file based dep, which to me implies that deps/root should work as it isn't stated otherwise#2023-04-2914:17imreBut it's quite possible that I'm missing something#2023-04-2914:18imreThanks for the Qs, Sean, I was watching the conj as I wrote this yesterday and couldn't fully concentrate 😅#2023-04-2914:59seancorfieldAh, yes... now I think about this, I've run into the same thing when I've been working with Polylith locally, and my test runner. :local/root is the full path to the folder containing deps.edn and tools.deps ignores :deps/root in that case.#2023-04-2915:00seancorfieldMight be worth posting something on Ask about it -- it would make life easier, switching between local and git deps when developing a library or tool.#2023-04-2915:00imreYeah that's what I was thinking#2023-04-2915:00imreBut wanted to check here first#2023-04-2915:00imrePerhaps it's intended#2023-04-2915:01imreBut I'll open an ask later#2023-05-0412:46imreOpened an ask at https://ask.clojure.org/index.php/12913/using-deps-root-with-local-root in case people want to check it out.#2023-05-0412:46imreOpened an ask at https://ask.clojure.org/index.php/12913/using-deps-root-with-local-root in case people want to check it out.#2023-04-2905:28jumarI don't use deps.edn that much so I forgot if this is a thing: can you use branch name as a git dependency version specification,
as opposed to using commit sha?
Looking at the https://clojure.org/guides/deps_and_cli#_using_git_libraries and the https://clojure.org/reference/deps_and_cli#_git it doesn't seem possible.#2023-04-2905:40seancorfieldNo, you can use tag and short sha or just full sha. You can resolve a branch to get the last sha tho'...#2023-04-2905:41seancorfieldFor example:
;; (require '[clojure.tools.gitlibs :as gitlibs])
;; (defn load-master [lib]
;; (let [git (str "" lib ".git")]
;; (add-libs {lib {:git/url git :sha (gitlibs/resolve git "master")}})))#2023-04-2906:11jumarI see, thanks!#2023-05-0221:55tcarlsWhat's the equivalent to -Atarget when using clj -X:deps tree instead of clj -Stree?#2023-05-0221:58seancorfieldclojure -X:deps tree :aliases '[:target]'
#2023-05-0221:58tcarlsThank you#2023-05-0221:58seancorfieldSeveral of the tools.deps functions understand :aliases#2023-05-0221:59seancorfieldFWIW, this will give you help and docstrings for all those functions:
clojure -X:deps help/doc
#2023-05-0222:00tcarlsnod; I found the doc string, but wasn't understanding it correctly#2023-05-0222:02tcarls(...hm; Also have a place where clj -Stree -T:build was being used instead of -A:build)#2023-05-0300:13tcarlsends up using #2023-05-0304:03tengstrandI have a problem. I can AOT compile and build an uberjar when I only include org.clojure/tools.deps {:mvn/version "0.18.1335"} and I can do the same if I only include clojure2d/clojure2d {:mvn/version "1.4.5"}} . But it doesn't work if I include both:
Execution error (FileNotFoundException) at .FileInputStream/open0 (FileInputStream.java:-2).
/var/folders/_d/7pv3mjnn18z3rkh7z7680xj40000gn/T/uber7113627005145822534/LICENSE (Is a directory)
I created the tiny project and a dummy component in the polylith repo, so people can reproduce it (instructions in the https://github.com/polyfy/polylith/blob/25dc11c62df9b7e7faac4da23b0daadd6443a06c/components/dummy/deps.edn file).#2023-05-0304:18seancorfieldThis has come up in a lot of threads here. Unfortunately, libraries are not very consistent about LICENSE as a text file and LICENSE as a directory so they conflict. You need to exclude the LICENSE pattern when copying the files, as part of building the uberjar...#2023-05-0304:19seancorfieldSee for example:
• https://clojurians.slack.com/archives/C6QH853H8/p1667576670174719?thread_ts=1645186448.329279&cid=C6QH853H8
• https://clojurians.slack.com/archives/C6QH853H8/p1658360455757139?thread_ts=1658360352.288139&cid=C6QH853H8
• https://clojurians.slack.com/archives/C6QH853H8/p1645186978713559?thread_ts=1645186448.329279&cid=C6QH853H8#2023-05-0304:21tengstrandThanks!#2023-05-0413:46pauldI can't seem to use compile-clj with certain dependencies but the project can launch the clojure repl and use the dependency as normal. I wonder if it is something to do with the fact that I'm on windows and using the scoop clojure via bash.
clj -T:build jar
Error: Could not find or load main class Dumais\AppData\Local\Temp\compile-clj6495804517722369898\compile-clj;target.classes;src;resources;C:\Users\Paul
Caused by: java.lang.ClassNotFoundException: Dumais\AppData\Local\Temp\compile-clj6495804517722369898\compile-clj;target/classes;src;resources;C:\Users\Paul
Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:114).
Clojure compilation failed, working dir preserved: C:\Users\Paul Dumais\AppData\Local\Temp\compile-clj6495804517722369898
#2023-05-0413:47pauldNotice how it's seems to be looking for main class that is broken by the fact that my user name has a space in it.#2023-05-0413:48pauldThis only happens when I depend on certain java dependencies found in a maven repo.#2023-05-0415:40pauldIt turns out I don't need this anyway! I wanted to call Clojure code from java but I can just do this: https://en.wikibooks.org/wiki/Clojure_Programming/Tutorials_and_Tips#Invoking_Clojure_from_Java#2023-05-0417:28seancorfieldThat is horribly outdated. (most of that wiki is objectively terrible). See https://clojure.org/reference/java_interop#_calling_clojure_from_java for the up-to-date way to call Clojure from Java.#2023-05-0417:38seancorfield(updated that Wikibooks page to link to that and remove the old clojure.lang.RT stuff)#2023-05-0417:38seancorfield@U0FH1UL84 ☝️:skin-tone-2:#2023-05-0419:16pauldDoes this method require a compiled clojure class?#2023-05-0419:18pauldNevermind, I read the docs.#2023-05-0419:55pauldHmm, I can't get this var to bind. I have a simple stub example in my maven project.
IFn test = Clojure.var("clojure.core", "+");
Object testResult = test.invoke(1, 2);
The above works, but I can't figure out how to get this to work:
IFn foo = Clojure.var("com.company.test", "foo");
Object result = foo.invoke("hi", "there");
I have test.clj file on the classpath at com/company/test.clj with this:
(ns com.company.test)
(defn foo [a b]
(str a " " b))
#2023-05-0419:57seancorfieldAre you sure your classpath for running your Java code actually includes your code?#2023-05-0419:58seancorfield(and it would help if you provided details of exactly what "doesn't work" -- i.e., what errors you are getting on which lines of code)#2023-05-0420:03pauldjava.lang.IllegalStateException: Attempting to call unbound fn: #'com/company/teast/foo
at clojure.lang.Var$Unbound.throwArity(Var.java:45)
at clojure.lang.AFn.invoke(AFn.java:36)
at clojure.lang.Var.invoke(Var.java:388)
#2023-05-0420:03seancorfieldYou're not requiring your namespace.#2023-05-0420:04seancorfieldPer the docs I linked to:
> Functions in clojure.core are automatically loaded. Other namespaces can be loaded via require:
IFn require = Clojure.var("clojure.core", "require");
require.invoke(Clojure.read("clojure.set"));
That includes requiring your own code before calling it.#2023-05-0420:05pauldomg, reading skill are worse than I feared! Thanks. Trying out now.#2023-05-0420:16pauldThat worked! Thanks a bunch.#2023-05-0420:29seancorfieldWe had a bunch of legacy ColdFusion apps and this was how we were able to slowly rewrite them into Clojure from the bottom up 🙂#2023-05-0516:46pauldNow, I want to make an uberjar but because of the original bug I mentioned, I will have to try leiningen to make an uberjar or perhaps use maven to and embed clojure files as I already demonstrated as working. Or use a linux vm... probably several options.#2023-05-0516:48pauldmaybe there is a windows channel...#2023-05-0519:12seancorfield#CFN4QDHPS @U0FH1UL84#2023-05-0622:25Eduardo LopesWhat is the most similar command to lein deps? clj -X:deps prep?#2023-05-0623:00hiredmanDepends on what you think lein deps does#2023-05-0623:23Eduardo Lopes:thinking_face: I was thinking about downloading project dependencies #2023-05-0623:39hiredmanI think prep would be fine for that, but I think -P is explicitly for that purpose#2023-05-0623:41hiredmanA long time ago lein would copy the dependencies jars into your projects directory tree instead of just using them out of m2, which if I recall was what lein deps originally did#2023-05-0706:38practicalli-johnclojure -P is the command you are looking for. The -P execution flag can also be used with other aliases via one of the other execution flags, e.g clojure -P -M:test/libs:build/libs to download the main dependencies and those for testing and building a project, as common in a CI workflow
https://practical.li/clojure/clojure-cli/execution-options/#prepare-dependencies#2023-05-0718:32Eduardo LopesThat doc is really awesome, funny that I discovered clojure -P after running clj -h that is not my usual workflow, I prefer reading a doc. Thanks lot, this will be useful for my learning 🙂#2023-05-0816:50pauldI'm trying to get an uberjar via windows clojure. I am getting this issue:
Execution error (FileNotFoundException) at java.io.FileInputStream/open0 (FileInputStream.java:-2).
C:\Users\Paul Dumais\AppData\Local\Temp\uber10211030562887780850\LICENSE (Access is denied)
#2023-05-0817:02Alex Miller (Clojure team)when compile-clj fails, it should leave the temp directory there for debugging#2023-05-0817:02Alex Miller (Clojure team)does that file exist? if so, can you read it?#2023-05-0817:09pauldThat directory "uber..." does not exist.#2023-05-0817:09pauldoops one sec.#2023-05-0817:11pauldyeah that directory does not exist.#2023-05-0817:11pauldno directories starting with that name exist.#2023-05-0817:12pauldI switched to :use-cp-file :always#2023-05-0817:12pauldSorry 🌄 ":never"#2023-05-0817:13Alex Miller (Clojure team)oh sorry, that's the uber building dir, my previous comment was re compile-clj so is not relevant#2023-05-0817:14Alex Miller (Clojure team)you mentioned modifying the options of compile-clj, but you should be modifying the options of uber to affect behavior#2023-05-0817:15pauldahhh#2023-05-0817:21pauldok tried that with same result:
(defn uber [_]
(b/copy-dir {:src-dirs ["src"]
:target-dir class-dir})
(b/compile-clj {:basis basis
:use-cp-file :never
:src-dirs ["src"]
:class-dir class-dir})
(b/uber {:class-dir class-dir
:uber-file uber-file
:exclude [#"(?i)^META-INF/LICENSE/.*"
#"^LICENSE/.*"]
:basis basis
...#2023-05-0817:34pauldI solved the issue by adding ".*LICENSE.*" to :exclude in above.
#2023-05-0817:35pauldThanks @U04V15CAJ.#2023-05-0817:35borkdudeThis may be caused by Windows \\ instead of / Perhaps tools build can normalize/unixify the paths before comparing or so#2023-05-0817:36borkdude@U0FH1UL84 So it worked with :use-cp-file :never?#2023-05-0817:37pauldyeah#2023-05-0817:37borkdudeah ok then#2023-05-0817:37pauldThe space in my user name (home dir) was probably an issue.#2023-05-0817:37pauldNot sure why it worked since I was likely over the 8k character limit.#2023-05-0817:38borkdudespaces in paths should be handled imo#2023-05-0817:38pauldcould be a problem with scoop clojure via bash.#2023-05-0817:38borkdudebash?#2023-05-0817:39pauldwindows git bash#2023-05-0817:39borkdudeok, maybe also try in cmd.exe and pwsh.exe then#2023-05-0817:39pauldok will try#2023-05-0817:41pauldNo, problem exists in powershell too.#2023-05-0817:42borkdudeCan you re-define what "the problem" is since that information is in #clj-on-windows in a thread over there, which was a different problem than the original post of this one#2023-05-0817:43borkdudeOr start a new thread over here about that one#2023-05-0817:44pauldSure I'll post here again to make it clear.#2023-05-0816:51pauldI tried adding this to options of compile-clj:
:exclude [#"(?i)^META-INF/LICENSE/.*"
#"^LICENSE/.*"]#2023-05-0817:50pauldHi folks, I just wanted to repost this issue and solution here just in case it is helpful to others. I was having an issue with compile-clj failing:
λ clj -T:build uber
Error: Could not find or load main class Dumais\AppData\Local\Temp\compile-clj5675381223217921312\compile-clj;target.classes;src;C:\Users\Paul
Caused by: java.lang.ClassNotFoundException: Dumais\AppData\Local\Temp\compile-clj5675381223217921312\compile-clj;target/classes;src;C:\Users\Paul
Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:114).
Clojure compilation failed, working dir preserved: C:\Users\Paul Dumais\AppData\Local\Temp\compile-clj5675381223217921312
Full report at:
C:\Users\Paul Dumais\AppData\Local\Temp\clojure-12040712073046716600.edn
It looked like it was not dealing with the space correctly in my user name (home directory). Changing the :mvn/local-repo to outside that directory with a short non-space name did not help. I solved the issue by adding :use-cp-file :never to the build-clj options map.#2023-05-0817:55seancorfieldI suspect there are quite a few tools out there that will fail on directory paths that contain spaces. I've seen a lot of people recommend to have user names without spaces on Windows because of that (`Program Files` can also be problematic if folks install non-native-Windows tooling there).#2023-05-0818:02pauldI'm using the clojure installed by scoop on windows.#2023-05-0817:52pauldI'm using the clojure installed by scoop on windows.#2023-05-0817:52borkdudeLet's use a thread for multiple messages about the same problem#2023-05-0820:17vemvmy tools.build uberjarring script looks approximately like this:
(b/compile-clj {:bindings {#'clojure.core/*assert* false}})
...yet I still see assertions enabled in production. What are some possible reasons?#2023-05-0820:24vemvMaybe the main (that with :gen-class) namespace isn't currently requireing the ns that shows this behavior?
Although I'm using :src-dirs so it should compile everything#2023-05-0820:25Alex Miller (Clojure team)do you have a user.clj?#2023-05-0820:25vemvno#2023-05-0820:26hiredmanhave you looked in the output jar to see if the namespace you want to be compiled without assertions is present in the output?#2023-05-0820:27Alex Miller (Clojure team)off-hand, don't know. you can fork tools.build, add any print debugging you need, and point to the fork via either :local/root or to a commit sha in your fork, if that helps#2023-05-0820:27hiredmanthe aot classes for the namespace are not present in the output, then when you load the code later it will be compiled then with whatever flags that are set then#2023-05-0820:50seancorfieldI'm always surprised when I see folks turning assert off in production... Don't you want your app to "fail fast" rather than potentially continue on with bad data and cause havoc? I've always preferred to keep assert ON in production for that reason...#2023-05-0820:55vemvI've shared this link a few times over the years :) https://en.wikipedia.org/wiki/Design_by_contract#Performance_implications
The liberating thing about disabling it is that you can go as creative and heavy-handed as desired with assertions, without worrying about performance.
It also helps drawing a clear line between assertionerror and ex-info (things that aren't dev sanity checks should be the latter). Particularly when assertionerror does not inherit from Exception#2023-05-0820:56vemvBack to the topic, the expected .classis there and (print *assert*) during compile-time printed false, as expected
Will investigate some other day#2023-05-0902:08hiredmanAnd how do you know asserts are being triggered(#2023-05-0902:09hiredmanGotta go put kids to bed, but wild guess you are seeing spec asserts fail, and if I recall spec asserts don't respect assert#2023-05-0902:22Alex Miller (Clojure team)correct, they have their own assertion flags#2023-05-1009:23vemv> And how do you know asserts are being triggered
a vanilla assert was.
for completeness, I ended up using the same solution I've used in past production deployments. It should be redundant given the build.clj, but this double setup leaves me more assured.
(ns my.app.launcher
"Please don't add any requires here."
(:gen-class))
(defn -main [& args]
(binding [*assert* false] ;; Bind *assert* before performing any requires
(let [f @(requiring-resolve ')]
(apply f args))))
(sadly I don't feel with the bandwidth for a full-blown investigation)#2023-05-1010:32yuhanDo :local/root deps always take precedence over (transitive) Maven and git deps? It's not explicitly mentioned in the reference.#2023-05-1012:03Alex Miller (Clojure team)Neither takes precedence. You should get an error during version selection#2023-05-1012:03Alex Miller (Clojure team)But a special case is that top level deps take precedence over transitive#2023-05-1118:54pauldI'm getting the following error when I add a second middleware option to my :main-opts:
Exception in thread "main" java.io.IOException: Cannot run program "C:\Program Files\AdoptOpenJDK\jdk-11.0.5.10-hotspot\bin\java.exe": CreateProcess error=206, The filename or extension is too long
at
Here is my main-opts:
:main-opts ["-m" "nrepl.cmdline" "--middleware" "[refactor-nrepl.middleware/wrap-refactor,cider.nrepl/cider-middleware]"]
I'm on windows 10 using scoop clojure. This only happens when I use a second middleware. It works fine if I only include cider-middleware.#2023-05-1118:58pauldThis is when I run clj -M:dev where the above options are part of the dev alias.#2023-05-1206:39dominicmAre you using the official CLI, or @U04V15CAJ's version?#2023-05-1206:40dominicmI vaguely recall that windows has a pretty limited cli length. Maven writes the classpath to a file to work around that.#2023-05-1814:02pauldI was using official CLI. I'm going to look at @U04V15CAJ’s version to see if that helps.#2023-05-1814:02pauldThe issue popped up again when I tried to add another dependency.#2023-05-1814:07pauldHmmm, using @U04V15CAJ’s deps.exe did not seem to resolve this.#2023-05-1814:07pauldIs there a work-around?#2023-05-1814:09borkdudeAre you sure you were using the official CLI?
Exception in thread "main" java.io.IOException: Cannot run program "C:\Program Files\AdoptOpenJDK\jdk-11.0.5.10-hotspot\bin\java.exe": CreateProcess error=206, The filename or extension is too long
at
Seems to indicate you are using deps.clj#2023-05-1814:15pauldahhh, I guess I'm not sure. I installed via`scoop install clj-deps`#2023-05-1814:16borkdudeThis isn't the official CLI. The official CLI is presented here:
https://clojure.org/guides/install_clojure#_windows_instructions#2023-05-1814:16pauldI should try installing a different way perhaps.#2023-05-1814:16pauldok, will give this a go.#2023-05-1814:16borkdudeI'll try deps.clj on Windows myself now too. Which version of nREPL were you using?#2023-05-1814:20pauldnot sure...#2023-05-1814:21pauldClojure CLI version (deps.clj) 1.11.1.1273#2023-05-1814:25borkdudeI can't repro this with deps.clj and this in :aliases:
Running (borkdude.deps/-main "-A:nrepl-test") works fine. But perhaps it's the space#2023-05-1814:28borkdudeSetting :mvn/local-repo to "C:/Temp/with space" also works just fine#2023-05-1814:31pauldI'll let you know if the official cli runs fine.#2023-05-1814:33borkdudeok, I tried various things set to the directory with spaces locally: DEPS_CLJ_TOOLS_DIR and CLJ_CONFIG, but everything runs fine here, in cmd.exe#2023-05-1814:33borkdudeHow are you invoking the CLI btw, from the command line or via other tooling?#2023-05-1814:35pauldjust via command line#2023-05-1814:35pauldThe official cli doesn't work for me either.#2023-05-1814:35borkdudeso something like clj -M:nrepl?#2023-05-1814:36pauldclj -M:dev#2023-05-1814:36borkdudethe official CLI: can you paste the output of clj --version?#2023-05-1814:36borkdudejust to verify that you are using it#2023-05-1814:36pauldProgram 'java.exe' failed to run: The filename or extension is too longAt C:\Users\Paul
Dumais\Documents\WindowsPowerShell\Modules\ClojureTools\ClojureTools.psm1:453 char:7
+ & $JavaCmd @JavaOpts @JvmCacheOpts @JvmOpts "-Dclojure.basis=$B ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At C:\Users\Paul Dumais\Documents\WindowsPowerShell\Modules\ClojureTools\ClojureTools.psm1:453 char:7
+ & $JavaCmd @JavaOpts @JvmCacheOpts @JvmOpts "-Dclojure.basis=$B ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : NativeCommandFailed
#2023-05-1814:37borkdudeok, similar error then#2023-05-1814:37pauldClojure CLI version 1.11.1.1165
#2023-05-1814:37borkdudeseems good. and what is your java -version?#2023-05-1814:38pauldopenjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)
#2023-05-1814:39pauldThis time I get the error when I add my 9th dependency in the toplevel deps.#2023-05-1814:39borkdudeyeah, too long classpath probably... passing the classpath via a file would be the solution here.#2023-05-1814:40pauldIs there a way to force that. I did use that option when I was making an uberjar last week.#2023-05-1814:41borkdudeit's not available in the CLI yet, but it would be fixed by adding an @ sign here:
https://github.com/borkdude/deps.clj/blob/4918fb6fb6f90019fbc0f5d3f3cda036fa864a4a/src/borkdude/deps.clj#L998#2023-05-1814:41pauld:use-cp-file :always#2023-05-1814:42borkdudeThe caveat is that this only works in Java 9+ so maybe hiding it behind an environment variable DEPS_CLJ_USE_CP_FILE or so would make sense. But I would be jumping ahead of @U064X3EF3 in this case.#2023-05-1814:43borkdudecan you paste me your deps.edn with top level deps? then I can try to repro on Windows over here#2023-05-1814:44pauldI can paste something close... I can't expose internal stuff at the moment.#2023-05-1814:44borkdudeof course#2023-05-1814:48pauld{
:paths ["src" #_"target/classes" #_"classes"]
:mvn/repos
{"gitlab-maven" {:url ""}
"e-iceblue" {:url ""}}
:deps
{org.clojure/clojure {:mvn/version "1.11.1"}
org.clojure/tools.namespace {:mvn/version "1.3.0"}
org.apache.commons/commons-imaging {:mvn/version "1.0-alpha3"}
io.github.clojure/tools.build {:mvn/version "0.9.4"}
io.github.cljfx/dev {:mvn/version "1.0.36"}
com.mycompany/a {:mvn/version "1.0.0"}
com.mycompany/b {:mvn/version "1.0.0"}
com.mycompany/c {:mvn/version "1.0.0"}
io.github.humbleui/humbleui {:git/sha "d7c2915b4900d34b727cd2006ff0f8cca1907242"}
}
:aliases
{
:dev {:extra-deps {cider/cider-nrepl {:mvn/version "0.30.0"}}
:main-opts ["-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]}
:build {:deps {io.github.clojure/tools.build {:mvn/version "0.9.4"}}
:ns-default build}
}
}#2023-05-1814:54borkdude com.mycompany/a isn't useful for me to repro, but I'll try the other deps#2023-05-1814:56borkdudeif you comment out the com.mycompany deps, do you still experience the problem? the other deps don't create the problem for me locally#2023-05-1814:58borkdudewhen you have a repro with only public libs, I'll try again locally#2023-05-1816:17pauldHmmm, yeah I try to get you something like that.#2023-05-1816:28pauldI'm having difficulty reproducing this issue without one of two particular internal dependencies that can only be found in our private maven gitlab repo.#2023-05-1816:29pauldThese have a large list of transitive dependencies that probably make the classpath very large.#2023-05-1817:22borkdudeI tried adding 100+ libs to the deps.edn and I still couldn't reproduce but someone suggested elsewhere (in #C03RZGPG3 ) that a too long filename on the classpath might trigger it.#2023-05-1817:23borkdudeif you remove your local .cpcache directory, and then run clj -M:dev do you see a .cpcache/<hash>.cp file and what is inside of that? any long paths? how many semicolons do you see? (this would count the total number of dependencies, about...)#2023-05-1818:01pauld329 semicolons#2023-05-1818:01borkdudeand how long (many characters) is the file?#2023-05-1818:04pauldwc -m .cpcache/<hash>.cp
returns 33653#2023-05-1818:04pauldthat's # of characters#2023-05-1818:05pauldwhich is also the byte count#2023-05-1818:07borkdudeAh, maybe it has to do with this: https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553#2023-05-1818:07borkdudesounds about right#2023-05-1818:08pauldahh#2023-05-1818:11borkdudeok, I'm at `user=> (borkdude.deps/-main "-M:nrepl-test")
22070`
now, so I should keep adding some more ;)#2023-05-1818:12pauldMine started working again at 32506 but was broken at 32800.#2023-05-1818:13borkdude👍#2023-05-1818:17borkdudeYep, finally I could reproduce it:
32900
Execution error (IOException) at java.lang.ProcessImpl/create (ProcessImpl.java:-2).
CreateProcess error=206, The filename or extension is too long
#2023-05-1818:17borkdude300+ deps in there ;)#2023-05-1818:18borkdudeI'll write a Windows-specific fix#2023-05-1818:23pauldThe joys of bringing clojure to a large legacy codebase.#2023-05-1818:49borkdude@U0FH1UL84 I think this version should work. Download the zip and run deps.exe -M:dev#2023-05-1818:49borkdudehttps://ci.appveyor.com/project/borkdude/deps-clj/builds/47083892/artifacts#2023-05-1818:57pauldgreat! repl starts now#2023-05-1819:10borkdudecool, I'll try to make a proper test for this and will try to release a new version tomorrow#2023-05-1821:13borkdude@U0FH1UL84 I released it now https://clojurians.slack.com/archives/C015AL9QYH1/p1684444369418759#2023-05-1821:14borkdudeit will take maybe a bit of time before clj-deps in scoop is updated but should be there in a few hours or so#2023-05-1909:08borkdudeThere is an issue with the scoop-clojure: it doesn't parse the suffixes I added to the version (`-4` ) so the package doesn't update currently. But you can use the manual powershell script and add --as-clj to install it as clj.exe and clojure.exe for now.#2023-05-1911:13borkdudeactually scoop should now work too, thanks to @UBLU3FQRZ#2023-05-1911:13borkdudescoop update
scoop update clj-deps
#2023-05-1408:26pezWhat’s the best way to get 1.12.0-alpha3 installed in an apt-get based Docker container? This doesn’t work:
root@da127afd0308:/app# curl -O
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 243 0 243 0 0 371 0 --:--:-- --:--:-- --:--:-- 371
root@da127afd0308:/app# tail -3 linux-install-1.12.0-alpha3.sh
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>QHDZR8BX0JE4XQS0</RequestId><HostId>6kINnasq/o/x2gOdNFw0J3HxPPk1Owa081QnyKtTj9yPEHsOB6G/5o2HbcX5iBlYQ07TZ/AkFBo=</HostId></Error>root@da127afd0308:/app# #2023-05-1411:26Alex Miller (Clojure team)There is no 1.12 version of the Clojure CLI. You need the latest version of the CLI (1.11.1.1273)#2023-05-1411:27Alex Miller (Clojure team)Any version of the CLI can use any version of Clojure - that depends on the Clojure version in your deps.edn#2023-05-1412:03pezThanks! Can I assume that any stable Clojure release has a corresponding CLI?#2023-05-1412:20Alex Miller (Clojure team)That’s not a meaningful question #2023-05-1412:21Alex Miller (Clojure team)So, no#2023-05-1412:25Alex Miller (Clojure team)The CLI version is actually meaningfully only the last part. The first 3 parts indicate the Clojure version it was built with and the Clojure version you’ll get from the root deps.edn if you don’t say otherwise#2023-05-1412:25pezHaha, thanks. Is there a way I can programmatically figure out the latest CLI download path?#2023-05-1412:25Alex Miller (Clojure team)Yes#2023-05-1412:27Alex Miller (Clojure team)There is a file https://download.clojure.org/install/stable.properties #2023-05-1412:28Alex Miller (Clojure team)That has the version and the sha of the download#2023-05-1412:29Alex Miller (Clojure team)Also if you use the url at the top but without the version, that’s always the latest#2023-05-1412:30Alex Miller (Clojure team)https://download.clojure.org/install/linux-install.sh#2023-05-1412:34pezAwesome! Exactly what I need. #2023-05-1415:14pezIs there a way to tell clojure to only consider the classpath that is specified in the local deps.edn file? So, nothing from user deps.edn and nothing that is not specified (like no implicit clojure dependency)?#2023-05-1415:16Alex Miller (Clojure team)-Srepro will omit the user deps.edn#2023-05-1415:17Alex Miller (Clojure team)You always need a Clojure dep, the default is in the root deps.edn, but overrides by project deps.edn#2023-05-1415:18Alex Miller (Clojure team)But maybe it’s more useful to tell me what you’re actually trying to do#2023-05-1415:21pezI am trying to feed ClojureCLR a classpath based on a deps.edn file. This is what I have now:
#!/usr/bin/env bb
(require '[clojure.string :as string])
(require '[babashka.process :refer [sh]])
(require '[babashka.fs :as fs])
(sh "clojure" "-P" "-A:dev")
(def original-classpath (string/trim (:out (sh "clojure" "-Spath" "-A:dev"))))
(def skip-deps ["org/clojure/clojure"
"org/clojure/core.specs.alpha"
"org/clojure/spec.alpha"])
(def skipped-deps-re (re-pattern (str ".*/\\.m2/repository/("
(string/join "|" skip-deps)
")/.*")))
(defn skip-path? [path]
(re-matches skipped-deps-re path))
(defn process-classpath [classpath]
(let [paths (string/split classpath #":")]
(->> paths
(remove skip-path?)
(map (fn [path]
(if (.endsWith path ".jar")
(let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
deps-dir (str "/app/dependencies/" deps-subdir)]
(sh "mkdir" "-p" deps-dir)
(sh "unzip" "-q" "-o" path "-d" deps-dir)
deps-dir)
path)))
(string/join ":"))))
(when (= *file* (System/getProperty "babashka.file"))
(-> original-classpath
process-classpath
println))#2023-05-1415:24pezI’ll add the -Srepro, even if it isn’t strictly needed because this is run in a Docker container that does not have a user deps.edn.#2023-05-1415:24Alex Miller (Clojure team)You can just call tools.deps directly #2023-05-1415:24Alex Miller (Clojure team)Instead of shelling out to use the CLI which uses tools.deps#2023-05-1415:27pezHmmm, not sure how. Babashka has tools.deps in there somewhere?#2023-05-1415:28Alex Miller (Clojure team)Call clojure.tools.deps/create-basis and pass it :root nil :user nil. That gives you a basis with a :classpath-roots, then call join-classpath#2023-05-1415:28Alex Miller (Clojure team)I think it does, but prob a question for the borkmeister#2023-05-1415:29Alex Miller (Clojure team)Alternately you can call the create-basis variant provided in the :deps alias#2023-05-1415:29Alex Miller (Clojure team)clj -X:deps create-basis :root nil :user nil#2023-05-1415:30Alex Miller (Clojure team)Something like that, on my phone#2023-05-1415:30pezI’ll experiment with this a bit.#2023-05-1415:31Alex Miller (Clojure team)Oh wait, that’s not there anymore but there is prob some path to doing it#2023-05-1415:32Alex Miller (Clojure team)But really, tools.deps is a library to make classpaths. Ideally you can just call it to do that#2023-05-1415:32borkdude@U0ETXRFEW babashka's classpath is empty by default and if you use (babashka.deps/add-deps '{:deps {}}) it will only add those deps to the classpath, not even clojure itself#2023-05-1415:33borkdudeYou can retrieve the bb classpath with (babashka.classpath/get-classpath)#2023-05-1415:36borkdudeThis is done via the :classpath-overrides option similar to this:
clj -Sdeps '{:deps {} :aliases {:rm-clojure {:classpath-overrides {org.clojure/clojure nil}}}}' -A:rm-clojure -Spath#2023-05-1415:37pezCan I get add-deps to use my deps.edn file?#2023-05-1415:39borkdude@U0ETXRFEW Yes: (add-deps (edn/read-string (slurp "my-deps.edn")))#2023-05-1415:45pezHow would I tell it to use the :dev alias in there?#2023-05-1415:48borkdude@U0ETXRFEW
(add-deps {:deps ..} {:aliases [:dev]})#2023-05-1415:51pezAwesome, works! Does it also download dependencies?#2023-05-1415:55borkdudeyes#2023-05-1416:10pezThis is great! Now I have this:
#!/usr/bin/env bb
(require '[clojure.string :as string])
(require '[clojure.edn :as edn])
(require '[babashka.process :refer [sh]])
(require '[babashka.deps :as deps])
(require '[babashka.classpath :as classpath])
(require '[babashka.fs :as fs])
(defn process-classpath [classpath]
(let [paths (string/split classpath #":")]
(->> paths
(map (fn [path]
(if (.endsWith path ".jar")
(let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
deps-dir (str "/app/dependencies/" deps-subdir)]
(sh "mkdir" "-p" deps-dir)
(sh "unzip" "-q" "-o" path "-d" deps-dir)
deps-dir)
path)))
(string/join ":"))))
(when (= *file* (System/getProperty "babashka.file"))
(deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
(-> (classpath/get-classpath)
process-classpath
println))
It does some work when running the script that I might want to have it do as part of building the container. Like downloading Clojure and stuff.
root@d278348ba283:/app# output=$(docker/cheap-deps.clj)
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.jar from clojars
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.jar from clojars
root@d278348ba283:/app# echo $output
src:dev:/app/dependencies/tools.nrepl-0.1.0-alpha1:/app/dependencies/tools.reader-1.3.7
What’s short command line to tell Babashka to do this initial work?#2023-05-1416:12borkdude> What’s short command line to tell Babashka to do this initial work?
I'm not sure what the question is#2023-05-1416:13borkdudebtw instead of (sh "mkdir" "-p") , you could use (fs/create-dirs deps-dir) and for (sh "unzip") you could possibly use fs/unzip#2023-05-1416:16borkdudeAlso instead of #":" I recommend using fs/path-separator for better cross-platform compatibility#2023-05-1416:16pezThe question. I want to populate the maven repo with the stuff that babashka needs at container build time. So that running my script doesn’t pay that overhead.#2023-05-1416:17borkdudebb already has babashka.classpath/split-classpath which can split the classpath for you in a cross-platform manner#2023-05-1416:18borkdude> I want to populate the maven repo with the stuff that babashka needs at container build time
Do you mean the "installation" of deps.clj, i.e. downloading the tools jar?#2023-05-1416:23pezYes, if that’s what causes all those downloads, then that’s what I mean. 😃#2023-05-1416:23pezThe script now looks like
(defn process-classpath [classpath]
(->> (classpath/split-classpath classpath)
(map (fn [path]
(if (.endsWith path ".jar")
(let [deps-subdir (string/replace (fs/file-name path) #".jar$" "")
deps-dir (str "/app/dependencies/" deps-subdir)]
(fs/create-dirs deps-dir)
(fs/unzip path deps-dir {:replace-existing true})
deps-dir)
path)))
(string/join ":")))
(when (= *file* (System/getProperty "babashka.file"))
(deps/add-deps (edn/read-string (slurp "deps.edn")) {:aliases [:dev]})
(-> (classpath/get-classpath)
process-classpath
println))
And runs crazy fast compared to what I started with!#2023-05-1416:26borkdude> Yes, if that’s what causes all those downloads, then that’s what I mean.
I don't know what you mean by "all those downloads", can you be more specific? If you download deps using this script, then it's expected that ... there are "all those downloads"?#2023-05-1416:27borkdude(.endsWith path ".jar") => (str/ends-with? path ".jar") would be my preference here#2023-05-1417:28pezSorry for being unclear. Of those downloads there, only the last two are from my deps.edn (one directly, and one transiently). The rest is something triggered by my script. #2023-05-1417:32borkdudeare you sure? e.g.:
Downloading: org/clojure/clr/tools.nrepl/0.1.0-alpha1/tools.nrepl-0.1.0-alpha1.pom from clojars
Downloading: org/clojure/clr/tools.reader/1.3.7/tools.reader-1.3.7.pom from clojars
#2023-05-1417:33borkdudeI'd do an -Stree or -X:deps tree to double check#2023-05-1418:28pezThose are the last two. My deps.edn looks like so:
{:paths ["src"]
:aliases {:dev {:extra-paths ["dev"]
:extra-deps {org.clojure.clr/tools.nrepl {:mvn/version "0.1.0-alpha1"}}}}}#2023-05-1418:31pez# clojure -Stree
org.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
#2023-05-1418:52borkdudeWell, those other deps aren't needed by bb so I think it's just the result of tools.deps downloading those as well#2023-05-1418:59pezIs there a way I can trigger the download of those dependencies without running my script?#2023-05-1419:02pezThis isn’t enough:
RUN curl -sLO \
&& chmod +x install \
&& ./install \
&& bb -e '(require (quote [babashka.deps :as deps]))'#2023-05-1419:03borkdudeI think you could even prevent downloading those dependencies with a hack:
clj -Sdeps '{:deps {org.clojure/clojure {:local/root "src" :deps/manifest :deps} org.clojure/core.specs.alpha {:local/root "src" :deps/manifest :deps}}}}' -Spath
src:/private/tmp/src/src:/private/tmp/src/src
So by setting clojure + core.specs.alpha to a local (existing!) dir it won't download those#2023-05-1419:04borkdudebut note that those libs will end up on the classpath, but combined with :classpath-overrides they will then go away again, I think#2023-05-1419:05borkdudeYou could force downloading those dependencies by just pre-fetching an arbitrary dependency with add-deps in the image build, probably#2023-05-1419:07pezI don’t have clj in the container any longer. Can I do this with Babashka?#2023-05-1419:08borkdudeyes, with add-deps#2023-05-1419:08borkdudebb -e '(babashka.deps/add-deps ...)'
#2023-05-1419:15borkdude$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}}))'
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.pom from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.pom from central
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.pom from central
Downloading: org/clojure/pom.contrib/1.1.0/pom.contrib-1.1.0.pom from central
Downloading: medley/medley/1.0.0/medley-1.0.0.pom from clojars
Downloading: org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar from central
Downloading: org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar from central
Downloading: org/clojure/clojure/1.11.1/clojure-1.11.1.jar from central
Downloading: medley/medley/1.0.0/medley-1.0.0.jar from clojars#2023-05-1419:15borkdudeso yeah, it seems to download clojure + spec etc even though they are cancelled out on the classpath with classpath-overrides 🤷#2023-05-1419:16borkdude$ bb -e '(babashka.deps/add-deps (quote {:mvn/local-repo "/tmp/src" :deps {medley/medley {:mvn/version "1.0.0"}}})) (prn (babashka.classpath/get-classpath))'
"/tmp/src/medley/medley/1.0.0/medley-1.0.0.jar"#2023-05-1617:14yuhanNot sure if this is a tools.deps specific question - what happens if two different libraries declare the same namespace, is it deterministic which one gets placed first on the classpath?
The library Medley recently changed its groupId from medley to dev.weavejester, and I realised after upgrading that my project is now pulling in two versions of the library, the newer 1.7.0 version (`dev.weavejester/medley`) and an older 1.0.0 version (`medley/medley`) from a transitive dependency that was previously shadowed. But the namespace medley.core hasn't changed, and :deps is an unordered map, so how can I be sure that (require 'medley.core) loads the correct jar?#2023-05-1617:19yuhanI tried an experiment with the root /deps.edn:
{:paths ["src"]
:deps {lib/old {:local/root "old"}
lib/new {:local/root "new"}}}
/old/deps.edn
{:deps {medley/medley {:mvn/version "1.0.0"}}}
/new/deps.edn
{:deps {dev.weavejester/medley {:mvn/version "1.7.0"}}}
Depending on the ordering of the lines in the root edn, clj -X:deps tree either shows the 1.0.0 version above or before the 1.7.0, but requiring the namespace always seems to load the 1.7.0 library.#2023-05-1617:21hiredmanas far as tools.deps knows, those are two different libraries#2023-05-1617:21hiredmanso it has no magic to pick the right one or anything#2023-05-1617:22hiredmanso you need to add an exclusion to the dep pulling in the old version with the old id#2023-05-1617:28yuhanIs this something that people have to regularly watch out for by screening their top level dependencies for 'clashing' transitive deps? I can imagine eg. something in lib/new trying to call a newer Medley function but breaking because lib/old came first on the classpath#2023-05-1617:30hiredmanit is something that comes up from time to time, more commonly with regular java deps where the same code is packaged in multiple jars#2023-05-1617:31hiredmansometimes you run into it with libraries that have been badly packaged (made uberjars)#2023-05-1617:32hiredmananother option would be to try and upgrade the dep that is using the old medly version, it looks like medly made the change more than a year ago#2023-05-1617:33hiredmanoh misread the date, 6-7 months ago#2023-05-1617:33Alex Miller (Clojure team)while you should avoid dependency with overlapping namespaces (this is almost always a mistake or an error), the classpath ordering is deterministic#2023-05-1617:34hiredmandeterministic in the sense that you will always get the same order, but not in the sense that it detects the same namepsaces in different libraries and somehow does some kind of resolution of that#2023-05-1617:35hiredmanthe classpath order ultimately determines which of the namespaces would get loaded when you do a require#2023-05-1617:36Alex Miller (Clojure team)correct, was just trying to explain why you get the same namespace loaded in the two cases above#2023-05-1617:37Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#_classpath_ordering#2023-05-1617:37yuhanThanks for the info! it's not an issue in this particular case but I was wondering in general, eg. with quite a few popular libraries having moved to clj-commons or changing ownership in the past, whether this is something that causes problems#2023-05-1617:39hiredmanthere is a ticket to maybe catch this kind of thing https://clojure.atlassian.net/browse/TDEPS-229#2023-05-1617:39yuhanaha, so in my example it was only the coincidence that "d" comes before "m"#2023-05-1620:25borkdude> The library Medley recently changed its groupId from medley to dev.weavejester
I've done this once and will not ever do it again, it causes more trouble than it's worth, unless clojars or tools.deps introduces groupid relocation (https://github.com/clojars/clojars-web/issues/801)#2023-05-1712:15kennytiltonThe Google offers two solutions for deploying a deps.edn based project to Clojars, https://github.com/slipset/deps-deploy and https://github.com/applied-science/deps-library#usage. Are there any others? Is one or another preferred?#2023-05-1716:02practicalli-johnPersonally I'd prefer https://github.com/slipset/deps-deploy as I've seen it used by others and the maintainer seems active (and a nice person in my interactions with them), but I havent pushed anything (useful) to clojars myself.
deps-deploy also supports signing (which may now be required - it should certainly be encourage). I didnt see a mention of signing on deps-library#2023-05-1716:10kennytiltonThx! The deps-deploy looked pretty accessible, and I am already doing the signing with Clojars so that should go smoothly. 🙏#2023-05-1712:46dominicm@alexmiller RE https://clojure.atlassian.net/browse/TDEPS-229
I wrote https://git.sr.ht/~severeoverfl0w/clj-classpath-duplicates to do that already. It uses https://github.com/classgraph/classgraph to scan the classpath, only for fun/speed/convenience reasons, not because of necessity.
I’m not sure I can be of any help, but happy to be if I can 🙂#2023-05-1712:48dominicmIt was convenience: ClassGraph has a “findDuplicatePaths” method.#2023-05-1712:47dominicmSomething that was very helpful in practice was including the digest. It allowed distinguishing “this is managed because I’ve got the versions lined up” vs “this is definitely different”#2023-05-1712:48dominicmIt was convenience: ClassGraph has a “findDuplicatePaths” method.#2023-05-1714:23lreadHello! I'm doing a little work on https://github.com/benedekfazekas/mranderson to more directly support deps.edn projects. For lein projects, MrAnderson supports marking deps specified in project.clj for inlining via metadata. An example:
:dependencies [^:inline-dep [rewrite-clj "1.1.47"]]
We are hashing out our options for deps.edn deps. I was thinking of using metadata like antq does (https://github.com/cljdoc/cljdoc/blob/08be85ad4074a54114d97d3423a597a9e21e5771/deps.edn#L13-L18), maybe something like so (1):
{:deps {^:mranderson/inline-dep rewrite-clj/rewrite-clj {:mvn/version "1.1.47"}}}
Or like so (2):
{:deps {rewrite-clj/rewrite-clj ^:mranderson/inline-dep {:mvn/version "1.1.47"}}}
But another option was proposed (3):
{:deps {rewrite-clj/rewrite-clj {:mvn/version "1.1.47"
:mranderson/inline true}}}
I'm guessing that the metadata approaches (1 and 2) might be less problematic to naive tooling because metadata would not be seen unless explicitly queried.
But is there anything technically problematic with the non-metadata approach (3)? Does (3) seem maybe a bit more human-readable-friendly?#2023-05-1715:00Alex Miller (Clojure team)I'm going to promise you nothing on whether that metadata gets transferred through any tdeps/cli machinery#2023-05-1715:00Alex Miller (Clojure team)please also consider a separate map of data in :aliases and/or a separate file#2023-05-1715:02Alex Miller (Clojure team)aliases data can be easily read from the basis and is arbitrary data that you completely control#2023-05-1715:04lreadThanks @alexmiller, much appreciated!#2023-05-1715:09lreadSo the approach taken by antq (https://github.com/cljdoc/cljdoc/blob/08be85ad4074a54114d97d3423a597a9e21e5771/deps.edn#L13-L18) is probably something that you'd discourage, ya?#2023-05-1715:17Alex Miller (Clojure team)yes#2023-05-1715:04grzmGood (java.time.Instant/now)! I’d like to create a classpath that just includes only paths in my project. clj -Spath -A:my-alias returns the full classpath: project paths, and paths to all of the deps as well. Right now I’ve been string-processing the full class-path, removing anything that’s an absolute path. Anyone have a better/another approach to recommend? (Now I’m kicking myself for not confirming whether inspecting the basis won’t provide something useful before asking the channel)#2023-05-1715:09Alex Miller (Clojure team)the basis has a map of :classpath parts, which have a value map telling you what it is and where it came from#2023-05-1715:09Alex Miller (Clojure team)so it would be quite easy to filter just the paths from the libs#2023-05-1715:16Alex Miller (Clojure team)(-> (td/create-basis {}) :classpath (get "src/main/clojure"))
;;=> {:path-key :paths}
(-> (td/create-basis {}) :classpath (get "/Users/alex.miller/.m2/repository/org/clojure/clojure/1.10.3/clojure-1.10.3.jar"))
;;=> {:lib-name org.clojure/clojure}#2023-05-1715:16grzmThanks, @alexmiller!#2023-05-1715:17Alex Miller (Clojure team)the ordering of the classpath keys is in the basis under :classpath-roots#2023-05-1717:37Dave OrmeHi everyone! I'm wondering: does anyone know the official status of add-lib in the add-lib3 branch? Are there plans to develop it more / integrate it?
Other threads have mentioned that it seems to have fallen out of sync with master though I confess I haven't tried a merge myself. I'm wondering if it would be supported if I did it?
### Background:
I've been playing with ideas to create a Smalltalkish dynamic runtime for Clojure/script, and would like to use tools-deps for dynamically adding libraries, assuming this would be the appropriate way to do it.
I have my own library (based on the technique Boot uses) but if there's going to be an officially-supported method, I would prefer to use that.
Thoughts, ideas, and feedback are welcomed.#2023-05-1717:38borkdude@U0587N61TMH Check out the clojure 1.12 alpha3 release notes, dude :)#2023-05-1717:39Dave OrmeOoooh. Okay; thanks!#2023-05-1717:53Dave OrmeFollow-up question is in #C03S1KBA2#2023-05-1720:52imreIs there a way to make the clojure cli (installed via brew) use a locally modified version of tools.deps? I'm planning to muck about with tools.deps a bit and would like to do some poking through the cli first#2023-05-1720:58Alex Miller (Clojure team)you can identify the script via which clojure, then edit the tools_cp variable to include a pointer to a local source dir in the front of the classpath (like /path/to/your/tools.deps/src/main/clojure). the big caveat is that the tools jar is direct linked, so it is possible to walk into some weirdness, but I do test things this way and it usually works ok#2023-05-1721:00Alex Miller (Clojure team)also, depending on what you're hacking, you might need more active use of -Sforce to force recomputation of classpaths#2023-05-1721:06imreThanks. I just want to do a bit of inspection to see if I could have a reasonable chance to add deps/root support to local/root. Should only be changes to clj files in tools.deps#2023-05-1721:07Alex Miller (Clojure team)go for it!#2023-05-1721:07Alex Miller (Clojure team)happy to have a patch on that to look at#2023-05-1721:08imreWill try to make some time for it, thanks!#2023-05-1811:47imrebtw, would it make sense to add a more graceful override for tools_cp for these kinds of thing? like an env var one could set?#2023-05-1823:10imresubmitted a patch 🎉#2023-05-2203:19Drew VerleeIs there a way to avoid declaring the mvn/root in a project, if the local/root dependency that uses it already has declared the mvn/root? For example, the example project below depends on local project f, which uses something found in the blah repo.
{:deps {f/f {:local/root "../f"}}
But if i try to build this, it fails, because it can't find blah. But if I add the mvn repo where blah is found, it works (as expected)
... :mnv/repo {"blah" {:URL "blahs url}}
But mvn/repo is already declared in the f project, so it feels like i'm declaring it twice for no added benefit.#2023-05-2203:24seancorfieldIt's to avoid possible hijacking of dependencies by a transitive dep.#2023-05-2203:26Alex Miller (Clojure team)The added benefit is that you know you declared it#2023-05-2203:27Drew VerleeGotcha. Thanks for the follow-up!#2023-05-2310:09borkdudeThis might be a silly edge case but... I'm sometimes switching between git/sha + deps/root to :local/root and I expected this to work:
{:git/sha "..."
:deps/root "render"}
=>
{:local/root "/Users/borkdude/dev/clerk"
:deps/root "render"}
but I think I also need to include the deps/root path into the :local/root here. Maybe something to consider?#2023-05-2311:21Alex Miller (Clojure team)TDEPS-246 is wip on this#2023-05-2314:25seancorfieldSee also https://ask.clojure.org/index.php/12913/using-deps-root-with-local-root#2023-05-2314:26seancorfield(it has bitten me too)#2023-05-2314:31borkdudecool, thanks#2023-06-1015:52imreI have a patch submitted for this. It hasn't been reviewed yet so I don't know whether the general approach is acceptable but I'd be happy to get feedback so feel free to give it a spin if you feel like.#2023-05-2521:05Eduardo LopesIs there a similar feature in Clojure CLI to Leiningen :injections? I'm using clj -e but I'm wondering if init-opt can be achievable from deps.edn#2023-05-2521:07seancorfield-e can be put in :main-opts in deps.edn but the specifics depend on exactly what you already have...#2023-05-2521:07seancorfieldIs this just to start a REPL?#2023-05-2521:10seancorfield(~/clojure/repl)-(!2009)-> cat deps.edn
{:aliases
{:repl
{:main-opts ["-e" "(ns foo.bar)" "-r"]}}}
Thu May 25 14:09:54
(~/clojure/repl)-(!2010)-> clj -M:repl
foo.bar=>
☝️:skin-tone-2: does that help?#2023-05-2521:13Eduardo LopesActually is a curiosity based on how githooks works, since we need to manually run git config core.hooksPath dir I can solve this on leiningen projects adding an injection like :injections [(require '[clojure.java.shell :refer [sh]]) (sh ....)]
I'm assuming that any lein anything I run will setup githooks for me. So adding this to a specific alias may not solve 100%#2023-05-2521:14Eduardo LopesMaybe adding as injections is not the best solution but I always try to "translate" from leiningen to clojure cli what I'm doing#2023-05-2521:15seancorfieldWith the CLI, you need an alias for this, or supply it on the command line. But you may be better off writing build.clj for the tasks you want to do since then you can run commands easily and compose things since "it's just code".#2023-05-2521:16seancorfieldI don't really understand your use case tho' -- you only need to run git config once for a given project, yes?#2023-05-2521:18Eduardo LopesYes, was just an idea to avoid manually doing this each time you download and be part of an “invisible” process #2023-05-2521:19Eduardo LopesI think using makefile is better for this also, is just a brainstorming #2023-05-2521:19seancorfieldIf you're interacting with git, I'd definitely suggest looking at tools.build since it has git functionality in its API. And then just do stuff via clojure -T:build <something> (like lein <something>) and it can do whatever you need...#2023-05-2521:20seancorfieldWe have pretty much all our dev/test/CI tooling built around tools.build and build.clj at work -- and I use that approach with most of my OSS projects on GitHub too.#2023-05-2521:26dpsuttonif it’s just once you could do something like
clj -X:dev dev/install-git-hooks and code it up yourself#2023-05-2522:00Eduardo LopesYes, both solutions seems better by adding to a makefile.
Another use case I remembered was a project using midje, when we executed eastwood and kibit the tests were running, since they load the files. The way to avoid this (don't know if it's the best solution either) was an injection :injections [(require 'clojure.test) (alter-var-root #'clojure.test/*load-tests* (constantly false))] in a linter profile, this could be 100% achievable with an alias on Clojure CLI.
The difference is that leiningen allows an injection on the defproject root, which may solve some better use cases than the ones I'm bringing here 😅#2023-05-2522:01Eduardo LopesBut thanks! I've learned a little more about clj cli with this conversation gratitude#2023-05-3004:14cflemingI’m trying to use a basis to figure out which paths were added from the :test alias. I’m not sure how to achieve this. I can get the paths that come from aliases easily enough (e.g. from :classpath where :path-key is :extra-paths), but I can’t figure out how to get the :extra-paths from :test. I can look them up under :aliases but if I understand correctly that can contain unexpanded substitutions. I could duplicate that logic but that’s what I’m trying to avoid. Is there a way I can do this?#2023-05-3005:01Alex Miller (Clojure team)I don't think that's available in the api right now#2023-05-3005:29seancorfieldIf you use (tools.deps/combine-aliases basis [:test]) and then look at :test in that result, is that what you need? Or is that the "unexpanded substitution" you mentioned?#2023-05-3007:26cflemingYeah, in the end I just reimplemented the expansion logic. It would be nice to not have to, though.#2023-05-3105:38twashingtools.deps isn’t pulling foo/bar from custom-mvn-repo.
There’s an entry for it in “~/.m2/settings.xml”.
I also specified it in deps.edn.
{:deps {foo/bar {:mvn/version "0.0.1"}}
...
:mvn/repos {"custom-mvn-repo" {:url ""}}}
But when trying to reference the :a alias, I still get the below error.
Why isn’t tools.deps trying to reference my artifact from custom-mvn-repo?
clj -M:a
Error building classpath. Could not find artifact foo:bar:jar:0.0.1 in central ()
#2023-05-3105:44seancorfieldWhat is the :a alias and why do you expect to change the repos that are searched for a basic dep you've specified?#2023-05-3111:20Alex Miller (Clojure team)Can you just share the full deps.edn?#2023-05-3111:21twashing@U04V70XH6 Thanks for the response.
:a is just an alias that starts an nrepl. Sorry, that nomenclature just confuses the issue.
Ultimately tools.deps doesn’t seem to search additional repos in :mvn/repos. And I’m unclear on why.
https://clojure.org/reference/deps_and_cli#_maven#2023-05-3111:23Alex Miller (Clojure team)It does search them#2023-05-3111:23Alex Miller (Clojure team)You really need to share more explicitly what you’re doing so we can help#2023-05-3111:24twashing@U064X3EF3 Because it’s a work repo, I technically can’t share that code. If it were OSS, I definitely would. Apologies.
But I hear you. Let me try putting together a test repo ☝️:skin-tone-5:#2023-05-3111:27Alex Miller (Clojure team)One thing you might try is generating a pom.xml with -Spom and see whether that looks right, then trying mvn dependency:tree#2023-05-3111:32twashingGood suggestion. I get this…
clj -Spom
Error building classpath. Could not find artifact foo:bar:jar:0.0.1-ace77e745-SNAPSHOT in central ()
#2023-05-3111:35twashing… “central” is specified in a custom ~/.m2/settings.xml.
That’s where I also tried to add my “custom-mvn-repo” (https://custom.maven.archive2/groups/public/).#2023-05-3111:37twashingThe only thing I can think of is to try with a simplified repo. More soon!#2023-05-3112:15Alex Miller (Clojure team)The error messages can be misleading here - if an an artifact is not found in any repo, only the first error is reported, which is typically central, but that doesn’t mean your other ones weren’t searched#2023-05-3112:16Alex Miller (Clojure team)But I notice this is a snapshot - are you sure the repo you’re adding is a snapshot repo?#2023-05-3112:18Alex Miller (Clojure team)Depending what kind of repo this is, you might also be able to look in that url at the desired path to verify it actually exists there#2023-05-3112:40twashing“but that doesn’t mean your other ones weren’t searched”
Ah ha!
“you might also be able to look in that url at the desired path to verify it actually exists there”
Spot on… It turns out that my Or g is migrating to a new Maven repository. So I need to update my settings.xml, and possibly a registration. So just going through that right now.
Really appreciate the feedback 👏:skin-tone-5:#2023-06-0502:08nivekuiltrying the new`add-libs` functionality, but because I have -Dcom.sun.management.jmxremote.port=17199 in JAVA_OPTS , when a new JVM is spawned it inherits that and tries to bind to that same port, so it throws an exception. Not sure how I can override this behavior, or if tools.deps should manage its own JAVA_OPTS#2023-06-0502:10seancorfieldIf you have that env var set globally, you can't start more than one JVM in general... How/where are you setting that?#2023-06-0502:12seancorfieldIt feels like something you should have under an alias, in :jvm-opts, so you can control when it's in effect. And then it won't interfere with other JVM instances.#2023-06-0503:43Alex Miller (Clojure team)I agree with Sean that that would be better (you can also pass on the CLI with -J-Dcom.... ), but this is something that we should at least you handle through a combination of https://clojure.atlassian.net/jira/software/c/projects/CLJ/issues/CLJ-2777 and https://clojure.atlassian.net/jira/software/c/projects/CLJ/issues/CLJ-2770 which are both things still in the queue#2023-06-1215:47borkdudedeps.clj downloads the tools jar if it's not installed yet on a system, but 1 in 100 (or fewer) times I get complains that deps.clj returns:
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main
because of an invalid downloaded jar which is then solved by deleting ~/.deps.clj so it forces a re-download.
I can repro this by messing with the tools jar:
echo '' > /Users/borkdude/.deps.clj/1.11.1.1347/ClojureTools/clojure-tools-1.11.1.1347.jar
A solution to solve this problem could be that I mirror the tools.zip in deps.clj github releases along with a .sha256 file so I can verify if the download was successful.
Or perhaps http://clojure.org could provide a checksum file (that I can verify in clojure/Java)#2023-06-1215:49borkdudeIs there anything against me mirroring the tools.zip on github releases of deps.clj? Or would you be in for solution 2, then I could just avoid doing so#2023-06-1215:51dominicmWhy is the jar invalid?#2023-06-1215:51dominicmHow do you download the jar?#2023-06-1215:52Alex Miller (Clojure team)there is a checksum file in maven already#2023-06-1215:52dominicmAre the clojure tools uploaded to maven?#2023-06-1215:52Alex Miller (Clojure team)oh nvm, you're talking uber jar here#2023-06-1215:53borkdudeyeah the zip file#2023-06-1215:54borkdudehttps://download.clojure.org/install/clojure-tools-1.11.1.1347.zip#2023-06-1215:54borkdudefor example#2023-06-1215:56borkdudefor zip and tar.gz files for other projects I now upload a .sha256 file for validation, e.g. see here:
https://github.com/clj-kondo/clj-kondo/releases/tag/v2023.05.26
I directly copied this idea from graalvm:
https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-22.3.2#2023-06-1215:57dominicmIf you checked the jar for clojure.main (or something) after downloading, that would serve the same purpose, right?#2023-06-1215:57dominicmI’m still wondering why the jar is invalid without you knowing, though.#2023-06-1215:58borkdudeyeah I think so, although there are more files in that .zip file#2023-06-1215:58Alex Miller (Clojure team)the .tar.gz sha is in https://download.clojure.org/install/stable.properties but we don't currently make or publish a zip sha#2023-06-1215:58dominicmWait, the zip unzips but the files that come out aren’t valid?#2023-06-1215:59borkdudeyeah that's weird. so it could also be an unzip problem perhaps#2023-06-1215:59dominicmDepending on what you’re using, I would have thought the checksumming in the zip would have prevented this.#2023-06-1215:59dominicmSomething smells fishy here.#2023-06-1215:59Alex Miller (Clojure team)maybe it's unzipping a partially downloaded zip?#2023-06-1216:00borkdudeor the user aborts while unzipping? dunno, but it sometimes happens#2023-06-1216:00dominicmDo you keep the zip file around?#2023-06-1216:00Alex Miller (Clojure team)like the first half of the zip. I don't know anything about zip but maybe it supports this so you can zip while still downloading#2023-06-1216:00borkdudedeps.clj first downloads, then unzips#2023-06-1216:00dominicmI vaguely recall that zip stores the file listing at the end just to be annoying.#2023-06-1216:01dominicmhttps://unix.stackexchange.com/a/125102#2023-06-1216:01borkdudethe .jar file does exist on disk, next time this happens for someone I should ask them to send this .jar file to me, but it's hard to know what exactly goes wrong. Getting rid of ~/.deps.clj solves the problem though. Perhaps I could also try/catch and check for clojure.main in the error message and offer a better suggestion#2023-06-1216:01Alex Miller (Clojure team)seems like it still needs a bit more examination (is there somewhere you can detect the problem and save off the badness when it happens) but happy to publish additional sha files if that's helpful#2023-06-1216:03borkdudeI guess I could add to the error message: please go to the #CLX41ASCS channel and post your .jar for examination ;)#2023-06-1216:03borkdudeRe-downloading automatically seems risky as if I make a mistake somewhere you could get into a loop#2023-06-1216:03dominicmIf you kept the zip around, you should be able to re-validate the jars against the zip.#2023-06-1216:04borkdudeyeah, good idea, I'll keep the .zip around. I currently delete it#2023-06-1216:08dominicmIf you wanted to check, you could .getCrc on your ZipEntry and calculate the Crc for the file on disk & compare them.#2023-06-1216:08dominicmNot sure how fast that would be, but maybe worth doing on failure to launch or something. Crc32 is pretty fast.#2023-06-1216:14borkdudeoh that might be a good check, I'll try it out#2023-06-1219:01borkdudeI'm getting -1 on all entries from the tools.jar zip#2023-06-1219:01borkdudewhich means "unknown"#2023-06-1219:04borkdudeah, the crc is known after you read it#2023-06-1220:14borkdudeSomething like this ought to do it.
https://github.com/borkdude/deps.clj/pull/103/commits/8627f15d2278e78bf949861c589fa40c8781fdc1#2023-06-1308:55borkdudeI wonder how I can purposely damage a .zip file to test this, because I'm still not 100% sure if what I did makes any sense. I'm getting the crc32 of the entry and then read it through a checkedinputstream and then compare the crc32, but who says that those aren't always the same, even in the case of a weirdly downloaded zip file#2023-06-1309:10borkdudeWhen I truncate some of the last bytes, I can't even unzip it:
dd if=clojure-tools-1.11.1.1347.zip of=tools-corrupted.zip bs=1 count=17999800
#2023-06-1309:11dominicmI would expect that the file either unzips falsely or is "damaged" after the fact.#2023-06-1309:11dominicmIf you keep the zip around you can revalidate the files against it.#2023-06-1309:12borkdudehow would that work?#2023-06-1309:12borkdudethe crc32 codes are -1 when I read them, which means they weren't in the zip file when the the file got zipped, right#2023-06-1309:12dominicmI'm not sure I follow#2023-06-1309:13borkdudethey only become some number after I've actually processed the entry, which tells me it's lazily computed based on the data that was already there, which is kind of pointless#2023-06-1309:13dominicmYou need to consume the stream for the CRC to be calculated, yeah#2023-06-1309:13dominicmSo you could just read the whole thing and then check it against the CRC of what's on disk#2023-06-1309:15borkdudeI'm checking this:
(= (.getCrc entry) (-> cis (.getChecksum) (.getValue)))
but you are suggesting calculating the crc32 of the file on disk, rather than from the checked input stream which was used to copy to disk?#2023-06-1309:18dominicmFile on disk compared with the one in the entry#2023-06-1309:18dominicmSorry, not both *#2023-06-1309:18borkdudeso not based on the checked input stream?#2023-06-1309:18dominicmI think they're both the same value tbh.#2023-06-1309:19borkdudeyeah, I think so too, the above comparison is always true, I think, no matter how corrupted#2023-06-1309:19dominicmYou can use the file Vs the zip as a checksum#2023-06-1309:19dominicmAlthough the CRC should be written in the zip somewhere, too#2023-06-1309:19borkdudeso comparing to on disk would detect some kind of data write failure?#2023-06-1309:20borkdudeyeah, I think you need to produce a zip file with explicit crc32 on#2023-06-1309:20dominicmOr if the file had been modified (eg by antivirus)#2023-06-1309:20dominicmOh I thought zips had them by default.#2023-06-1309:20dominicmIt could be a limitation of the Java zip library, too#2023-06-1309:21borkdudeah right it has crc32 by default#2023-06-1309:26borkdudeok, I added the extra check:
https://github.com/borkdude/deps.clj/commit/82adf0cb15249a753851579bc9fac0db52921778#2023-06-1309:26borkdudewill keep an eye out until the next failure and will ask for the zip file so I can double-check if this really helps#2023-06-1409:22borkdudeI guess if I could manually change the crc code of the tools jar in an existing .zip file, I could test if the current deps.clj would detect this (and give the better error message)#2023-06-1409:22dominicmYou could also create a new zip with a different tools jar in ☺️#2023-06-1409:23borkdudeand then?#2023-06-1412:10dominicmThe CRC of the tools jar in the zip would be different than the one on disk.#2023-06-1412:11borkdudebut not if you unzip that zip with the different tools jar#2023-06-1412:11borkdudeI just want a zip file that I can throw at this function and then the function complains#2023-06-1412:33dominicmAh, then you’re probably into some kind of malicious crc fiddling of the zip file, yeah.#2023-06-1412:34dominicmTime to pull out your hex editor 😄#2023-06-1620:26grzmShould --report stderr work with an alias that includes :main-opts? For example, I'd expect this to echo to stderr rather than writing to a file:
% clj -e "(-> (slurp \"deps.edn\") ((requiring-resolve 'clojure.edn/read-string)) (get-in [:aliases :main]))"
{:main-opts ["-m" "my.main"], :jvm-opts ["-Dclojure.spec.check-asserts=true" "-Djdk.internal.httpclient.disableHostnameVerification=true"]}
% clj -M:main --report=stderr
Execution error [redacted]
Full report at:
/var/folders/p0/qtwr_lhx6dlf8kddzgrbsn180000gp/T/clojure-11739203642877558907.edn#2023-06-1620:27grzmPassing the clojure.main.report Java property does what I expect:
% clj -J-Dclojure.main.report=stderr -M:main
Caught error [reacted]
{:clojure.main/message ,,,,#2023-06-1620:28grzmBut --report stderr is so much nicer to type at the command line (if it works)#2023-06-1620:39hiredman--report stderr is an option for clojure.main so it would go in main-opts, my guess would be --report stderr is being interpreted here as an argument for your code and being passed to my.main#2023-06-1620:43grzmYeah, that's likely what's happening. Just would be nice to be able to toggle it at the command line more easily.
% clj --help
Version: 1.11.1.1347
...
Run main clojure [clj-opt*] -M[aliases] [init-opt*] [main-opt] [arg*]
...
init-opt:
-i, --init path Load a file or resource
-e, --eval string Eval exprs in string; print non-nil values
--report target Report uncaught exception to "file" (default), "stderr", or "none"
#2023-06-1918:42seancorfieldGiven the number of people (mostly beginners) who seem to run into problems with the CLI because they have an older version and are following either newer instructions or trying to use libraries that rely on newer features, I wonder whether it would be worth adding a check to the script that would output a warning if the current stable version is more recent (from the download site)?
I know that won't help people who already have an old version installed but it would at least help future people...
...or is the command-line "API" stable enough at this point that it wouldn't be worth it?#2023-06-1918:47Alex Miller (Clojure team)Probably worth doing, feel free to add an ask for it#2023-06-1918:50seancorfieldhttps://ask.clojure.org/index.php/13020/could-the-cli-check-for-the-latest-stable-version#2023-06-1920:08pavlosmelissinosI've been thinking about this but never asked the question. Thanks, upvoted 🙂#2023-08-1200:11jasonjckni think this is already on your radar… but be mindful of folks working in network secured environments , definitely needs a cli flag to disable , probably timeout the connection very quickly and/or do it async #2023-08-1200:14jasonjcknperhaps clojure and clj would have different defaults here #2023-06-2118:01cflemingIs clojure.libfile still a thing?#2023-06-2118:01cflemingI had a request a while ago to add it, but it looks like it doesn’t do anything any more?#2023-06-2118:02Alex Miller (Clojure team)no#2023-06-2118:02Alex Miller (Clojure team)it doesn't exist#2023-06-2118:03Alex Miller (Clojure team)it was replaced with clojure.basis property (which is now accessible in 1.12 via new clojure.java.basis api)#2023-06-2118:04cflemingOk, thanks.#2023-06-2203:06cflemingI’m feeling a bit silly about this, but I can’t figure out the incantation to get a REPL using a tool classpath. Say I would normally do clojure -T:build <something> to run a tool, how do I start a REPL with that classpath?#2023-06-2203:38seancorfieldThis made me curious. In theory you can run clojure -Tmy-tool some-fn where some-fn will start a REPL... but the gotcha is that it needs to accept a single hash map argument -- for -X style invocation -- and I couldn't find anything that fit the bill...#2023-06-2203:39cflemingDoes this mean that no-one is using REPLs to develop their builds? It makes me feel better about not doing it myself, but surprises me.#2023-06-2203:40seancorfieldYou said "tool" so I assumed you meant -Tmy-tool not -T:some-alias#2023-06-2203:41cflemingIn my case it’s an alias, with a :deps and a :ns-default#2023-06-2203:41seancorfieldclojure -A:the-alias#2023-06-2203:41cflemingWhich is a tool, right? Perhaps I’m not getting the terminology right.#2023-06-2203:41cflemingBut that will also include the project classpath, and not just the tool classpath.#2023-06-2203:42cflemingi.e. it’s the -T that does the replacing of the project classpath and paths.#2023-06-2203:42seancorfieldYeah, I start a "build REPL" at work with clj -M:build -i build.clj -r I think#2023-06-2203:43seancorfieldLet me see what's in the script I use for that... I suspect it works well because it's a Polylith project and there are no dependencies unless you specify the :dev alias 🙂#2023-06-2203:45seancorfieldOh, I only have that on my other machine so I can't check right now.#2023-06-2203:46cflemingSo I just tried that in my project, and I get the project paths, at least (`src` etc, not .)#2023-06-2203:46seancorfieldMostly when I'm working with my build.clj, I just use -M:build or similar and don't care whether the rest of the project is accessible or not.#2023-06-2203:47cflemingThat seems odd, particularly since the paths are incorrect. I’m curious what others do.#2023-06-2203:49cflemingSo weirdly, the deps are correct but the paths are not. I suspect this is because the alias uses :deps rather than :extra-deps. I guess the paths are not a problem as long as your build is in a single file.#2023-06-2203:49seancorfieldI guess it depends on how isolated you need it to be while working on it. You could use -Sdeps '{:paths ["."]}' I suspect.#2023-06-2203:50cflemingI guess so. Thanks for the info!#2023-06-2203:52seancorfieldI tend to work on my build.clj files alongside the project files so I want "src" etc as my paths -- nothing in my build.clj files relies on the paths being just "."#2023-06-2203:54cflemingThat wouldn’t work for me without something like the -Sdeps trick, since I slurp config files from the path.#2023-06-2203:54cflemingActually, that would work as long as the CWD is set correctly, I guess only resource loading would be affected.#2023-06-2204:05seancorfieldHow do you normally start your REPL for working with code? Via Cursive, I'd assume, with nREPL etc?#2023-06-2300:10cflemingYes, either nREPL or socket REPL. But right, this is all part of a large change I’m making to how Cursive uses deps. I’m going to use deps.clj, which will make it easier to just allow people to enter the same things they’d enter on the command line to start their REPLs, and it’ll work more like the CLI than my current frankenintegration.#2023-06-2922:52isakIf I have this alias in my deps edn: :setmain {:ns-default foo.foo}, when I run clj -A:setmain, why does it start a repl in the user namespace instead of foo.foo?#2023-06-2922:59hiredmanhttps://clojure.org/reference/deps_and_cli#_arg_map_keys#2023-06-2922:59hiredman:ns-default tells -X and -T to lookup unqualified symbols passed as arguments in the given namespace#2023-06-2923:02isakI see, thanks#2023-06-2923:06isakDo you know if there a way to set the default namespace when you are starting a REPL?#2023-06-2923:06hiredmanthere is none#2023-06-2923:07isakah, ok#2023-06-2923:09hiredmanyou can pass expressions to be pre-evaluted
% clj -M -e '(ns foo)' -r
foo=>
which I believe in tools.deps would be maybe a main args, no idea how any of that would be quoted/escaped#2023-06-2923:24isakNice, that approach works. I'll try to see if there is a way to put it in an alias.#2023-06-3000:16Alex Miller (Clojure team)You can put that in :main-opts#2023-06-3000:38seancorfieldIf it's an existing ns you want to load and switch to at startup, make sure you require it and then in-ns it!#2023-06-3007:34practicalli-johnRather than try to set the namespace when starting the repl, a common practice is to use a custom user namespace to load and run code
https://practical.li/clojure/clojure-cli/repl-startup/#2023-07-0115:43Ian FernandezIs there a way to import java classes from a git deps?#2023-07-0116:04Alex Miller (Clojure team)if the git dep has a prep step that compiles the classes to a folder, that's really the only way#2023-07-0116:05Alex Miller (Clojure team)https://clojure.org/reference/deps_and_cli#prep#2023-07-0116:22Ian Fernandezthanks!#2023-07-0220:31Martynas MaciulevičiusHey.
Is this relevant to when a package uses a git repository directly?
https://news.ycombinator.com/item?id=36556436#36558002#2023-07-0221:45Alex Miller (Clojure team)maybe? is there a clearer explanation somewhere?#2023-07-0221:53Alex Miller (Clojure team)tdeps git libs are bare repositories, but always used in explicit mode, I can't tell if that eliminates the issue or not#2023-07-0222:02Alex Miller (Clojure team)I'm not sure I fully understand every detail, but it does not seem like tdeps git libs does the things that would be a problem because it never runs command in a sub dir and always uses explicit bare repos#2023-07-0223:41didibusI feel in general you should trust your git deps, they can use prep or as soon as you run your code, they'll be loaded in and can evaluate anyways no.
It's still be nice to remove any possible exploit, but I wouldn't consider it safe personally to depend on an untrusted git deps no matter what#2023-07-0223:43didibusThat's why I personally find Maven Central the safest. They go out of their way to validate the source and make sure nothing can be hijacked.#2023-07-0301:11Alex Miller (Clojure team)Do they? They validate that you control the groupId domain somehow. I’m not sure that they validate anything about a repo at all#2023-07-0305:36didibusSorry, I meant the source of the package, like who published it, they don't validate the source code. But you can be sure that you are getting the authentic package as published by the actual owner, since they have to prove domain ownership, and every file needs to be signed, and the groupId prevents hijacking, etc.#2023-07-0305:36didibusI think Clojars does a bit more validation now as well.#2023-07-0308:50Martynas Maciulevičius> maybe? is there a clearer explanation somewhere?
I don't know. This is the only post that I encountered.
I think that the attack is simply that the repository could execute code on behalf of user's account. And this means that it could interact with ~/.bashrc and other important files.#2023-07-0312:01Alex Miller (Clojure team)No one checks anything about the signed package, just that you uploaded a signature file. A consuming client can verify the jar is signed and check that the publishing identity matches the expected identity, but this is optional (no one does it) and manual (it’s up to you to determine whether the key is the expected key)#2023-07-0312:03Martynas Maciulevičius> No one checks anything about the signed package, just that you uploaded a signature file.
I was not thinking about signed packages.
I was only thinking about source dependencies that aren't compiled. You have to use git underneath so this is why I wrote about this.#2023-07-0319:29timothypratleyHi 👋
I'd like to make use of some files from the [JSON-Schema-Test-Suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite),
(specifically, it has some test/ref.json etc files).
I added it as a git/sha dependency in my deps.edn:
io.github.json-schema-org/JSON-Schema-Test-Suite {:git/sha "7950d9e0579382103031ef3cfcdc37e7c58b2d1f"}
Error:
Manifest file not found for io.github.json-schema-org/JSON-Schema-Test-Suite
Which makes sense... this project is not a Clojure or Java project (it's just a bunch of data files).
But, on the other hand, it would be very convenient if I could specify it as a dependency in this way and make use of those files.
Is this something that can be achieved?#2023-07-0319:31seancorfieldYou can specify :deps/manifest :deps in the coords to tell tools.deps "treat this as if it were a deps.edn project" (with an empty deps.edn file).#2023-07-0319:31timothypratleyOh, wonderful. Thank you Sean!#2023-07-0319:32seancorfieldIt probably still won't let you load that JSON file since it won't seem to be on a classpath...#2023-07-0319:33seancorfieldYou're talking about the stuff in this folder, yes? https://github.com/json-schema-org/JSON-Schema-Test-Suite/tree/main/tests/draft2020-12#2023-07-0319:33timothypratleyyes 🙂#2023-07-0319:34seancorfieldNot sure how you'd tell tools.deps that would be on your classpath for it... you might be better off just pulling the file(s) direct from GitHub with an http client.#2023-07-0319:36timothypratleyI kinda wish I could do:
:deps/manifest {:paths ["tests"]} 🙂#2023-07-0319:38timothypratleyooo maybe I can :override-deps#2023-07-0319:40dpsuttonwould just using a traditional git submodule be simpler and straightforward in this instance?#2023-07-0319:42timothypratleyYes; using a git submodule is the recommended way to depend on the test suite, but I was curious 🙂 IMO submodules aren't much fun.#2023-07-0319:42dpsuttonyeah. i’ve never actually used one personally. so i can totally understand wanting to add it into other tooling to hide it#2023-07-0319:54Alex Miller (Clojure team)submodules don't work with tdeps git libs#2023-07-0319:55dpsuttonyeah. just suggesting it being a submodule of the project he’s working on so then he could just add it to the classpath and use git to fetch as normal without tdeps involved#2023-07-0319:55Alex Miller (Clojure team)yes, that should work afaik#2023-07-0319:56Alex Miller (Clojure team)I think this is not a code dep and not really the case git libs was designed for#2023-07-0320:15timothypratleyThis worked:
(slurp (io/file (System/getProperty "user.home") ".gitlibs/libs/io.github.json-schema-org/JSON-Schema-Test-Suite/7950d9e0579382103031ef3cfcdc37e7c58b2d1f/tests/draft7/ref.json"))
😎#2023-07-0501:59cflemingIs it documented anywhere that tool aliases can use :deps and :paths, and that this will replace the project deps? https://clojure.org/reference/deps_and_cli#_arg_map_keys doesn’t mention it, and I can’t find it explicitly mentioned in the reference, but there’s an example at https://clojure.org/reference/deps_and_cli#_using_tool_aliases. What is the difference between doing this and using :replace-deps?#2023-07-0504:07Alex Miller (Clojure team)No difference, they are aliases#2023-07-0504:07Alex Miller (Clojure team)Same with paths and replace-paths#2023-07-0504:08cflemingOk thanks.#2023-07-0504:10Alex Miller (Clojure team)Kind of a historical artifact at this point#2023-07-0504:10cflemingIs one or the other the preferred way going forward?#2023-07-0504:39seancorfieldI'll be interested to hear it there's a preference at this point... The official guide suggests :deps (and paths is rarely needed but I'd expect :paths to be suggested), but I don't remember the history...#2023-07-0505:25cflemingDoes it? The deps and cli ref mostly mentions :replace-deps, and only has :deps in examples. I can’t see anywhere there where it’s even documented that :deps is a thing you can use (although the doc is big and dense, I might have missed it).#2023-07-0505:30seancorfieldThe official tools.build guide, I meant, shows :deps.#2023-07-0505:30cflemingAh, ok#2023-07-0505:33seancorfieldYes, the CLI/deps guide/reference talks about :replace-deps and :replace-paths in https://clojure.org/reference/deps_and_cli#_replace_project_environment_tool and https://clojure.org/reference/deps_and_cli#_arg_map_keys but that was about how to get tool-like behavior before -T existed I think? But https://clojure.org/guides/tools_build#_setup talks about -T and :deps and :paths.#2023-07-0505:48cflemingWell, the deps and cli ref definitely discusses -T, as well as installing tools and running them via aliases. I don’t think it’s a big deal, but the inconsistency makes it more confusing than it needs to be right now.#2023-07-0506:06seancorfieldSure, but that reference has this example:
:aliases {
:linter {:deps {cool/lint-tool {:git/tag "..." :git/sha "..."}}
:ns-default cool.lint.tool}
With :deps, not :replace-deps.#2023-07-0506:07seancorfieldI agree it could be cleaned up. I'm not sure it's "inconsistent" tho'...#2023-07-0506:07cflemingYes, I know, I linked that above. But having it appear like that in examples, when there’s no mention in the doc that that is even possible, is confusing.#2023-07-0506:07seancorfieldIt omits quite a bit of nuance (but that's nothing new for Clojure docs).#2023-07-0506:08seancorfieldNo mention of what in the doc?#2023-07-0506:08cflemingThat you can use :deps in aliases instead of :replace-deps. Under “Arg Map Keys” only the replace- versions are shown.#2023-07-0506:10seancorfieldYou're not joining the dots tho'... All of the tool examples use :deps and the narrative around -T says that project :deps / :paths are ignored so you only get :deps and :paths from the alias (or the tool) -- and that if :paths is omitted you get ["."]. That's all consistent.#2023-07-0506:11cflemingI guess we’ll have to agree to disagree. Anyway, at least I understand what’s going on now.#2023-07-0506:11seancorfieldThe :replace-deps / :replace-paths stuff is not inconsistent with that: it says that in any context (`-M`, -X, or -T), those override the other deps/paths.#2023-07-0506:13seancorfieldI'm not sure what we're disagreeing on -- but I'm glad you understand the docs, I think?#2023-07-0506:13cflemingNo, I understand what is going on after Alex clarified it for me above 🙂#2023-07-0508:04practicalli-johnPersonally I find the :deps and :path shortcuts make learning and maintaining Clojure CLI projects more involved. deps and paths do different things depending on if they are at the top level or in an :aliases definition.
I encourage use of the explicit names to make it obvious as to what the configuration will do.#2023-07-0508:19cflemingSounds very wise to me.#2023-07-0513:06dpsuttonI’m having an issue using a dep from maven. The dep is https://central.sonatype.com/artifact/org.nd4j/nd4j/1.0.0-M2.1 (org.nd4j/nd4j). I can browse to it at https://repo1.maven.org/maven2/org/nd4j/nd4j/1.0.0-M2.1/ . But there are no jars in that folder, just a pom and checksums. Looking at the pom I don’t see it labelled as a bom, but I do see <modules> that makes me think it is using a feature that tools.deps doesn’t support.
> ❯ clj -Sdeps ‘{:deps {org.nd4j/nd4j {:mvn/version “1.0.0-M2.1”}}}’ -P
> Error building classpath. Could not find artifact org.nd4j:nd4j:jar:1.0.0-M2.1 in central (https://repo1.maven.org/maven2/)#2023-07-0513:17Carsten BehringFigure out which modules you need and then add them one-by-one ?
This is one of them:
https://repo1.maven.org/maven2/org/nd4j/nd4j-tensorflow/1.0.0-M2.1/nd4j-tensorflow-1.0.0-M2.1.pom#2023-07-0513:18dpsuttonok. so modules are opaque to tools.deps and i can unfurl them manuall?#2023-07-0513:06dpsuttonIs there a way to consume this dependency with tools.deps?#2023-07-0513:27Alex Miller (Clojure team)pom-only or bom dependencies are not currently supported as top-level deps, this is a wip area (https://clojure.atlassian.net/browse/TDEPS-202)#2023-07-0513:34dpsuttonhow can i recognize that this is “pom-only” dependencies? Is it <modules>?#2023-07-0514:04Alex Miller (Clojure team)<packaging>pom</packaging>
#2023-07-0514:04dpsuttonah. thank you#2023-07-0514:09Alex Miller (Clojure team)modules is about maven build time, doesn't affect anything on consumption side#2023-07-0813:07bobcalco@alexmiller I finally got around to the feature request we discussed on the #clr channel a couple weeks back: https://ask.clojure.org/index.php/13072/feature-request-clojure-specify-conditional-values-local#2023-07-1114:25Sagura091hey so I have a question and I may not be in the right channel, but I am trying to package all of my non-compiled cljs code into a maven local repo and use it like a library in my deps.edn file.#2023-07-1114:29borkdudeIs there a question?#2023-07-1115:46Sagura091sorry, can I do that using maven without having .js files? right now we have a git sha lib system but we want to transfer it over to maven, and its dependencies.#2023-07-1115:47borkdudeyes, there is no difference between a mvn or git dependency when it comes to what files are in your dependency#2023-07-1116:03Sagura091how do I put my entire project into a local mvn repo with out using clojars?#2023-07-1117:05borkdudea local mvn repo? why would you do that?#2023-07-1119:24Sagura091Because some of our content has to be private and we want to add our dependencies to be compiled and included without uncompiled cljs. the issue is we use Git/sha and it's supposed to be a library. but if a developer brings down the library they will have to find the shad include that then do a npm install that may that multiple dependency includes. we want to have self contained dependencies for our lib.#2023-07-1120:19borkdude> Because some of our content has to be private
What does this have to do with git or mvn? Both git or mvn can be private or public#2023-07-1120:19borkdude> we want to add our dependencies to be compiled and included without uncompiled cljs
ok...#2023-07-1120:20borkdudeI'm not sure I understand the rest of what you said. git, mvn and now npm... Perhaps you can try to focus on one clear problem at a time? #clojurescript might also be a good channel to ask questions about this#2023-07-1205:25jjttjjCheck out the "Local Maven install" section on this page https://clojure.org/reference/deps_and_cli#2023-07-1207:10thhellerif your intent is to have shadow-cljs install npm dependencies of the library you can add a deps.cljs file to that library including {:npm-deps {"that-dep" "1.2.3"}}. when shadow-cljs is started it'll look for those files and install the dependencies via npm install #2023-07-1207:11thhellerdoesn't matter at all whether thats a git, local or maven dep#2023-07-1207:12thhellerjust needs to be at the root of the classpath. so any of the lib :paths.#2023-07-1215:54Sagura091@U05224H0W Hey I am not trying to be confusing. the git/sha we have is fine, but it is clunky. in our main project, we have packages that we cannot upgrade because it will break our build. the lib has all the current up-to-date npm packages it needs. when we add the lib to the project shadow or clj says "hey can't do this because it, is not the right version" what I am trying to look for, when add our lib to the main project and we start to build it with shadow and web-pack, the lib it does an npm install from its own repo and we don't have to maintain the libs dependencies.#2023-07-1216:47thhellerI don't really understand your issue then. I assume you wanted to transfer npm dependencies from the lib to the main project, which I described. how you resolve conflicts is really up to you and a npm issue, neither deps.edn or shadow-cljs will help with that in any way.#2023-07-2102:19jasonjcknHow can I create a (non uber) jar release with a lib/ directory #2023-07-2102:41seancorfieldCan you explain what you mean about the lib/ directory?#2023-07-2102:41jasonjcknlike the datomic sdk , all the dep jars in lib/#2023-07-2102:41jasonjcknor like lein-libdir plug-in #2023-07-2103:44seancorfieldYou can't redistribute those so you can't (it shouldn't) try to bundle them with your library.#2023-07-2103:44seancorfieldI assume you're asking about building a library from your own code that depends on those local libs?#2023-07-2103:46jasonjckn‘lib’ is maybe a misnomer , im just trying to distribute my application and uberjar has some unfavorable characteristics #2023-07-2103:46jasonjckninstead of messing with all the logic in trying to merge every dependency jar into a single fatjar #2023-07-2103:46seancorfieldUberjar is normally how you build an application - can you explain why it doesn't work for you?#2023-07-2103:47jasonjckni wanted just java -cp lib: app.jar #2023-07-2103:48jasonjcknmy specific use case is probably not great, im getting security scanned and the fatjar has vulnerable code that i can’t tell which dependency it’s coming from via Stree#2023-07-2103:48seancorfieldSo... Build the (Uber) jar without those deps?#2023-07-2103:48jasonjcknbut uberjar have always had problems as i recall #2023-07-2103:49seancorfieldHuh? That is the standard way to build an app for deployment...#2023-07-2103:50jasonjckni remember running into issues i how it attempts to merge jars together, it was so long ago though, i wish i could be more specific #2023-07-2103:50jasonjcknanyways i get your instincts #2023-07-2103:50jasonjcknlet me
figure out a different path , thank you 🙏 #2023-07-2103:51seancorfieldI would say: try the standard approach and if you have a specific issue, ask about that specific issue...#2023-07-2104:55favilaFwiw I don’t think this request is weird, and I too wish “dir of libs” was more standard because of uberjar edge cases. Datomic does it after all! #2023-07-2104:55favilaI asked about this earlier https://clojurians.slack.com/archives/C6QH853H8/p1640216122102000#2023-07-2105:06favilaIf you look in the channel at that date but outside that thread, you see some suggestions along these lines https://clojurians.slack.com/archives/C6QH853H8/p1640218580108200#2023-07-2105:06favilaEssentially use create-basis then copy. Git deps require some extra care though#2023-07-2609:30dominicmpack.alpha is designed to support this kind of workflow#2023-07-3117:59jasonjcknanother fantastic juxt library , ty @U09LZR36F #2023-07-3108:02jumarI'm trying clojure -X:deps aliases but getting this error
clojure -X:deps aliases
Namespace clojure.tools.cli.api loaded but function not found: aliases
What am I doing wrong?
I noticed the namespace is somewhat different from what I found on github: https://github.com/clojure/tools.deps.cli/blob/main/src/main/clojure/clojure/tools/deps/cli/api.clj#2023-07-3108:03jumarCommands like tree and list work, though.
I found those commands here: https://clojure.org/reference/deps_and_cli#_other_programs#2023-07-3108:37Alex Miller (Clojure team)probably an older version of clojure?#2023-07-3108:38jumarThis is what I've got
which clj
/usr/local/bin/clj
ll /usr/local/bin/clj
lrwxr-xr-x 1 jumar admin 37B May 19 2022 /usr/local/bin/clj -> ../Cellar/clojure/1.11.1.1113/bin/clj
#2023-07-3108:40jumarIf I do brew upgrade
✗ brew upgrade clojure
Warning: clojure/tools/clojure 1.11.1.1113 already installed
#2023-07-3108:40Alex Miller (Clojure team)was added in 1.11.1.1119#2023-07-3108:40Alex Miller (Clojure team)current version is 1.11.1.1347#2023-07-3108:45jumarOk, I thought I'd been using the latest version until today.
I don't know why brew upgrade clojure doesn't work so I did this:
brew install clojure/tools/clojure
Error: clojure 1.11.1.1113 is already installed
To install 1.11.1.1347, first run:
brew unlink clojure
brew unlink clojure
Unlinking /usr/local/Cellar/clojure/1.11.1.1113... 4 symlinks removed.
brew install clojure
...
clj --version
Clojure CLI version 1.11.1.1347
Now the aliases command works.#2023-07-3108:59Alex Miller (Clojure team)you should be using the official Clojure tap with brew upgrade clojure/tools/clojure (but we push the same versions to homebrew-core, so what you were doing should work too)#2023-07-3108:45jumarOk, I thought I'd been using the latest version until today.
I don't know why brew upgrade clojure doesn't work so I did this:
brew install clojure/tools/clojure
Error: clojure 1.11.1.1113 is already installed
To install 1.11.1.1347, first run:
brew unlink clojure
brew unlink clojure
Unlinking /usr/local/Cellar/clojure/1.11.1.1113... 4 symlinks removed.
brew install clojure
...
clj --version
Clojure CLI version 1.11.1.1347
Now the aliases command works.#2023-08-0720:29flowthingSo the invocation to serve a clojure.main REPL over a network socket is a bit of a mouthful (`clj -J-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"`).
I’m wondering whether it would make sense to add a built-in alias (like -X:deps) for launching a socket-based clojure.main REPL. Ideally, you’d be able to do something like clj -X:repl, which would launch a socket server serving a clojure.main REPL with :port 0. It’d then write the port number into a file (like Lein does with .nrepl-port). I believe Cursive can read a socket REPL port number from a file now, too, for example. Any trailing args would get passed to clojure.core.server/start-server.
I know I’m probably the only person in the world who wants this, but still, figured I’d ask. 🙂#2023-08-0720:32hiredmanthe jvm has built in support for unix domain sockets now, seems a shame for new stuff for local tooling to continue to use network sockets#2023-08-0720:33seancorfieldBear in mind that -J... combines with any options -- as soon as you tie it to -X:deps, you've excluded a whole bunch of invocations (including -M based stuff).#2023-08-0720:35dpsuttonfrom my .zshrc:
# sample usage: clj -J"$(socket-repl 6000)" ... or java "$(socket-repl 6000) -jar ...
socket-repl()
{
echo -n "-Dclojure.server.repl={:port $1 :accept clojure.core.server/repl}"
}#2023-08-0720:35seancorfieldYou can always put an alias in your user deps.edn: https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L55#2023-08-0720:35Alex Miller (Clojure team)you can vote for adding something to the cli for this here: https://ask.clojure.org/index.php/7830/first-class-options-for-socket-repl?show=7830#q7830#2023-08-0720:35seancorfieldThen you can use that with -A or -M or -X 🙂#2023-08-0720:36flowthingYeah, I’m aware of those approaches (although neither automatically assigns an available port), but it’s an extra hurdle for brand new users, I think.#2023-08-0720:36dpsuttoni use my socket-repl shell function all the time#2023-08-0720:36flowthingThanks, Alex! I’ll do that.#2023-08-0720:36flowthingI do something like this:
(defn start-server
[_]
(let [server (server/start-server {:name "server" :port 0 :accept `server/repl :server-daemon false})
port (.getLocalPort server)
host (-> server .getInetAddress .getCanonicalHostName)
port-file (io/file ".repl-port")]
(.deleteOnExit port-file)
(spit port-file port)
(printf "Socket server listening on %s:%s\n" host port)))#2023-08-0720:36Alex Miller (Clojure team)anyone who wants to suggest things for this could also add those to that question if we wanted to collect ideas there#2023-08-0720:37flowthingI juggle so many projects that I don’t want to think about whether a port number’s free.#2023-08-0720:39flowthingI also considered going with Unix domain sockets earlier, but I didn’t want to require Java 16. Also, it’s handy to be able to connect to a remote socket, too. I’m also not sure whether there’s any benefit to using Unix domain sockets over a network socket (other than avoiding the port number file hassle).#2023-08-1117:04telekid@seancorfield I'm curious to know where you've landed with regard to https://clojure.atlassian.net/browse/TBUILD-6#2023-08-1117:06telekidI was reading through https://clojure-doc.org/articles/cookbooks/cli_build_projects/#:~:text=or%20in%20containers.-,tools.,jar%20and%20. and generally liked the pattern established here, but ran up against the issue of not seeing a clear path forward to pass :exec-args to cognitect.test-runner.api/test#2023-08-1117:10seancorfieldWe ended up backing away from it altogether and went back to :main-opts stuff via aliases. The cookbook pretty much lays out exactly what we do at work and what I do in all my OSS projects these days.#2023-08-1117:13seancorfieldI'm still interested in TBUILD-6 but we have the new stuff in 1.12 that makes tool invocation easier which would get us part of the way there...#2023-08-1117:21seancorfieldBTW, there's a #C02B5GHQWP4 channel (for future tools.build Qs).#2023-08-1117:46telekidah, great. Thank you!#2023-08-1405:41uochan@alexmiller
I'm not sure this channel is suitable for repoting this problem, but the linux installation script seems breaking.
For example, https://download.clojure.org/install/linux-install-1.11.1.1369.sh has the following command.
> curl -O https://github.com/clojure/brew-install/releases/download/1.11.1.1369/clojure-tools-1.11.1.1369.tar.gz
This command will create 0 byte file since https://github.com returns 302 .
So the installation script should have curl -L -O ... command to download file correctly.#2023-08-1405:50Alex Miller (Clojure team)yes, I am working on moving towards this atm - there is now a newer version 1.11.1.1379#2023-08-1405:56uochanThank you for your rapid fixing!#2023-08-1722:25greg@UBL24PLE6 have a look at the asdf.
There is https://github.com/halcyon/asdf-clojurewritten by @U08JQ8RD0 to asdf fetching clojure binaries from github.
You can have a globally set version of clojure, and locally set different versions
(although the latter is less relevant for development with Clojure)#2023-08-1813:59Martynas MaciulevičiusHey.
What should I do if I want to clean the built jars and dependencies but there is no target/ folder?
I have this error but it worked a moment ago:
Caused by: java.lang.Exception: Cyclic load dependency: [ /clojure/spec/alpha ]->/clojure/walk->[ /clojure/spec/alpha ]->/clojure/main->/clojure/core/server
I try to start my REPL and this error happens.
Should I nuke my ~/.m2 folder?#2023-08-1814:05Martynas MaciulevičiusMoving ~/.m2 and removing ~/.clojure/ helped. Hm.#2023-08-1814:06Martynas MaciulevičiusRestoring .m2 directory restores the crash.#2023-08-1814:18Alex Miller (Clojure team)I think you'll have to provide more information, this is pretty devoid of context#2023-08-1814:22Martynas MaciulevičiusI was running my REPL just fine and then added a dependency of etaoin/etaoin {:mvn/version "1.0.40"}
I decided to restart my REPL and got that error.
I don't have a specified clojure version in my deps.edn file and I think that there could've been an upgrade of Clojure jar when I restarted my REPL.
I don't have the log of newly started REPL that crashed.#2023-08-1814:48Alex Miller (Clojure team)"added a dependency" how?#2023-08-1815:01Martynas MaciulevičiusI added it to deps.edn file and saved the file. I didn't try to add it via dynamic way.#2023-08-1815:05Alex Miller (Clojure team)if you remove it, do you still have the problem?#2023-08-1815:12Martynas MaciulevičiusI had the problem still.#2023-08-1815:12Martynas MaciulevičiusThis is why I started to think about caches and removing them. And it worked when I removed ~/.m2 cache and ~/.clojure cache.
And then I restored the error by moving old .m2 directory back into its place.#2023-08-1815:14Alex Miller (Clojure team)well, that doesn't make much sense to me for .m2, it's going to just download the same exact jars#2023-08-1815:15Alex Miller (Clojure team)if it's classpath related, then .clojure might help, but you would want to understand why the classpath is causing the problem#2023-08-1815:15Alex Miller (Clojure team)are you compiling your current project such that it has classes that are produced?#2023-08-1815:15Martynas MaciulevičiusOh right. Right after the error happened I also removed project-dir/.cpcache directory first.
Before removing all other ones.
And then the error was still there.#2023-08-1815:16Martynas Maciulevičius> are you compiling your current project such that it has classes that are produced?
I only ran REPL. I don't even have the config for that yet.#2023-08-1815:16Martynas Maciulevičiusdev {:doc "Start REPL"
:task (if (fs/windows?)
(clojure "-M:nrepl")
(shell "rlwrap bb clojure -M:nrepl"))}
This is what's run:
clojure -M:nrepl
:nrepl ;; added by neil
{:extra-deps {nrepl/nrepl {:mvn/version "1.0.0"}
cider/cider-nrepl {:mvn/version "0.28.7"}}
:extra-paths ["resources-test"]
:main-opts ["-m" "nrepl.cmdline" "--interactive" "--color"]}
Edit: I updated the snippet to match the versions I had during the error.#2023-08-1815:19Alex Miller (Clojure team)can you look at clojure -X:deps tree :aliases '[:nrepl]'#2023-08-1815:20Martynas MaciulevičiusBut now I don't have the old .m2 dir anymore. I deleted it 😕#2023-08-1815:20Martynas Maciulevičiusorg.clojure/clojure 1.11.1
. org.clojure/spec.alpha 0.3.218
. org.clojure/core.specs.alpha 0.2.62
babashka/babashka.curl 0.1.2
clj-tagsoup/clj-tagsoup 0.3.0
. org.clojure/data.xml 0.0.3
. org.clojars.nathell/tagsoup 1.2.1
. net.java.dev.stax-utils/stax-utils 20040917
org.clj-commons/hickory 0.7.3
. org.jsoup/jsoup 1.15.2
. quoin/quoin 0.1.2
dev.weavejester/medley 1.7.0
etaoin/etaoin 1.0.40
. org.clj-commons/clj-http-lite 1.0.13
. org.clojure/tools.logging 1.2.4
. org.clojure/tools.cli 1.0.214
. slingshot/slingshot 0.12.2
. clj-http/clj-http 3.12.3
. org.apache.httpcomponents/httpcore 4.4.14
. org.apache.httpcomponents/httpclient 4.5.13
X org.apache.httpcomponents/httpcore 4.4.13 :older-version
. commons-logging/commons-logging 1.2
X commons-codec/commons-codec 1.11 :older-version
. org.apache.httpcomponents/httpclient-cache 4.5.13
. org.apache.httpcomponents/httpclient 4.5.13
. commons-logging/commons-logging 1.2
. org.apache.httpcomponents/httpasyncclient 4.1.4
X org.apache.httpcomponents/httpcore 4.4.10 :older-version
. org.apache.httpcomponents/httpcore-nio 4.4.10
X org.apache.httpcomponents/httpcore 4.4.10 :older-version
X org.apache.httpcomponents/httpclient 4.5.6 :older-version
. commons-logging/commons-logging 1.2
. org.apache.httpcomponents/httpmime 4.5.13
. org.apache.httpcomponents/httpclient 4.5.13
. commons-codec/commons-codec 1.15
. commons-io/commons-io 2.8.0
. slingshot/slingshot 0.12.2
. potemkin/potemkin 0.4.5
. clj-tuple/clj-tuple 0.2.2
. riddley/riddley 0.1.12
. cheshire/cheshire 5.11.0
. com.fasterxml.jackson.core/jackson-core 2.13.3
. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.13.3
. com.fasterxml.jackson.core/jackson-core 2.13.3
. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.13.3
. com.fasterxml.jackson.core/jackson-core 2.13.3
. tigris/tigris 0.1.2
. babashka/process 0.3.11
X babashka/fs 0.1.6 :older-version
. babashka/fs 0.2.12
nrepl/nrepl 1.0.0
cider/cider-nrepl 0.35.0
X nrepl/nrepl 1.0.0 :use-top
. mx.cider/logjam 0.1.1#2023-08-1815:20Alex Miller (Clojure team)I'm on vacaction, so have to run, but I suspect something in the nrepl deps that is affecting your classpath#2023-08-1815:21Martynas MaciulevičiusIt's fine.
Thanks anyway.
Edit: Have a good vacation :))#2023-08-1816:09zarkonehi hi 👋 is it possible to specify a certain alias of a local/root dependency in deps.edn ? something like
{:local/root "../project-a"
:aliases [:dev]}#2023-08-1816:25seancorfieldNo, only top-level aliases can be used with the CLI -- no transitive aliases.#2023-08-1816:40zarkoneGot it, thanks!#2023-08-1904:15vemvHow can I create a basis for an absolute path if the JVM's current directory is unrelated?
e.g.
user> (tools.deps/create-basis {:project "/Users/vemv/foo/bar/deps.edn"})
Execution error (ExceptionInfo) at clojure.tools.deps.extensions.local/ensure-file (local.clj:41).
Local lib yyy not found: xxx
what's failing is that a dependency is defined like this: foo/bar {:local/root "../../baz"} so it's tripping up due to my unrelated CWD.
(names redacted)#2023-08-1904:21Alex Miller (Clojure team)You can wrap in c.t.deps.util.dir/with-dir#2023-08-1904:21Alex Miller (Clojure team)It will use that as if it was the cwd#2023-08-1904:22vemvThank you! 🙏
You might want to consider an addition to the create-basis docstring#2023-08-1904:22Alex Miller (Clojure team)You shouldn’t need to set the :project then either#2023-08-1904:43vemv> You shouldn’t need to set the :project then either
I tried this and my tests began failing 🥲#2023-08-1904:58Alex Miller (Clojure team)Well I could be crazy but I would expect it to use deps.edn as the relative path, resolved against the with-dir#2023-08-1905:17vemv(Sorry if FAQ)
Given :deps {x/x {:local/root "../../x"}}, can I pull in its aliases too?
e.g. it defines :dev, I'd like to use its :dev alias from the dependent project
Some extra context: ../../x is not under my control (not without forking) so I can't simply re-architect things#2023-08-1905:30Alex Miller (Clojure team)Not currently#2023-08-1905:33vemvThanks, I voted on https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects#2023-08-2200:14seancorfieldhttps://stackoverflow.com/questions/76943471/clojure-deps-edn-not-resolving-dependency -- after a brief poke at this, it seems that something, somewhere in the chain of dependencies is requesting a version of babel__highlight that doesn't actually exist (7.x seems to be the highest version on Maven) and it looks like the waltid deps have a lot of <scope> specifiers in their various POM files so I'm wondering if this is due to tools.deps trying to follow a dependency that is marked with a <scope> that it should perhaps be ignoring?
What would be an "easy" way to debug this, given that resolution ultimately fails?#2023-08-2200:22Alex Miller (Clojure team)tdeps should take compile and runtime scopes#2023-08-2200:23Alex Miller (Clojure team)I would probably debug it by loading tdeps in the repl and using a modified version of one of those commented out examples with a debugger#2023-08-2200:24Alex Miller (Clojure team)https://github.com/clojure/tools.deps/blob/be30a1ae275eabfd1eba571080e039451c122c69/src/main/clojure/clojure/tools/deps/extensions/pom.clj#L75 is where the scope filter is set#2023-08-2200:24seancorfieldThere's no "magic" debug or trace option that makes it write out resolutions as it goes?#2023-08-2200:24Alex Miller (Clojure team)no#2023-08-2200:25seancorfieldLooks like the waltid ssikit pom is all compile/runtime so that's safe. But it depends on a ton of stuff so figuring our where in the full tree it breaks is going to be tedious 🙂#2023-08-2200:25seancorfieldIt's not my problem -- I just wanted to help the person on SO 🙂#2023-08-2200:27seancorfieldIt feels like some sort of "debugging mode" for deps resolution might be useful for situations like this -- instead of just "error! failed!" -- would you entertain a patch that added some sort of logging/trace mode that could produce output during resolution to help debug situations like this?#2023-08-2201:00seancorfieldWith your pointer, I modified trace+ to print entry which helped... turns out that because tools.deps picks the most recent version available on transitive deps, it managed to find org.webjars.npm/babel__code-frame 8.0.0-alpha1 which depends on a version range for babel__highlight that has not been released yet!#2023-08-2201:02seancorfieldSo I pinned that to 7.21.4 but I'm seeing quite a few
Download corrupted: Checksum validation failed, no checksums available
which doesn't seem to stop the CLI from continuing... I assume that can be "safely" ignored?#2023-08-2201:18Alex Miller (Clojure team)Re debug, yes, but just has not seemed worth doing as I rarely need it (and I usually end up debugging such things anyway)#2023-08-2201:28Alex Miller (Clojure team)re checksums, I assume that's continuing because it's only a warning to be missing the checksums, but I haven't seen that before#2023-08-2201:30seancorfield(the printout from trace+ "helped" but still needed a fair bit of intuition about "how this all works" for me to narrow down the problem to a handful of dependencies that I then had to go check on Maven manually -- so I suspect you're right that this isn't widely useful)#2023-08-2201:30seancorfieldThank you for the valuable pointers tho'!#2023-08-2201:33Alex Miller (Clojure team)the info you need is often context dependent when debugging a problem and too much can be more harmful than helpful so... not an urgent need#2023-08-2419:36agI want the following structure of my project:
src
└── some_proj
├── backend
│ └── system.clj
├── dev
│ ├── config.edn
│ └── user.clj
└── frontend
└── app.cljs
And I want to separate frontend, backend, dev dependencies. Like so:
{:paths ["resources"]
:deps {org.clojure/clojure {:mvn/version "1.12.0-alpha4"}}
:aliases
{:backend
{:extra-paths ["src/some_proj/backend"]
:extra-deps {,,,}}
:frontend
{:extra-paths ["src/some_proj/frontend"]
:extra-deps {,,,}}
:dev
{:extra-paths ["src/some_proj/dev"]
:extra-deps {,,,}}}}
And now, if I do clj -A:backend, in the repl I should be able to (require '[some-proj.backend.system]). But it doesn't see it in the classpath. I think I miss something in my understanding of extra-paths.#2023-08-2419:41hiredmanPaths are classpath roots#2023-08-2419:41dpsuttonif the classpath root is src/some_proj/backend, if you are looking for some-proj.backend.system it will look at <classpath-root>/some_project/backend/system.clj#2023-08-2419:41hiredmanThe way namespaces are mapped to files is relative to roots#2023-08-2419:42dpsuttonwhich would be src/some_project/backend/some_proj/backend/system.clj#2023-08-2419:43agCan you guys please tell me what am I doing wrong?#2023-08-2419:49favilaYou cannot do this with paths: where the path “ends” becomes the root for class lookup, which in your config will never match the namespace. You need multiple roots if you want to do this with paths#2023-08-2419:51agSo what's the idiomatic way of splitting dependencies with deps.edn?#2023-08-2419:51favilamore deps.edn#2023-08-2419:52favilaand :local/root#2023-08-2419:53favilaif you want to keep them all in the same dir, you can do this, but you may have trouble consuming artifacts made by this (maybe that doesn’t matter if it’s an application)#2023-08-2419:53favilasrc-backend
└── some_proj
└── backend
└── system.clj
src-dev
└── some_proj
└── dev
├── config.edn
└── user.clj
src-frontend
└── some_proj
└── frontend
└── app.cljs
{:aliases
{:backend
{:extra-paths ["src-backend"]}
:frontend
{:extra-paths ["src-frontend"]}
:dev
{:extra-paths ["src-dev"]}}#2023-08-2419:54favilabut at this point, why separate classpaths at all?#2023-08-2419:55seancorfieldI guess the question is really: what are you trying to achieve (and why)?#2023-08-2419:55agSo I can deploy front-end completely separately from the backend#2023-08-2419:56seancorfieldWell, that's about what goes into your deployable artifacts -- so it's all about how you build things, not how the source code is structured.#2023-08-2419:58dpsuttonand if you want stuff in separate folders, just use a structure that has backend, frontend , and dev folders and make each of those an item in :extra-paths ["backend"] …#2023-08-2420:00dpsuttonand then make sure the namespaces are in the correct structure underneath backend/ frontend/ or dev/#2023-08-2420:01ag@seancorfield so like for example, if I want to have a "test build" that can separately build front-end and test it without bundling dependencies required for the server. And vice-versa, I don't want to include the test scripts into the main (prod) front-end build artifact.#2023-08-2420:02dpsuttonnote this is exactly @U09R86PA4’s suggestion above. except he used src-backend#2023-08-2420:02favila@U0G75ARHC You’re skipping a step: how will you make those builds? deps.edn doesn’t do this automatically#2023-08-2420:03dpsuttonyou can pass aliases into it right? basis takes aliases and everything should work fine from there on out?#2023-08-2420:03dpsuttonor will copy source still have some issues#2023-08-2420:03favilathat’s definitely a way to go, and works for application artifacts#2023-08-2420:04favilait doesn’t work if it’s supposed to be a git-consumed or local-root consumed things because aliases aren’t transitive#2023-08-2420:04dpsuttonyeah. can’t specify aliases for dependencies. i think there’s an http://ask.clojure.org ticket for it though#2023-08-2420:04favilabut the point is many folder and deps.edn structures can support many different artifact variants and vice-versa#2023-08-2420:06ag> deps.edn doesn’t do this automatically
so like if I use clojure.tools.build.api and all my code (front, back, dev, tests) are all in the same place, wouldn't it make it difficult to build multiple artifacts? i.e., I wouldn't want to include compiled and minified .js files in my server-side uberjar#2023-08-2420:07dpsuttonyou are pretty easily in control of what exactly goes in your jar. you build it, clean the directory, copy sources, etc into your jar.#2023-08-2420:10agyeah, good point. Maybe why bother then by separating the files at the root level?#2023-08-2420:10agThank you, everyone. You are all very helpful. As always.#2023-08-2421:09genekimAfter years of being dazzled by people using add-libs (looking at you, @seancorfield), I decided to give it a try.
I first tried using Clojure 1.12.0-alpha2, but got this error:
(deps/add-libs '{babashka/fs {:mvn/version "0.4.19"}})
Namespace clojure.tools.deps loaded but function not found: resolve-added-libs
I then tried using Clojure 1.12.0-alpha4, and got a different error:
(deps/add-libs '{babashka/fs {:mvn/version "0.4.19"}})
Execution error (ExceptionInfo) at clojure.tools.deps.interop/invoke-tool (interop.clj:49).
null
Has anyone else had this error, or have any recommendations on how to make this work? Thank you!#2023-08-2421:10hiredmanyou need a newer version of the cli#2023-08-2421:11genekimWhoa. I’m using this inside of Cursive/IntelliJ. That never occurred to me — trying that now!#2023-08-2421:11hiredmanadd-libs shells out to the cli#2023-08-2421:12genekim(While I’m watching homebrew upgrade/update install…. @U0NCTKEV8 that this depends on shelling out to CLI would have NEVER occured to me!)#2023-08-2421:13seancorfieldIt requires CLI 1.11.1.1347 (or later)...#2023-08-2421:16seancorfieldhttps://clojurians.slack.com/archives/C06MAR553/p1681479874896649?thread_ts=1681479653.194879&cid=C06MAR553 -- looks like 1.11.1.1273 or later from that announcement but I think on Windows it needs 1.11.1.1347 for the "arguments passed via stdin" feature.#2023-08-2421:17seancorfieldHere's the invoke-tool function it uses under the hood (mentioned in one of your errors): https://clojure.github.io/clojure/branch-master/clojure.tools.deps.interop-api.html#2023-08-2421:24genekimOMG. It works!! Thank you, all!!!!#2023-08-2421:24genekim(need to step out for a sec, but will respond more fully later today. 🙏 )#2023-08-2421:26seancorfieldhttps://ask.clojure.org/index.php/13196/the-new-add-lib-etc-fns-not-make-it-clear-they-shell-out-the-cli#2023-08-2609:12vemvI can ask it but perhaps it's easy enough
What I am doing wrong here?
user> (tools.deps/resolve-deps {:deps '{com.google.cloud.opentelemetry/exporter-trace {:mvn/version "0.25.2"}}}
{})
Execution error (ArtifactNotFoundException) at org.eclipse.aether.internal.impl.DefaultArtifactResolver/resolve (DefaultArtifactResolver.java:496).
Could not find artifact org.apache:apache:pom:21
When depending on com.google.cloud.opentelemetry/exporter-trace via deps.edn, it works fine and it gets added to the classpath.
But with programmatic usage, it chokes on https://mvnrepository.com/artifact/org.apache/apache/21 , which to be fair looks like an odd dependency:
[org.apache/apache "21" :extension "pom"]
I have tried different variations of programmatic usage, no luck#2023-08-2612:32Alex Miller (Clojure team)The last argument of resolve-deps is the procurer config - you’re passing an empty map which means you are defining no maven repos, so nothing will be found#2023-08-2612:35Alex Miller (Clojure team)You can instead pass the var defining the standard repos https://github.com/clojure/tools.deps/blob/52a1e172457fe934bbcfb8e5a216ecd1f37590fa/src/main/clojure/clojure/tools/deps/util/maven.clj#L61#2023-08-2612:36Alex Miller (Clojure team)Or rather, those need to be in the first arg (see https://github.com/clojure/tools.deps/blob/52a1e172457fe934bbcfb8e5a216ecd1f37590fa/src/main/clojure/clojure/tools/deps.clj#L955 example)#2023-08-2613:54vemvThanks, Alex! specifying :mvn/repos worked.
Perhaps the behavior when they're nil/empty could be improved? In this thread I showed one way in which errors won't point in that direction. In this other question I posted another https://ask.clojure.org/index.php/13201/clojure-tools-deps-calc-basis-unable-resolve-version-range
It would probably make sense to fail-fast when performing maven ops with no repos backing them? Or at least, include that info prominently in the ex-msg / ex-data.#2023-08-2616:09Alex Miller (Clojure team)It's important to separate the CLI (which provides a lot of assumptions and defaults) from the lib. The lib is a more general tool and you have to provide more context. You don't need to pass any maven repos if you're doing git-only stuff for example, so it's not an error to not have repos. If you get unable to resolve, that's what it means - the lib you asked for can't be found in the repos given. There's no way to know if that's because you didn't provide the right repos or if the lib doesn't exist.#2023-08-2616:09Alex Miller (Clojure team)even if you don't have repos you might be able to serve libs already in the local cache#2023-08-2619:06vemvYes, I understand those principles, and in particular that there isn't a hard-and-fast way to determine the cause of a failure.
However by including the :mvn/repos in the ex-data, users can make a more educated guess of what's going on.#2023-08-2619:19Alex Miller (Clojure team)well, I think you're probably wrong because no one actively looks at the ex-data#2023-08-2619:20vemvIDK, we Clojurists are a diverse and generally smart bunch :)#2023-08-2619:52Alex Miller (Clojure team)well that is true#2023-08-2615:31vemvDoes the clojure.tools.deps ns have a function that given a "-M:what:ever" , will return (or at least include somewhere) the main program that will be executed? i.e. the name of the underlying java class#2023-08-2615:56Alex Miller (Clojure team)https://clojure.github.io/tools.deps/#clojure.tools.deps/create-basis is really the primary entry point now and it takes :aliases vector. all merged and resolved alias options will be in the :argmap of the return map#2023-08-2615:57Alex Miller (Clojure team)so (-> (create-basis {:aliases [:what :ever]}) :argmap :main-opts) should be something like ["MainClass" ...]#2023-08-2619:33vemvThanks! I got it working#2023-08-2719:26rsI'm trying to create a new project with clj-new. In ~/.clojure/deps.edn [copied from practicalli] .. the exec fn looks like this
:project/new
{:replace-deps {com.github.seancorfield/clj-new {:mvn/version "1.2.399"}}
:exec-fn clj-new/create
:exec-args {:template app :name practicalli/playground}
:main-opts ["--main" "clj-new.create"]}
The following command does not work
clj -M:project/new :template lib :name mygroup/myproject
The following does
clj -M:project/new lib mygroup/myproject
But as per https://clojure.org/reference/deps_and_cli
I should be using
clj -M:project/new :template lib :name mygroup/myproject
to override arguments
Any thoughts on what's going wrong?#2023-08-2720:04Alex Miller (Clojure team)"does not work" == ?#2023-08-2720:04Alex Miller (Clojure team)if you're using :exec-fn and :exec-args, then those are only used with clj -X#2023-08-2720:05Alex Miller (Clojure team)perhaps you just need to s/-M/-X/#2023-08-2720:08rsI see. Let me try it out. Thks!#2023-08-2720:55seancorfield@U0AHTPQBG You might want to look at the clj-new documentation itself: https://github.com/seancorfield/clj-new#2023-08-2813:26practicalli-johnThe alias is working correctly when the appropriate argument types and execution option is used
-M uses string arguments
-X uses key value pair arguments
Unless a specific Leiningen template is required, I suggest using the deps-new project.
Practicalli Clojure CLI Config defines the :project/create alias to use this project, which also includes additional templates
https://practical.li/clojure/clojure-cli/projects/templates/#2023-08-2820:08rs@U05254DQM thks yes that was my experience as well as u specified.#2023-08-3119:55vemvAre there deps that don't have a mvn/version, git/whatever or local/root?
i.e. is there some other, less-obvious type#2023-09-0100:48rsivy repositories but I don't think deps support it#2023-08-3119:58Alex Miller (Clojure team)no#2023-08-3120:00Alex Miller (Clojure team)at the lib level it's an open system so you could use tools.deps and add your own#2023-08-3120:03vemvthanks!#2023-09-0114:18Noah BogartI have been editing a java file in combination with clojure code, which means i have to recompile and restart my repl. is there a way to run clojure -T:build compile-java && clojure -M:dev:test:repl in a single call? it's annoying to have to wait for clojure to spin up twice#2023-09-0114:22Alex Miller (Clojure team)have you heard the good word about the repl?#2023-09-0114:22Alex Miller (Clojure team)you could run the build in your repl, then restart#2023-09-0114:23Noah Bogartcare to explain a little? I thought i had to restart the repl to get the latest version of the compiled java class#2023-09-0114:23Alex Miller (Clojure team)you do (well, depends on your editor - Cursive can hot reload classes) but the tools.build call is ultimately just running a function#2023-09-0114:24Alex Miller (Clojure team)you might (depends on your deps) be able to include the :build alias in your repl (+ "." in :paths) and then just (build/compile {})#2023-09-0114:25Alex Miller (Clojure team)b/c tools.build will bring in a lot of deps, these might conflict with your app, but also they might not :)#2023-09-0114:25Alex Miller (Clojure team)OR you could do the same in a separate repl that you keep running for hte purpose of rebuilding#2023-09-0114:26Alex Miller (Clojure team)this is where "builds as programs" comes home :)#2023-09-0114:31Noah Bogartcool, that all makes sense#2023-09-0118:26seancorfieldFWIW, I run a REPL for my build script 24x7 separate from my application so that I can run pipelines of tasks easily. I start my "build REPL" like this:
clj -M:build -i build.clj -e "(in-ns 'build)" -r
and I make sure all my build.clj functions return their options hash map argument so I can thread them easily. Although you'd still have to wait for your clojure -M:dev:test:repl command to spin up, at least you could use the "build REPL" to run:
build=> (compile-java {})
and not have to wait for it to spin up every time. An example of something I run in the "build REPL":
(->build=> {} (check-all) (ancient) (cve-check) (cold-start) (test-stable) (build-uberjars))#2023-09-0120:23Noah Bogartthat's smart#2023-09-0816:05danielszhttps://meyvn.org will watch a directory of Java source files, and when modification occurs, it will recompile the affected class and its dependent classes. It achieves this by doing static analysis not on source files but on compiled classes. By keeping a dependency graph, it knows which classes need to be reloaded and in what order. This is a seamless experience. Here's a https://www.youtube.com/watch?v=xIuJ0f1Vqek.#2023-09-1209:36vemvVery late to the party, but this is a perfect use case for Makefile:
.javac: $(wildcard java/foo/*.java)
clojure -T:build compile-java
repl: .javac
clojure -M:dev:test:repl
You will only compile java if some java file changed.#2023-09-1213:20Noah Bogartis that what wildcard does? I've never heard of that before#2023-09-1213:31vemvwildcard allows you to depend on multiple files (using glob-like syntax), or files that may be absent (which, if present, would invalidate the 'cache')#2023-09-1213:31Noah Bogartthat's very cool#2023-09-0906:26markbastianIs it possible to add a dependency from the command line and invoke -X with an alias at the same time? I have the following invocation (generalized):
clojure -Sdeps '{:deps {com.github.markbastian/myproject
{:git/url ""
:sha "SOMEGITSHA"}}}' \
-X:myalias myproject.main/run
When I run the command, I get the message: WARNING: Specified aliases are undeclared and are not being used: [:myalias]. How might I do this? I can invoke clj -X:myalias myproject.main/run from within the project just fine. The alias is definitely in the deps edn associated with the sha of the commit.#2023-09-0911:47dpsuttoni think because you are clobbering the deps.
❯ cat deps.edn
{:deps {}
:aliases
{:demo {:exec-fn clojure.core/println}}}
❯ clj -Sdeps '{:aliases {:crit {:extra-deps {criterium/criterium {:mvn/version "LATEST"}}}}}' -X:crit:demo :a 1
{:a 1}#2023-09-0911:47dpsuttonput the extra stuff in an alias#2023-09-0911:50dpsuttonactually, it seems fine in your original case for me:
# a dep in the deps.edn file
❯ cat deps.edn
{:deps {com.github.seancorfield/honeysql {:mvn/version "2.4.1066"}}
:aliases
{:demo {:exec-fn clojure.core/println}}}
# check the classpath and both are there
❯ clj -Sdeps '{:deps {criterium/criterium {:mvn/version "LATEST"}}}' -Spath
src:/Users/dan/.m2/repository/com/github/seancorfield/honeysql/2.4.1066/honeysql-2.4.1066.jar:/Users/dan/.m2/repository/criterium/criterium/0.4.6/criterium-0.4.6.jar:/Users/dan/.m2/repository/org/clojure/clojure/1.11.1/clojure-1.11.1.jar:/Users/dan/.m2/repository/org/clojure/core.specs.alpha/0.2.62/core.specs.alpha-0.2.62.jar:/Users/dan/.m2/repository/org/clojure/spec.alpha/0.3.218/spec.alpha-0.3.218.jar
# can run the exec fn with the `demo` alias
❯ clj -Sdeps '{:deps {criterium/criterium {:mvn/version "LATEST"}}}' -X:demo :a 1
{:a 1}#2023-09-0913:28dpsuttonOh. Are you wanting to use an alias defined in a dependency?#2023-09-0913:29markbastianYes, the alias is defined in myproject above.#2023-09-0913:29dpsuttonGotcha. I don’t believe that is supported#2023-09-0913:30markbastianAh, 😞#2023-09-0913:30markbastianI can just rearrange my deps file for this.#2023-09-0913:30dpsuttonPut an alias with the dep and exec fn info in your .clojure/deps.edn file and you can invoke it easily#2023-09-0913:31dpsuttonNo need for inline deps and such at that point#2023-09-0913:31markbastianCool, will do!#2023-09-0913:32dpsuttonI imagine the reasoning is that aliases are not namespaced. So your class path could have arbitrarily many aliases with the same same#2023-09-0913:32markbastianMakes sense, I'm sure there are a lot of :build aliases out there.#2023-09-0913:33dpsuttonAnd lots of :dev. It would be cool if you could specify them with the dep name. But I suppose you’d have to be careful with which version is selected if multiple are specified by your dependency tree#2023-09-0913:35markbastianYeah, my thought last night (exactly what you suggested) was something like:
clojure -Sdeps '{:aliases {:my-tool {com.github.markbastian/my-app
{:git/url ""
:sha "..."}}}}' \
-X:my-tool:my-alias my-app.main/run#2023-09-0913:36markbastianBut that has some ambiguity about the nested aliasing. For now I'll just do exec fn.#2023-09-1209:40vemvAre repls bad to launch from a Makefile? I found this SO thread with that repeated piece of advice https://stackoverflow.com/questions/70267017/no-repl-prompt-when-running-from-make
(my question isn't tools.deps-specific per se, but Makefiles aren't rare here)#2023-09-1209:43dpsuttoni think the main thing to consider is that repls can take lots of arguments. aliases, main opts, etc. socket repl config, env variables before the command, etc. For people using nrepl, they have nrepl options and middleware to dynamically add it.
For these reasons, I usually find that being unable to modify how a repl is started very frustrating. And the more layers of tooling involved, the more documentation you have to read. You need to understand how tool X preserves information to pass through tool Y so that the repl is started as clojure -A:dev . To me, clojure repls are so simple to start that I don’t usually see benefit from other tools between me and the repl.#2023-09-1209:47vemvGot it. However one can parameterize clojure aliases e.g. ALIASES=:dev:test:cider make repl .
(Granted, that's a layer of indirection, but please assume there's a good reason to do this whole thing)
Most of all, I'd be interested in a strong technical reason why running an interactive progam through Make would be problematic.#2023-09-1209:49dpsuttonyeah. i do that with clj -M:"$ALIASES". I just don’t see what make would give me. and i have to know which env vars it expects and propagates.
I don’t know anything about how make interacts with long running interactive processes. I see the one point about std in being randomly allocated to one of the threads which seems interesting. No idea otherwise#2023-09-1209:55vemvThat's when using make -j (parallel execution), personally I've never even used that#2023-09-1209:57dpsuttoni’ve seen that as almost always used when making projects. no reason to leave idle threads when building large projects#2023-09-1210:00vemvYeah, that's fair. Most Makefiles I've seen in Clojure contexts are concerned with standardizing usage patterns - there's rarely much to be 'built'.#2023-09-1212:27practicalli-johnI've worked a lot with developers who are not using Clojure regularly and make is a simple enough tool to create a consistent command line UI across a wide range of languages and tooling.
A Makefile with standard task was actually required in my previous commercial role. A project wouldnt be deployed to test or production without those make tasks.
I like the convenience of make repl , make test-watch , make docker-build, etc so I dont have to remember all the commands and aliases I crafted for the specifics of the current project (especially all the docker stuff)
The Makefile also provides a concise way to document common commands. The makefile can then be used as a reference for commands that can be used outside of make (also using the Shell history to minimise typing once the commands are used). Of course commands could be put in a well organised readme laughcry
I've also never knowingly used the multiple jobs approach with make, especially for Clojure (although I assume they are more common building larger C/C++ code bases, perhaps make for Emacs uses multiple jobs). I didnt see any other reasons to avoid make for interactive commands offered in the Stack Overflow discussion.
So in summary, I dont see any problems using make to start a repl. If you find make valuable, then use it. Providing make tasks doesnt prevent anyone from using (or modifying) the underlying commands if they prefer.
I also include a Makefile with standard tasks in projects created with https://github.com/practicalli/project-templates#2023-09-1213:18vemvThanks much! That's reassuring if you've used make repl substantially.#2023-09-1213:30Ingy döt Netfwiw here's an example of something I do quite a bit:
LEIN := clean compile deploy install jar \
release repl retest run test uberjar
$(LEIN):
lein $@
will add those lein commands as Makefile targets, so you can make repl etc.#2023-09-1213:43Ingy döt NetThose lines would just be a part of my Makefiles which typically do much more.
This technique can obviously work with any kind of CLI tooling that has subcommands that you want to expose as make targets.
I use Makefiles extensively and even though I'm pretty new to Clojure they have worked out well there.#2023-09-1213:43vemvThanks!#2023-09-1213:46Ingy döt NetAlso probably better to write it as
$(LEIN)::
lein $@
the :: allows you to add to a target's commands by later doing
test::
another-test-command#2023-09-1213:58Ingy döt NetPS. Here's a shell trick that often works for me when a process has lost its tty stdio for some reason; like being call from a process whose stdio is already being piped or redirected elsewhere:
lein repl < /dev/tty &> /dev/tty
you can test it with
echo | (lein repl < /dev/tty &> /dev/tty) > /dev/null
which should work as normal, but these:
echo | (lein repl) > /dev/null
echo | (lein repl)
(lein repl) > /dev/null
do not work as normal.#2023-09-1214:03Ingy döt Netbut this isn't necessary for running shell commands in make (at least GNU make)#2023-09-1214:10vemvThanks, I've saved it :)#2023-09-1214:24practicalli-johnFor reference, this is the basic approach I take with a Makefile
https://practical.li/engineering-playbook/build-tool/make/
There is also a great tutorial at https://makefiletutorial.com/#2023-09-1221:25pithylessIf you'll allow a small tangent in this thread, the arguments around using Makefiles as an easy-to-expect "least common denominator" of CLI tooling, reminds me of the https://testanything.org/ for trying to standardize test outputs across languages. I see parallels to both their pros of standardization and cons of inflexibility to language-specific tooling and granularity.#2023-09-1222:01Ingy döt NetI use Makefiles and TAP testing often in at least half a dozen languages. Big fan of both.#2023-09-1222:03Ingy döt NetAs another tangent this is a pretty cool extension to GNU make that I whipped up a few years back http://makeplus.net/#2023-09-1222:04Ingy döt NetI've gone pretty deep into make and bash. If anyone has issues with either ping me. I love that stuff.#2023-09-1222:05pithylessI will freely admit, that my lizard brain quickly reaches for babashka if at all possible. :)#2023-09-1222:06Ingy döt NetI made a library distribution system for bash last year called bpan. Mostly for reusing the bash libraries I've created over the years.#2023-09-1222:09Ingy döt NetWell as I understand it but babashka is just a way to run bash commands generated from a Clojure syntax.
One of the things that attracted me to Clojure was the way it interopped with existing technologies. So far my ambitions are to use it to get good ideas out to other languages, not to avoid other languages by doing everything from a clojure point of view.#2023-09-1222:12pithylessYou're right; but I specifically use it so I don't have to think about and improve my bash-fu (and can just pretend it's another Clojure program). 😅#2023-09-1222:13pithylessBut now we're really on a tangent of a tangent; apologies to others subscribed to this thread. :)#2023-09-1222:13Ingy döt NetI don't know anyone who does more bash than me and I'm still learning every day so your approach makes sense to a lot of people I would think.#2023-09-1222:14Ingy döt Net@U05476190 as long as we're this far off track https://github.com/bpan-org/test-tap-bash is my bash tap framework 😉#2023-09-1221:03Joshua Suskalohttps://ask.clojure.org/index.php/13278/tagged-literals-in-clojure-cli-arguments-and-deps-edn#2023-09-1406:35seancorfieldIs there an easier way to do this?
clojure -A:deps -Sdeps '{:deps {my/lib {:local/root "."}}}' -X help/doc :ns next.jdbc.sql
(assuming I'm in the next.jdbc repo and want to display all the docs for the specific ns)
The "problem" is that the :deps alias removes the project context:
https://github.com/clojure/tools.deps/blob/master/src/main/resources/clojure/tools/deps/deps.edn#L9-L10
I'm not sure what the "best" solution would be here but perhaps a variant of the :deps alias that used :extra-deps (and no paths modifiers)?#2023-09-1406:40seancorfieldWith a user alias like:
:deps+ {:extra-deps {org.clojure/tools.deps.cli {:mvn/version "RELEASE"}}
:ns-default clojure.tools.deps.cli.api
:ns-aliases {help }}
it becomes much more manageable:
clojure -X:deps+ help/doc :ns next.jdbc.sql
#2023-09-1412:15Alex Miller (Clojure team)What are you actually trying to do? View the docstrings of namespaces in your own project?#2023-09-1416:19seancorfieldYes, I saw some toot (on Mastodon) about the power of clj -X and I was going to respond with "did you know you can get the docs about a namespace / function from the command-line?" but then I tried it and found it was harder than I expected.#2023-09-1416:21seancorfieldIn my various projects's build.clj files, I have a note in the docstring that you can get help on the build tasks with clojure -A:deps -T:build help/doc and my various tools projects say how to get help on the tool, e.g., clojure -A:deps -Tnew help/doc so I was hoping there was an "equivalent" for a project...#2023-09-1416:24seancorfield(I "solved" this by adding :deps+ as shown above to my user deps.edn file but I think it would encourage folks to start using help/doc more if it was built-in, to make it "easier" -- right now, a lot of folks using the CLI don't even know about help/doc, it seems, and have the complaint that discoverability of tasks to run is "hard")#2023-09-1620:08jumarIs anybody building WARs with tools.deps?
I found an older question about it here: https://ask.clojure.org/index.php/11341/possible-build-uberwar-using-deps-using-tools-build-other-tool
• links https://clojure.atlassian.net/browse/TBUILD-21 which doesn't show much progress.
I'm looking for something like this plugin for leiningen: https://github.com/luminus-framework/lein-uberwar#2023-09-1620:25Alex Miller (Clojure team)I don't know of anyone that's made this, I am not actively working on it. war files are just jar files with special stuff in it. the hard parts of making a war are gathering the dep jars, and the jar packaging, which are both pretty solved problems in tools.build. the presumably not hard (but context I lack) is what other files people typically bundle into the war and where they would get them from#2023-09-1621:44seancorfieldI was just looking at lein-uberwar inspired by the update to that Ask Q... it wouldn't be hard to turn it into a build.clj utility that could be used easily with the CLI... but I haven't needed to use WAR files for years so I don't have any motivation to tackle the conversion, I'm afraid.#2023-09-1622:25Alex Miller (Clojure team)well, me too :)#2023-09-1708:39practicalli-johnIf all the required files have been created to deploy the web app on a specific application server, then they could be zipped up into a war file
The only thing specific about a war file is its contents, which can often require web application server specific info (although most of the config will be quite similar)
I assume war deployment is for a specific server (e.g. Tomcat, jboss, jetty, websphere, weblogic, glassfish, etc...)
The web application server should have docs to show the expected directory structure of a war and specific config (for entry points, etc)
Once the precise structure and config is known, then I assume tools.build (or java jar command, Ant, Maven, Eclipse, IntelliJ) could be used to generate the war file.
Given that most Clojure projects embed the app server rather than package a war file for deployment, I'd suggest using one of the many existing tools from the Java world (maybe using tools build to still create the jars).
Or consider the value/feasibility of moving the Clojure projects to an embedded web server approach (if not constrained by the business to only use war deployment approach)#2023-09-1718:38seancorfield> Or consider the value/feasibility of moving the Clojure projects to an embedded web server approach
Especially with the sunng87 Ring adapter that now has support for Jetty 12 and virtual threads. Jetty 12 has a stripped down core that doesn't even include the Servlet API and some folks are saying it's 10% faster than Jetty 11 just because of being so stripped down.#2023-09-2216:12seancorfieldI tried to install Polylith as a tool using install-latest and got the following error:
> clojure -Ttools install-latest :lib io.github.polyfy/polylith :as trial
trial: Did not find versions for io.github.polyfy/polylith
I was expecting it to find https://github.com/polyfy/polylith/releases/tag/v0.2.17-alpha as the latest version.
Is this because the top-level deps.edn does not include :tools/usage? (the actual poly tools needs :deps/root "projects/poly" for an install with a specific version -- this started out with me wondering if install-latest could take :deps/root as an option in order to make this work 🙂 )#2023-09-2913:07lispycloudsHello, I'm trying to add https://mvnrepository.com/artifact/com.kohlschutter.junixsocket/junixsocket-core/2.7.2 as a dep as {:deps {com.kohlschutter.junixsocket/junixsocket-core {:mvn/version "2.7.2"}}} and it can't seem to find it:
Error building classpath. Could not find artifact com.kohlschutter.junixsocket:junixsocket-core:jar:2.7.2 in central ()
this is a pom only lib and works in leiningen with [com.kohlschutter.junixsocket/junixsocket-core "2.7.2" :extension "pom"]. do i need to do something different for tools.deps?#2023-09-2913:11Alex Miller (Clojure team)Pom-only libs are not currently supported, but you can include it’s deps instead#2023-09-2913:11lispycloudsyeah thats what i ended up doing. thanks for the conformation!#2023-09-2914:44dergutemoritzYou can vote on this at https://ask.clojure.org/index.php/10892/add-support-for-bom-bill-of-materials-dependencies#2023-10-0217:16karol.adamiecHI,
I am trying to use hashp (https://github.com/weavejester/hashp)
Sadly there is no instruction how to add this when using tools.deps.
I add the lib to my ~/.clojure/deps.edn fine as dependency, and then in cljs file i can add it to current namespace and use.
What i would like is for the #p macro to be available without including the require. Lein has :injections, is there similiart thing with tools.deps and clojurescript?#2023-10-0217:59nbardiuk{:aliases
{:nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "RELEASE"}
cider/cider-nrepl {:mvn/version "RELEASE"}
hashp/hashp {:mvn/version "RELEASE"}}
:main-opts ["-e" "(require,'hashp.core)"
"-m" "nrepl.cmdline" "--middleware" "[cider.nrepl/cider-middleware]"]}}}
I've added a require for hashp with -e flag in the alias I use to start my repl#2023-10-0218:13karol.adamiecthat is a good path i think...
the issue seems to be that i would like that in cljs#2023-10-0218:13karol.adamiecmight be a figwheel main option#2023-10-0407:54kwladykaHow do you deal with Bill of Materials?
https://cloud.google.com/java/docs/bom
{:tag :dependencyManagement, :attrs nil, :content ("\n " {:tag :dependencies, :attrs nil, :content ("\n " {:tag :dependency, :attrs nil, :content ("\n " {:tag :groupId, :attrs nil, :content ("com.google.cloud")} "\n " {:tag :artifactId, :attrs nil, :content ("libraries-bom")} "\n " {:tag :version, :attrs nil, :content ("26.22.0")} "\n " {:tag :type, :attrs nil, :content ("pom")} "\n " {:tag :scope, :attrs nil, :content ("import")} "\n ")} "\n ")} "\n")}
I know there is https://clojure.atlassian.net/browse/TDEPS-202
Because BOM is not supported I have deps manually, and I have Bad type on operand stack
on (.build (.setData (PubsubMessage/newBuilder) (ByteString/copyFromUtf8 (json/write-value-as-string body)))), but only in JAR.
I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar.
I tried clojure.tools.build.api/write-pom and versions look the same as in deps.edn.
Any hints what I can do to fix this? How to debug this to find the issue?
PS I can’t be sure if the issue is luck of BOM or something else.#2023-10-0410:50Alex Miller (Clojure team)if you're not using BOM, then that can't be the problem - just work this as a java interop issue - what is the full error when it occurs?#2023-10-0410:52kwladyka#error {
:cause Bad type on operand stack
Exception Details:
Location:
com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
Reason:
Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
Current Frame:
bci: @4
flags: { }
locals: { 'com/google/protobuf/ExtensionRegistryLite' }
stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
Bytecode:
0000000: 2ab2 0002 b600 03b1
:via
[{:type java.lang.VerifyError
:message Bad type on operand stack
Exception Details:
Location:
com/google/api/AnnotationsProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
Reason:
Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
Current Frame:
bci: @4
flags: { }
locals: { 'com/google/protobuf/ExtensionRegistryLite' }
stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
Bytecode:
0000000: 2ab2 0002 b600 03b1
:at [com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]}]
:trace
[[com.google.pubsub.v1.PubsubProto <clinit> PubsubProto.java 570]
[com.google.pubsub.v1.PubsubMessage$AttributesDefaultEntryHolder <clinit> PubsubMessage.java 112]
[com.google.pubsub.v1.PubsubMessage$Builder internalGetAttributes PubsubMessage.java 910]
[com.google.pubsub.v1.PubsubMessage$Builder buildPartial PubsubMessage.java 694]
[com.google.pubsub.v1.PubsubMessage$Builder build PubsubMessage.java 682]
...#2023-10-0410:53kwladykaWhy it work in REPL and doesn’t in jar?#2023-10-0410:55kwladykaI don’t see any conflicts or double versioning etc. in pom.xml or -Stree#2023-10-0412:09Alex Miller (Clojure team)not sure, but "Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'" makes me think of either getting mismatched lib versions, or a classloader issue#2023-10-0412:09kwladykaAny actions which I can do to verify it?#2023-10-0412:10Alex Miller (Clojure team)a VerifyError generally means the class bytecode is invalid. I assume here it's a class generated by protobuf#2023-10-0412:11Alex Miller (Clojure team)when you say "in jar" - what does that mean?#2023-10-0412:11Alex Miller (Clojure team)are you building an uberjar out of this?#2023-10-0412:11kwladykaI assume it is about:
> I see ByteString/copyFromUtf8 is com.google.protobuf.LiteralByteString in jar, but com.google.protobuf.ByteString$LiteralByteString in REPL. Why it works in REPL, but doesn’t in jar.#2023-10-0412:12Alex Miller (Clojure team)I don't understand what you mean by that sentence#2023-10-0412:12kwladyka(b/uber
{:class-dir class-dir
:uber-file jar-file
:basis basis
:main 'system.main})
To be precise, this is not my project and I am still learning this project.#2023-10-0412:13kwladykaI mean (ByteString/copyFromUtf8 (json/write-value-as-string body)) in REPL is com.google.protobuf.ByteString$LiteralByteString, but in jar is com.google.protobuf.LiteralByteString which suggest me there is an issue about type.#2023-10-0412:14kwladykaand this is the input for later function which fail#2023-10-0412:15Alex Miller (Clojure team)don't know why that would be unless the libs are different. could be you have two jars on your classpath with the same classes and on the repl, the ordering puts the "correct" one first, but after building the uber jar, the "wrong" one ends up in the uberjar#2023-10-0412:16kwladykathis was my first idea, but I don’t see any duplicate of libs in -Stree or pom.xml#2023-10-0412:16kwladykaDo you suggest any actions, code, commands, check which I can do?#2023-10-0412:18Alex Miller (Clojure team)it would not be duplicate lib, it would be two different jars that include the same classes. since you have two classes above, I would look for those in the jars you have (com.google.protobuf.LiteralByteString and com.google.protobuf.ByteString).#2023-10-0412:19Alex Miller (Clojure team)maybe something in the google-cloud-pubsub includes protobuf instead of depending on it#2023-10-0412:20Alex Miller (Clojure team)that kind of problem could definitely cause the error you're seeing#2023-10-0412:51kwladyka> com/google/protobuf/ByteString$LiteralByteString.class
> com/google/protobuf/LiteralByteString.class
I see both in Jar, but still I don’t know what actions I can do to fix it.#2023-10-0412:55kwladyka> maybe something in the google-cloud-pubsub includes protobuf instead of depending on it
Do you mean the pubsub (or dependency of pubsub) is builded as uberjar instead of jar and include other lib jar with protobuf while at the same time have protobuf as dependency?#2023-10-0413:48Alex Miller (Clojure team)Yes#2023-10-0413:54kwladykaI don’t think there is any way to fix it in deps.edn, is it? Perhaps BOM fix it.#2023-10-0414:00Alex Miller (Clojure team)Bom has nothing to do with this - it’s just a way to specify multiple deps together#2023-10-0414:01kwladykaI mean maybe BOM set the same version in dependency as it is in jar#2023-10-0414:06Alex Miller (Clojure team)Ultimately you’re getting some set of jars that is on your classpath in repl case. In uberjar, that set of jars is being collapsed into a single jar. If the jars do not overlap, this identical from classpath perspective. If jars overlap, then ordering matters, in both cases.#2023-10-0414:36kwladykainteresting, I have tried to isolate problem (libraries and small code) to separate in small project:
1) with deps.edn clj -T:build ci I have
> Cannot write META-INF/license/LICENSE.aix-netbsd.txt from io.grpc/grpc-netty-shaded as parent dir is a file from another lib. One of them must be excluded.
2) with lein uberjar it compiles and work in jar as expected#2023-10-0414:38kwladykaboth have the same versions of the same libraries in the same order#2023-10-0414:44kwladykaalso -Spath show io.grpc/grpc-netty-shaded only once#2023-10-0414:44kwladykaeverything is only once#2023-10-0414:47kwladykaCan it be clj build bug somehow?#2023-10-0414:49Alex Miller (Clojure team)that error means probably means you have both a META-INF/license/ dir and META-INF/license file from different jars during uber jar'ing - in uber task you can add :exclude ["META-INF/license"] to exclude the file#2023-10-0414:51kwladykaoh sorry, I misinterpret it as a part of the libraries conlict#2023-10-0414:55kwladykaok, it works both from lein and clj when isolated from the project.#2023-10-0414:55kwladykafinally, something#2023-10-0414:55kwladykathank you for your help#2023-10-0415:15kwladykayou were right. This is one of third party library which has to include protobuf in hidden way. Just the presence of the library in deps.edn make it fail after build to jar.#2023-10-0415:16kwladykaI did it by commenting half of the deps, later other half etc. in experimental project. Is there any other way to detect such conflicts in the real project and real code?#2023-10-0415:26kwladykaI mean in the real project I can’t comment any library, because system will not build, because code needs each of them.#2023-10-0415:29kwladykaSo I am thinking if there is any reasonable way to detect such conflicts and display them.#2023-10-0415:37Alex Miller (Clojure team)jars have files, jars with the same .class file are in conflict - there are some java tools out there that can help identify stuff like this#2023-10-0415:38kwladykaCan you give any name of such tool to start reading in google?#2023-10-0415:38Alex Miller (Clojure team)don't know one offhand#2023-10-0415:40kwladykathank you, I will find something#2023-10-0620:10isakWith you are working on a library and generate a pom with deps, is the intention that this pom is modified manually and checked in?#2023-10-0620:15isakOh I think I might see it now, there is a "source pom" and generated ones#2023-10-0620:18Alex Miller (Clojure team)some tools can work from the generated pom.xml, so in some cases you might find it useful to check in the generated pom.xml (and in that case, I would name the generated one something like template-pom.xml)#2023-10-0620:18isakOk, good idea, thank you#2023-10-0715:27seancorfieldIn all my projects, I used to keep the src-pom checked in (in a subfolder but it could easily have been just a different name) and then just generate into target as part of the jar build.
Most of my projects had a src-pom that was generated by clj-new or deps-new which has more sections in it than the basic tools.deps (or tools.build) generated pom.xml.
Yesterday, with the release of tools.build 0.9.6 and the addition of the pom-data option, I got rid of the src-pom and checked in file and switched to providing all that extra information in build.clj via pom-data.
You can look at HoneySQL or next.jdbc for examples.#2023-10-0715:30lread@U04V70XH6 cool! I'm gonna take a peek at your changes. I only had what I was calling "anemic" pom.xml files checked into git because tools.build couldn't fill in some blanks for me.#2023-10-0715:47seancorfieldhttps://github.com/seancorfield/next-jdbc/blob/develop/build.clj#L35-L60 is how I'm doing it now (I can post a link now I'm not just on my phone). Here's the commit where I deleted the old template/pom.xml and switch to that Hiccup-style data: https://github.com/seancorfield/next-jdbc/commit/35eb931877c337103aae6f7145c168b31b635696#2023-10-0709:11borkdudeWhile I was trying to get to the root of a problem that @sibuna had in #clj-on-windows, we found that:
clojure -Ttools install-latest :lib io.github.seancorfield/clj-new :as new
clojure -Ttools install-latest :lib io.github.seancorfield/deps-new :as new
i.e., overwriting a tool by a different tool under the same name, results in an error. Could this be a bug in tools-deps?#2023-10-0715:21seancorfieldWhat's the error?#2023-10-0716:19borkdude$ clojure -Ttools install-latest :lib io.github.seancorfield/deps-new :as new
Execution error (ExceptionInfo) at clojure.tools.deps.extensions.git/coord-err (git.clj:45).
Library io.github.seancorfield/deps-new has invalid tag: v1.2.404
Full report at:
/var/folders/j9/xmjlcym958b1fr0npsp9msvh0000gn/T/clojure-888812946633046028.edn
The issue is not specific to your tools, just happens when you install tool X under name N and then a completely different tool Y under the same name N#2023-10-0716:45seancorfieldAh, because it uses the existing alias EDN file to determine whether the newest version of the :lib it can find is actually older than the currently installed one. That's an interesting puzzle.
I guess if it cannot compare the two versions (for any reason that leads to an exception) then you have two possible choices:
• Report the old coordinates, say it can't be compared, and skip the install.
• Go ahead and overwrite it, possibly with a message about the old coordinates.
The former is probably safer and would, overall, be a better UX.
As it stands, the behavior is "correct" but not very helpful to folks in terms of diagnosing what they actually did wrong.#2023-10-0718:01Bastianwell, if one uses install + version number instead of install-latest it has no problem in overwriting the name.
not sure if "folks did something wrong" according to the api docs, in any case a better error message could save some time#2023-10-0718:18seancorfieldIf you use an explicit version, it doesn't have to compare for latest: you can install any version, even older versions.
If you want latest, it has to compare the current version to see whether to install or not. If you've already installed, say, some specific branch version that you're testing, you can overwrite it with another specific version, but you probably don't want some "random" latest version overwriting that specific version?#2023-10-0718:22borkdudewhy does it have to compare something if you want to install the latest. it could just decide to overwrite the installed one with the latest it can find?#2023-10-0718:22borkdudewhat does "install" even mean#2023-10-0718:22Bastianthats my point ( i just expected install-latest to install the latest )#2023-10-0719:39seancorfieldThis should be a safe, idempotent operation:
> clojure -Ttools install-latest
antq: Skipping, newest installed com.github.liquidz/antq 2.7.1133
clj-new: Skipping, newest installed com.github.seancorfield/clj-new v1.2.404
clj-watson: Skipping, newest installed io.github.clj-holmes/clj-watson v4.1.3
new: Skipping, newest installed io.github.seancorfield/deps-new v0.5.3
poly: Did not find versions for io.github.polyfy/polylith
tools: Skipping, newest installed io.github.clojure/tools.tools v0.3.1
#2023-10-0719:41borkdudeDo you mean, safe as in, when people remove the latest version from the internetz you will still have the latest version locally?#2023-10-0719:42seancorfieldStrawman. "People" (Clojurians) don't randomly remove the latest version from the internetz.#2023-10-0719:42borkdudeThen what do you mean?#2023-10-0719:42seancorfieldI mean, safe in terms of not overwriting any special version I have installed as I already explained above.#2023-10-0719:43seancorfieldFor example, if you're working on a tool and have it installed via a :local/root dep, you don't want that overwritten by install-latest.#2023-10-0719:43borkdudeSorry, I didn’t understand that#2023-10-0719:43borkdudeGot it #2023-10-0719:43seancorfieldOr you're depending on a Pull Request, or non-merged branch...#2023-10-0719:44seancorfieldYou want all of those to be not comparable and therefore not overwritten.#2023-10-0719:44seancorfieldI think we all agree that the error message could be improved in this case 🙂#2023-10-0719:44borkdudeRight, fully agreed#2023-10-0719:47Bastiani dont't understand how the example of install-latest without any arguments relates to the above problem and think install and update are confused here.#2023-10-0719:48borkdudeHow does install-latest detect that you don’t want to upgrade a specific version?#2023-10-0719:49seancorfieldI assumed you're asking Bastian their opinion since I've already explained why (and how) above?#2023-10-0719:52borkdudeI don’t fully understand yet but I’m also ok with dropping this conversation for today#2023-10-0719:53borkdudeI’m kinda tired from COVID so I’ll check back in later#2023-10-0720:31borkdude@U04V70XH6 I guess what I mean is this scenario:
This is in my ~/.clojure/tools/new.edn
{:lib io.github.seancorfield/deps-new, :coord {:git/tag "v0.5.3", :git/sha "c899135"}}
How does clj know that this was the result of install-latest or that I just put it there to stick to this specific tag + sha and I want to stay on those, thus install should skip in case there is a new version in the future. Is there a way to tell the difference?
You might have already explained this, but then I'm just very sorry for not understanding and I'll just finally give up for today ;)#2023-10-0720:40borkdudeWhen I change new.edn to {:lib io.github.seancorfield/deps-new, :coord {:git/sha "5d0e47d05dc828703c23253ed9fc57c2d61f7848"}} and then run install-latest, it happily changes it back to {:lib io.github.seancorfield/deps-new, :coord {:git/tag "v0.5.3", :git/sha "c899135"}}#2023-10-0910:30borkdudeAaaanyway, @U064X3EF3, if you want an issue for the problem "overwriting a tool with an existing name but from another project fails with cryptic error" on ask.clojure, I will create one#2023-10-0912:12Alex Miller (Clojure team)Yes#2023-10-0912:15borkdudehttps://ask.clojure.org/index.php/13368/install-latest-already-another-results-cryptic-error-message - hopefully sufficient#2023-10-0912:18Alex Miller (Clojure team)If you install-latest of a lib, it should install the latest of that lib and ignore the current tool named that#2023-10-0912:19borkdudeI'd say so too#2023-10-0916:07seancorfield@U064X3EF3 would it still skip the download/install if the currently installed version looked the same as the latest version determined? (rather than trying to compare for greater than / less than - just use Clojure equality on the hash map)#2023-10-0916:12Alex Miller (Clojure team)yes, that's where it's getting tripped up, but it should just ignore in the case where they are different#2023-10-0916:14borkdude> would it still skip the download/install
FWIW, I demonstrated above that install-latest went back to the latest tag even though I had a newer SHA than the tag specified in the tools/new.edn file. Not sure what logic is happening here, but also, it's not so important to me because for development I would probably just rely on :aliases + -X to test#2023-10-0916:47seancorfield☝️:skin-tone-2: That was unexpected to me. I had assumed the whole point of install-latest doing that version check was so that it would not overwrite a more recent SHA install of a tool with an earlier tag version, so that you could install from a PR and not worry about it getting overwritten until that PR was merged and a new version released? @U064X3EF3#2023-10-0916:53Alex Miller (Clojure team)no, that was not the point, it was to avoid doing an install if latest was already installed. If you say "install-latest", then it will install latest.#2023-10-0917:04seancorfieldOK, as long as the docs make it clear that install-latest could overwrite an install of the latest SHA with an earlier tagged version... 🙂#2023-10-0917:11Alex Miller (Clojure team)tools only consider tags as versions. if you "install latest" version, then you are installing the latest tagged version. other shas are not considered#2023-10-0917:12Alex Miller (Clojure team)the https://clojure.github.io/tools.tools/clojure.tools.tools.api.html#var-install-latest imply all of this, but maybe but do not say it clearly enough#2023-10-1119:10Alan BirchenoughI have a noob question about preparing libs to be :local/root source dependencies. I have such a lib, but I can't get the dependency to work because no matter what I do, clj still thinks the library has not been prepped.
It is a tiny library consisting of one source file plus a couple of Maven deps on Commons IO and Clojure Data CSV. I have run clj -X:deps prep on this library, and that process added the expected required content to deps.edn, to wit:
{:paths ["src" "target/classes"]
:deps/prep-lib {:alias :build
:fn compile
:ensure "target/classes"}}
I then go to another tiny project that specifies my library as a :local/root dependency and try to start it up, but I always get a classpath error that the lib is not prepped. What am I (probably) missing?#2023-10-1119:11Alex Miller (Clojure team)is there a target/classes in the dir?#2023-10-1119:12Alan BirchenoughNo.#2023-10-1119:12Alan BirchenoughIt was not clear to me how that should be created.#2023-10-1119:13Alex Miller (Clojure team)to translate {:alias :build :fn compile :ensure "target/classes"} into words...
if this project does not have "target/classes", run clj -T:build compile (which should put something in target/classes)#2023-10-1119:14Alan BirchenoughI would literally run the command in red?#2023-10-1119:14Alex Miller (Clojure team)you don't run it - prep will run it for you when you use it from another project#2023-10-1119:14Alex Miller (Clojure team)I mean, you can run it, and then it should appear prepped#2023-10-1119:15Alan BirchenoughDo I run prep while in the other project, as opposed to running it while in the library project dir?#2023-10-1119:15Alex Miller (Clojure team)yes#2023-10-1119:15Alex Miller (Clojure team)clj -X:deps prep means "prepare my deps to be on my classpath"#2023-10-1119:15Alan BirchenoughWhen I run the prep command in the other project, it complains that it doesn't know what compile is.#2023-10-1119:16Alex Miller (Clojure team)we may have run out of specific-enough information at this point#2023-10-1119:16Alan BirchenoughLOL - I would be happy to provide more specifics.#2023-10-1119:16Alex Miller (Clojure team)https://clojure.org/guides/deps_and_cli#prep_libs hopefully says that same stuff#2023-10-1119:17Alan BirchenoughIt does, but not as explicitly. It was not clear to me which deps.edn would acquire those new entries and which dir I should be in when I ran the prep command, for instance.#2023-10-1119:18Alan BirchenoughProbably because I don't really understand what I am / it is doing.#2023-10-1119:19Alex Miller (Clojure team)that example has:
• needs-prep - a local lib that needs prepping. it has the :deps/prep-lib in its deps.edn
• cool-lib - a project using needs-prep, includes needs-prep as a :local/root dep. clj -X:deps prep is run here#2023-10-1119:20Alan BirchenoughI will give this another try based on that understanding. Thank you.#2023-10-1119:21Alex Miller (Clojure team)I can see why that example does not fully specify, will try to improve it#2023-10-1119:22Alan BirchenoughGreat.#2023-10-1119:22Alan BirchenoughThis is the error I get when I run prep in my equivalent of cool-lib:
Prepping csvlib/core in /Users/alanbirchenough/clojure/csvlib
Execution error (ExceptionInfo) at clojure.tools.deps/qualify-fn (deps.clj:654).
Unqualified function can't be resolved: compile
#2023-10-1119:25Alan Birchenough@U064X3EF3 Can you please suggest what I should do to diagnose this error, now that I appear to be executing the process correctly? Thanks.#2023-10-1119:29Alan BirchenoughI think I know what is causing this.#2023-10-1119:36Alan BirchenoughClosing the loop, then I'll shut up: Because I prepped while in my-lib instead of in cool-lib, I had a bogus reference to compile in my deps.edn.#2023-10-1120:26Alex Miller (Clojure team)so, working now?#2023-10-1121:00Alan BirchenoughSeems to be, thanks.#2023-10-1121:01Alan BirchenoughThank you very much for your assistance with this process.#2023-10-1710:18stathissiderisamazing, came to this channel to ask about this exactly, and the thread was right here ❤️ thank you!#2023-10-1710:52stathissiderisSince git deps are checked out before being used, could clj -X:deps prep also work on them?#2023-10-1712:53Alex Miller (Clojure team)Yes, it does#2023-10-1718:54danielcomptonPerhaps a dumb question, but is it possible to run a tool without installing it first? I’m imagining something like this:
clojure -Ttools run nvd-clojure/nvd-clojure '{:mvn/version "3.4.0"}' nvd.task/check
This is for CI/scripting, where I’d prefer not to have a separate install step from running the tool#2023-10-1718:55Alex Miller (Clojure team)Named tools are installed by definition #2023-10-1718:55Alex Miller (Clojure team)You can use a tool alias though #2023-10-1718:56Alex Miller (Clojure team)Do you need -T or just -X?#2023-10-1718:57Alex Miller (Clojure team)If you want to both install and run, that’s going to be two commands#2023-10-1718:58danielcomptonI want to run https://github.com/namenu/deps-diff, without using the content of my deps.edn file in the classpath, so I think as -T?#2023-10-1719:03seancorfield@U051KLSJF There's no entry point to run that via -T or -X with paths to EDN files. It has a GH action that invokes the CLI twice to produce deps trees and then compares those as EDN: https://github.com/namenu/deps-diff/blob/main/action.yml#L64-L89#2023-10-1719:04seancorfield(I mean, yes, there's a -X entrypoint the action uses but you need to run two CLI commands first to prep the data)#2023-10-1719:11danielcomptonYup, I need to fork it to get it to run its own diff as a tool. Currently, it is running as clojure -Sdeps which means that everything in the deps.edn is also included in the classpath, which is causing issues in our case#2023-10-1719:12danielcomptonThe other issue with installing as a tool in a GitHub action is that the author of the GitHub action needs to pick a name that doesn’t collide with the names of tools that any consumers have installed. So I guess you should just pick a long unique name?#2023-10-1719:25Alex Miller (Clojure team)Can you use :deps in the alias to replace the deps?#2023-10-2001:45danielcomptonYeah, that might be an option, I’ll take a look#2023-10-1906:43Ovidiu StoicaHey guys! I posted this in #C03S1KBA2 but I think this is the best place to ask:
I’m trying to install the google cloud java lib. Here’s the maven code:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>26.25.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
</dependency>
After reading through the docs I got it to
com.google.cloud/libraries-bom {:mvn/version "26.25.0"
:deps/manifest :pom}
but it still does not install correctly using intelliJ Cursive. However I know its there: https://mvnrepository.com/artifact/com.google.cloud/libraries-bom/26.25.0
Anybody has any ideas?#2023-10-1906:45lispycloudscheckout: https://clojurians.slack.com/archives/C6QH853H8/p1695992844961919#2023-10-1906:48Ovidiu StoicaOh I didn’t know that! Thanks a lot. Should’ve asked earlier!#2023-10-1912:13Christian JohansenIs there a way to specify a default namespace to be loaded when I start a REPL?#2023-10-1912:15vlaaadThere is a default namespace loaded — user . You can create user.clj and require all necessary namespaces from there.#2023-10-1912:16Christian JohansenAh, of course#2023-10-1912:16Christian JohansenThanks 🙂#2023-10-1913:48Noah Bogartyou can also say -e "(require 'some.ns) (in-ns 'some.ns)" which will call that when the repl loads#2023-10-1916:55practicalli-john-e option is the only reasonably reliable way to set the namespace, although personally I do not recommend this approach.
The user namespace is an effective way of interacting with the program when using a command line REPL prompt, as it contains additional tools useful for development. Requiring namespaces into the user namespace keeps those tools available.
NOTE: Using an editor and evaluating source code is automatically done in the relevant namespaces, so no switching is required and this approach is generally recommended.
Using a custom dev/user.clj file can be used to load libraries and run tools, e.g. https://practical.li/clojure/clojure-cli/repl-startup/#calling-functions . The custom user namespace is loaded via an alias that adds the dev directory to the classpath on startup, so that this custom user is loaded during development (and not released product).
I use a custom user namespace to launch Portal data inspector, configure Portal to listen to tap> values over nREPL and send all mulog event logs to Portal for a complete repl history. Many developer oriented tools and libraries can be made available using a custom user namespace.
https://practical.li/clojure/clojure-cli/projects/templates/practicalli/ includes a custom user namespace in the projects it generates, including component lifecycle REPL tools when selected as an option.#2023-10-1917:09Noah BogartHere’s an example of dev/user.clj from my linting library: https://github.com/NoahTheDuke/splint/blob/main/dev/user.clj. Has some side effecting dev requires and potemkin imports dev helper.#2023-10-2313:51Hannu HartikainenEDIT: :deps/root already does this!
~It looks like the git procurer requires a manifest at repository root. I feel like adding support for subdirectories with ~
~Full disclosure, I'm considering porting a bunch of libraries in a multi-module repo from leiningen to tools.deps.~#2023-10-2313:52Alex Miller (Clojure team)There is already support for this with :deps/root#2023-10-2313:52Hannu HartikainenGreat! Thanks!#2023-10-2313:55Hannu Hartikainensorry for not RTFM'ing, I did look at that list of common attributes but still missed it 😬#2023-10-2720:42Alex Miller (Clojure team)@seancorfield (or others), do you know of places where polylith or anything is creating a tools.deps basis using a custom :project deps.edn, particularly in a directory other than the current directory?#2023-10-2720:46Alex Miller (Clojure team)I'm looking at https://clojure.atlassian.net/browse/TDEPS-256 where I believe the problem is that if a :project deps.edn is provided in some other directory, local deps are resolved relative to the current directory, not relative to the project deps.edn, which seems wrong. I would like to fix this but want to make sure I'm not going to horribly break something someone is depending on#2023-10-2720:50seancorfieldI'll check our repo... just a sec...#2023-10-2720:50seancorfield...nope, not in our Polylith repo...#2023-10-2720:54Alex Miller (Clojure team)I imagine someone probably would have mentioned this if so, but trying to be cautious#2023-10-2808:20namenuwe use polylith too and every .cpcache/**.basis is created in the same path where deps.edn is located.#2023-10-2813:56Alex Miller (Clojure team)Thanks, good to know#2023-11-0814:13fabraoHello, anyone is having problems to use https://github.com/nnichols/clojure-dependency-update-action ?
It started failing with this error 2 weeks ago:#2023-11-0816:30seancorfieldHow is it "failing" and what is the relevance to tools.deps? I'm tagging @UCZ5ZDJKF to see if he has any insight.#2023-11-0816:52fabrao@U04V70XH6 sorry posting here, but I didn't find a place to include this problem, and I saw someone posting something about this lib here.#2023-11-0817:23seancorfieldWell, the author isn't in this channel and I must admit I've never heard of that GH action...#2023-11-0903:39Cora (she/her)you'd be a lot better off switching to Renovate#2023-11-0905:18Cora (she/her)but I'd wager that the error is due to a variable not being quoted in that shell script and that causes splitting when running commands which makes for extra arguments#2023-11-0905:19Cora (she/her)I spent a little bit of time writing a replacement for dependency_check.sh but I hit my timebox for it https://gist.github.com/corasaurus-hex/bd8b85a46d1ab7cad05c49cad4519bd6 -- feel free to take that and fix the github action if you feel like it#2023-11-0913:05fabrao@U02N27RK69K thank you for the information. The good part of that lib is that it creates a PR for validation.#2023-11-0913:08Cora (she/her)https://github.com/renovatebot/renovate#2023-11-0913:12Cora (she/her)you can make that work with clojure, and it opens PRs#2023-11-1015:21fabrao@U02N27RK69K Hello, I setup one project with renovate and I liked the way it works. Thank you for the advice.#2023-11-1916:54nnicholsI just got back from vacation today. Once I catch up on things, I'll see if I can make a fix for existing consumers. That said, I agree with @U02N27RK69K - Renovate is a much better tool for the job, and the action was originally only meant as a stopgap until Clojure got Dependabot/Renovate/etc support#2023-11-0922:09socksyjust found that a Clojure project I maintain has a transient dependency to shadow-cljs due to a badly written :deps map in a cljc local/root lib dependency. But I am wondering — is there actually an idiomatic way to have a cljc lib have different dependencies for clojurescript vs clojure?#2023-11-0922:15favilaI don’t think so. I routinely have to do manual :exclusions. Sometimes the lib will use :provided and make you manually require.#2023-11-0922:17socksyare there any docs on that?#2023-11-0922:17faviladocs on what?#2023-11-0922:18socksyI don't know what :provided means exactly, and my googling isn't helping#2023-11-0922:18favilaoh. This is something the lib would do.#2023-11-0922:19favila:provided is a maven-ism that means this thing is required by the lib at runtime, but won’t be included via transient deps retrievial#2023-11-0922:19socksywell i have commit access to the lib too so this is totally doable for me. If I were doing a normal project I might split these out into different aliases, but then I believe I would break the cljs project that does want it to depend on some cljs libs. But if it's a maven-ism does that mean that it'll be ignored for a local/root require?#2023-11-0922:20favilayeah it would be#2023-11-0922:20favilathe equivalent would be hiding it in an alias#2023-11-0922:21socksybut then for the project that does need those dependencies you won't be able to specify the alias in the :deps map, right?#2023-11-0922:21favilaright, you would need to know (e.g. reading the lib docs) what it needs and provide it yourself#2023-11-0922:22socksyalright, a list of 10 :exclusions it is then 😅#2023-11-1217:59vemvCurious, is the syntax described in If used, alias names should refer to a path vector in the alias data relatively new? I don't think I had seen it till today
https://clojure.org/reference/deps_and_cli#_paths#2023-11-1218:04seancorfieldIt's been supported for quite a while.#2023-11-1218:04seancorfieldI use aliases-as-data for several things (but not paths).#2023-11-1218:08seancorfieldLooking in the source, around the chase-key function which is part of the implementation for that, it's been in the code for at least 3 years @U45T93RA6#2023-11-1218:11vemvThanks much!#2023-11-1619:59simonHi all, I saw that when downloading dependencies the repositories are tried in order, from src/main/clojure/clojure/tools/deps/alpha/util/maven.clj
(defn remote-repos
[{:strs [central clojars] :as repos}]
;; always return central, then clojars, then other repos
(->> (concat [["central" central] ["clojars" clojars]] (dissoc repos "central" "clojars"))
(remove (fn [[_name config]] (nil? config)))
(mapv remote-repo)))
There are some libraries though which use a different id for maven central and then the ordering is not applied (`maven-central` instead of central). One of these libraries is https://github.com/googleapis/google-cloud-java/blob/main/google-cloud-pom-parent/pom.xml.
Is this something that has changed or is it just a loose convention to use central as id? And do you think it's a good idea to include maven-central in this ordering?#2023-11-1621:11Alex Miller (Clojure team)central and clojars are built in to the root deps.edn and so are "special"#2023-11-1623:02simonah ok, that’s why they’re hardcoded, got it, thank you