0:00:00Clojure Made Simple
0:00:00 I want to thank everybody for coming the title is talk is closure made simple on the brochure they left out the comma0:00:09so it's not closure made simple in other words a tutorial in closure or an easy explanation of closure it's not actually a comprehensive explanation of closure0:00:18at all but a look at a slice of what closure is about a way of thinking about why you might want to use it so I'm the0:00:27person who made closure I currently work on a database called de Tomic which is kind of a functional database it's written in closure and runs on JVM0:00:37architecture I'm a co-founder of daytime of cognate equi build stay Tomic and sponsors the development and stewardship0:00:46of closure but the main point I wanted to make about myself to this audience was because in this talk I might seem somewhat skeptical of Java and0:00:56object-oriented programming is that I've done an absolute ton of that that's what I did for two decades before I said if I0:01:05still want to be a programmer I don't want to do it this way anymore so I know exactly how apps are built using Java and C++ and C sharp because that's what0:01:15I used to do that doesn't mean what I think about them is correct but that's my experience but I'm wondering about0:01:24you how many people program in Java here how are you happy about that how you were actively unhappy about that looking for alternatives okay great how many0:01:35people have tried closure at all great how people never heard of closure and are in the wrong room okay how people have tried closure and are trying to get0:01:45it to use it at work but not yet a few maybe this talk will give you some some ways of talking about closures value0:01:54proposition that could help you how you will actively use closure and somehow are accidentally at JavaOne okay0:02:03and the rest couldn't get into Brian gets a sock I shouldn't even mention that Brian gets us to talk right now because we could have people filing or maybe you're just tired of Brian gets talking about immutability in which case0:02:13you're definitely in the wrong room now I like them Brian's a good friend and his talks are great so I appreciate your0:02:22bringing this one so if you know very few people had never heard of closure so I'm not going to spend a lot of time on oh I had one more one more question how many people have seen my talk simple0:02:32made easy and people not okay a few so I made spend a minute describing what I mean when I say simple closure is a0:02:42programming language it runs in the JVM and JavaScript and pretty much a substantial a subset of closure runs on JavaScript so it's a program lines with0:02:51which you can target both but originally it only targeted the JVM and the CLR it's they're still a port to the CLR that's maintained but does not seem C0:03:00wide use released at first in 2007 it's had you know surprising adoption especially from my perspective since0:03:09then given given its characteristics because it's a lisp it's functional its data oriented and it has a lot of things that make it seem not like the kind of0:03:19language that would succeed and this talk will really be about the data orientation of closure so a lot of the0:03:29best programmers and the most productive programmers I know are writing everything in blank and swearing by it and then just producing ridiculously sophisticated things in a very short0:03:39time and that programmer productivity matters so Adrienne Cockcroft was an architect and Netflix now is the battery ventures how many people think Java goes0:03:49in the blank okay so we know we know this there's something about Java that makes0:03:59it not suitable for this blank - maybe we can tease that apart yes of course he was saying closure and this talk is0:04:08about maybe why like how could this be true what is it that makes closure different and possibly a better fit for0:04:18that blank so the first thing I want to talk about is is that I think we have this tendency in programming to think about ourselves just a ton and our0:04:29languages our tools and our techniques and me me me me me us what we're doing whatever we lose track of the fact that we're all working for somebody else who's or ourselves but for a business or0:04:39an organization that's trying to accomplish something in the world and the software is completely secondary to that task right it should be measured0:04:48always in terms of the cost-benefit ratio the return on investment right how quickly can we get a product to market and is what we're doing profitable right0:04:57if we're not doing that we're not really being good participants on our in our businesses or organizations so what do0:05:06the stakeholders want they really want two things they want something good and they want it soon so something good we0:05:21think we know it's something good we have you know we know how to make things good right we have these techniques and some things things are good when the techniques are successful with them0:05:30right so when our types check on our test pass we have something good but of course we all know that with our best efforts in those things which are I'm0:05:40not saying they're bad activities by the way but no matter what we do there we end up with programs that don't work we all know we have programs of type check and test pass and they don't work and0:05:50when they don't work from the perspective of the stakeholder in other words they don't do what the program was supposed to do and what's supposed to do was something that was conveyed between people or through documentation or0:05:59papers or things that are not in program languages they have to meet operational requirements and they have to be flexible okay now there are some times0:06:09where people just want something soon and they don't want something good there are actually better languages enclosure for that right just give me something fast that I'm absolutely definitely0:06:19going to throw away will not grow will not expand will not will that take me further so these first two things are0:06:28means right they're good but they're only good insofar as they help ensure the latter three things so we break it down what is it supposed to do again0:06:37it's a prospective thing if the stakeholder thinks assume what's supposed to do they're fine of course they're going to have expressed concerns about software what it's supposed to do0:06:46it should do this when I push this button and they're going to have unexpressed presume things like it should be secure it shouldn't you know cause the computers ago on fire because0:06:55it's so slow it shouldn't require three new data centers it should keep running and not stop for an hour every day those0:07:04are sort of the unstated presumptions of something being good but it ends up that if you build large elaborate stateful programs it's extremely difficult to0:07:14ascertain whether or not they are going to do what they're supposed to do and in fact if you build any one of those things if you just build a very large program or a very elaborate program or a0:07:24very stateful program it will be as hard to figure out if it's one if it's going to do is supposed to do so one of the things closure is oriented at is making0:07:34it easier to understand whether or not your program is going to do it's supposed to do mostly by making it substantially smaller and also by making it more functional in terms of0:07:44operational requirements is a boatload of things as a boatload of unstated requirements of software you know - are you cantide apply it in the normal way with all my stuff with the people who0:07:53know how to run my machines and everything else and and that's one of the targets of closure closure was meant to be hosted it's just a jar it runs in0:08:02the environment it's easy to sneak in right let's just add this to one more jar and then we're and then we're running but but it's not a small thing0:08:11right if you wanted to adopt calm lisps a or Haskell you would be asking your urops team and your deployment team0:08:21to start manipulating something completely alien whose characteristics they don't understand in terms of security everything that's available0:08:30from the JVM for security is available via closure and there are performance and other concerns a very important0:08:39thing though is that now we can also reach the browser so I think you know how many people write applications where some part of the overall system touches a browser yeah so right now you use two0:08:50different things I almost definitely use two different things and I think it's a strength of closure that we're delivering the same value proposition in both places both on0:08:59the server and in the client even if they're separate devs the value proposition is necessary I'm almost more so in the browser which is one of the most complex places ever in terms of0:09:11performance you know a lot of times you might look at a dynamic language and say you know how could it be whatever but you know closures right down there with the fast languages on admittedly the0:09:23benchmark game which is a benchmark and a game so but it says we can reach that on JavaScript we have a very interesting0:09:32result here so ohm is a closure script library so closure script disclosure on on JavaScript and ohm is a library that actually wraps react which is the new0:09:43hotness what's the really interesting things here ohm wraps react and then spanks it in performance how's that0:09:52possible it ends up that that ohms use of persistent data structures which we'll talk about in a minute make it faster0:10:01than react because the big part of react is doing change detection and change detection for immutable things is identity comparison so it's super fast0:10:11and in fact they are reacting their whole thing to use persistent data structures in JavaScript now the react dev came up to me at strange loop and0:10:22said shook my hand and said you're saving us a ton of money because we're switching to that strategy so the other part of the value0:10:32proposition I said was flexibility right people that that you know have a stake in software know that they're building a system now but tomorrow things are going0:10:42to change requirements going to changes we have to do something different change is inevitable so can we change the program can we make it more flexible and ends up there's a0:10:51lot we can learn from bigger system design in the small to make the sub components of systems more flexible and of course this is old this is the oldest0:11:02thing right loose coupling right but we talked and talked and talked about it we continued to use techniques that thwart it like every single day you pull out a technique you do something that's that0:11:11makes this harder so what makes it easier so the my talk was done and0:11:21somebody said you know I have this quote from Walmart Labs guy you might want to put it in your talk and it was like wow this is great because he's telling my story right with0:11:30closure we get to market faster with better quality we avoid unintended interruptions in java apps from code in one area impacts the application in another closure shrinks our code base to0:11:40about one-fifth the size it would have had we written in Java right and these are the points of my talk all right this0:11:49is what I want this is a stakeholder saying this is what we get by by choosing closure faster time-to-market better quality right we avoid coupling0:11:59problems that make it difficult for us to change and we have a smaller code base so how does closure do this there0:12:08are many many different characteristics to closure but I only want to talk about sort of two today mostly one one is and because this is one I think that sets closure apart it makes it somewhat0:12:19different is that it has data orientation through and through and the other is simplicity so when I say simplicity I mean the opposite of0:12:28complexity I do not mean ease and I mean closure is an easy language that's easy only you know your type and everything magically happens or some sort of easiness metric0:12:39complex things are intertwined and simple things are not right they're more independent they're separate even if they have as many things going on this0:12:48is simpler than this right for things like this is complex and for things like this is simple so that's what I mean when I say simple unentangled so the0:13:01thing is that you know all these languages can do everything right we can do the same stuff and people are like why you know I know I can do the same stuff in c-sharp in Java and and Scala0:13:11and and closure and you know anymore any general-purpose language you can accomplish the same things at the end of the day right so what differentiates languages is what they make practical0:13:21and what they make idiomatic and in closure we focused on something making something idiomatic that I think is not and should be more so that's this before0:13:34we had all this high fluting you know opinions of ourselves as programmers and computer scientists and stuff like that programming used to be called data processing how many people actually do0:13:44data processing in their programs you can raise your hands we all do right this is what most programs do you take some information in somebody type some0:13:53stuff somebody sends you a message you put it somewhere later you try to find it you put it on the screen you send it to somebody else that's what most0:14:04programs do most of the time sure there's a computational aspect to programs there's quality of implementation issues to this but there's nothing wrong with saying programs process data because data is0:14:14information information systems I mean this is should be what we're doing right we're the stewards of the world's information and information is just data0:14:23it's not a complex thing it's not an elaborate thing it's a simple thing until we programmers start touching it so we have data processor processing0:14:33most programs do this there's very few programs that don't write and data is a fundamentally simple thing data is just0:14:43raw immutable information so that's the first point data is immutable if you make a data structure you can start mess with that but actual data is immutable0:14:55so if you have a representation for it that's also immutable you're capturing its essence better than if you start fiddling around and that's what happens0:15:04languages fiddle around right they elaborate on data they add types they add methods they make data active they make data immutable they make data0:15:13movable they turn it into you know an agent or some active thing and at that point they're ruining it at least0:15:23they're moving it away from what it what it is and in object orientation I think this is rampant because our fundamentals construct the object conflates two0:15:34things it doesn't give us two separate ways to talk about process for which objects are an okay approach and information for which they're a terrible0:15:43terrible terrible approach right but every time we have a problem we pull out an object we make a new object we make a new class we make a new instantiation of something right and and this makes our0:15:53programs more about themselves again and less about the information programs are more increasingly about code and decreasing lis about data and I think0:16:03that's a mistake so closure embraces data that's like the simplest idea of my enclosure it's just there's nothing wrong with data data has these great0:16:12properties let's use it in fact let's make it a really important first class in your face kind of thing so closure embraces it first and foremost by having0:16:22strong data literals which I'll show you in a minute and they're plane right plane it just means flat level unadorned no extra0:16:33stuff code and this is an old listing in closure is represented as data that's important for a lot of reasons it0:16:42enables macros and a lot of sophisticated program transformation things but it also means that you don't have different stuff right the majority0:16:51of functions in closure just take data and by data I mean immutable unadorned stuff and they return that same thing giant library functions hundreds and0:17:00hundreds and hundreds of data manipulation function ready to go they take data they return data so if you have anything that's data you can use all those functions on it0:17:09which tends to press you towards making everything data because then you have this giant library which you learn once and you can apply to every problem you0:17:18have and in particular every time we encounter information in closure systems because there's different parts of systems right there's a part of your system that manipulates information is a0:17:27part of your system that's sort of plumbing right or the machinery of your program like you know a socket or communications endpoint is more like a machine than it is like information0:17:37right so there's going to be active parts of your program but whenever you're dealing with the part of your program that's just about representing facts and information we will always0:17:47enclosure choose plain data to do that so there there's a small set of data literals enclosure they're relatively0:17:56obvious right you can write integers in the normal way and doubles the normal way and big decimals with an M at the end there are ratios and these are proper ratios that don't lose precision0:18:05strings are in double quotes and strings are java.lang string when you write that you get a java.lang strength and so it's just a literal for a string there are0:18:14the rules for characters because we use data structures for programming right there's a couple of extra things we need but if you look at Java code is it0:18:24written all with quotes around every every word no so job I need something besides strings in order to be a successful programming language it needs0:18:34symbols and it needs identifiers so if you're going to properly represent your code as data those things have to be0:18:43first-class data structures or atomic data types that are different from strings so there are two in enclosure one are symbols and0:18:52the other are keywords and they're use in closure which I'm not going to you know dive into is that symbols are generally used to reference something0:19:01else so like variables and things like that they name something and keywords name themselves then we're like enums if you want they're very useful as keys in0:19:12maps for instance that's why they're called that there are true or false boolean literals and there's nil which is not it's the same that's a Java null and those boolean's0:19:22are Java new Java boolean's those characters or Java characters etc etc and there's also literals for regex and0:19:31then we have data structures some fundamental data structures we have the singly linked list right it's in parens those are all lists their list of0:19:40numbers a list of symbols a list that has a symbol and then some numbers that's okay can be heterogeneous and they grow at the front and they have linked lists kind of performance0:19:49characteristics which means it's fast to put something at the front and it's slow to find the 57000 thing it's linear time to find stuff in the middle then we have0:19:58vectors there in square brackets right again they can be heterogeneous this aims to be one with numbers and one with symbols but you can intermix them they0:20:07grow at the end in constant time but they also offer fast access anywhere in the middle so they're different from linked lists then and then there are0:20:16maps their key value key value key value the commas are optional the keys not need not be key words so this first one uses key words mapping to integers and0:20:25the second one uses integers mapping to strings but you can be heterogeneous in both the key and the value and then we have sets which are just Curly's0:20:35preceded by a hash and you can have sets again heterogeneous of anything and all of the stuff nests one of the important things about this is that these maps0:20:45they scale and are efficient from the very small all the way to the very large so you can use them as sort of pseudo objects and the small and have you know four or five fields if you will or0:20:56entries or you can have a giant map that has you know millions tens of millions hundreds of millions of things in it the same data structures used throughout that range enclosure we don't0:21:07distinguish those two uses at all so all of the data structures I showed you are immutable there's no way to change them0:21:16just like there's no way to change 42 there's no way to change a vector what you can do is make a new vector that's slightly different from the vector you0:21:25started with and you both vectors and there's a technique called persistent data structures which makes this making of a new slightly0:21:35different version efficient the two versions will have a substantial amount of structural sharing going on under the hood and that's possible because they're0:21:44immutable right they can share structure because no one can change them and that's what makes it practical to use immutable data structures all the way0:21:53through the range from tiny things through very very large things it's not copy-on-write right these new modifications that you make still comply0:22:03with the big o expectations you have for the data structure in hand and this is the key to practical functional programming is having this so the idea0:22:13behind closure or one of the ideas behind closure is just this right I'd I had done so much object-oriented programming and like it's just so much0:22:22busy work and so much extra stuff and when I finally later in my career learned Lisp I saw people building very very interesting systems out of much0:22:32much simpler stuff and I tried it and guess what you can build exactly the same systems out of much much simpler stuff and I said well I can't repeat it0:22:43but I was very unhappy basically something to the effect of I have been wasting my time in my career doing what I've been doing I need to do something I0:22:52need to change what I'm doing because I'm wasting my time I'm wasting my life doing it this way because you can build programs the same programs better programs that do the same things with0:23:03substantially simple stuff in fact you can build them out of the data structures I just showed you plus pure functions that take those things and return those things most of your program0:23:13you can build that way a little tiny bits of your program you'll have state you'll have communication you'll have the other the other aspects I like runtime polymorphism enclosure has it0:23:22I'll talk about it in a second but you can build programs out of this and you can build big programs out of it you can build databases out of it right I've done it I've built a database out of it when I made closure I was targeting0:23:32being able to do everything that I used to do in Java and C++ so I'd built broadcast automation systems scheduling systems yield management systems0:23:43election projection systems and exit poll systems in although you know C++ and Java and c-sharp I believe that I could have used closure to build0:23:52anything I ever built right with maybe a tiny little bit of lower-level code in small places but that's the target for closure I wanted to replace what I was doing I0:24:03think the programs are substantially smaller they're simpler and they're much more robust than an object program or address oriented programming I was doing0:24:13to accomplish the same jobs so the idea behind closure is let's make that de Matic let's make that the first customer we do things so the syntax of closure there's no more that you just saw it0:24:24it is those data structures that set of data structures and knows those fundamental things are a format called Eden which stands for extensible data0:24:34notation but it's just a grown up version of s expressions which have been used in the list community for years basically you build programs out of data0:24:44structures and I'll show you some of that in a second so the data structures are the code the syntax is not based around characters it's based around data structures you know the definition of0:24:53what a function is is the list whose head is a symbol called you know fun and whose next thing in the list is a vector of arguments which are themselves0:25:02symbols etc etc but the syntax is described in terms of data structures it's not like there's no syntax but that's that's where the interpretation0:25:12happens and everything that would be special in an ordinary programming language declarations control structures function calls operators etc etc they're0:25:22all just represented as lists with the verb at the front that's it that's the Lisp way also everything is an expression and that's typical of0:25:32functional programming languages that's about all I'm going to say about closure the language but I'm going to show you a lot more about the use of these aspects0:25:42of it I have a little bit more later so Eden this extensible Data notation I showed you some built-in things there's also a way to extend it to new tags that are namespace to allow you0:25:53to describe something new in terms of anything that's already known so you can't make up arbitrary stuff you can't say we'll build a new parser to parse characters and get this thing but you0:26:02can say I have a new interpretation of a vector of two numbers and we're going to call that a point and so on and so forth and you can cascade these extensions and0:26:11build richer things but one of the important things about Eaton is it's meant to be useful as for data and code so you saw in a previous slide I said commas are optional that's kind of0:26:22critical imagine if in between you know for add line in between everything you said in Java you had to put a comma who would like that that would be awful0:26:32right so you can't have stuff like that so I mean how many people have ever used any system that tried to encode programming in like JSON or XML yeah how0:26:44fun is that yeah so this is a little bit of like AWS cloud formation has some functions in there syntax which is normally0:26:54declarative so this is a nested function call but we can't you can't do this right you can't program like this so you get a program using data structures you0:27:03have to have a data structure format that's amenable to that so this is what closure looks like I don't really expect you to be able to read it but I just told you before that's a list right Kirk0:27:13you know a paren defin right it defines a function the name of the function is0:27:22words takes an argument called text and there's implementation and so on and so on and so forth so this is a port of a Peter Norvig spy thong code it's as short as the Python program right and0:27:33and that's not like a contest right it's it it's more about this everything that's in here is about the problem like0:27:44all the words you're reading and everything it's all about the problem there's no extra stuff there's no static import blah type this type that yadi ah0:27:54extra control stuff it's all about the problem 100% about the problem and that's what you want right so it's short it's free of ceremony but most0:28:04important is it's about the problem which means it's a lot easier to look at now and later and see what you're trying to do what you're trying to accomplish0:28:13so once we have this format Eden and we use it for code and we can obviously use it for data you use it for everything right so let's say you want to have a0:28:23DSL to represent HTML right because we know HTML on its own is kind of gross to generate and manipulate because it's XML plus some randomness so this is one0:28:35of many dsls for representing HTML enclosure but it's Eden this is the same stuff that the program was made out of the same reader reads this if I if I0:28:44call read which is available a la carte enclosure I get a vector that has a keyword that then a vector that has a keyword a vector blood I get data I you0:28:53know exactly what I'm gonna get the data structures I'm going to get I have a special thing I have a Dom blah blah blah you know I get data when I read this and that's great0:29:02it means it's easy for me to process this it's easy for me to write a program that produces this so I can produce HTML without an extra special different thing of course we would use this for our0:29:11configuration files right why not we can read it we can generate it we can process it we can manipulate it everything we know all those hundreds of functions we know how to do we can use0:29:21to do this now add up the stuff in Java how much have you got you have syntax for what you use I don't know what some parcel thingy right for that from Java C0:29:31or something to manipulate you have annotations right you have JSON here you have XML there right maybe for writing your own DSL you use antler something0:29:40keeps adding up more and more more different things and if you read stuff with an API what do you get some API authors idea of the Dom for this kind of0:29:50thing right what do you get when you read XML well it depends on how you read it but you could get this you know machine right that calls you back every time it gets a new element Wow0:30:02this is a Hadoop programming Netflix has quite profitably used closure to build a very succinctly DSL0:30:12doing Hadoop and big data processing it looks exactly like closure it is closure sort of embedded in closure and the0:30:21thing is you can run this locally and then push a button and it will distribute it over your Hadoop cluster and run it there same thing and you can0:30:30run your own closure functions there too and they'll ship them and everything else so we just do this we do this everywhere everywhere we want there's a type annotation system for closure that0:30:39uses data there's a schema language for closure that uses data there are many kinds of logic dsls that all use data as0:30:48their as a representation and and this allows you to do something that's very interesting which is to write an embedded DSL now how many people have0:30:57written a DSL and had more and more pressure to make it taurine complete and you know general-purpose alright you start with the DSL and it did X&Y and then people like could use e could you0:31:07have conditionals could you have case could you have blown you know they always want more stuff so one of the cool things about doing dsls in a0:31:16language like closure is that you can sort of you know co-opt all of closure inside your DSL it's like oh you want to do arithmetic well sure you know all you0:31:25have to do is expose something that you're going to you're going to flow through to closure and let close your eval --it so it's very powerful you keep doing this the other big point of0:31:35programs as data is that allows you to write program generating programs and that takes a few forms in the small there's a you know capability and0:31:44closure and other list was called macros they're nothing like C macros from from your past maybe depending on how old your there are functions of data0:31:54structures to data structures but basically it says the compiler for closure says if you declare something as a macro then if I see that in the program I will call your code I will0:32:04give you the form as data that I encountered and you give me back a different form you can do any transformation you want which is you can build your own syntax you can build your0:32:14own constructs you can extend the language however you want you do not need to wait for me or for anybody else or for the Java jsr whatever you just go0:32:25so we're doing all this stuff with raw data structures and and how do we contrast that with objects right this is0:32:35the ranty part of the talk just warning in advance so objects are like marionettes every has their burrito0:32:45analogy but I couldn't connoisseur not like burritos objects are like marionettes okay they have all these methods on them right and anybody who has access to the object it's like0:32:55they have that that control stick thing wikipedia says that the person who has that is called the master mind ER I0:33:04couldn't find anybody else who agree with them but that's a cool name like you can remote control the object which you can write if you have a reference to the object you know it's type you can call any of its methods whenever you0:33:13feel like it and whatever thread you want right just have at it you can do anything you want because you can call those things so whoever writes the object their class they have to defend0:33:22against that because what's going to happen well in the real program you can start passing around references that object well now you have more than one master binder now you have as many0:33:33masked reminders as you've shared references to the thing and anybody can call something at any time and what do you end up doing the end up saying well maybe they're friends and they're standing next to each other and they're0:33:42like okay let's make the fancy horse you know dance and I'll do the front legs and the back legs and that happens in puppetry right but in programming you0:33:52know sometimes they just go off and like somebody's trying to make the front of the horse go this way and somebody's trying to make the back of the horse go that way and it doesn't work and so you have to have all kinds of protection and0:34:02you can't you actually can't effectively do this Java and languages like it don't give you an effective way to do this so you're suffering from this constantly as0:34:11soon as you have a reference leak you're suffering now I know you say oh I use value objects some level you know you don't everywhere you could but you don't0:34:21and it's not ego Matic and it's still hard and you still don't even know like if somebody gives you a reference right to an interface can you know that it0:34:32will start dancing on you is there anything in Java that will tell you it's not going to move around it anyway any construct anything that type system0:34:41anything no nothing so every day you have this unhappy face and then write0:34:51the final problem right we how many people write a program it's just the program I like it sits by itself and it reads you know standard and spit standard out right compiler writers0:35:01right that's that right and they write these great languages are really good at that right but that's not the real world right how do people write programs I talk to other programs routinely yeah we0:35:11do that all the time how people put objects on the wire Wow that is terrifying all right but just so you so you know a0:35:22long time ago we decided that was a bad idea so I don't know you got the memo or whatever but yeah objects don't travel0:35:31on wires so you can fake it you can pretend you can make all these elaborate things but most of most of these things have have failed in practice right you0:35:41can you can get away with it in small circumstances but it's not how things work right we don't actually give somebody a reference to something allow them to sort of remote-control us right0:35:52so you know this is why I was saying before if your API takes an object especially by a reference to an interface right so you don't have the concrete classes definition to sort of make you feel better about it not0:36:02changing do you know if it's going to mutate right can your type system help you with this which is really it's such an important thing for the robustness of your program to control this the answer0:36:11is no you're getting no help at all so your default idioms leave you completely on your own to deal with these problems and really in a position to encourage it0:36:20encourages you to create these problems just by accident right the other thing is so let's say you know let's say RMI is not allowed at your company how many0:36:32people are not allowed to use our my intercompany everybody's allowed to use our Wow götze yeah that's actually not as bad0:36:46but but let's say let's say you're allowed right and somebody says I want you to service eyes your thing how many people would choose RMI over something0:36:56else like HTTP or anything else all right nobody nobody would right because and then what do you have to do0:37:05right you have this interface you said I took an object I expected all these methods on the object and now I have to talk over a wire what do you have to do0:37:15map mapping right object relational mapping object blah map object something map every time you want to get to and from the outside world especially stuff0:37:26you're not writing so RMI is out of the question right you're not going to get like some webs website to like accept your are my calls or make our my calls0:37:35to you right anytime you have to go outside of your box out of your world view your object world view you have to map right so you know at Java0:37:46programmers and object-oriented programs have been like you know kicking sequel saying oh it requires object relational mapping and that's like a problem with sequel no it's a problem with objects right that's objects are not the way the0:37:57world works nothing in the world works that way people do not hand their strings out to other people to like start yanking on them and like that's how we're going to build this is that so we're going to have like a soccer team0:38:06right it's like I'm really gonna have a reference to somebody else and like you know and you call pass to me and like we build this big spaghetti nightmare that's not how the world works it's0:38:16completely not the way the world works it's not how physics works so we say objects on their way to model the real world it's not at all it's a complete programming fabrication0:38:25it's not very realistic it's not a good fit for almost anything that's in the outside world so you can build your own world where all this stuff makes sense0:38:34but it's inherently I would call it idiosyncratic but in particular it's not the way that systems work right systems0:38:43in the large right so what are systems well the word system means to cause to stand and I love that idea I mean I0:38:52always think of with these legs sort of self-assembling to try to make something that stands up this independent parts that you connect0:39:01together substantially independent parts right right because you don't need to cause something to stand that's like one thing that already has three three legs you're not you're not causing it to0:39:11stand up it's the independence of the things that matters and in general we try to build systems in a way that makes them independent right do we want to care if0:39:21another server is using the same programming language that we are or the same runtime in the same version of the Java Runtime or the same type system do we want to build a system like that0:39:30where we care no we don't why because it's going to be brittle right it's going to be hard to make changes we have to agree with the other person right0:39:39tomorrow we're going to have this thing and everything's going to be different you know three to one you know we don't do that the internet does it work that0:39:48way no right we don't we don't do specific stuff we do general stuff and we try to be as independent as possible0:39:57most things that happen between systems use one of two techniques right they use RPC with plain data out and back or they use queues where you send0:40:07data and somebody shows up later and gets the data you just send and you just flow data around this is the way that systems are built big systems big successful systems like the internet and0:40:17most systems right and those systems are flexible you measure their flexibility in terms of how much independence they support can you independently develop0:40:26these parts of system can somebody upgrade they are part of the system and not mess up the other person right because like Twitter is not going to tell your web browser when they change0:40:36their homepage and like Safari you know do something special that's not how it works right everybody has independent development independent time frames then0:40:45the other thing that's critical to this is that if somebody else on the other end is going to change you have to be tolerant of things being different but you can't say well here's our contract you know the nine hundred things out to0:40:54be exactly this way and then I'll work and then you'll work then if we're going to change anything we have to you know have lunch and have a meeting and and0:41:03and decide this stuff again you have to be tolerant and accepting some more things so we don't have to change the lockstep and so these systems are inherently dynamic and they're inherently extensible right that's what0:41:14it that's why I just said systems are made with dynamic types extensible types right they can accept data that you weren't expecting to see and it won't make them fall over and hopefully0:41:23they'll do a good job of propagating it right they're all made this way so this is the other fundamental idea of closure0:41:32we should build the insides of our systems like we build the outsides of our systems all those value propositions that accrue to systems we want them how0:41:41many people want to have a meeting every time they change a class or a subsystem how people have meetings every time they change your class yeah you have to write this go break stuff is going to break so0:41:52we should we should communicate using immutable data inside our systems for the same reason we do outside it makes our systems more robust it makes them easier to change it makes the0:42:02independent parts separate it makes it easier to move them around right we get loose coupling we get subsystem independence we get flexibility and0:42:11what's the mapping well there's no real mapping right this doesn't need mapping right RPC becomes PC right we can we can0:42:20call functions we were calling functions before there was our right then they had our we had PC before we had our PC we can go back to PC we can do that right0:42:30we can pass data to functions and get data back we used to be able to do it then we had all this elaborate stuff now I forgot how to do it right and we can0:42:39implement queues and flow inside our inside our programs using queues or channels or things like that so that's the other key idea now there's going to0:42:50be process and state closure is not a closure as a practical language of course you can have processing state or there's no reason to run your program I0:43:00just make the computer hot and you go home right you're going to have state and effects but this is another area where we're left just totally with nothing in in object-oriented languages0:43:11like Java you have nothing here like there are very fancy functional languages that have purity although they will force you through this system to identify and isolate all the0:43:22parts of your program that could do IO or have any kind of effect they'll either do a vie a purity or V effects systems right and then there's the0:43:31alternative to that is nothing by Muslim most people have is absolutely nothing and then you could also have reified constructs that at least make a state0:43:42change explicit and that's where closure sets because in Java and C++ and C sharp you have nothing we just have nothing you have some0:43:51really raw constructs like mutexes and you know a pat on the back and good luck buddy and read Brian's book so so0:44:02closure doesn't have any purity to it but it has explicit constructs for State these are like you can imagine them being variables that have semantics to them so it's not just like anybody can0:44:11come in and whack on this variable at any time instead you say I'm going to give you a function and you somehow apply that function to that variable to move it from one state to another but in0:44:22doing so you can ensure it's free of conflict and free of races and it's never going to become half of a thing and these variables always refer to values so you're always able to observe0:44:32them or dereference them and get out a value there's nothing else there's only these reference cells that point to values and values there's no mutable object that half of0:44:43which could be whatever or you know like a date class that you could set the month or things like that right a date is a value you can have a reference to a date you can make that reference point0:44:52to another date you can't change a date and see things you can change which are references and you have values which are not you can't change dates anymore you can change 42 maybe you can change these0:45:03references but their atomic they just point to one thing so between those you can get a whole bunch of different variants right Kaz implements a0:45:12successorship model you say only make this new thing if it was the thing I'd you know if my presumption is still valid right since the tiniest version enclosure has a construct that wraps0:45:21that so you don't have to write the right to loop or anything else say here's my function apply it to the inside of that use Kaz do the loop for me and I know I'll get0:45:31a clear successorship there with no race and no conflicts there's also an STM enclosure that allows for bigger transactional kinds of modifications to0:45:41occur but the point is this construct is doing the job and the construct is calling out here is where the mutation is in the system here is where the state is and the thing is you have a way to0:45:52get out of it right if I give you something that you don't know if it's going to change how can you save its value like if I give you a reference by0:46:01an interface to some composite type you don't know if it could mutate how can you save its value what's the safe way to do that you don't know the clone is0:46:12going to work at all sorry does not work what else what's role that gets ok this get that you got nothing you have absolutely0:46:21nothing so this is like a critical thing for making a system that works you have absolutely nothing to do this with you have to build up your own convention0:46:30around this so like I think you should pick right you should either have explicit constructs or go all the way to Haskell because everything in between is catastrophe and then we have a enclosure0:46:42we have something called core async which is a channel model it's a little bit richer than queues because you have to set up the threads and do all the0:46:51micro rights they have semantics that are based around something called communicating sequential sequential processes but the basic idea is that you're going to try to encourage0:47:00especially when you're trying to convey values through a system instead of saying I'll put the Acorn behind this tree and you come by later and find it0:47:10behind the tree you say I'm going to put the Acorn on the conveyor belt and you can take it off the conveyor belt and this is a big difference between those two things0:47:19because if you put something on a conveyor belt and then go back to it what do you expect nothing comparable0:47:28it's moving then that's that's it that flowed so you can't write kind of the logic you can write with variables going back and re-examining a place to update0:47:37it in place and you know try to read it again it flows and so flow is a much more robust way to build a system data clothes a much more robust way than variables so we want to emphasize flow0:47:48over places so program size matters right smaller is better right there there's app this is one of the few areas0:47:57where we have like research right people have done research and said smaller programs have fewer bugs it's just that simple it doesn't matter what programming language of this smaller0:48:06programs are fewer bugs write bigger programs have more bugs longer time to market they're harder to maintain and they're more brittle right but what I0:48:16think is interesting is that there's two flavors of small a lot of languages focus on concision writes which is size in the small like how small is your if0:48:26statement you know how small as a function call how you know tiny are your constructs how much overhead how much syntactic stuff is there and there's a lot of languages that focus on that Ruby0:48:36and Python a lot of languages are actually very good at concision but the the bigger impact on a program overall right it's0:48:47not moving from you know 42 characters to 20 characters that only gets you to X right the biggest thing is moving from more specificity which bloats your0:48:57program to more generality which shrinks it that's the big payoff that's the kind of that's the area where you're going to get a payoff much higher than 2x so one0:49:08of the other things I think we suffer from in object orientation is death by specificity right all the time we have a new thing we have a new idea a new piece0:49:17of data boom we have a new class get this get that get whatever I don't care if their value types whatever right it's just a glorified map except you can't0:49:26even use it as a map in Java right there's no generic way to manipulate something that says get this get that get that so new type new language gots own little vocabulary right so you're0:49:38going to have more code you can have much less for use you can have more coupling right because essentially what's happening is every object has its own little language my class my0:49:47interface my own language this is the my biggest pet peeve I want to get away from this when we saw it right and get this get that it's like this is there's0:49:56no purpose to this this is just life-sucking so let's look at life sucking in there in this is this is just a tiny part I actually skipped this is just the servlet request and I0:50:07have a little bit of httpservletrequest which really doubles the size of this thing but my question to you is how many maps do you see here we're like give a name0:50:19you get a value I got 100 I actually can't do the auctioneer thing how many got yeah alright so first of all this0:50:39game is hard right there there's some I got I got three inside and the overall0:50:48thing is a map too so I got four right off by you know picking it apart what's really interesting is look at these map interfaces they're all ad hoc guess what0:50:58else they're all different one has setting one you can actually get the map when you can get a list some you can get out with types there's four different0:51:07maps in this one class this is crazy right in closure we just use maps right this stuff came over wire in HTTP as0:51:18text how did we turn it into this what happened what happened why you know this is crazy now who can I told you the0:51:28curly braces are maps who can see how many math you want to you know you see all the maps there's like there's because there's still maps right if you're going to write code that0:51:37manipulates that other stuff every single line of code you write is going to be special has to you know use whatever Java X servlet bla right and0:51:48your rank codes explicitly to this thing if there's another way to do HTTP not that the risks or if what's will probably the only way it sort of to HTTP but if there was another way to do HTTP0:51:58will you be able to reuse that code know it's all written it's all hardwired to this person or persons idea of like what0:52:08an HTTP request is right so you okay this is a tiny little benefit right dot works in your IDE whoo oh my goodness because I could never remember that so I bet something better happen0:52:20when I press stop because I'm doomed otherwise of course I could look at the HP spec and you know like we could agree on these names and I just don't get it0:52:29you know you can tell a kid not to put a spoon in a blender and turn it on and like they will remember that for their entire lives they will never make that0:52:38mistake but grown-up adult programmers like weed we need protection right for from this stuff but the protection we0:52:47get is really minimal what's the cost it's huge right that's an idiot in inconsistent interface it's incredibly idiosyncratic the interface is huge so0:52:57if you wanted to like have a second implementation you know get to work there's a ton more code to consume it you can't use any of the libraries you already have right with the closure0:53:06version all the math code like I said those hundreds of functions they work on this you can create this with them you can read this with them you can merge two of these with that and like you have0:53:16no new code to manipulate this no new code all the functions you already know manipulate HTTP requests as soon as you represent them as data which they were by the way before we map them right the0:53:29testing right it's easier to test data can you make a program that makes this yeah can you make a program X one of these yeah okay and then certain the0:53:43other problem is your typical Java program has two to three orders of magnitude more of that more of this0:53:52right 100 classes to a couple hundred how many people programs with more than a thousand classes yeah that's a party0:54:02all right so some closure programs are smaller in both ways they're more concise and they support generic programming because we just program with0:54:11these data abstractions we represent information as plain data so I mean this is always the biggest reservation I love my types I like my dot I like my IDE you0:54:20know I I can't I can't deal with something and it's true right if you have the types of Java right now fancier types0:54:30fancier type systems can do more but if you have the types of Java you can you can catch you know typos and pressing in so when you're supposed to pass strings but it's really likely that your tests0:54:40or your ripple interaction is going to catch that stuff that is not a quality metric that is not sufficient for quality it's part of quality right no typos right but it's not sufficient for0:54:49quality right the quality is and all this other stuff you have where did we say we have no way to deal with state management we're encouraged to write highly coupled programs we're inflexible0:54:58we're not meeting the customer stakeholder quality metric at all in fact we're pointed against it time and time again we're pointed at the wrong0:55:08thing and because our code is so huge we can't even really understand what it does anymore so the biggest source of bugs and programs which is misconceptions right everybody gets it0:55:17wrong I talked to the stakeholder they told me this I didn't think of one of the situations when I wrote the program that's the act those are the real bugs and programs by everything else their0:55:27superficial bugs those are the real bugs they're harder to see so I think this default idioms are a big one unlike I spend much time on this but it was interesting to me because I always like0:55:36to look up words what this economic mean and actually means relating to household management right so the idea of home economics is kind of redundant it's what0:55:46the word means and I would say that you know sort of our programming house is just like it's like a hoarders delight everything there's too much stuff in it0:55:55everything is too big we need too many people to do basic things there's a lot more to closure this was not a tutorial on closure but the important thing is that most of it is in libraries closure0:56:06grows file libraries the core is really pretty vigorously protected against growth it's not like a you know an0:56:15experiment and language design so the one of the part of closure I'd like to talk about is polymorphism and it just because it's another example of simple I0:56:25haven't talked a lot about simple but one of the cool things about closure is that polymorphism is independent in other words it doesn't require inheritance you can imagine has0:56:35something called protocols they're a set of functions work together there polymorphic on the first argument so that's like the same kind of single dispatch you have in Java0:56:44so you can imagine them as interfaces but they don't require inheritance and the beautiful thing about not requiring inheritance is you can have a protocol and you can extend it to something0:56:53that's finished you know something that son wrote a long time ago and is never going to change and certainly is never going to implement your interface right you can also take stuff from two vendors0:57:03right right because what we usually have we have the framework problem there's a privileged framework in Java the one that comes with it and people implement those interfaces but if you have a piece0:57:13of software from vendor a that has an interface and stuff from vendor B like objects from vendor B that you want to use how do you get vendor be to implement vendor A's interfaces it just0:57:23doesn't happen which is even worse than the C++ world but in Java we still have this problem there's a privileged framework that people invent the interfaces and otherwise implement interfaces are parochial or small so we0:57:36have polymorphism enclosures is a la carte and that reduces couple of coupling because you don't need to derive alright I know Mabel maybe closure seems more0:57:45interesting now so it's not just about technology right it's not even about these programming things it's also about ecosystem things like that and the first thing that's great about using closure0:57:54is that you get to keep connection to the ecosystem you already know not only the runtime and the deployment environment but those libraries right the interrupts in both cases is0:58:04extremely good all those closure data structures I showed you they all implement the appropriate Java util map whatever they all implement all the Java0:58:14interfaces you can just take one of those things you wrote in square brackets that was pretty easy and pass it to something that expects a Java util list that implements random access right0:58:23that works ready to go the closure is also very stable you know I value that in Java I think it's important part of0:58:32why Java grew and closure takes the same approach is not like kids on github hacking away adding every new idea or it's not a think-tank experiment and0:58:41it's made for production use and it's very stable all the programs from a long time ago still run there are books if you wanted to get started there's a lot0:58:50of books now foreclosure you know I spoke here last five years ago and people have never heard of closure and now does plenty there are tools oh look does an IDE that0:59:01looks like eclipse with code highlighting foreclosure and like structural NAB and oh that's IntelliJ same thing breakpoints look and you type0:59:12and it starts popping stuff up this is good we're good and you have a ripple down there which is even better once you get used to that there's a ton of tools0:59:22in various areas there are lots of libraries that says 12,000 repos on github there are lots of users the0:59:32mailing list has almost 10,000 users on it and they're all happy nice people I promise now they are I think that matters if you've ever seen the old list0:59:41community they weren't all happy nice people but close your users are happy and nice people that's where closures add in red monks where our language is that like way up0:59:51there this functional Lisp what is happening in the world look at it there it is on the tech radar adopt it's actually gone off the tech writer the1:00:00like of course you should be using closure already and and there right right people are using closure already a lot of people are already using closure banks use closure plenty of startups use1:00:10closure plenty of analytics houses use closure so people are being successful with it so I think that's the short1:00:21message for today the idea behind closure is to get you to better and more flexible programs sooner and the way it approaches that is by1:00:31being data oriented and simple and I really appreciate your time thanks0:00:00Simple Made Easy 2012 - Rich Hickey
0:00:00 thank you very much thanks for inviting me to give this talk I know everybody's0:00:09thinking the same thing which is uh what's with that hair and I'm trying to get into the Foo Fighters but so far0:00:20they haven't called me back so we're gonna talk about simplicity today and I think it's super critical in fact it's0:00:32often said everybody agrees it's a it's the most important thing but it never gets its own track for some reason so0:00:41try to make up for that and Dykstra said it's a prerequisite for reliability but you if you replace easy easiness ease is0:00:52a prerequisite for reliability I don't think anyone would agree so these words mean different things and the point of0:01:01this talk is to point out our easiness culture and try to give people who are trying to pursue simplicity vocabulary0:01:11for talking about it in their own organizations and pursuing it in their own software so it's always fun to look at words and if we look at simple the0:01:20root of the word is simplex which means one fold or braid or in twining and of course it doesn't make much sense to0:01:29have only one braid so something with only one braid is unencumbered with anything else but complex means to twist0:01:38the braids together or two braids together and obviously that's the source of complexity if we look at easy the0:01:47root of this is a little bit vague er it comes from some french word and then it's speculative whether or not that comes from this latin word but i I like0:01:56this derivation because I think it really points to something critical right so this in this derivation it says the easy means near by or at hand you0:02:06know to lie near you and I think that's very critical we care so much about that now it's like what can I do in 10 seconds and0:02:15it's hurting us so simple means one fold what does that mean what does it mean to fold something in software well you0:02:25might think about it as one roll one task that something has to do one idea that something is about one dimension in0:02:35which something focuses like security would be an idea of a dimension well it's critical to understand about simplicity is that it's not about just0:02:47one instance you know counting something or one one operation you know having an interface with one method it's about not0:02:56having the interleaving okay it's not about the cardinality but the the other thing that's interesting about simplicity is that its objective people0:03:06are like oh that's easy that's hard it gets it gets wishy-washy but simplicity is actually pretty straightforward things are either twisted together or they aren't so that's a great property0:03:18if we look at easy again we said it means to be near and how do we translate that to software and development well0:03:27one one thing is it's like it's already installed that's it that's an idea of something being near it's in our tool set it's what we always use it's our IDE we can get it easily by using some0:03:37command-line tool the second notion of something being nearby is that it's something that we already understand it's familiar to us0:03:47the third notion which i think is also interesting but somewhat challenging to talk about is to be near our0:03:57capabilities this is a sensitive subject right because we're in the problem-solving brainy smart people business and so to acknowledge that0:04:07something is not near our capabilities is is an ego hit and to say that if someone else is very inappropriate but0:04:18the fact of it as we'll see in this talk is uh from a capability versus problem size standpoint we're all in the same boat no one's0:04:29really significantly smarter than anyone else so but but it's important to understand that we have limits to our capabilities and finally as I said0:04:41simple is objective and easy is particularly not and it's not just subjective its relative once you talk about nearness right you know it's you0:04:52know to say something is near implies near to what so there's always this relativity to easiness you're somewhere the other thing is over somewhere else0:05:02how far apart are you is there is a measure of how easy it is or easy it will be for you and and I think that we0:05:11we have real challenges here we repeatedly choose the familiar and to the extent we do that we're never going to learn anything new and we're really never going to have significantly0:05:20different results so how do we how do we pull this stuff apart well I think one way to look at it and the most important one you know we often talk about tool0:05:30fixation or methodology fixation is to distinguish the construct that we're using in our program or the tool or the0:05:39method or the language from the artifact right we're all fixated on what is it like how many characters do I have to type can my IDE automatically redo it0:05:49the convenience aspects from a business standpoint actually the ability to replace programmers is a driving force for repeating the same old stuff all the0:06:00time it actually shouldn't be a programmer objective at all versus the artifact and what I mean by artifact I mean the0:06:09program the thing we're actually writing the thing that has to run that has characteristics to in terms of complexity and and as Dijkstra was0:06:19trying to point out if we want to make software that's reliable that works well it needs to be simple and furthermore if we want to be able to change our0:06:29software or maintain it either to correct problems or address new requirements it's got to be simple so we have to stop assessing the0:06:39the tools we use and the constructs for use and the languages we use from there from our own si ooh this is fun for me oh I can do this with two fingers it's0:06:49not about that it's about what we deliver and how that is in practice over its lifetime so I talked a little bit0:06:58about cognitive limits and we have to recognize that they exist right we can only make things reliable that we can0:07:07understand and we can only juggle so many balls and somebody got the0:07:16misapprehension that I was going to implying that software is like juggling software's not like juggling it has the same sort of cardinality to the limit0:07:26when you're trying to consider a problem or a bug that involves X if X is complex that means it's somehow intertwined with Y it means that when you're trying to0:07:36understand what's going on over here you've got a load into your head what's going on over there because they're both interacting with each other so intertwine things have to be considered0:07:45at the same time that means that you know we have a big problem with complexity right every time you combine two things you twist two more things you might say oh I need to I need to think0:07:55about this because we haven't a problem here and pull it up and get this rat's nest of stuff that's been attached to it and everybody's been there the other0:08:05critical thing about things being simple is how easy are they to change right so we need to solve problems with them so we need to be able to think about them to do that the other thing is we need to0:08:14be able to enhance them and again understanding is critical here if we can't understand our software we can't move it forward we can't say okay it0:08:23does x and y in order to make it do Z the implications are gonna be this or that how do we make these decisions if it's if it's too hard for us to understand I think that's all very0:08:34difficult I don't think having a great test suite is the way to be able to change your software without fear I don't think that at all it's your0:08:44ability to understand your software that allows you to change it without fear right because if you didn't think about it for your tests they're not thinking about it and I'm not talking0:08:56about reasoning and sort of the proof sense I'm just talking about casual reasoning to be able to sit down in a room with your teammates and talk about why something is going to work or not so0:09:09in addition to our typing fixated culture and IDE fixated culture I also think we have sort of a test and safety0:09:18oriented culture where people are there's nothing wrong with testing and safety but people are believing that it0:09:27does more than it does okay so what we like to say what's true of every bug in the field that you found in the field right it passed the type checker its0:09:36type safe right to the extent you're using a language that does that checking and it passed all the tests so when you pull it back from the field you're not0:09:45gonna be like well let's run those tests again because you know maybe they didn't work the first time and we can find our problem you know they're not going to ever say anything that they didn't say0:09:55before you shipped so you need to be able to think about your program you're you're done it comes back to you it isn't working the stuff you use to make it you already0:10:04used it you got nothing else but your brains and your program to try to figure out your your problem so again our ability to reason about things kicks in0:10:15we talked a little bit about development speed because here too I think we're making a short-sighted decision right we0:10:24definitely are emphasizing early speed and there are certainly projects for which you don't care you have to get done in two weeks no one's going to use it in three weeks does not matter but in ignoring0:10:36complexity is definitely going to slow you down what happens is you know you start off you have your first meeting you have there's no software you've got your0:10:46methodology you've got your plan you know you're using some techniques you love your tool set but that you're it's just dreamland0:10:57there nothing's happening and you feel really agile because the first week obviously you're going to make something and it's going to be more than and better what you have the second week but as0:11:06time passes what's going to happen is your program is gonna be you know a new team member and it's going to grow and grow and grow it's going to turn into an elephant right and if it's not simple0:11:17it's gonna come into your stand-ups and just trample everyone because you're your artifact actually dominates what you can do my agility is not about0:11:27process right it's about doing right not doing over and at a certain point in time your program your artifact is going0:11:37to dominate what you can do so you want to keep it keep it simple one of the problems we have I mean there's nothing0:11:46wrong with things being easy in fact the point of this talk is to take simple things and make them at hand so you can use them the problem is a lot0:11:55of things that are easy are complex or complicating they're easy they're easily described for instance assignment would0:12:06be an example of this right everybody understands assignments oh look it's a new x equals five we're done what could be easier everybody knows that but the implications of x equals five in the0:12:16middle of a block of code are profound and really difficult to manage so you have to you know sort these two things out they're readily available these0:12:26kinds of tools and they're easy to use so I think we have to revisit our tool set to assess each construct that we use0:12:38in terms of what it produces and how complex is that is that result because every bit of complexity that's introduced by a programming construct or0:12:51an approach to things is incidental complexity right we're an incidental is a latin word it means that it's your fault we don't have we don't have anyone0:13:04else to blame right so this guy is using a loom he's only going to one kind of result out of using a loom if you're using a loom I don't expect to get0:13:13something other than a knitted thing out of it so now we have the two kinds of things you could make you have Lego castles and you0:13:22have a knitted castle at castle on the top there is actually knitted out of yarn all right which one's going to be0:13:31easier to understand right which one's going to be easier to change or debug it's definitely the Lego one right0:13:40trying to change the knitted castle I you know it's just really really difficult you get a lot more flexibility also right so there's this whole0:13:49maintain maintainability I think everybody understands that but the other thing you get out of not having things all intertwined is flexibility right as0:14:00you want to move forward right so you obviously have maintenance tasks you know go back and move the you know the parapet from one thing to another or you know add a moat but you also have0:14:10architectural flexibility that you that you lose when you've walked away from simplicity things like being able to change how the policy of some part of0:14:19your application works or as we'll see later in this talk whether or not you can move it move things move subsystems is is a good example of the difference0:14:30between those two things and again I'll point out the fact that testing and and and type systems are completely orthogonal to this right like you can0:14:41test a knitted castle right but it doesn't it's not going to make changing it any easier or go any faster so how do0:14:52we make things easy right because we want to maybe want to choose some simpler things but we want them to be as easy as the things we consider too easy today right the first thing is just you0:15:03know get it on your box install it you know download it or get it approved the second thing is to learn it learn about0:15:12it take some take some courses or go to a seminar or read a book these first two aspects of making something easy are they're up to you and you can easily do0:15:23this you just choose to do it and you spend your time on it and you've made something easier for yourself totally up to you but there's that third part of easiness0:15:32right we say that hand familiar then near our abilities this third part I think is is pretty tricky right because0:15:42how quickly can you change your mental capability now very and as I said before it's not as if it's a race between smart0:15:54people you know we're all smart the the problem is it's it's the the problem is this big and you know our brains are that big so you're not going to move0:16:03them so again easiness we said was relative right it's the distance between your problem and yourself right so if0:16:13you can't become significantly smarter by orders of magnitude in a short period of time what are you going to do to change this distance this can't move you0:16:22have to move this over right so the thing that you have to do is simplify things in order to get them to become easier right you can learn about them0:16:32that's up to you if you want to understand them better you can't get smarter so you got to make them simpler and move them towards you and you know0:16:44again the juggling analogy I think is is sound in fact the numbers are pretty pretty close right how many how many ideas can you simultaneously keep in your head that's like seven plus or0:16:53minus two right and like the best jugglers in the world I don't know what they can do is it's not more than probably the dozen balls so the the cardinality of those two operations is0:17:03is similar so let's look at a simple example I have a couple of more involved ones later which is you know it's a it's0:17:12a frequent complaint of people who are asked to learn or look at Lisp is that parentheses are hard and it's true they0:17:23are hard right from the easy hard perspective they're not nearby if you haven't been using them they look like they're far away you you don't have them0:17:32installed you're not familiar with them unless you you know took a course in in college so from the easiness at hand perspectives they are far away0:17:43but then there's the you know the last one right cognitively are they simple or complex and it ends up that they're not simple and the reason why I use this0:17:52example is not it has nothing to do with lists but it's a great example of that cardinality aspect of simplicity that simplicity is not about having just one0:18:01thing right because in Common Lisp in and scheme you know they did everything with lists and they did everything with parens so they only had one thing you know what could be simpler than having0:18:12one thing well it ends up that having one thing is not simple right because this is these languages that had only only parentheses and only lists they had to use them to represent everything so0:18:23they're very overloaded right they use them for calls using for grouping they use them as data structure it ends up that overloading that one thing made it0:18:32more complex and the thing that's simpler then one thing is more than one thing so for instance in closure one of the things that we did was use vectors0:18:43for grouping instead of parens again and now as it was much clearer parens almost always mean a call and vectors now mean grouping so by having more than one0:18:53thing it ends up being simpler and this is a this is a general principle that can apply to your own stuff if you find yourself doing a lot of overloading0:19:02you're introducing complexity right because you're saying oh look there's only one thing yeah there's one thing that somebody types in and there's two things that somebody looking at what0:19:11they typed in has to juggle to try it what is this what is this this case or that case every time they look at an overloaded thing they see two possibilities or more depending on how0:19:21much overloading you're doing so again this is this is sort of a general principle it's not about cardinality it's about how much intertwining there0:19:30is an overloading is essentially intertwining so let's talk about0:19:39programmers this is an old quote talking about lists programmers who knew the value of all the constructs they were0:19:48using but not the performance implications or costs or memory costs of them but I think we can move this phrase right to the modern error and apply it to0:19:57all programmers programmers know the benefits of everything but the trade-offs of nothing right all I see is0:20:06so look at this it does this oh look it has that oh look it's you know it's it's this fast never do you see but in order to be that fast it does this or in order0:20:16to be that simple it has this other thing it's all benefits and no trade-offs and that really is hurting us so I'd like to go and look through some0:20:27of the things that we use and see where the complexity is in these we have state0:20:37and we have objects and we have an alternative which is values we have methods built into objects and instead0:20:46we could be using functions and namespaces we use variables but we could use constructs that are that have stronger state transition semantics we0:20:58use inheritance and switch statements and matching but we do have choices in some languages to get polymorphism that's not that doesn't pollute your0:21:11types the way inheritance does more Alec Hart we have syntax often when we should have data we have actors and a simpler0:21:24alternative would be queues yeah this one is interesting we have0:21:34conditionals and and we could have rules and I think I think doing more rule oriented programming is is definitely in0:21:43our futures if we want to make more reliable programs and finally we have inconsistency right or eventual consistency which is a fancy word two0:21:53words for inconsistency and consistency now I don't mean to say that everything in this simplicity column is inherently simple some of it is just simpler than0:22:04the thing in the other column but you got to look at your tool kits right because you have choices so I want to0:22:17bring back a dead word called complex it means to intertwine or braid things because you know we saw simplicity means0:22:28one one braid and complexity means two to entwine things but I don't think these are good words for software and you know like it you braided that on me0:22:38so we're going to bring back this archaic word it's a verb complex and when you when you unnecessarily combine two things together or intertwine them0:22:48you're going to say you complected that and when you're sitting in a meeting and somebody ruined your software by doing that you could say were you complected my thing with this other thing you don't0:22:57want to do this right this is where complexity comes from from complected and we complex we do it we guess we just choose to do it and when we go and we do0:23:06it we have to be more cognizant of when we're doing it so we have this pretty picture here where we see four strands0:23:17drape straight down and if I ask you to say you know where does the third strand end up looking at the first picture you know that in a second right in a flash if I say where does the third strand end0:23:27up in the last picture what do you have to do who's got it right what are you doing what are you doing you're like following going behind0:23:38looking through the twists this is what we do right every should recognize this you know this should be touching the same parts of your brain that you use when you're programming right this way I0:23:47do all day long I'm trying to like figure out what's gonna happen and it's more difficult the more complected it gets so and what's the other problem0:23:58right if this has already happened what do you have to do if you want to move forward you have to fix it this is one of the things we often have to fix right0:24:07it's not like we got we made a mistake oh we said plus instead of - that's a mistake we have to fix our mistakes this was not a mistake right this was like we0:24:18went out and did this we we took out our knitting needles and went woo and then we need to fix our software and the first thing we have to do is undo this disentangle right the whole notion of a0:24:29knot in softwares you know I think well well understood so we know how to fix this right we have another great word0:24:38which is composed which means to place together right and composing simple components is the way to build robust0:24:47software that's it yeah this is talk right this is like all0:24:56right yeah everybody knows that's it yes everybody knows all of this but we sit down to write software and we don't0:25:05think about it anymore so let's look more carefully at this component based approach right so we know these things right modularity is the key to making things simpler all we0:25:16need to do is segregate our modules and we'll end up with simpler software but we know it's not that straightforward right because the blue piece could have0:25:26some knowledge about how the yellow piece works and the yellow piece could have some knowledge about the blue piece works and they're both sitting there in their own modules and you're like what what look this is separate but they're not unless you are are making sure to0:25:40make sure they're both only ever thinking about some abstraction that they share you could end up with things in separate modules that are not simple0:25:49they're actually very complected so the physical either partitioning or stratification it does not imply simplicity right you can you can after0:26:00you've made something simple that's a realization of it right but this doesn't go the other way right the implication only moves in one direction if it's simple it'll be easy to make it modular0:26:09if it's modular eyes it doesn't mean that's inherently simple so you have to be careful of your current organization state however is unambiguously never0:26:19simple right it essentially is complex there's no way to uncomplete it it0:26:28because it combines value and time together all right programming languages make this extremely easy to do but the0:26:38fact is that once you've used state and a particularly pervasive use of state it's going to cause everything that touches it to become interleaves and0:26:48everything that touches it either directly or indirectly right if you have a stateful thing it doesn't matter if the things that are interacting with it do so through an encapsulated interface0:26:58or through modules or anything else like that right they talk to you at one point and you know they give you X and you say Y and some other point they give you X and you say Z0:27:07Wow how Nelson what's their decision-making process they can't treat you as a very stable constant thing that you're now this the source of of change0:27:18there will be sources of change you can't build systems that don't have any change right but this is not something to slather all over your application it's something to say I am really scared of0:27:28this I'm going to make sure when I use it I'm going to be very careful and explicit and put warning signs around it and say look there's some state here0:27:37here are the rules about it this is what we're gonna do state is this problem has nothing to do with concurrency nothing's0:27:46zero to do with concurrency state makes your program more complicated even if you don't have any threads so some new0:27:56languages have constructs that make it more explicit right so you can say this this is a vowel or this is a var or this is a ref and it's important to note that0:28:08none of these things actually make state any simpler I said it's essentially complex but they do help you know put0:28:18signs on it look here's here's the state and when and when they have a way to say this is never gonna change right like final is a great thing you say final people don't0:28:27have to worry about that anymore ooh that's final that can't be my problem that can't be changing that's really great a couple of languages have0:28:36reference types that actually put more constraints over the assignment to variables such that they can with a0:28:45combination of an API and the pattern of use make it so that the only thing you ever put in a variable is a value that can't change another thing you can ever get out of a variable is a value right0:28:55as soon as you do that it's very simple it's much simpler just to manipulate state because there's the state you get something out of it now you're out of0:29:04the state game you're not constantly referring to something that can change you now have the value and you can use that and the critical thing I think for a lot of programs is you know if you're0:29:15using more than one variable to represent a thing you're doing it wrong you you have got a problem right because now a sudden you have a big coordination0:29:25problem right if you have a single variable that holds a composite immutable value you can easily atomically transition from one value to the next if you have four variables to0:29:35make up a thing how do you how do you coherently transition from one state to another now you get into locking and all those other kinds of problems so I would0:29:46look for constructs that help you with that okay so let's dig in a little bit more to these constructs and and how do they0:29:55how are they complex I just said they were so how is that is that the case we already said of state right it can't it0:30:04complex everything that touches it right because I'm counting on you and you say something different every time then somebody's counting on me how could I not say something different every time because I'm counting on you0:30:14something's counting on them and it's just a complex your whole system objects especially in the very traditional sense0:30:23or is delivered by languages like Java boy it's just a pie I mean you can't you just go on and on and on one is a class it's got state it's got an identity it's got values got operations it's like a0:30:33kitchen sink of stuff poured together and every time you take one out to solve a problem you're starting you with three or four strands already wrapped together0:30:43you've already got that much just when it's blank you know class X open closed you know what does it have a final0:30:53constructor how do you make it how do you copy it what are its concurrency semantics what are the set of methods that can do what if it doesn't do what I0:31:02wanted to do how can I make it do something else it's derivation stuff all baked in blob I mean it just goes on and on and on these are terrible constructs from a complexity standpoint methods are just a0:31:12subset of that but the same kind of thing to the extent you have a method on a stateful object you're complected us to things syntax is an interesting one0:31:23right we love our many languages and and whatnot but syntax is again is it by definition is a complexity of two things because syntax means0:31:32to derive meaning from the arrangement of something so the meaning of something and the arrangement of something are combined and we'll see in an example0:31:42later how subtle that is but it comes up all the time inheritance complex types right you're saying this type is complected with that0:31:51type and so it's what you're saying when you say inherits or extends switching and matching right complex a set of inter entry points every says or you0:32:01should prefer you know polymorphism to that because that is complected right that it makes a point in your program that's baked in as a set of decisions and the more you do that the more places0:32:11you have to change when you change that decision variables we talked about already complexing value in time loops0:32:20complex what you want to do with how you're doing it or do this now do this now do this now do that you've not walking away you're not up a higher level saying I just really want to map0:32:30this function across this collection that's a way to say it say what you want to do without saying how you want to do it but when you look at a loop those two things are together right constantly in0:32:40Java for instance where we're writing loops and every time somebody walks up to a loop they have to figure out is this loop summarizing what it's going over is approaching another collection0:32:50that's a modification of what it started with is it changing the thing it's you know you have to reread it because there's no way to say up at the top without a comment this is a mapping this0:32:59is a filtering this is a reduction you want to start using functions like map and filter and reduce so that you don't have to parse your loops because they've0:33:08complected these two things actors complex what has to be done with who's going to do it yeah again0:33:18and and conditionals are also interesting you know so we say if this blah blah blah and we have you know these kinds of conditionals and we put a0:33:27lot of important business logic into that and that one of the problems with conditionals is that they're situated right if I say if this is a good customer do whatever you know that0:33:36expression is like in the middle of part of a method of in the middle of my program and and everything about what makes that true has has to do with like0:33:46where it's situated it's great if we can somehow lift that stuff up right because of what we've done here is we've complected why we're making you know0:33:56doing this or that with where it sits in our program so what about some alternatives and in what ways are they0:34:05simpler values I mean if I could say anything it's that you should be programming with values as much as you0:34:14possibly can because they're dead simple they're essentially simple like we said some things were essentially complex they're essentially simple right so you0:34:23can get this from using final pervasively or using persistent collections which give you collections that are actually immutable functions0:34:34are simple right when you do when you try to test a method what do you have to do you have to set up the whole context the state of the thing and everything on the whole world right the method is0:34:44related to the whole world when you test a function anybody function I mean you know a stateless method that has no side effects okay it's the same every time you can test it completely in isolation0:34:54it's a much simpler thing namespaces are simpler and if your length language has them that's where you get them another big big one and we'll see this later is0:35:04data data is is essential we were supposed to be writing programs that manipulate it and the first thing we do is we ruin it instead of just using it0:35:14directly so use your data structures in your languages PI morphism a-la-carte is great this is harder to get if you don't have these kinds of constructs like0:35:24protocols or type classes things like that we talked about manage references you can get set functions right that kind of like map filled reduce that you can get libraries that0:35:33let you say map instead of making you write a loop you should use those cues again you can get from libraries declarative data manipulation again is a0:35:43little bit harder but certainly things like sequel and data log are great and if you have the ability to express some of your program using these things you0:35:53should do it because they they separate what you want to accomplish from how it gets done and that's big rules again you0:36:02can get from libraries and consistency you get from transactions so you have to be careful when you walk away from consistency because it again is going to0:36:12affect the rest of your program now there are some kinds of complexity that we can't do anything about I would call environmental complexity a lot of these have to do with the fact that our0:36:21programs are parts of our programs share stuff with either other programs or other parts of our programs right they're sharing resources memory CPU and0:36:32they're all contended for and that yields all kinds of problems right you get garbage collection problems or memory out of memory problems the0:36:43problem here is that we don't really have good answers for these right if you try to segment your shared stuff then you have waste right I said you get half I get half you know if you're not using0:36:52your half we're wasting our computer but then so far most of the solutions to the environmental complexity problems don't0:37:02compose well right so like I have a thread pool that's great ok this like 14 other thread pools in this application now so how's that gonna work right in0:37:12other words individual good decisions don't combine to be a good set of decisions this is a tough area so I would say that this kind of complexity0:37:21is inherent and you know that's Latin for that's not your fault you do need to think about these things but there's no getting around them this0:37:30stuff you're gonna going to encounter so we had abstraction bashed pretty well0:37:40yesterday and III think that there's sort dangers and over abstracting but I think abstracting for simplicity is never0:37:50wrong right if you're if you're abstractions are driving you down right driving down the sets of things you have to deal with you're you're gonna be better off and0:38:01maybe one of the simplest things is to say okay we know this danger isn't over abstract thing you get ten abstractions your team's gonna have ten abstractions0:38:11only ten pick the ten that you want it's going to drive you to a lot of things I showed you on the previous slides right it's gonna drive you towards well I wanted instructions that let me map and0:38:21filter I want abstractions around your data if I'm only gonna have ten I can't have this elaborate I can't make I process factory collection of resource manager0:38:31or whatever abstraction right because if that's one of my ten I'm doomed but having ten abstractions will make your life substantially simpler so I0:38:42think one of the things about abstraction is to recognize the difference between abstracting in order to simplify and abstracting in order to hide right if abstracting is like you know putting0:38:52paint on it or you know just like putting it in a box and closing it because it reeks too bad that's not good that's not actually you're not accomplishing anything there it's not about hiding things I used to0:39:06have a t-shirt the student made for me that said I don't know I don't want to know because I said it's so often in my C++ class this is the approach you want to have this is really what you want to0:39:16be doing you want to be doing you want to say so he says oh I have this thing you just need you to blah blah blah you're like I do not want to know that you have to make it possible for me to use your thing and not know that and0:39:26when you do that you'll be forced to decent abstractions as opposed to sort of fabricate abstractions solving problems that nobody has all right so0:39:35I'd like to do a couple of examples to make this more concrete and I love this quote I found it as I found everything when I'm doing talks by you know like0:39:44just I'm doing talking about excellently just looking stuff on the internet so it's not like a bread francuzzi but it's a great thing because it says that there0:39:54is there are essentials to things and when you find them and learn to recognize them things will become simpler so I want to talk about two0:40:04things in the last two examples here the first is order and the second is this information so we're gonna talk about0:40:13lists and order a list we all know what that is it's a sequence of things right one thing comes before the next thing but any time you encounter a list you0:40:22have this question does the order of things in this list matter right because you can have different kinds of lists like you might have a list like this it's all the same stuff there's one0:40:32thing and two things and there might be three things or five things or seven things you get a sense of this as being sort of a uniform collection of stuff an order doesn't matter right it's just0:40:41this is the stuff I put it in a list just you might want to access it that way versus a more couple like use of a0:40:51list that says well the first things going to be the depth and the second thing is going to be the width and the third thing is going to be the height and immediately you're starting to say wow I'm not sure I would hope you starts0:41:02to say I'm not sure I love that way of talking about this because like I'm gonna forget or the order of these things right and it's easy when when you0:41:11have undifferentiated sets of things to just use sets instead of lists right if you use set it advertises to people the order doesn't matter this is a collection of things and I'm telling you0:41:21the order doesn't matter and that they're unique so that's it that's a nice thing but we're still faced with sometimes we don't need sets we need we0:41:30need lists and sometimes we have named parts what should we do well why do we even care about this why make a point out of it right it is a complected right0:41:40to complex one thing with the next what's the next thing you're going to counter and if you moved one or inserted something what would happen it also infects every use right if I'm using0:41:50your with depth height loop I already made a mistake because I think it was depth width height or something I forgot already so the usage points get infected0:42:00it inhibits change right if you start doing this and inhibits change right if you have name and email and you say well I'd like to make phone the second thing what's gonna happen to everybody0:42:10was using your stuff they're gonna break now at this point everybody have IDs now so like my ID will fix it oh man it may0:42:20fix it for you but if this is your contract over service interface it's not gonna fix it for them so everything all right but this is obviously dumb right0:42:31we're not going to do this we're not gonna put you know particular things in particular places in the order of the list right we never do that we would use a map or hash or associative thing right0:42:42so we never do this right we do this all the time it's baked into a ton of things that we do a ton of things right we do0:42:52this every time we use padishah positional arguments right that's I can have a function that took with that tight right every is familiar with that you might never use it as an interface to a to a service but you do it every0:43:02day when you write it functions right now I'm not saying of all these things that I'm criticizing as being complex that they're bad and you should never use them right I'm what I'm saying is0:43:12they had these complexities and you should understand them before you apply them and you should know I am making a trade-off right I may make this trade-off I'm for instance closure made0:43:21this trade-off it has positional arguments a lot of languages have positional arguments and few have named arguments but you know you'd have to do that with eyes wide open you're making a0:43:32choice that's going to introduce some complexity syntax the same thing we said order matters it's an example of this sort of fundamental nature right the0:43:41ordering problem we could call it product types if you use languages that have them are exactly like that with depth height you know the type of that thing could be float float float you're0:43:51lost you got nothing right you could rearrange those things or get them wrong they're telling you nothing about the semantics of what you're doing imperative programs earnings above the0:44:00order problem right can I take X equals 5 and move it up three lines or down three lines no it means something different when I move it0:44:09around Prolog suffers from this problem and it's not as if I feel like you know people are using Prolog but you know if you're using some declarative language0:44:18or something somebody's given you a DSL or something this is a question you should ask about it does the order of the in which I say things matter because if0:44:27it does that's an that's a negative cold chain or an example of this right so this object object X called object Y calls object Z that's an example of with0:44:38death pipe you've baked into your program this chain of things right and if you want to say oh I want to add a step you can't just add it right you0:44:47have to change the guy before and potentially the guy after but at least the guy before he needs to know he shouldn't be calling X shouldn't be calling Y actually be calling W and W0:44:56needs to know to call Y and then Y call Z right XML is another great example right what is the M stand for an XML0:45:06markup what are they marking up documents what do documents have they have ordered order matters tremendously two texts0:45:15right I can't just take the words in a sentence or sentences in a paragraph or paragraphs in a document and switch the order and end up with the same meaning documents really have this thing but0:45:25we're using this for non documents all the time how many people are actually using XML for text as markup anymore nobody but tons of people are using it0:45:34for data where it stinks right because even if by convention you've agreed that you're going to use XML in it in a date away in a data like way maybe even a0:45:44very rigid way you're gonna say we're gonna make the elements in each guy as if they were entries in a map now you can agree you can by convention0:45:53say we're going to take a simpler use of XML but your XML tool you're gonna pull out does it know that you're doing that no it doesn't know that you're doing0:46:03that because it has to support that order problem it's got the nastiest stupid Dom or sequential event triggered interface which you now have to use on0:46:13every piece of data you choose to represent in XML it's an example of the ordering problem so we have choices0:46:23right instead of using positional arguments we could use named arguments if our programming language supports it or we could pass arguments in a map which gives us again that nice named0:46:33labeling and independence over where things appear instead of using texts for things we can use data right we don't tend to use syntax right when0:46:42you think about going out of the box right going out of the box and making certain things service-oriented there's really helped us because we've seen all the goofiness we do inside our programs0:46:51that's no longer tolerable when we want to talk service wise right I can't say you can talk to my service here's my programming language for talking to me here's my language who does that who has0:47:02I'm gonna make a service I'm gonna give consumers of the service a language they can talk to me in no one but inside our programs we're happy to do you know all kinds of syntax stuff product types you0:47:13can replace with records imperative programs so you can replace with declarative programs in the keynote yesterday morning we're talking that it was said you know it can be difficult for people to know how to map their0:47:22problem to Map Reduce right why is that because right now if you use like the Java interface to Map Reduce you know how things are done is like in your face0:47:31you want to be talking about what you want to do there are other interfaces to Map Reduce that use data log where you have no idea how Map Reduce works you0:47:40say I want to accomplish this go and it figures out how to do Map Reduce moving to declarative programming will really improve your programs and take away these problems because you shouldn't0:47:49need to care about how Map Reduce works you can replace call chains with cues right X was calling why I was calling Z if extra spending those results on on a0:47:59cue and Y was wired up to consume from that cue then what would the person who wants to have X send stuff to W need to0:48:08do just rearrange the cues not the not the caller's right rearrange the consumption of the cues you've now gotten this policy0:48:17independence right because you have less complected you had that degree of independence I talked about before architectural II my things that are simpler can be rearranged in a more0:48:27straightforward way and XML we can place and sort of have already with a more explicit representation I mean JSON is not the best thing in the world but it's got at least you know0:48:38this list type is a list and this map type is a map it's clear that it's a map all the parses can give you back a map they don't have to say well you might0:48:47have interleaved some text in there so I can give you a Dom instead of a map and that's a beautiful thing and it's sort of winning I think not because people are being explicit about that because that value0:48:56proposition is there and they're saying they're realizing maybe just incidentally that's making their lives simpler so I can't say enough about0:49:06using maps you want maps if your programming language doesn't have first-class support for maps I mean idiomatic support you can write map0:49:15literals you can you have syntax support for accessors you have the ability to do symbolic keys and a library for doing generic manipulation you either hat0:49:26should push for that to be added or get out of there you're wasting huge amounts of time it inordinate amounts of time taking stuff in and out of objects that0:49:36are adding nothing so that's the order problem and generally it's solved by using associative like structures we can0:49:46talk now about the information problem this is actually not a problem we don't have a problem information it's them it0:49:55starts off simple right what is our problem we're not satisfied with that or our programming language can't handle it it's just you know it's amazing how bad0:50:05some of our languages are at manipulating data right what do they force us to do oh I I can't actually have a map with with the name and a0:50:14dresser or with depth and height I can't do that it's cumbersome for me to do I have no syntax for interacting with it I've got to make a class as soon as I made a class I say class box or whatever0:50:25that was now what have I got nothing as blank I have to define a whole language for interacting with boxes now if I look0:50:35at box and I look at like person record I don't see two different things I see two instances of the same thing these are two associative types to have different fields in it but no we add0:50:45these classes right and there's big cost to this you can't write things that generically manipulate classes right and although you know reflection exists for0:50:54a reason too but it's extremely cumbersome right it ties it to the logic ties the logic to the representation you're using right now right that class is a very specific0:51:04thing it's tied to your programming language for instance I think we should start representing data as data in our programs so let's look a little bit0:51:14about what we think we're doing when we're wrapping information we think we're doing encapsulation this is not what encapsulation is encapsulation means hiding0:51:23implementation details right this is how I work and I don't want you to know how I work right information doesn't work information isn't something whose it has innards0:51:32there's no innards right there's no implementation of information you added it why did you do that you should really0:51:44have an answer to that question why did you do that it might the answer might be because I'm using Java and I had no choice that's not a great answer but it0:51:56might be it might be the truth okay information will have representation that there's no getting around this you no amount of classes you add is gonna change the fact that eventually there's0:52:05going to be a face to that and that face is going to be what somebody else consumes right so you know you don't solve anything by wrapping information up in stuff you're still gonna have I'm0:52:16going to see information I'm going to know about XY and Z but you may add a lot of other stuff so what else do we bring in when we take this approach we're going to make classes for our0:52:25information all right so what does this look like it looks like this and we're being good so we're using interfaces but you're not going to marry my concrete person class have a person info0:52:36interface I have get name and other hideousness right then there's going to be a consumer right somebody that knows0:52:46how to do something we're going to pass these things around so 9 implementation detail I'm using it and so the consumer has you know some some activity it does on your behalf and it takes this person0:52:56info I was at a talk where someone was advocating it was a good talk building systems out of systems you know building0:53:05systems out of sets of systems out of services and taking that architectural approach from the get-go like when you design a new system0:53:15you should compose it out of services and really architect that from the from the beginning and I had a question about that which was you know if if we had0:53:24designed the first you know it was in contrast to a monolithic system that eventually you grow and say well now we got to break this apart and that's really hard would there be a way to0:53:34start with a single process and using the right architectural principles end up with something that when we decided to service eyes parts of it that0:53:43wouldn't be a big deal and and I got back a great answer which was that's a great idea but the programming languages0:53:52don't let you do that and I realized that was the perfect example of this problem of wrapping information because0:54:01because he was right one way to it's a sort of litmus test whether or not you've got things simple is is can you move it right can you move0:54:14your subsystem I don't care like how you move it right can you move it out of process can you change it to a different language can you move it into a different thread the fact that you know you might have to move it is a great way0:54:24to think about things like what if this wasn't exactly what I'm thinking about now but somebody had to move it over there I'm not talking about inventing abstractions to do this or predicting0:54:33the future right but your ability to move something really impacts really indicates whether or not what you built0:54:42was simple not being able to move it is the knitted castle effect right if I say well now I'm going to take this thing and it was in process and it was just0:54:51your you know subsystem and and I want to make it a service that we call if that's a big job you didn't have it simple in the first place so what would0:55:00a subsystem have to have in order to be easily moved it should have well-defined boundaries right it should have an abstracted operational interface right0:55:09there should be an interface that encapsulate the purpose and how they work that way we can move it somewhere else they could work differently we can move it to another language it can work differently move it to another threat to0:55:18have different threading policy it's not there it's just some general way of doing handling and without getting into the that's a deep problem but certainly0:55:28there are simple ways to get this wrong checked exceptions would be an example of that there like a poster child of complected right because what's going to happen if you move this out of proc your0:55:39implementation is now going to have a different kind of exception right now you could have i/o exceptions that you didn't have before and that's going to poison the whole call chain the fact that that happens due to checked0:55:48exceptions indicates check tip set check exceptions are a disaster they're inherently complected it's a terrible terrible idea so this is the critical0:56:00thing though where do we actually go wrong before it's that subsystem should have data as the interface for what they take in return right we didn't do this right we0:56:12passed I person infos they are not data and we're not going to solve this by just putting in some serialization why0:56:22not because the consumer of this thing didn't just get data here they got code who knows what else was an0:56:31i person if those stuff they could call if I want to now put this over in a service am I gonna can do something you can call we're gonna start making like circular HTTP calls to get stuff done0:56:42no in fact you're saying wow that's that's stupid right it's stupid one two processes but it's okay and one it's not you should be taking and returning data0:56:52you wouldn't have this problem even inside so in the end I think we have a0:57:01lot of tools some of the tools inherently generate complexity some are inherently simpler but nothing makes nothing manufacturer's complexity0:57:13manufacturers simplicity right there's nothing you can do there's no technique there's no methodology you can adopt or cult that you can join that will force0:57:22your things to be simple this is a decision-making process you have to make over and over and over again you have to become vigilant about it and it says often a sensibility thing like I talked0:57:31about this order problem that's the kind of thing you want to develop sensitive sensibilities around right you need to be able to smell that coming in saying whoo I see the order0:57:41problem here maybe I should do that maybe I should do something about that you want to sensibilities to be based around entanglement not ease-of-use you0:57:50can make these things easy to use in fact the vast majority of those simpler things are as easy are easier to use and the complicated things there may just be less familiar the other thing is to make0:58:02sure you don't overestimate your reliability tools right they are not simplicity tools you can test you can0:58:12type check complex things the same way you can test and type check simple things it's orthogonal it doesn't mean those things are bad but they're not0:58:22helping you in this area you need to do this by choice so to make simplicity easy right to bring it nearer yourself0:58:32you need to choose simpler constructs you need to especially avoid complexity generating constructs ones that just start you off behind the behind the0:58:428-ball to focus on the fact that it's what you're making not how much of a party you're having making it you want0:58:52to create abstractions that are focused on simplicity right abstracting is not about fabricating universes it's about boiling things down you often want to0:59:02try to simplify the problem space before you start that would be a whole other talk about analysis but simplifying during analysis is critical here again if you start with something that's all0:59:12tied together and say you implement this knot if somebody ask you to implement a knot you shouldn't say okay you should say that's a knot we're gonna tie it before0:59:21we start and also to remember that simplicity often makes means making more things not fewer it's not about counting necessarily although generally simpler0:59:31things are smaller and have smaller interfaces and stuff like that so I'll leave you with this quote and you should0:59:40use this whenever somebody tries to sell you something sophisticated and that's it you [Applause]0:00:00Simple Made Easy 2011 - Rich Hickey
0:00:00 who's ready for some more character category theory you're all in the wrong0:00:10room so this talk I hope seems deceptively0:00:21obvious one of the things that's great about this conference is this is a this is a pretty cutting edge crowd a lot of you are adopting new technologies a lot0:00:30of you are doing functional programming and you may you may be nodding saying yeah yeah yeah through parts of this and if some of its familiar that that's0:00:39great on the other hand I think that I would hope that you would come away from this talk with some tools you could use to help conduct a similar kind of0:00:49discussion to this talk with other people that you're trying to convince to do the right thing so I'll start with an0:01:00appeal to Authority simplicity is a prerequisite for reliability I certainly agree with this I don't agree with everything Dykstra said and I and I0:01:10think he might have been very wrong about proof in particular but I think he's right about this we need to build simple systems if we want to build good0:01:19systems and I don't think we focus enough on that I love word origins there are tremendous fun one of the reasons0:01:29why they're fun is because words eventually become come to mean whatever we all accept them to me you know whatever is commonly understood to be the meaning is what it means and it's0:01:39often interesting to say well I wish I could I wish we'd go back to what it really means and use that and I think there's a couple of words that I'm gonna use in this talk that I would love for0:01:48you to come away knowing the origins of and try to use more precisely especially when talking about software so the first0:01:57word is simple and the roots of this word are sim and Plex and that means one fold or one braid or twist and that0:02:10characteristic about being about one literally fold or twist of course one twist what's one twist look like no twists0:02:19right and actually and it the opposite of this word is complex which means braided together or fold it together0:02:30being able to think about a software in terms of whether or not it's folded together sort of the central point of this talk the other word we frequently0:02:39use interchangeably with simple is the word easy and the derivation there is - a French word and the last step of this derivation is actually speculative but I0:02:49bought it because it serves this talk really well and that is from the latin word that is the root of adjacent and0:02:58which means to lie near and to be nearby and the opposite is hard of course the root of hard has nothing to do with lying there it doesn't mean lie far away0:03:08actually means like strong or torturously so so if we want to try to0:03:17apply simple to the kinds of work that we do we're going to start with this concept of having one braid and look at0:03:27it in a few different dimensions I thought was interesting and Eric's talk to talk about dimensions because that's definitely a big part of doing design work and so if we want to look for simple things we want to look for things0:03:36that have sort of one of something they do they have one role they fulfill one task for a job they're about accomplishing sort of one0:03:45objective they might be about one concept like security and and sort of0:03:56overlapping with that is they may they may be about a particular dimension of the problem that you're trying to solve the critical thing there though is that when you're looking for something that's simple you want to see it have focus in0:04:06these areas you don't want to see it combining things on the other hand we can't get too fixated about one in0:04:16particular simple doesn't mean that there's only one of them right it also doesn't mean an interface that only has one operation so it's important to0:04:27distinguish cardinality right counting things from actual interleaving what matters for0:04:37simplicity is that there's there's no interleaving not that there's only one thing and that's very important okay0:04:48the other critical thing about simple as we've just described it right is if something isn't relieved or not that's0:04:57sort of an objective thing you can probably go and look and see I don't see any connections I don't see anywhere where this twist was something else so0:05:06simple is actually an objective notion that's also very important in deciding the difference between simple and easy0:05:16so let's look it easy I think this notion of nearness is really really cool in particular obviously there's many ways in which something can be near0:05:26right there's sort of the physical notion of being near right is something you know like right there and I think that's where the root of the word came0:05:35from you know this is easy to obtain because it's it's nearby it's not in the next town I want to take a horse or whatever to go get to it we don't have the same notion of physicality0:05:44necessarily in our software but we do sort of have you know our own hard drive or our own tool set or it's sort of the ability to make things physically nearby0:05:54getting them through things like installers and stuff like that the second notion of nearness is something being near to our understanding right or in our current0:06:05skill set and I don't mean in this case near to our understanding meaning of capability I mean literally near something that we already know so the0:06:16word the word in this case is is about being familiar I think that collectively we are infatuated with these two notions0:06:26of easy we are just so self involved in these two aspects it's hurting us tremendously right all we care about is0:06:35you know can I get this instantly and start running it in five seconds it could be this giant hairball that you got but all you care is you can you get it0:06:44in addition we're fixated on oh I can't I can't read that I can't read German does I mean German is unreadable no I0:06:55don't know German so you know this this sort of approach is is definitely not helpful in particular if you want0:07:04everything to be familiar you will never learn anything new because it can't be significantly different from what you already know and not drift away from the familiarity there's a third aspect of0:07:15being easy that I don't think we think enough about that's going to become critical to this discussion which isn't which now is being near to our0:07:24capabilities and we don't like to talk about this because it makes us uncomfortable because what kind of capabilities are we're talking about if we're talking about easy in the case of0:07:33violin playing or piano playing or mountain climbing or something like that well you know I don't personally feel bad if I don't play the violin well0:07:43because I don't play the violin at all but the work that we're in is conceptual work so when we talk to start talking about something being outside of our0:07:52capability it was you know it really starts trampling on our egos in a big way and so you know due to a combination0:08:01of hubris and insecurity we never really talked about whether or not something is outside of our capabilities it ends up that it's not so embarrassing after all0:08:12because we don't have tremendously divergent abilities in that area the last thing I want to say about easy and the critical thing to distinguish it0:08:22from simple is that easy is relative right playing the violin and reading German are really hard for me they're easy for other people certain other0:08:32people so unlike simple where we can go and look for interleavings look for braiding easy is always going to be you know easy for whom or hard for whom it's a0:08:43relative term the fact that we throw these things around sort of casually saying oh I'd like to use that technology because it's simple and when I'm saying simple I mean easy and when I am saying easy I mean because I already0:08:53know something that looks very much alike that is how this whole thing degrades and we can never have an objective discussion about the qualities that matter to us in0:09:04our software so what's one critical area where we that where we have to distinguish these two things and and0:09:14look at them from a perspective of them being easy and being simple it is it has to do with constructs and artifacts right we program with constructs we have0:09:24programming languages we use particular libraries and those things in and of themselves when we look at them like when we look at the code we write have0:09:33certain characteristics in and of themselves but we're in a business of artifacts right we don't ship source code and the user doesn't look at our0:09:43source code and say oh that's so pleasant right now they run our software and they run it for a long period of time and over time we keep climbing more0:09:54stuff on our software all that stuff the running of it the performance of it the ability to change it all is an attribute0:10:03of the artifact not the original construct but again here we still focus so much on our experience of the use of0:10:13the constructs who look only heads type 16 characters wow that's great no semicolons or things like that this whole notion of sort of program or0:10:23convenience again we are infatuated with it not not to our benefit on the flip side it gets even worse our employers0:10:33are also infatuated with them right those first two meanings of easy what do they mean right if I can get another programmer in here right and they look0:10:44at your source code and they think it's familiar right and they already know the toolkit right so it's near at hand0:10:53they've always had the same tool in their toolkit they can read it I can replace you it's a breeze especially if I ignore the third notion of easy right0:11:02which is whether or not anybody couldn't stand your code right because that don't actually care about that they just care that someone can go sit in your seat start typing so again as sort of business0:11:14owners there's sort of an again the same kind of focus on those first two aspects of easy because it makes program is replaceable so we're going to contrast0:11:23this with the impacts of long term use right what does it mean to use this long term and what what's there what's there0:11:33is all the meat right does the software do it is supposed to do is it of high quality can we rely on it doing what's0:11:42supposed to do can we fix problems when they arise and if we're given a new requirement can we change it these things have nothing to do with the0:11:53contract as we typed it in or very little to do with it and have a lot to do with the attributes of the artifact we have to start assessing our0:12:03constructs based around the artifacts not around the look and feel of the experience of typing it in or the cultural aspects of that so let's talk a0:12:15little about bit about limits oh look it doesn't move this is just supposed to sort of lull you into this state where everything I say seems true because I0:12:27can't use monads to do that this stuff0:12:36is pretty simple logic right how can we possibly make things that are reliable that we don't understand it's very very difficult I think a professor Sussman0:12:47made a great point saying there's going to be this trade-off right as we make things more flexible and extensible and dynamic in some possible futures for0:12:56some kinds of systems we are going to make a trade-off in our ability to to understand their behavior and make sure that they're correct but for the things0:13:07that we want to understand and and make sure are correct we're going to be limited we're going to be limits our understanding and our understanding is is very limited right there's a whole0:13:16notion of you know how many how many balls can you keep in there at the time or how many things can you keep in mind it's a it's a limited number and it's a very small number all right so we can0:13:25only consider a few things and when things are intertwined together we lose the ability to take them in isolation so if every time I think I pull out a new part of the software I need to0:13:35comprehend and it's attached to another thing I have to pull that other thing into my mind because I can't think about the one without the other that's the nature of them being intertwined so0:13:45every intertwining is adding this burden and the burden is kind of combinatorial as to the number of things that we can we can consider so fundamentally this0:13:56complexity and by complexity I mean just braiding together of things is going to limit our ability to understand our systems so so how do we change our0:14:09software apparently I heard in a talk today that agile and extreme programming have shown that refactoring and tests allow us to0:14:18make change with zero impact I never knew that I still do not know0:14:27that that's that's not actually a knowable thing it's that's phooey right0:14:36if you're going to change software you're going to need to analyze what it does and make decisions about what it ought to do yeah I mean at least you're going to have to go and say what is the impact of this potential change right0:14:47and what parts of the software do I need to go to to affect a change and you know I don't I don't care for using XP or0:14:57agile or anything else you're not going to get around the fact that if you can't reason about your program you can't make these decisions but I do want to make0:15:07clear here because a lot of people as soon as they hear the word reason about they're like oh my god are you saying that you have to be able to prove programs I am NOT I don't believe in that I don't think that's an objective0:15:16I'm just talking about informal reasoning the same kind of reasoning we use every day to decide what we're gonna do we do not take out category theory and say you know we actually can reason0:15:28without it thank goodness so what about what about the other side right there's two things you do with the future of your software one is you add new capabilities the other thing is you fix0:15:38the ones you didn't get you know so you know done so well and I like to ask this question what's true of every bug found in the field it got written yes what's a0:15:52more interesting fact about it it passed the type checker what else did it do it passed all the tests okay so now what do0:16:05you do right I think we're in this world I'd like to call guardrail programming right it's really sad we're like I can0:16:17make change because I have tests wait who does that who drives their car around banging against the guardrail saying well I'm glad I've got these guardrails because0:16:27I'd never make it to to the show on time right and and do the guardrails head0:16:37help you get to where you want to go like the guardrails like guide you places no there's guardrails everywhere they don't point your car in any particular direction so again we're0:16:50going to need to be able to think about our program it's going to be critical all of our guardrails we'll have thought will have failed us we're gonna have this problem we're gonna need to be able to reason about our program say well you0:16:59know what I think because maybe if it's not too complex I'll be able to say I know through ordinary logic it couldn't0:17:08be in this part of the program it must be in that part and let me go look there first things like that now of course everybody's gonna start moaning but I have all this speed I'm0:17:19agile and fast you know this easy stuff is making my life good because I have a lot of speed so what kind of runner can run as fast0:17:30as they possibly can from the very start of a race right only somebody who runs really short races ok but of course we0:17:43are programmers and we're smarter than runners apparently because we know how to fix that problem right we just fire the starting pistol every hundred yards0:17:53and call it a new sprint all right I don't know why they haven't0:18:04figured that out but right it's my contention based on experience that if you ignore complexity you will slow down0:18:14you will invariably slow down over the long haul of course if you are doing something that's really short-term you don't need any of this you could write it you know win ones and zeroes and this0:18:26is my really scientific graph you notice how none of the axes are there's no numbers on it because I just I completely made it up it's a it's an0:18:36experiential graph and what it shows is if you focus on ease and not an ignore simplicity so I'm not saying you can't0:18:45try to do both that would be great if you focus on ease you will be able to go as fast as possible from the beginning of the race but no matter what technology you use or Sprint's or firing0:18:54pistols or whatever the complexity will eventually kill you it will kill you in a way that will make every sprint accomplished less most Sprint's be about completely redoing things that you've0:19:04already done and the net effect is you're not moving forward in any significant way now if you start by focusing on simplicity why can't you go0:19:13as fast as possible right at the beginning because some some tools that are simple are actually as easy to use as some tools that are not why can't you0:19:23go as fast then you have to think you have to actually apply some simplicity work to the problem before you start and that's gonna give you this ramp up so0:19:36one of the problems I think we have is this conundrum that some things that are easy actually are complex so let's look there are a bunch of constructs that0:19:48have complex artifacts that are very succinctly described right some of the things that are really dangerous to use are like so simple to describe they're incredibly familiar right if you're0:19:58coming from object orientation you're familiar with a lot of complex things they're very much available right and they're easy to use in fact by all0:20:08measures conventional measures you look at them and say this is easy right but we don't care about that right again0:20:17the users not looking at our software and they don't actually care very much about how good a time we had when we were writing it right what they care about is what the program does and the0:20:28and if it works well it will it will be related to whether or not the output of those constructs were simple in other words what complexity do they yield when0:20:38there is complexity there we're going to call that incidental complexity right it wasn't part of what the user asked us to do we chose a tool it had some inherent0:20:48complexity in it it's incidental to the problem I didn't put the definition in here but uh incidental is Latin for your fault and it is and I think you really0:21:03have to ask yourself you know are you programming with a loom you know you're having a great time you're throwing that shuttle back and forth and what's coming out the other side is this knotted you0:21:15know mess I mean it may look pretty but you have this problem right was the problem the problem is the knitted castle problem right do you want a0:21:29knitted castle so what benefits do we do we get from simplicity we get ease of understanding right that's sort of definitional I contend we get ease of0:21:39change and easier debugging other benefits that come out of it that are sort of on the secondary level are increased flexibility when we talk more0:21:49about modularity and breaking things apart we'll see where they were that Falls like the ability to change policies or move things around right as0:22:00we make things simpler we get more independence of decisions because they're not interleaved so I can make us a location decision it's orthogonal from0:22:10like a performance decision and I really do want to make you know ask the question agile Astorga is is having a test suite0:22:20and refactoring tools going to make changing the knitted Castle faster than changing the Lego castle no way0:22:30completely unrelated okay so how do we make things easy i'm presumably you know the objective here is not to just be moan that's their0:22:41software crisis right so what can we do to make things easy so we'll look at those parts those aspects of being easy again there's a location aspect making something at hand putting it in our0:22:50toolkit that's relatively simple right we just install it right maybe it's a little bit harder because we have to get somebody to say it's okay to use it then0:23:00there's the aspect of how do I make it familiar right I mean I've ever seen this before that's a learning exercise I've got to go get a book go take a tutorial have somebody explain it to me0:23:10maybe try it out write both these things we're driving we're driving we install we learn where it's totally in our hands then we have this other part though0:23:20right which is the mental capability part and it does the part that's always hard to talk about the nuts capability part right because the the fact is we0:23:32can learn more things we actually can't get much smarter we're not going to move we're not going to move our brain closer to the complexity we have to make things0:23:43near by simplifying them but the truth here is not that that's like they're these super super you know bright people who can do these amazing things and0:23:52everybody else is stuck because the juggling analogy is pretty close right the average juggler can do three balls the most amazing juggler in the world0:24:02can do like nine balls or twelve or something like that they can't do twenty or a hundred right all of we're all very0:24:11limited you know compared to the complexity we can create we're all you know statistically at the same point in our ability to understand it which is0:24:20not very good so we're going to have to bring things towards us and because we can only juggle so many balls you have to make a decision how many of those0:24:29balls do you want to be incidence of complexity and how many do you want to be problem complexity alright how many extra balls you want have somebody throwing you you have to try to incorporate in here0:24:39Oh use this tool you're like whoa you know more more stuff who wants to do that all right so let's look at a let's0:24:49look at a fact so I've been on the other side of this complaint and I like it we0:24:59can look at it really quickly only because it it's not this analysis has nothing to do with the usage this complexity analysis is just about the0:25:09programmer experience right so parens are hard right they're not at hand for most people who haven't otherwise used it and what does that mean it means that like they don't have an editor that0:25:21knows how to do you know paren matching or move stuff around structurally or they have one and they've never loaded the mode that makes that happen totally given right it's not at hand0:25:31nor is it familiar I mean everybody's seen parentheses but they haven't seen them on that side of the method I mean0:25:42that is just crazy but you know I think this is this is0:25:52your responsibility right to fix these two things as a user as a potential user you got to do this but we could dig deeper let's look at the third thing0:26:01did you actually give me something that was simple is a language built all out of parens simple in the case I'm saying right is it free of interleaving and0:26:12braiding and the answer is no right Common Lisp and scheme are not simple in this sense in their use of parens0:26:21because the use of parentheses in those languages is overloaded right parens wrap calls they wrap grouping they wrap0:26:31data structures right and that overloading is a form of complexity by the definition you know I gave you right and so if you actually bother to get0:26:42your editor set up and learn that the parentheses goes on the other side of the verb this was still a valid complaint now of course everybody was0:26:51saying easy it's hard it's complexing they were using his words really weakly right but it was hard for a couple of reasons you could solve and it was not0:27:00simple for a reason that was the fault of the language designer which was that there was overloading there right and we can fix that right we can just add another data structure it doesn't make0:27:09Lisp not Lisp to have more data structures right it's still a language defined in terms of its own data structures but having more data structures in play means that we can get0:27:19rid of this overloading in this case which then makes it your fault again right because now this the simplicity is back in the construct and it's just a0:27:29familiarity thing which you can solve for yourself okay this is an old dig at0:27:38Lisp programmers I'm not it's totally sure what the what he was talking about I believe it was a performance related0:27:47thing that lispers was just they Kahn stop all this memory and they did all this evaluation and it was it was a big list programs at that time were list programs at that time were complete pigs0:27:57relative to the hardware so that you know the value of all these constructs right this dynamism you know dynamic nature these things are all great they are valuable right but there0:28:07was this performance cost I'd like to lift this whole phrase and apply it to all of us right now right as programmers we are looking at all kinds0:28:16of things and I just see it you know we read hacker news or whatever it's like oh look this thing has this benefit oh great I'm gonna do that oh but this has this benefit oh that's cool oh that's awesome0:28:27you know that's shorter you never see in these discussions was there a trade-off is there any downside you know was there anything bad that comes along with this0:28:37never nothing it's just like we look all for benefits right so as programmers now I think we're looking all for benefits and we're not looking carefully enough at the the byproducts so what's in your0:28:50tool kit I have a you know I have these two columns one says complexity and one0:28:59says simplicity the simplicity column just being simpler it doesn't mean that the things over there are purely simple0:29:08now I didn't label these things bad and good I'm leaving your minds just do that [Laughter]0:29:20so what things are complex and what are the simple replacements I'm going to dig into the details on these so I won't actually explain why they're complex we're going to say state and objects are0:29:29complex and values are simple and and can replace them in many cases I'm gonna say methods are complex and functions0:29:38are simple and namespaces are simple and the reason why methods are there because often the space of methods in the class0:29:47or whatever is also a mini very poor namespace VARs our complex and variables0:29:56are complex managed references are also complex but they're simpler inheritance switch statements pattern-matching are0:30:06all complex and polymorphism a la carte is simple okay now remember the meaning of simple meaning of simple means unentangled0:30:18right not twisted together with something else doesn't mean I already know what it means right simple does that mean I already know what it means okay0:30:27syntax is complex data is simple imperative loops fold even which seems0:30:36kind of higher-level still has some implications that tie two things together whereas set functions are simpler actors are complex and queues0:30:45are simpler ORM is complex and declarative data manipulation is simpler okay even Eric said that in his talk he0:30:55said it really fast near the end oh yeah an eventual consistency is really hard for programmers0:31:07conditionals are complex in interesting ways and rules can be can be simpler an inconsistency is very complex it's0:31:17almost definitionally complex right because consistent means to stand together so inconsistent means to stand apart and that means taking a set of things that are standing apart and0:31:26trying to think about them all at the same time it's inherently complex to do that a neighbor who's tried to use the system that's eventually consistent knows that okay0:31:36so there's this really cool word called complex I found it I love it it means to0:31:45interleave or entwine or braids okay I want to start talking about what we do to our software that makes it bad I don't want to say braid or entwine0:31:54because it doesn't really have the good bad connotation that complex has complex is obviously bad right it happens to be an archaic word but you know there's no0:32:04rules that say you can't start using them again so I'm going to use them for the rest of the talk so what do you know about complex is bad don't do it right0:32:15this is where complexity comes from like complected that's very simple0:32:24right and in particular it's something you want to avoid in the first place right look at this diagram look at the first one look at the last one right0:32:33it's the same stuff in both those diagrams the exact it's the same strips what happened they got complected and now it's hard to0:32:45understand the bottom diagram from the top one but it's the same stuff you're doing this all the time you can make a program a hundred different ways some of0:32:54them it's just hanging there it's all straight you look at yourself oh I see it's four lines this program right now you can type in four lines in another language or with a different construct0:33:03and you end up with this not so you gotta take care of that so complex actually means to braid together and0:33:12campos means to place together and we know that right everybody keeps telling us what we want to do is make composable systems we just want to place things0:33:21together which is great and I think there's no disagreement right composing simple components simple in that same respect is the way we write robust0:33:32software so it's simple right all we need to do is everybody knows this I'm0:33:41up here just telling you stuff you know we can make simple systems by making the modular right we're done I'm like halfway through my talk I don't even know if I'm gonna finish it's so simple0:33:51right this is it this is the key no it's obviously not the key right who has seen who has seen components to have this0:34:00kind of characteristic I'll read my head twice because not enough people are raising their hands it's ridiculous right what happens you can write modular0:34:09software with all kinds of interconnections between them right they may not call each other but they're completely complected right and we know0:34:20how to solve this it has nothing to do with the fact that there are two things it has to do with what those two things are allowed to think about if you want to really answer for more Phi's and what0:34:30do we want to make things allowed to think about and only these things some abstractions I thought that's coming out that well that's a dashed0:34:40white version of the top of the Lego right that's all we want to limit things to because now the blue guy doesn't really know anything about the yellow guy and the yellow guy doesn't really0:34:49know anything about the blue guy and they've both become simple so it's very it's very important that you don't associate simplicity with partitioning and stratification right they don't0:34:59imply it right they are enabled by it if you make simple components you can horizontally separate them and you can vertically stratify them right but you can also do0:35:10that with complex things and you're going to get no benefits and so I would encourage you to be particularly careful not to be fooled by code organization0:35:20right there's tons of libraries that look oh look there's different classes there's separate classes they you know they call each other in sort of these nice ways right then you get out in the0:35:30field and you're like oh my god this thing presumes that that thing never returns in number 17 what is that okay0:35:40I'm not going to get up here and tell you state is awesome I like state I'm not a functional whatever a guy whenever I'm gonna say instead I did this and it sucked0:35:51right I did years and years C++ you know heman stateful programming it's it's really not fun it's not good it's it's0:36:04never simple having seen your program is never simple right because it has a fundamental complected that goes on in its artifacts right a complex value in0:36:14time you don't have the ability to get a value independent of of time and sometimes not an ability to get a value in any proper sense at all but again0:36:25it's a great example this is easy it's totally familiar it's at hand it's in all the programming languages this is so easy this complexity is so easy and0:36:37you can't get rid of it everything so I'll happen I have modularity that that assignment statement is inside a method right well if every time you call that method with the same arguments you can0:36:47get a different result guess what happen that complexity it just leaked right out of there it doesn't matter that you can't see the variable right if the thing that's wrapping it is stateful0:36:56or thing that's wrapping that is still stay full in other words by stateful I mean every time you ask it the same question you get a different answer you have this complexity and it's like poison it's like dropping you know some0:37:07some dark liquid into this - of Oz it's just gonna end up all over the place the only time you can really you know get rid of it is when you put it inside0:37:16something that's able to present a functional interface on the outside a true functional interface same input same output you can't mitigate it0:37:25through the ordinary code organization things and note in particular I didn't talk about concurrency here this is not about concurrency this has nothing to do0:37:35with concurrency it's about your ability to understand your program your program was out there it's single threaded it didn't work all the tests passed it made it through the type checker figure out0:37:45what happened right if it's full of variables what are you gonna need to try to do recreate the state that was happening at the client when it went bad0:37:55is that gonna be easy no but we fix this right your language your new shiny0:38:05language has something called var or maybe has refs or references none of these constructs make state simple0:38:14that's the first primary thing I don't want to say that even of closures constructs they do not make state simple in the case I'm talking about and in nature of simple I'm talking about but they're not the same right they all do0:38:25warn you when you have state and that's great most people who are using a language where mutability is not the default you have to go out of your way to get it finds that the program's end0:38:34up writing have dramatically like orders of magnitude less state than they would otherwise because they never need it all the other state in the first place so0:38:43that's really great but I will call out closure and Haskell's references as being particularly superior in dealing with this because they compose values in0:38:54time there are actually little constructs that do two things they have some abstraction over time and the ability to extract the value that's really important because that's0:39:06that's your path back to simplicity and if I have a way to get out of this thing and get a value out I can continue with my program after that pass that variable to somebody else or a reference to0:39:15something that's going to find the variable every time through the varying thing poisoning the rest of my system so you know look at the VAR in your language and ask if it does the same the0:39:25same thing alright let's see why things are complex state we already talked about it's complex everything it touches0:39:34objects complex state identity and value they mix these three things up in a way that you cannot extricate the parts0:39:43methods complex function and state ordinarily right in addition in some languages they complex namespaces right0:39:52derive from two things in Java they have the same name method and right doesn't work syntax interestingly complex meaning and0:40:03order often in a very unidirectional way professor Sussman made the great point about data versus syntax and it you know0:40:12it's super true I don't care how much you really love the syntax of your favorite language it's inferior to data in every way0:40:21inheritance complex types right so these two types are complected that's what it means inheritance complected is like it's it's0:40:30definitional right switching and MACT matching right they complex multiple pairs of who's going to do something in0:40:40and what happens right and they do it all in one place in a closed way that's very bad VARs and variables again0:40:50complex value in time often in an inextricable way you can't obtain a value we saw a picture during a keynote yesterday this amazing memory right0:41:01where you could dereference and address and get an object that I want to get one of those computers right have you ever used one of those computers I can't get0:41:11one I called Apple and they were like no those you can never get a have a memory is a word a scaler right the thing that was all derided right recovering a0:41:21composite object from an address it's not something computers do none of the ones that we have so variables have the same problem you cannot recover a0:41:30composite mutable thing with with one dereference loops and fold loops are pretty obviously0:41:39complected what you're doing and how to do it fold is a little bit more subtle right because it seems like this nice somebody else is taking care of it but it does have this implication about0:41:48the order of things this left-to-right bit actors complex what's going to be done and who's going to do it Oh0:42:00now professor Sussman said all these talks have acronyms and I couldn't actually my slides in time so object0:42:09relational mapping has oh my god complexing going on you can't even begin to talk about how how bad it is right and and and you know if you're0:42:20going to do like duals right what's the dual of value is it ko value what's a KO0:42:29value it's an inconsistent thing whom what's that and conditionals I think are interesting right this is this is sort0:42:39of more cutting edge area we have a bunch of sort of rules about where a program supposed to do it's strewn all throughout the program can we fix that0:42:49because that's complected with the structure of the program and the organization of the program all right so if you take away two things from the0:42:59stock one would be the difference between that word simple and easy the other I would hope would be the fact that we can create precisely the same0:43:10programs we're creating right now with these tools of complexity with dramatically drastically simpler tools right I did C++ for a long time I did0:43:22Java I did C sharp I know how to make big systems in those languages and I completely believe you do not need all that complexity you0:43:31can write as sophisticated a system with dramatically simpler tools which means you're gonna be focusing on the system what it's supposed to do instead of all0:43:40the gook that falls out of the constructs you're using so I'd love to say the first step in getting a simpler life is to just choose simpler stuff0:43:49right so if you want values usually you can get it most languages have something like values final or Val cut you know0:43:58lets you like declare something is being immutable you do want to find some persistent collections because the harder thing in a lot of languages is getting aggregates that are values right0:44:09you got to find a good library for that or use a language where that's the default functions most languages have0:44:18them thank goodness if you don't know what they are they're like stateless methods namespace this is something you really need the language to do for you0:44:27and unfortunately it's not done very well in a lot of places data please we're programmers we supposedly write data processing programs there's always0:44:37programs they don't have any data in them have all these constructs we put around it and globbed on top of data data is actually really simple there's not a design a tremendous number of0:44:47variations in the essential nature of data right there are maps there are sets there are linear sequential things there's not a lot of other conceptual categories of data we create hundreds of0:44:58thousands of variations have nothing to do with the essence of the stuff and make it hard to write programs that manipulate the essence of the stuff we should just manipulate the essence of the stuff it's not hard it's simpler0:45:09also same thing for communications right are we all not glad we don't use the UNIX method of communicating on the web0:45:19right any arbitrary command string can be the argument list for your program and any arbitrary set of characters can come out the other end0:45:28that's all right parsers no I mean it's it's a problem it's it's a source of complexity right so we can get rid of that just use data the0:45:37biggest thing I think the most desirable thing the most esoteric this is tough to get but boy when you have it your life is completely totally different thing is0:45:46polymorphism Alec Hart right closure protocols and Pascal type classes and and constructs like that give you the0:45:55ability to independently say I have data structures I have definitions of sets of functions and I can connect them together and0:46:04those are three independent operations and their words the generosity is not tied to anything in particular it's available a la carte I don't know I've a0:46:14lot of library solutions for languages that don't have it I already talked about manage references and how to get them maybe you can use closures from different Java languages set functions0:46:24you can get from libraries queues you can get from libraries right you don't need a special communication language you can get declarative data manipulation by using sequel or learning0:46:33sequel finally or something like link or something like data log I think these last couple of things are harder right0:46:43we don't have a lot of ways to do this well integrated with our languages I think that's currently linked isn't as an effort to do that rules right declarative rule systems0:46:53instead of you know embedding a bunch of conditionals in our raw language at every point of decision it's nice to sort of gather that stuff and put it over someplace else and you can get0:47:03rules systems in libraries or you can use languages like Prolog if you want consistency you need to use transactions and you need to use values okay there0:47:13are reasons why you might have to get off of this list but boy there's no reason why you shouldn't start with it okay0:47:23there's a source of complexity that's really difficult to deal with and not your fault I call it environmental complexity right our programs end up running on machines0:47:32next to other programs next to other parts of themselves and they contend they contend for stuff right memory CPU0:47:41cycles and things like that everybody's contending for this is an inherent complexity inherent is Latin for not your fault in the0:47:51implementation space and though this is not part of the problem but it is part of the implementation right you can't go back to the customer and say the thing you wanted is not good because I have GC0:48:01problems but did you see problems and stuff like that they come into play there's not a lot of great solutions right you can do segmentation you can say this is your memory this is your0:48:10memory this you were me this is your your CPU and your CPU but there's tremendous waste in that right sweep reallocate you don't use everything you don't have sort of dynamic nature but0:48:20the problem I think we're facing and it's not one for which I have a solution at the moment is that the policies around this stuff don't compose right if everybody says ultra size my thread pool0:48:30to be the number of you know of course how many times can you do that in one program not a lot and have it still work out so unfortunately a lot of things0:48:41like that splitting that stuff up and making an individual decision is not actually making things simpler it's making things complex because that's a decision that0:48:51needs to be made by someone who has better information and I don't think we have a lot of good sources for organizing those decisions in single0:49:01places in our systems this is hugely long quote basically it says programming is not about typing like this it's about0:49:15thinking so the next phase here I got to move a little bit quicker is how do we design simple things of our own all right so the first part of making things0:49:24simple is just to choose constructs to have simple artifacts but we have to write our own constructs sometimes so how do we abstract for simplicity all0:49:33right an abstract again here's an actual definition that made up one means to draw something away and in particular it means to draw away from the physical nature of something I do want to0:49:44distinguish this from the sometimes people use this term really grossly to just mean hiding stuff as not what abstraction is and that's not going to0:49:53help you in this space there's two you know I can't totally explain how this is done it's really the job of designing but one approach you can take0:50:03it's just to do who what when where why and how if you just go through those things and sort of look at everything you're deciding to do and say what is the who aspect of this what is the what aspect of it this can help you take stuff apart0:50:13the other thing is to maintain this approach that says I don't know I don't want to know I once said that's so often0:50:22during a C++ course I was teaching that one of the students made me a shirt it was a boots diagram so we didn't have whatever it is now the unified one and0:50:33every line just said that that's what you want to do you really just don't want to know all right so what is what what is the operations you know what is0:50:42was that what we want to accomplish right we're going to form abstractions by taking functions or and more particularly sets of functions and giving them names in particular so and0:50:53you're going to use whatever your language lets you use right so if you only have interfaces you'll use that if you have protocols or type classes0:51:02you'll use those so all those things are in the category of the things you use to make sense of functions that are going to be abstractions and they're really sets of specifications of functions the0:51:12point I'd like to get across today is just that they should be really small much smaller than what we typically see Java interfaces are huge and the reason0:51:22why they are you just because Java doesn't have union type so it's inconvenient to say this function takes you know something that does this and that and that you have to make of this0:51:31and that in that interface so we see these giant interfaces and the thing with those giant interfaces is that it's a lot harder to break up those programs so you're gonna represent them with your0:51:41polymorphism constructs their specifications right they're not actually the implementations they should only use values and other abstractions0:51:50in their definitions so you can define interfaces or whatever type classes that only take interfaces and type classes or values and return them and the biggest0:52:00problem you have when you're doing this part of design is if you complex with how right you can complex' it with how by jamming them together and saying here's just a concrete function instead of having an interfere0:52:09so here's a concrete class instead of having an interface you can also complex' it with how more subtly by having some implication of the semantics of the function dictate how it is done0:52:19unfold as an example of that the strictly separating what from how is the key to making how somebody else's0:52:28problem right if you've done this really well you can you can pawn off the work of how on somebody else you can say database engine you figure out how to do this thing or a logic engine you figure0:52:38out how to search for this I don't need to know who is about like data or entities these are the things that are abstractions going to be connected to eventually depending on how your0:52:47technology works you want to build components up from sub components in a sort of direct injection style right you don't want to like hardwire with the sub components are you want it as much as0:52:57possible take them as arguments because that's going to give you more programmatic flexibility and how you build things you should have probably many more sub components than you have0:53:07so you want really much smaller interfaces and you have and you wanna have more sub components than you probably are typically having because usually you have none and then maybe you have one when you decide to I need to0:53:16farm out policy if you go in saying this is a job and I've done who what when where why and I found five components don't feel bad that's great you're winning massively by doing that you know0:53:27split out policy and stuff like that and the thing that you have to be aware of when you're building you know the definition of a thing from sub components is any of those kind of you0:53:37know red and yellow thinking about blue blue thinking about yellow kind of hidden detailed dependencies so you want to avoid that how things happen this is0:53:47the actual implementation code the work of doing the job you you strictly want to connect these things together using those polymorphism constructs that's the0:53:56most powerful thing yeah you can use a switch statement you could use pattern matching buts glomming all this stuff together if you use one of these systems you have an open pi Morse's and policy0:54:06and that is really powerful especially if it's runtime open but even if it's not it's better than than nothing and0:54:15again beware of abstractions that dictate how in some subtle way because when you do that you're really you're nailing the person down the line us to do the implementation you're tying0:54:25their hands so the more declarative things are the better the better things work and the thing that I mean how is sort of the bottom right don't mix this0:54:34up with anything else all these implementations should be islands as much as possible when and where this is pretty simple I think you just have to0:54:43strenuously avoid complected this with anything I see it accidentally coming in mostly when make people design systems0:54:52with directly connected objects right so if you know the your program is architected such that you know this thing deals with the input and then this thing has to do that the next part of0:55:02the job well if if thing a calls thing B you just complected it right and now you have a when and where thing cuz now a0:55:11has to know where B is in order to call B and when that happens is whenever a does it right stick a cue in there make0:55:22use of the way to just get rid of this problem if you're not using cues extensively you should be you should start right away like right after this talk and then there's the y part this is0:55:34sort of the policy and rules this is this is I think this is hard for us we typically put this stuff all over our application and if you ever have to talk to a customer about what the application0:55:44does you it's really difficult to sit you know with them in source code and look at it now if you have one of these pretend testing systems and let you write English strings so the customer0:55:53can look at that that's just silly right you should have code that does the work that somebody can look at which means to try to you know put this stuff someplace outside try to find a declarative system0:56:04or a rule system will let you let's you do this work finally in this area information it is simple right the only thing you can possibly do with0:56:13information is ruin it right don't do it right don't do this stuff we I mean we got objects I've just were made to like0:56:22encapsulate IO devices so there's a screen but I can't like touch the screen so I have an object right there's a mouse I can't touch the mouse oh there's an object right that's all they're good0:56:31for they were never supposed to be applied to information and we apply them to information that's it's just wrong it's wrong and but I can now say it's wrong for a reason right0:56:41it's wrong because it's complex in particular it ruins your ability to build generic data manipulation things if you leave data alone right you can0:56:53build things once that manipulate data and you can reuse them all over the place and you know they're right once and you're done the other thing about it which also applies to ORM is that it0:57:03will tie your logic to representational things which again tying complected intertwining so represent data is data please start using maps and sets0:57:13directly don't feel like I have to write a class now because I have a new piece of information it's just silly so the final aspect right so we choose0:57:24simple tools we write simple stuff and then sometimes we have to simplify other people's stuff in particular we we may have to simplify the problem space or0:57:33some code that somebody else wrote this is a whole separate talk I'm not going to get into right now but the job is essentially one of disentangling right0:57:42we know what's what's complex its entangled so what do we need to do right we need to somehow disentangle it right you're gonna get this you're gonna need0:57:53to first sort of figure out where it's going you're gonna have to follow stuff around and eventually label everything right this is the start this is roughly0:58:03what the process is like but again it's a whole separate talk to try to talk about simplification all right I'm gonna wrap up a couple of slides the0:58:13bottom line is simplicity is a choice it's your fault if you don't have a simple system and and I think we have a culture of complexity to the extent we0:58:23all continue to use these tools that have complex outputs we're just in a rut we're just self reinforcing and we have to get out of that rut but again like I0:58:32said if you're already saying I know this I believe you I already use something better I've already used that whole right column then hopefully this talk will give you the basis for talking with somebody else who doesn't believe0:58:41you right talk about simplicity vs. complexity right but it is a choice right it requires constant vigilance we already saw0:58:50the guardrails don't yield simplicity they don't really help us here right it requires sensibilities and care your sensibilities about simplicity being0:58:59equal to ease-of-use are wrong they're just simply wrong right we saw the definitions of simple and easy they're completely different things right so0:59:08easy is not simple you have to start developing sensibilities around entanglement that's what you have to just you have to have entanglement radar right you want to look at some software0:59:18and say ah you know not that I don't like the names you used or the shape of the code or there was a semicolon I mean that's also important too but you want to start seeing complected you want to0:59:29start seeing interconnections between things that could be independent that's where you're gonna get the most power all the reliability tools you have right since they're not about simplicity0:59:39they're all secondary right they just do not touch the core of this problem right they're safety nets but they're nothing0:59:50more than that so how do we make simplicity easy right we're gonna choose constructs with simpler artifacts right1:00:04and avoid constructs that have complex artifacts it's the artifacts it's not the authoring as soon as you get in an argument with somebody about Oh should1:00:13we should be using whatever get that sorted out you know because however they feel about the shape of the code they type in is independent from this and1:00:22this is the thing you have to live with we're gonna try to create abstractions to have simplest city as a basis right we're gonna spend a little time upfront1:00:31simplifying things before we get started and recognize that when you simplify things you often end up with more things1:00:41right simplicity is not about counting right I'd have rather have more things hanging nice straight down not twisted together then just a couple of things1:00:52tied in a knot and the beautiful thing about making them separate is you'll have a lot more ability to change it which is where I think the benefits lie so I think this is a big deal1:01:04and I hope everybody's able to bring it into practice or use this as a tool for convincing somebody else to do that so1:01:13I'll leave you with this this is what you say when somebody tries to sell you a sophisticated type system thank you1:01:22[Applause]0:00:00Are We There Yet - Rich Hickey
0:00:00 so I'm gonna talk about time today0:00:13how we treat time in object-oriented languages generally and and maybe how we how we fail to so I'm trying to provoke you today to just reconsider some0:00:23fundamental things that I I just think we get so entrenched with what we do every day we fail to step back and look at what exactly are we doing so are we0:00:32being well served by object orientation as commonly embodied right the concept is pretty broad and there are multiple possible embodiments but the ones that0:00:41we have have a lot of consistent attributes do we all agree this is the best way to write software so we think this will continue to be the best way0:00:53certainly today this is a really entrenched model it doesn't matter which language you're using and we're a bunch of everybody has different language of0:01:02groovy and whatnot Scala and Java and people use C sharp and and they love the differences between these languages and I want you0:01:11to focus on the similarities between these languages which is they're all single dispatch stateful object-oriented languages and they have a lot of the0:01:21same kinds of things in classes some notion of classes inheritance fields or interesting concept methods are more interesting we'll talk about them later they're all garbage collected and they0:01:32have a heritage that goes back to languages like small talk they're not significantly different in some dimensions right they're superficially0:01:42different they might have mix-ins I might have interfaces even static and dynamic typing I think is not nearly as0:01:51important as some of the underpinnings that they share you know everybody's so excited because now there are languages without semicolons and other great0:02:00choices that we have but they have more more to do with the sensibilities of the programmer than they have to do with significant differences in the0:02:09programming model okay so they're all different cars they're all in the same road is this the end are we done are we0:02:20going to keep making languages that are just very very slight incremental differences to the things that we know is certainly one thing is undeniable people0:02:31like object orientation on the other hand I think we've gotten increasingly conservative and which makes sense of course you get adopted by large companies they have big investments0:02:41people know how to do it it's not something you're gonna move move away from any too readily and certainly I want to emphasize the purpose of this0:02:50talk is not to beat up on oh whoa but to have everybody just take a step back just imagine you don't love it if you do and think about whether or not it's it's0:03:03perfect when we look at languages and trying to think of what should be if I could write another language or if I could fix this language or if I could0:03:13make something add a feature to the next version of the language what would we do why do we add things what drives us to make changes well what drives us to change cars you0:03:23know to say I'm going I'm gonna stop using this language I adopt this other language and what what things should drive us to that I don't think a lot of people say oh I'm tired of semicolons0:03:33I'm not I just like I can't do it anymore or curly braces or something I'm gonna switch switch to something easier I think static and dynamic may cause people to switch but I think there are0:03:43there are examples already in our history that show us what what causes us to switch so the things I'm going to talk about today are a small subset of0:03:53the kinds of things I think you should think about when you look back at the language you're you're using and try to decide whether or not you want to do something differently I want to talk0:04:02about complexity today I want to talk about time mostly about time and then about models we can use to better implement time and and some of the0:04:14principles that underlie object orientation it's a modeling concept right it's based around we can sort of do things in our programs that are similar to what we do what we see in the world though and that0:04:24helps us understand our programs so the hero of the talk today is Alfred North Whitehead right he's the famous guy who0:04:33with Russell wrote principia mathematica subsequent to that he also became a philosopher and he wrote some great things and I'm just gonna put them up0:04:42here because they're great so the first thing is distrust simplicity I don't want to talk actually about the complexity of the problems we're trying0:04:51to solve we all know we're given increasingly more complex problems to solve bigger problems more data more0:05:00flexibility expectations of people for software will only ever increase the complexity I want to talk about today is the incidental complexity the complexity0:05:10that arises from the way our tools work from the ideas that embody our tools for the ways from ways our tools don't work from the ways our approaches don't work these things all become problems that we0:05:20have to solve and you have a certain number of hours in the day you have to solve problems other problems you're solving the problems of the application domain all the problems you've said in front of yourself by choosing a0:05:31particular language or tool or development strategy so that's an incidental complexity it's coming along for the ride it's not part of the problem you're trying to solve and it's0:05:42worse I think I mean everybody knows when something's complication you look at it says are you know calm complex and every says okay well I see that that is0:05:53scary I know that's a danger zone I know I'm going to be careful with that the worst kind of incidents of complexity is the kind that's disguised as simplicity look how easy this is0:06:04there's no semicolons I don't wanna beat up on no semicolons it's just an easy way to say look at some superficial aspect of language I'm using this seems easy this seems familiar0:06:13but is there incidence of complexity hiding underneath it so this is an example again not to beat up on C++ but0:06:22I spent more than a decade doing this so it's it's not that hard I mean if you get into template metaprogramming it can0:06:32get it can get hard but the basics are pretty simple right you can write a function that returns the pointer what's wrong with that it's pretty simple you0:06:41know there's there's new in the lead and this pointers and you can pass them around you can dereference them there's are like five things you need to know you can learn them in an afternoon0:06:50so it is it really simple for instance the same syntax is used to refer to things on the heap and things that are0:06:59not on the heap these pointers but it gets worse right and the real problem with that function signature is what do0:07:08you do with the thing that you get when you call it is it yours is it now your responsibility you know do you have to delete it later is it even something0:07:17that can be deleted can you hand it to somebody else is that allowed could you save it so the problem there was there was no standard automatic0:07:27memory management right there's no garbage collection and this was and still is for people using this language a big source of incidental complexity0:07:36right because managing memory is on you you don't see that there's not a sign on the top of your source code don't forget managing memory is on your head right this is incidental complexity0:07:47you have to just know that it's not in the source code and it's a big problem I think the lack of garbage collection really impedes C++ and one of its design0:07:56objectives which is supposed to be a library language all the original design stuff and anytime you heard some talk about is like C++ is going to be a library language but it only ever ended0:08:06up being a parochial library language every shop had a library but there were not a library and still are not a lot a lot of libraries that go between places because of this problem and we know that0:08:18Java having garbage collection has a huge library infrastructure so I think people that moved from C++ to Java did0:08:30so in no small part due to the fact that they were no longer willing to bear this implicit complexity I don't want to do0:08:39manual memory management it's not part of the problem trying to solve at all it's just another problem on my plate every day when I go to work and I want to do it so let's look at Java it's0:08:52easier there's no asterisk this is like even better so what's the problem with this it's it's simpler it's definitely0:09:01simpler right now we only have references to manage memory and we have automatic memory management we have garbage collection this is much much0:09:10better it's much easier except again we have this hidden complexity right is0:09:19this a mutable thing or not can I you know when will I see a consistent value if I look at this right0:09:29now and I and I walk through its fields will the some of the things I've seen represent a consistent value all right this isn't just a concurrency problem0:09:39right there is a concurrency problem and it's a big one but but even before we had threads and all that part this is a big source of incidence of complexity in0:09:50programs because we don't know when we have a stable value right can I store this date off and look at it later and0:10:00know I'm gonna see what I saw when I was handed it you don't know in addition if you hand a date or some mutable thing I0:10:09mean I know the mutable things have been all deprecated I know they're fixing data I'm not trying to beat up on date but if you hand a mutable thing to somebody or and they may hand it to0:10:19other people and then you need to change it who is going to be affected by that you have no idea so this looks really easy now this is true listen this is not0:10:28just Java this is every single language I listed that allows for mutable objects has this problem and there's no way to0:10:37fix it so what's the problem here I'm gonna say the problem here is we don't have any standard time management okay0:10:47that may be a really confusing thing hopefully it won't be as we go along so this is kind of a little bit of really0:10:56reiteration of the points I was making before I think that because we're so familiar with this it's we're absolutely completely blind to it right and and and0:11:06when we choose languages or when people choose different languages a lot of times they make the decisions I'm very superficial differences like the syntax or perhaps0:11:16sort of this makes me feel good expressivity differences which I admit completely are real and valid but they're somewhat emotional in the0:11:26meantime our systems are getting very very hard to build maintain and make correct and in no small part that's due0:11:35to this incidence of complexity we can't understand big programs right we have these giant test Suites right because and we run them every time we changed0:11:44any little thing because we don't know if we change something over here that it's not going to break something over there and we can't know and I think you know0:11:55for me and I think for many people we're gonna find concurrency just is the straw that breaks the camel's back in this area so so we're programmers you know we0:12:07don't use assembly language anymore we have languages right each time we build a new language or we use a new language we're expecting some benefits in this area right we want to hide chunks of0:12:20stuff name them encapsulate them get them out of our way so we do not have to think about them and we can build something on top of that I mean somebody0:12:30who's building houses out of bricks does not need to worry about the inside of bricks right they have certain properties they have certain expectations and I think it's one of the0:12:40selling points of object orientation that this is a way to make these kinds of units that we can combine to make programs that are easier to understand because we understand the pieces and we0:12:50put the pieces together and get something we can understand it ends up that that they're really not the best unit for that the best unit for that are0:13:01functions and in particular pure functions right if you want something you do not have to worry about you should love the pure function the pure0:13:10function takes immutable values it does something with them the stuff it does has no effect on the world and no connection on the rest of0:13:19the outside world then it returns another immutable thing so the entire scope of its activity is local0:13:28it has no notion of time that's going to become important later but it's definitely easy to understand it's easy to change right there's some signature0:13:38that's the only thing about it anybody else knows we change the signature wait and we change the insides nobody cares pure functions are and should be the0:13:48bricks that we use because they are the things we can use without worrying about them most readily there are definitely huge benefits from doing this I think0:13:59they're there you could easily do it in object-oriented languages what people don't in contrast objects and methods do not0:14:08have this property they do not have the I don't need to think about them property they definitely don't and we're gonna see why in a minute on the other0:14:18hand as great as functions are as building blocks our programs in general are not functions okay there are programs that are functions right there0:14:27are compilers and theorem provers you know take the stuff convert it whatever but a lot of programs run indefinitely long and people have an expectation of being able to see their behavior to have0:14:37inputs as the program runs right and get something different every time I want I don't want Google to return the same result every time I type the same word0:14:46into it Google wouldn't work for me that if Google was a function it would be no good okay Google is a process it's connected to the rest of the world0:14:55it's scouring pages and integrating them and forming algorithms which hopefully also should change as a whole it feels much more like a participant in the0:15:04world than a function anymore it's not an idealized calculation so we can say that you know that the entire program0:15:14has this behavior we can observe over time although you'll see I don't like the word behavior so so most programs and let most programs that most people0:15:24work on in industry our processes so so we see maybe we haven't seen the value0:15:35of functions I certainly don't think we have but we also have seen the limitations object-oriented was a way to say well you know I have functions are great and they're great for calculations and all the stuff but0:15:44then I see the real world and there are objects in there you know there are windowing systems and there are things and you know object orientation was a way to say alright well how do we take our mental model for the processes we0:15:54see in the world and embody them in some kind of programming model right and so the essence of the object-oriented programming model is not encapsulation blah blah blah it's really that that0:16:04behavior that flow like thing you know we have these entities we see doing things in the world we should have entities that do things in our programs so with the first thing we should0:16:16realize that any programming model is going to be that tries them all the real world isn't in essentially going to be a simplistic thing okay but again there's that be where simplicity you know is0:16:27this thing too simple to do the job correctly right one of the problems with object-oriented time is that you know0:16:37that we talk about behavior and state and things like that that really really loosely these terms are almost completely meaningless and in addition0:16:47even though objects you know putative lis are about process there's no notion no concrete notion of time in objects no0:16:56more so than there are in functions but lease functions aren't pretending to play with time functions say there's no time right there's my inputs my outputs I'm not I'm not pretending to deal with0:17:07time objects are pretending to deal with time and yet our object systems don't have any reified notion of time there's nothing you can talk about explicitly because most of them were born in a day0:17:18when you know your program ruled the computer you know you had a single monotonic execution flow and it just did what it wanted do this do that0:17:27you know there was a single universal process controlling everything now that that's no longer true you know we try to use locks to restore that vision of the0:17:38world but that vision in the world was never correct and and you can tell in one key way because we still even with0:17:49all the locks and everything else we still don't really have a concrete representation we can use for perception you know can I look at something and see it be stable or memory can I remember0:18:00that right these objects are all live their time bottles right we have gotten this wrong odd the object oriented model0:18:10has gotten time wrong and we've done so in a couple of ways the first is we've made objects that can change in place0:18:19and we've made objects that we could see change in place right as I said we left0:18:28out any concrete notion of time and there's no proper notion of values okay you can fabricate values right you can0:18:37make a class that has all immutable components and that would constitute a value but there's no proper notion of value in a lot of these languages the0:18:48biggest problem we have is we've conflated two things we've said the idea that I attached to this thing that that0:18:57lasts over time is the thing that lasts over time and that's not actually true in addition I'll as I said before our0:19:08ability to perceive is fragile so I have the hero of the day Whitehead up here who subsequent to do in all the principia mathematica stuff as I said0:19:17became a philosopher and he tried to concern himself with how does the world actually work informed by the current knowledge which this was back in the 20s0:19:27of quantum mechanics and relativity and one of the things that he came up with was the fact that time must be atomic0:19:40and move in chunks and in fact time isn't actually a real thing you can touch but it's something that you derive from seeing these epical transitions so0:19:55I'm going to explain that more but this is a great quote right no man can ever cross the same river twice right just what's a river we love this idea of0:20:05objects like there's this thing Changez right there's no River right there's this water there at one point in time and another point in time there's0:20:14other water there right River this river is all in here so how do we how did we0:20:25make this mistake what's the real nature of this mistake right it looked like we could change my rain place right we were doing it it's peek and poke and it0:20:35looked like we could see that yeah we could read but there was nothing about what we were putting in memory that had any correlation to time right it was0:20:45live again and now we're finding what look at these new computer architectures where is the variable well there's one version of it over here0:20:54from one point in time right and another one over here and that's on its way to a place that this over there might see at some point it's live now we see the0:21:05problem right there are no there are no changing values there's values at points in time and all you're ever going to get is the value for a point in time and0:21:17values don't change all right so the biggest key insight of Whitehead was there's no such thing as a mutable0:21:27object we've invented them we need to uninvent them okay in whiteheads model which I am grossly0:21:36oversimplifying okay I don't even understand it the book is completely daunting but it's full of really cool insights and and what he's built is a0:21:45model that says there's there's this immutable thing then there's a process in the universe all right that's going to create the next immutable thing and0:21:55entities that we see as continuous are a super imposition we place on a bunch of values that are causally related we see0:22:04things happen over time we say oh that's Fred or oh that's that's the river outside the back of my house that's the cloud right we know you can0:22:13look at a cloud for enough time and also it's like well now there's three clouds or the cloud disappeared right there is no cloud changing right you superimpose the0:22:23notion of cloud on a series of related cloud values so here are the rules again0:22:33I am NOT restating Whitehead I'm making this up now okay actual entities are immutable right when you have a new0:22:43thing it's a function in that pure functional sense that I just talked about of the past so the future is a function of the past and processes and0:22:52the notion of process is what creates the future from the past identities are mental constructs okay we call it a0:23:01cloud we call it a river we called him Fred it's an extremely useful psychological artifact that's why we have object-oriented languages this is0:23:10useful to us it helps us understand things but we have to make sure we understand that objects are not things that change over time all right we0:23:20superimpose objects on a set of values we saw over time that's an object so just because we can we like to think of0:23:29it this way because it's important to us to understand the causality you lyin lyin lyin lyin lyin you know I better go right that doesn't mean there is a line0:23:40that's changing there isn't and then time then is strictly again a derivative0:23:49of this series of of events okay so whiteheads great quote which is extremely confusing but I think it's0:23:58something that you could try to try to get right now and remember as I keep going is that there's a becoming of continuity right there's this process in0:24:07the universe that's creating successive values right and that allows us to say oh continuity great it's not the other0:24:19way around so now we're completely out of Whitehead terms he has a whole bunch of his own0:24:28terms but this is these are the terms I want to use to talk about the rest of this this problem the first is the notion of a value we need a very proper notion of a value right we we tend to0:24:39have a decent notion of a value when we say 42 we have a much weaker notion of a value when we talk about dates so the0:24:48key characteristic of a value is that it's immutable okay it could be a magnitude it could be something like that or any composite of those things that's also immutable is a value these0:24:59are extremely important to us right then we have identity identity again is the psychological construct we're going to see a succession of values who's who's a0:25:09causation is is related right well was the was caused from the previous was caused from the previous and we're gonna say Fred Fred again is a label right the0:25:21important thing is this identity which is just a contract we used to collect the time series a state right it's not0:25:34something you can change the state is a snapshot this entity has this value at this point in time that's state so the0:25:44concept of mutable state it makes no sense mutable objects they make no sense finally we have time time is a0:25:54completely relative thing all time can never tell you is this thing happened before or after that other thing or at the same point okay it's not a measurable thing it doesn't have0:26:03dimension this all sounds kind of highfalutin why do we care about this0:26:12we care about it because we're trying to make programs that make decisions but we have logic in our programs you can't have logic on top of rivers that can0:26:22change okay you can only have logic on top of values right so we need stable values and we need to collect them from0:26:32other parts of our program we need to see stable values we need to be able to remember the so I'm using the word perceived I understand completely perception is an0:26:41incredibly intricate and unresolved mental phenomenon but I like it better than just observe because then I can observe the entire room but perception0:26:51really is kind of that division in two entities it's a little bit a little bit finer on the other hand I do think we0:27:00need identity I mean I think that the appeal of object orientation is is is valid right we care about this because it's the way we're thinking about the0:27:10world all the time if I have to change completely the way I'm thinking about the world in order to write a program my life is going to be hard if I can somehow carry over from the way I think0:27:19about the world something to the way I write my program it will be easier okay but we can't screw up time and and and0:27:28and state the way we have and have it still be easier because it's now wrong so it looks like oh I understand the0:27:37objects I understand but it's not right so I saw this great talk at JavaOne where the the people who wrote head0:27:48first Java which is fantastic book talked about the guy talked about I forget his name I'm sorry you should put a lot a slide of a lion in your talk0:27:57because it'll get everybody like scared and then they'll be more receptive so this is my lion okay so the let's let's0:28:09try to like pull that theoretical mumbo-jumbo down to something we can use to write programs the first thing we need to understand is we don't make decisions about the world by by directly0:28:19by direct cognition we know take our brains and rub it on the table we don't rub it on the you know Fred there's a there's a disconnect between our our0:28:32logical system and the actual world okay it's not live right this whole liveness we have from I can see memory that's0:28:41that's not how it works the other thing we don't get to do in the real world right we're going to model the real world we don't get to do this wait0:28:50okay okay we don't get to stop the world especially not to observe it okay but what are we doing our programs all the0:28:59time stop wait stop wait hold on you know everybody's trying to stop the world so they can control it completely as we get more concurrent we're going to0:29:08need to learn to live in a world that's going to proceed in spite of our intention or desire or best wishes that it would not because it would be a lot easier for us if it wouldn't it's gonna0:29:19we're not going to achieve the degrees of parallelism and the concurrency we want and so we can accept this and embrace it so we need to look more0:29:29carefully at well how does perception actually work we don't rub our brains on it we don't stop the world it is incredibly parallel right there's empty and people in this stadium they0:29:39can all watch the game they don't say whoa whoa whoa let me look at you they'll say hang on let me take a picture they don't need to right they can pick a picture in the game can keep going right so the first thing is0:29:50perception is uncoordinated okay it's massively parallel it is not message passing there's no communication between0:30:02the people want to see the game and the game so we can again look again we're trying to model reality so we can look0:30:11at reality a little bit how do we how do we do it how does the wetware do it well it ends up that the first thing you have0:30:20to realize is we're always considering the past we never proceed perceiving the present right there's the propagation of light it hits my sensory system is0:30:30incredibly slow system that carries that's when my brain by the time I'm making the decision about anything I am using the past I am always calculating with the past0:30:39because we're not able to impede time right we can't stop the world right so the world is absolutely continued I have some you know it seems instantaneous I0:30:48you know I see the person in the front row here but you know they could leave depending on how much time and how much distance because flight is pretty fast again it's like tricky like like you0:30:59know electrons and makes us think that when we're looking at memory right now but it's really not it's always the past we're always perceiving the past the other thing to pick up from looking at0:31:09our sensory systems is the fact that they're incredibly oriented around discrete events okay we have neurons0:31:18that carry chemical signals which could be continuous and we could have built brains that were continuous right that somehow took the world and and and0:31:28consider it like this this moving thing and the moving thing comes into our brain and it's all moving around okay guess what we didn't do that evolution0:31:38did not do that why because that's a mess right weak you cannot do logic if everything you're trying to consider is moving around so what do our neurons do0:31:49they build stuff up and then they go fine they discretize the input what's the next thing that we do we say whoa0:31:59ten things happen at the same time right so we discretize things and then we love simultaneity we have simultaneity detectors that's0:32:08what our brains are at a lower level okay so coarsely we like snapshots okay0:32:17snapshots are good they help us think they're like values another thing we've done an object orientation said your0:32:26methods and methods are way to read things and perceive things and the way to make things happen well making things happen and perceiving things are completely different they're completely0:32:35different they shouldn't be in the same construct there are two different things right because action is this other property right no two things can you0:32:44know effect the same thing at the same time we have to sort of take turns that succession of values that we're going to use to to to understand the world right0:32:57is atomic it's an atomic succession and while we've grouped it into threads that it helps make make it easy to understand it's not actually that way we certainly0:33:08understand the fact that you know there can be only a certain amount of stuff in one place at one time and when you're trying to act on that0:33:19stuff you're gonna have to be there so action has to be sequential an action and perception are two different things so now I'm gonna put this up I'll put it0:33:31up again later this is a model this is not a picture of some software this is a model for how to think about time the0:33:42first thing we need is we need a value that's a point in time we said a point in time is is a value right it can't be changed so we'll use values to represent0:33:51points in time we will still probably organize our programs by identities as long as you remember the slide from before that the identity is a derived0:34:01notion it isn't a thing that's doing stuff right it's a derived concept we get from this process we can still use0:34:10identities to organize things because that's going to be useful to us object orientation has shown us that's useful for us to understand processes but how0:34:21does how do we get through these epical atomic successive events we use functions right we take a function of0:34:30the past we produce the future so the apps on the top or pure functions right they take the state of the universe or bless to say the state of an eye and0:34:39that identity at one point in time and produce the next one what's inside them is individual indivisible unperceptive0:34:48all its atomic the functions are atomic and that's the that's the process of the0:34:57world right we say behavior in object-oriented systems there really is a behavior that says you know I'm driving right I'm doing this right but0:35:06when you get hit by lightning who's behaving there's no behavior but there are processes in the world may affect things0:35:15so those are those functions right we're going to call you know any one of those relative to an entity all right an identity it's state right again it's0:35:26just a label of a value of an identity at a point we'll call the state and the identity itself again is a derive thing the succession of states0:35:35is Fredd or the river the important thing also here is that people can be0:35:44looking at this right there can be observers light can bounce off the river it can bounce off a Fred Fred doesn't need to do that Fred doesn't need to0:35:53drive that we can look at that so we can observe things and it's very important that observers are not in the timeline0:36:02and then the blue stuff in there it's not actually reified anywhere but that's time again it's another derived thing so0:36:12the box around all the states that identity is derived the notion of time it's only because at one point we looked at this another point we looked at that0:36:21that we know that there is time things don't come with labels this was you know September 22nd yeah all right so how do0:36:31we do this if we wanted you know take things apart like this and then put them back together what how do we gonna do it well we're going to need two things we0:36:40looked on a diagram before we saw functions pure functions I think we know how to do that I think we're all agreed we have the technology to write pure0:36:50functions so that leaves with two other things on a diagram one was values the other was somehow we managed that0:36:59succession right some sort of time constructs so we need a way to efficiently create values right save them maybe we'll use them as percepts0:37:10later and we need something that's going to coordinate the the succession of values right so we'll call them time0:37:20coordination constructs so we need those it ends up that we can and maybe some theoretician will prove we have to0:37:31consume memory in order to model time we certainly can I don't know that we need to but I'm figuring out a way to deal0:37:40with that so what do we do we say we pass the old value to a pure function we produce a new value non-destructively0:37:50right that's going to consume some memory and we know that but that's gonna let us make this correct those values0:38:01have other have other value because they can serve as our perceptions right what this whole system you know the whole visual system is about making these0:38:10snapshots all right well in a program I mean admittedly the snapshot in my mind is not the audience here they're two different things but in a program they're not really two different things0:38:20right if you had a thing if you had a value in the program and another part of the program wanted to perceive it they'd love a copy of it that would be great that's so good that's a good enough0:38:29record for them so we could use these values as as our per steps right we can also use them as our memories right if we have a portion of a program it needs0:38:38to remember something this value would also serve that purpose so we have a good system for doing values we can do that and then the beautiful thing is if we're consuming memory to model time GC0:38:47will erase the past and the memories that nobody cares about anymore so the contract I think we need to do0:38:57values is our persistent data structures I've talked about them before I mean if anybody doesn't know really quickly we're not talking about being able to put stuff on disk here a persistent data0:39:06structure is immutable right when you make a new version of it when you try to change it you get a new thing both on the old and the new thing are available0:39:15after you've made it and both have the same performance characteristics and then make that characteristics of the data structure and the production of the new version also has the same0:39:24performance characteristics so that's quickie persistent data structure so what good are they in particular they're immutable okay so they're great for the purposes that we0:39:34need memories and perceptions snapshots essentially they're stable another beautiful just practical aspect of them0:39:43is they never need synchronization which is back that's just like the baseball game right that's good just 19,0000:39:53memories there or 19,000 perceiver no synchronization the other nice thing about persistent data structures is in0:40:03their implementation generally the next version of the value shares a lot of structure with the prior version so that makes them more efficient the other0:40:13thing that's important is when we make the new value we don't disrupt anybody who's looking at the old value we don't need to say wait stop hang on even if we're not going to0:40:22destroy it we don't need to do anything and that goes back to the synchronization and and if you have not ever used a functional language or ever0:40:31use persistent data structures in a non functional language just take my word for it this is so much better if you write a program that uses data structures like this you will just be0:40:40able to sleep at night you're gonna be happier you know your life is gonna be better because there's a huge quantity of things you will no longer have to0:40:50worry about all right so this persistent0:40:59data structure sano involved there you know it's just oh this is really old it's the stuff is so old it's almost embarrassing to put it up here right0:41:09trees and all the persistent data structures essentially under the hood are trees because trees have these properties that allow you to share structure right and and do updates but0:41:19in particular I think you know in a practicum a practical sense you can implement the kinds of things you're used to having like vectors and hash hash maps and things like that using0:41:30trees with a couple of properties at least this has been my experience one is that they have very high branching factors and so therefore they're very shallow and that gives you good0:41:39performance you can implement vectors you can implement hash maps and I think the the world of things you can do here is still open but the bottom line is0:41:49they're all trees and they're their trees for this reason trees support structural sharing so the tree rooted in past there is immutable it's never going0:41:59to be changed when we need to make a new version say add a and a new node we're going to use something called path copying right we're going to copy the path from the root to the two0:42:09we need to change right so make copies of those over here on the right well this new copy will have the new node we want you know the leaf node we want and0:42:19we get a new root but that new tree rooted at next shares everything with the old tree except those three red notes so that's good0:42:34moving forward as we try to make programs that we can paralyze we have to stop writing loops right I think everybody understands it's a whole0:42:43separate talk that you know we're going to get our future performance gains from parallelization right which means going to have to write more declarative programs and those parallel those0:42:55declarative programs are going to need to be able to take data structures and do parallel transformations on them and produce new data structures and we want to stick with this model we want them to be persistent right so how do these0:43:06persistent data structures serve that purpose very well it ends up because they're already divide and conquer I mean half half the work is already done0:43:16they're sitting there divided okay they're pre partitioned in addition if you if you do it right you also have the ability to construct them in a0:43:25compositional way without any collisions so you can avoid synchronization in the building of the of the new versions so they're they're they're pretty well set0:43:34up for doing parallel algorithms I think persistent data structures should be the default data structure I wish there was a language where0:43:43persistent data structures were the default data structure okay so I mean I'm not gonna lie to you0:43:52right we everybody's like performance models everything else they're slower they are slower especially for serial use and especially for writing for0:44:03reading you will be very much surprised at how good the performance can be some of the good performance I see I completely do not understand but it's0:44:13there reading is actually pretty solid writing those a problem you have that path copy and everything else but you know I am NOT a fundamentalist I'm a pragmatist0:44:23so if there's this path right and if no one can ever see what happens there right in other words if it's gonna take0:44:32something immutable and it's going to produce something immutable and those are two discrete instances of time and everything else about this is atomic then nobody cares what happens inside F0:44:41okay you probably do care if it's a big involved thing you probably still want to do it with pure functions for sanity preservation reasons but for this time0:44:51modeling reason you can do whatever you want okay which means that when you're bringing you know when you're birthing the next version of a persistent data0:45:00structure you can do the same old good stuff you know how to do right you can allocate an array and you can bash on it because no one has yet seen that array0:45:09you can use fork/join but it works great and these things will eventually bridge0:45:18the gap already on you know my quad core a parallel version of map on a persistent vector is as fast as the loop0:45:28that bangs on an ArrayList same speed so more cores we start winning because we0:45:37have all these other great benefits no synchronization required for this persistent data structure share it all you want rest easy comes with all those benefits that ArrayList doesn't the other thing0:45:50that's possible is you can make what I call transient versions of these persistent data structures and that's something I've been working on recently which have nearly the same speed of the0:46:00good old data structures you're using and in particular support constant time creation from a persistent data structure and constant time restoration0:46:09as a persistent data structure and can be made safe they're like 90% as fast as a mutable thing so obviously this is0:46:20something you should care about on the other hand I wouldn't deny the power of this model because you're afraid of this0:46:29okay so that's about values now let's look at the time model again so remember what we're talking about when we talk about time okay so we just said we now0:46:39know what's that what the V's are they're going to be these persistent data structures right so so how do we make sure that there's you know only one blue arrow train for any particular0:46:49identity how do we coordinate time again remember identity is as a as a side effect right we see that later0:46:58same thing with time we see that later but it's convenient to us when we're trying to model in a program to pretend we're driving it forward so what is the0:47:09time kind of truck do its main job is to make sure that you have atomic succession of values okay that's its0:47:18main purpose that we go from one value to another uncorrupt ibly and that there's no in-between right that's what ethical means right0:47:30the other thing that time contract has got to do is it's got to provide some way for us to see the identity to see the thing that it's managing has to provide visibility and again has to do0:47:40that atomically right because what really happens is the baseball game right and there's photons and then there's a point in time and the photons are the same place the baseball game is0:47:49and then they go their own separate ways again right so there was a moment there where those things could connect to each other but what was represented was that0:48:00snapshot again a value at a point in time so they need to provide that we want to have multiple timelines again this whole I am the program I control0:48:09the universe you know I'm stopping everything around them the only thing that isn't working anymore we need to have lots of threads of control which means we want multiple0:48:18timelines the nice thing about this whole thing is that there's no inherent semantics to this other than you know0:48:27complying with these couple points there's a variety of different semantics you can apply you can use Kaz which is essentially saying there's one timeline per identity right and it's0:48:37uncoordinated it's impossible to coordinate two things that are using Kaz timelines but Cass landlines are still useful they have semantics you can understand there are agents or actor systems which are0:48:48also one-to-one there's one timeline per entity so they can't be coordinated but they're asynchronous so that they're not connected to the timeline of the person0:49:00enacting the event there are things like STM which allow you to do timeline to coordinate timelines and maybe even0:49:11there can be new constructs based around locks right because you can look at lots of saying well you know that's the way to enforce timelines it definitely is a way to enforce timelines if you have a0:49:20way to automate that and package it up into one of these time constructs that's great and you should the difference between them and SDM would likely be that they have fixed reasons as opposed0:49:30to ask them which is arbitrary regions but if you said well you know all these timelines are a really timeline X X to be represented by a lock if you have0:49:39some sort of time just construct that ensures lock acquisition order you can play this game so let's look at cached in the time country because it's the0:49:48easiest possible thing so you have some calves like thingy like atomic reference right that's going to store your timeline right there's no history in it0:49:57which means essentially that each successive value will replace the other but what we care about is that there is a timeline in this case the wrist represents one identity the thing that's0:50:06in it that's always going to be an immutable value right cast ensures atomic states accession right if two things the two processes decide I'm0:50:16going to move value two forward I am the process that's going to do that only one of them can succeed right so this red line that will be prevented by Kaz right0:50:26and you can wrap up the the logic associated with doing that correctly with calves right which is that spinning thing and just package it in the0:50:36constructor so it could look something like this right swap some calves based reference using this function which will be the function0:50:46of the past maybe plus some extra information the args and and what happens in any time construct is this latter point here you're gonna call the function on the0:50:57current state also passed the args if you want and that will become right that's what the construct does it makes it allows that to become the next value0:51:06time is derived from that identity is derived from that but that's what's really happening so it looks like that and again under the hood we can automate0:51:15the spin the other thing you know atomic references allows is the ability to atomically look at what's inside of it and as well as what's inside of it is a0:51:24value we have good point in time perception I don't want to talk to spend too much time on agents I'm there a lot like has except that there's no longer a0:51:34coordination in in in Kaz when somebody's calling this function when someone's saying swap there are actually two timelines right there's a timeline0:51:44of the identity they're trying to manipulate and there's the caller they have their own time line right those two timelines meet at swap with an actor or0:51:55an agent system they don't meet right you initiate some energy force and it flows out towards that thing and you0:52:04walk away and eventually that energy force hits that thing and whatever the result is is the result and the thing changes right so there's now an0:52:13asynchrony between the caller's time line and the time line of the identity but otherwise it's still doing all the same work it's one-to-one relationship between the time line an identity right0:52:22atomic states accession falls out of two things the succession falls out of the fact that everything's being put in a queue and the atomic falls out of the fact that there's only one reader and0:52:33they can also provide point in time value perception the reason why I called these things agents and not actors actors typically do not in fact they0:52:42definitely do not but in an in process model I think perception should always be supported alright so what happens when0:52:51he needs coordinate two things or more than two things these things these Casas and other things are not going to work right because you can't coordinate them so you need something else0:53:01one possible other thing it's probably not the only one is software transactional memory right or any kind of transactional thing which0:53:11allows you to coordinate the activities of multiple arbitrary regions so multiple timelines we're going to say okay this this action um I'm invoking is0:53:23going to affect three things which means somehow their timelines have to have to meet they have transactional0:53:32capabilities which are not really interesting for this but the most important thing is we're not walking away from the epical time model this is still the epical time model for any0:53:41value that's going to participate in an SDM transaction it's still the same thing you're going to have some function on the past produce the future a pure0:53:52function and values in and out so what does this look like now there's multiple identities right potentially or places0:54:01or whatever you know whatever whatever contract is meaningful to your program you still have that and any particular transaction is going to you know take an0:54:10arbitrary set of these and atomically do that function transformation so it's a way of collecting a bunch of little micro processes and making them one0:54:20process internally each one works exactly the same way as before I just didn't put all those arrows because it would be unworkable and the set of0:54:30transactions themselves feels like a like a timeline in particular you know if blue and yellow don't overlap they0:54:39technically happen at the same time really they happen at times there's no time right because there's no succession between those two things there is no0:54:48time you'd have to superimpose it because we said time only is a derived concept from one thing happening after another so if they're unrelated it0:54:57really gets messy about time that's physicists will so I left perception out of this because0:55:07this is too too messy so so what's what's the perception story for for STM can we look it can we look at the whole0:55:16stadium at one time there can be glance and see multiple entities and it ends up that you can build systems that do that in particular an STM that uses0:55:27multi-version concurrency control can do it and I'll explain that more later but I just want to show it to you first right essentially what happens is there0:55:36can be perceivers what's really important about this diagram is they're still not in the timeline there's never been somebody perceiving who got up into0:55:46this box right perception does not interfere with process you cannot lunge those two things together so relative to0:55:58these atomic events that in fact more than one thing any perception is either going to occur completely after one or0:56:07completely before it if it's transactional itself all right that's what STM's provide you can still do a0:56:16non transactional scan you can you can pan you can look at this part of the stadium and then go over here and look at that okay or you could you know you can look at a car in the road and you0:56:26can look up at the clouds and you see the red car here and look over the cloud you look over here and you see the red car right but you know when you're doing that what that may be the same red car0:56:35now you realize when you're panning like that you're not seeing a point in time but you have the choice yes0:56:50now we'll have to save that now they0:57:04don't need to agree they don't I mean but you you can you could have multiple STM's right this STM sort of constitutes a little universe okay so we have0:57:17transactional viewing which is like glimpsing we have non-transactional viewing which is like scanning so one0:57:26way to do this is using multi-version concurrency control right this is the same all this all the stuff is old alright this is the same old stuff from0:57:35databases but so multi-version concurrency control means that you're keeping some history in order to satisfy readers that's the that's the database0:57:44thing but but there's a way to think about it in this model as well right that very critically one attribute the key attribute of multi-version0:57:53concurrency control is that readers don't impede writers that perception doesn't impede process that's huge I think you cannot do without that0:58:03everything about everything I've shown you before if you stick perceivers in the middle of those timelines your life is gonna get way more complicated stop the baseball game so we we don't want to0:58:15do that and no and so the you know in this modeling you know let's pretend we're we're we're doing the real world in our programs you could say that0:58:24multi-version concurrency control models light propagation or sensory delay right that whole chain that if if somehow the light bouncing off the baseball game is0:58:33capturing its value you know enough good enough for us that transmission delay means that you know that value has got to be somewhere well0:58:43it's being transmitted while process keeps going game keeps going so we do this by keeping some history quite interestingly and fortuitously0:58:53persistent data structures make keeping that history cheap there's something profound about that that I don't understand the other cool thing0:59:05about multi-version concurrency control is it allows readers to have their own notion of a timeline I saw this then and then later I saw that that is really0:59:17important to some decision-making in fact when our brain reconstructs behavior that's exactly what it does we just looked at the sensory system its0:59:26discretizing and snapshot izing everything okay well but well we definitely have per steps for you know there lion is running towards me we're gonna go running we have to read arrive0:59:36that that running and we do that by saying you know by saying by a mental process sets that somehow allows us to compare a snapshot from before to a0:59:47snapshot we knows later right and see the deltas of that and say running lion in addition we again we know the0:59:57difference between the visual scan we know when we've looked at something and we and we've carelessly looked at something else over here we are not allowed to correlate those things and say they happen at the same time so you1:00:09know this is really not talk about STM but I do think that the one takeaway I'd really like you to have is that STM's are different from each other there's no one STM if you want to beat up on STM1:00:20you know pick one pick its attributes and you know find out what's wrong with it because there there are some that I think really get time wrong still get1:00:31time wrong right so if you don't have multi-version concurrency control you either are gonna be limited to scans non temporally related I looked at this1:00:41thing I looked at that thing I have no idea I have no ability to look at two things at once or you're gonna have some right you're gonna be back - wait wait stop the process so I1:00:53can get my perception in the middle of it right without multi-version concurrency control that's where you are the other thing about STM's I think is1:01:02super super critical is that granularity matters okay if you're using an STM that either that forces you or you're incorrectly using1:01:12an STM and you find yourself requiring a transaction in order to see a consistent value you have got time wrong again okay so STM's that require1:01:24transaction to read you know four fields of an object consistently or not doing time right they're not really solving1:01:33this problem okay so in conclusion sometimes I mean all the time I think if1:01:43you're suffering from excessive complexity you got to think about you know changing something and sometimes people actually do change might remove four languages of work garbage collected1:01:52substantially to ones that are because that reduces our implicit complexity there's no other reason okay but in the1:02:02current state of the art object-oriented languages this conflation of behavior and time and identity and state is just making our lives much much harder and1:02:12it's going to get worse okay we need to become explicit about time in our programs we really need to pay attention1:02:21to the functional programming people who are saying you know look at all these great properties of pure functions they are there they definitely you know1:02:30satisfy whiteheads you know we we move forward by taking away they need to understand the insides of things okay I believe1:02:40that this epical time model is is worth trying I think it's general model it supports multiple implementation ideas and it1:02:50will work in the local process I'm not you're talking about distributed computing at all the other thing I can tell you is that the current infrastructures that we have are1:02:59sufficient for experimenting with this for doing the implementation so you know so what what you know it's still unresolved here well coordinating this1:03:09internal time with the external world it's going to become an important thing and it's a hard problem and you know tying STM transactions to IO you know1:03:20transactional IO would be a very interesting and I think possible thing again overall though you want to move away from transactionality1:03:30transactionality is you know control control of control you want to become as happy as you can with a lack of control that will give you more concurrency1:03:39there always could be more parallelism and the more performance and you know there definitely could be more work done on parallelism inside these data1:03:48structures there are more data structures to be done I'm sure in better versions there are definitely going to be other time constructs I think doing1:03:58moving locking under this model is extremely interesting because locking has some particular efficiencies that we'd want to leverage and there's a way1:04:07to understand it in terms of this and I'll leave as an open question for everybody else you know is there a way to reconcile this with object orientation it could we separate1:04:17perception of an object from its identity enough so that we'd still get the benefits of objects but we don't get a mess later1:04:37it's not for questions or I know where right away okay any questions yes well I1:04:59don't know what those talks were but you know from what I've seen functional programming you know in many respects just tries to get time out of the way1:05:08get time out of my well that's not what this is this is about time is a important part of programs you have to contend with it's not advocating purely1:05:17functional programming here at all I'm saying you know there are programs there are programs that are done on one of those boxes right one transition from value to another right that kind of1:05:26program is a calculator right most programs have to deal with that that progression of time and that's a hard problem so I'm not trying to walk away1:05:35from it I'm trying to walk towards it1:06:10yeah well I mean read whiteheads he really did that he really did exactly what you're saying I'm just trying to to program without going crazy but I but I1:06:20am inspired by am inspired definitely by by that by those notions that I think they're really important for this for fixing the model we have yes yes yes I1:06:46mean I think I think there's lots of cool things I think that's interesting I think the whole wave you know what's happening in wave right now those operational transforms are a really interesting way to to think about this I1:06:56mean there's a lot more to this for instance the the composability of transformations and things like that that are very interesting but yes it is1:07:29I mean the communications part gets tricky I think I like the fact that this is sort of communication free but maybe1:07:38time constructs different form of communication I know there's different they may be well it is sort of in the1:07:50flow of time so that's that is definitely in the coordination aspect that's communicating to people you next now you know you yes1:08:07I'm having a good time right now I don't1:08:17the garbage collection pressure of this is going to be you know significant so just keep making everything you have faster that's for ten times that's1:08:37pretend time did you know I don't want to characterize functional reactive programming I will say this I brought my worked in broadcast automation systems for a long time I read that book I saw1:08:49absolutely no correlation between that and what I actually had to do in the real world at all right turning no but fabricating time and turning it into your own arguments of1:08:59functions that's that now you're punting again you're pretending right that's not time that's again punting and soon as1:09:10you connect it to the outside world you'll see that that's the case1:09:23move back you can store two tables or aggregation as long as all your data is1:09:34immutable I think you're on the right track you know I think the key takeaway here is there's no such thing as a mutable object you know if you if you can really believe that you can build1:09:43better systems which is not which again is not to disagree I mean obviously I like functional programming right but I1:09:52don't see them talking about a lot of the problems that I think real people have Thanks1:10:01[Applause]0:00:00Design, Composition, and Performance Short - Rich Hickey
0:00:00 thanks for having me it's nice to see everybody here bright-eyed after a night0:00:09of heavy thinking this is the obligatory legal disclaimer okay so we're gonna0:00:22talk about design and of course as you all know by now I don't actually write talks I just look up stuff in the dictionary and so I looked it up in the0:00:31dictionary I saw this great definition prepare the plans for a work to be executed especially a plan to plan the form and the structure of that thing so0:00:41this is there's a lot of really interesting stuff in this definition the fact that there's a plan the fact that upon having the plan something's going to be done and the plan is going to be0:00:51executed and that's about form and structure that's really good there are other definitions they are one of which is this there's nothing wrong with this definition it's just this is not what0:01:00I'm talking about today we're not talking about how things look at no point today when I say design am i using this meaning of the word design and the0:01:09route goes back to des ignore II which is which means to mark out and it's also a very interesting thing it has to do sort of with writing but also with sort0:01:18of demarcating right to designate or to sort of set something aside but you know most simply we can just say it's about0:01:28making plans and and also writing them down unfortunately I think that you know with all our effort to become agile0:01:37where we're letting design become less of what we do and and so there are a lot of butts when people say oh design you0:01:46know it's like you know we already write down code do we need to write down design you know do we still need to do designs because obviously it makes sense0:01:55if you're going to build a house right you can't just say let's just go build the house you have to write something down so that somebody can go build the house because the realization of the plan has a different form than the plan0:02:04right the plan is something written down but the realization is made of wood and nails and things like that but codes already written down so we have something to read you know do we still0:02:14need designs or can we just generate them right can we just write our program and then generate some documentation from the implementation and the answer is no0:02:23that's that's not a plan it would be something written down but it's not it's not a plan the other argument you get against design is oh my god you know I've lived0:02:33in the 80s and design stunk you know there's people who thought they could just do everything they were do it's all top-down monolithic designs these giant plans you know etched in foam books that0:02:45never came true and it's true that those were plans but they're those are not good plans and that doesn't mean planning is bad it means that that that style of planning is it's not not good0:02:54so what do we mean when we say what do I mean when we when I say good design I think one of the most interesting things about design is that you know people think it's it's generating this0:03:05intricate plan but designing is not that designing is fundamentally about taking things apart it's about taking things apart in such a way that they can be put0:03:16back together if that makes sense so separating things into things that can be composed that's what design is if0:03:27you just if you just make this intricate thing you really haven't designed in a way that's going to support the things designs need to support like change0:03:36everything every component of a design should be kind of about one or very few things sort of the nature of it breaking things down until they are nearly atomic0:03:47and only then do you take those things that you've broken apart and compose them to solve the problem you set out to0:03:56solve but the first job is to take things apart in such a way that you can do that and and a good design process is iterative this isn't a grand plan you do0:04:06once and then you go and and and finish obviously there are some kinds of design that you do have to sort of etched in stone because you're going to you know build a fabrication plant and have a few0:04:15few options later but in software we know the the materials are working with us so malleable we can get some cycles back to iterate so when I say taking0:04:26things apart what kinds of things can we take apart we can take apart all things in this list and I'm gonna break them down one by one so what does it0:04:35mean to take apart requirements this is actually quite important obviously there's a job another job also sort of gone wanting these days called analyst0:04:44there used to be analysts and designers and you know this whole water Waterfall model but that but there are requirements and usually we get these0:04:53now directly from customers and they often take the form of I want this I need that I want I want I need I want I want and the first job we have to do is0:05:02decompose those wants and needs into problems because obviously underneath all those wants and needs are some problems the the customer wants to solve0:05:11and understanding those problems is the key to designing something that solves the problems because oftentimes the design that solves the problems is not what the customer said they wanted or0:05:20needed right we separate requirements into knowns and unknowns from you know we know how to do some part of this job and we don't know how to do this other0:05:29part that's quite important there are problems you know now again we're dealing with problems we've already sorted things so there are problems that are on the domain side might possibly0:05:39some domain experts going to need to help us solve them and other problems that are on the solution side how will it scale where we run it how much will it cost to operate how much energy will0:05:48use things like that the other thing that we have to take apart are the difference between causes and symptoms so sometimes a customer will say I have0:05:57this problem right the problem is you know my screen is black and that's not actually the problem you know that's not the problem the problem is that's a0:06:07symptom of a problem the problem somewhere underneath it so another thing you have to do when you're when you take things apart is take apart causes and root causes from symptoms because you0:06:17want to get two causes because the thing your design needs to address are the causes of problems not the symptoms because you could just throw up you know and add a JPEG over that black screen0:06:28and you like its face the other requirements are the unstated requirements which are always present there are problems that nobody wants to have in the future0:06:38like I want I don't want the system to be something I can't maintain I don't want it to run out of memory I don't want it to run really slowly they often don't even say that these are problems0:06:47because they're just problems you know their future problems they don't want to have but they end up being part of the requirements set other things we can0:06:56take apart we can take apart time and order and flow the use of cues the use of idempotency commutation and transactions or all ways in which we can0:07:07separate apart when things happen and and often these are the paths to separating apart who does these things and these terms are become more and more0:07:18important to see than more and more in systems level design right commutativity is going to become a huge thing for us to be thinking about and then there are0:07:27times when we really need to know that a bunch of things are going to happen together and transactions help keep things separate by keeping things that are supposed to be together together we0:07:37take apart places and participants where things are going to happen and what components or processes are going to do them right this classic quote you know0:07:46just add some indirection and there is a lot of that a lot of design is just putting in appropriate levels of indirection but there this kind of indirection happens at all all levels0:07:55for instance part of a design is incorporates how it's going to be built you know is this pot is it possible for more than one person or more than one0:08:05team or people working in more than one language to build this system altogether or you know does everybody have to work in the same space at the same time with the same tool then that your your0:08:14process for building it isn't going to scale because you haven't taken that apart it's another kind of thing you take apart this next one I think is really super critical and I think we0:08:25don't understand it or think about it enough in our software which is the difference between the information our systems is going to manipulate and the0:08:35mechanisms by which we're going to manipulate it right so so to just talk about it simply here we'll say the set of logged in users my system is an idea and it would be0:08:46information that my systems gonna need to manage but if I have a set class or some sort of set construct in my programming language that's a mechanism0:08:55by which I might achieve representing that information but unfortunately because we only have you know our programming language and our programming0:09:04language constructs to represent both these things we often conflate things that might be appropriate as mechanisms as being appropriate for information and0:09:13they are desperately not so so we have a lot especially in an object-oriented language you have a lot of very mechanical kinds of classes and when you0:09:22use them to represent information like for instance any kind of mutable information object is an absolutely atrocious idea it's really really bad0:09:32and it comes out of the fact that we're not separating these two things when we're doing our designs finally after we've gotten something we think might be an answer we need to take that apart so0:09:43I have I have maybe one or more possible designs that I think address the problems that that I'm trying to take on at this point I have to look at each of0:09:52those solutions and take them apart from a bunch of perspectives right do that what benefits do they provide and this I think we have no problem with everybody looks at the library libraries like it0:10:01does this it does that you know it's a floor wax it's a dessert topping you know it's all benefits it's all benefits it's very rare that you see somebody say and here's the trade-offs of using this0:10:10here's that here's the here's what's not good about it here's where I decided to do X and I and I'm not going to be able to do Y and beyond being honest about0:10:20that to ourselves about our own designs is really important it also helps us communicate to stakeholders right you are going to get this and you're not going to get that okay there's nothing0:10:30wrong with that we have to see what the costs are and the other thing we have to do is determine problem fit sometimes you can take on this big solution you know part of which solves your problem0:10:39but you've taken on this big thing you know and only part of it really addresses your problem do you want that whole big thing is it really a fit is there a smaller0:10:48solution or answer that's a that's a closer fit because both may solve the problem you can have a set of choices here so I think design is really important I0:10:58mean that's what I do and and I know a lot of other people do it and I and it but I think it's important to enumerate why it's it's good to spend time on this0:11:08a design helps you understand a system right without a design you're sort of flailing around wondering why is this this way a design helps us coordinate0:11:17that becomes obvious as you get into you know having teams you can't build two things and never have talked to each other if you don't have an agreed-upon plan written down0:11:26somewhere it's unlikely that your two things are going to plug in together and work you know it's like finding a napkin what we wrote it on design helps0:11:37extension right to the set that you've broken things into separate parts with it with an eye towards connecting them back together it means that you're resulting design is going to have0:11:46connection points and when you want your your system to do something new you'll it would be possible to do it because there's something there that's why0:11:55design is not just about you know accreting up to an answer because when you do that you don't end up with any connecting points you don't end up with any building blocks and you can't really0:12:04extend that thing similarly the flipside of that is to the extent you've broken your problem down into reusable parts and compose them those parts may be0:12:13separable from your design and useful in another context right and that's how we get reuse reuse comes from design it doesn't come from language constructs or0:12:22anything like that of course everyone does design driven testing right because that's the right kind of testing if you have designs and0:12:33they specify things well and you have some automated way to go from that specification to a test that's that's good testing everything else is backwards and the last thing I think0:12:44that one of the main reasons people say oh I don't I can't do design is I don't have time I don't have time to do design but but I will make the argument that design is the key to more efficiency0:12:55because it's a lot easier to iterate a design than it is to iterate a solution or an implementation and even after0:13:04you've got an implementation right because I said it's iterative so you're gonna have some form of it or it generation after you've done this kind0:13:13of design and you're iterating and you need to do more design that design often takes the form of ah you know what does this part of the system I actually didn't breakdown enough I0:13:22need to break it down more I need to take this thing and split it in half and that's where I find most often when I have more more to do in an iterative0:13:31design process it's almost always because I look at something and say that's still doing more than one thing I just need to cut it in half and it ends up that that kind of further0:13:41decomposition is really easy to integrate in an ongoing process in in further development so it is possible to do in you think about design is taking0:13:52things apart you can always take something more apart it's great okay so this talk is going to be about design composition and performance and I know0:14:02you know when you think about composition performance you think about bar talking culturing I know I do so this is the great thing0:14:13about the dictionary is that you know there's the first definition and then there's like more definition and so there are more definitions obviously and more notions of the word composition0:14:23performance and this is near and dear to me because as intro said this is my background how many people don't know who bar truck bar taco culture in our0:14:33it's okay if you don't so Bartok was a Hungarian composer it sort of bridged the Romantic era in the modern modern era and a phenomenal0:14:43composer who had a great interest in the folk music of Hungary and and really advanced I think still to this day0:14:52modern composition in very interesting ways and John Coltrane and and by the way was also a pianist and performer0:15:01right he in and teacher of piano John Coltrane is you know possibly one of the best improvisers humanity has ever0:15:11produced a fantastic saxophone player played fundamental you know mostly tenor and hear soprano sax he was also a0:15:21composer and and wrote very interesting tunes and advanced jazz harmony in those in those tunes but he's going to play the role of a performer today so we have Bartok the0:15:31composer and culturing the performer so when we talk about composition and in particular music composition you know what are we talking about now we're into the arts right and it's quite0:15:41interesting right because the arts usually are not about solving real world problems at all but if you look at any of the art forms where there's somebody0:15:51like a composer it could be a choreographer or anything like that the first thing these people do they have they have blank slate they have like blank staff of music0:16:00blank page empty stage the first thing they do is they make problems for themselves they set up a set of constraints under which they're going to0:16:09form an artistic work it's quite interesting that they create their own problems to solve from there and this0:16:20happens again for all these different forms and for they're from there they now have a design problem they're like oh great I just made myself a design problem now I can go and design now I can go try to solve this problem solve0:16:31those constraints I've set up for myself and make a plan which will be realized by performers so when we look at music0:16:42composition it's it's quite interesting that there's a variety of specificity and scale to the designs that we that are created often you'll see fully0:16:51orchestrated pieces so classical music is is typical typically this way where every every note that's going to be played by every instrument is specified0:17:00in a score and what's what's particularly interesting I think about music composition here is that this specificity is much more common the0:17:09larger the scale so they're taking on bigger problems and the bigger the problem they take on the more they specify it's kind of an unusual I think0:17:18the other kinds of compositions you see are like songs and we'll take the lyric side of the picture and just say melody and changes and here you're giving the0:17:28suggestion of what the piece is going to be but the specifics are left out this kind of design just says here's the melody here's the harm and everything else is gonna be left to0:17:37the performers they have a lot more latitude and with that a lot more responsibility I think one of the things that's interesting is that software strategies straddles these two worlds we0:17:48tend we'll work on the smallest part of the system to have the most specificity our designs we're working at the largest scale in our systems we have the least specificity but I think that one of the0:17:58things that that programmers are afraid of it are these large scale designs that are going to completely tell them what to do they're like oh you know don't0:18:08repress me man I don't want to see this and certainly that was well-founded not in the 80s people thought you know people would you know make pictures and push buttons and get COBOL to come out0:18:18and that would be what software was going to become so programmers were legitimately skeptical if not afraid of that so here we have two examples of0:18:28what I'm talking about the piece on the left is the segment of bartók's concerto for orchestra and you can't see the details but you know this there's all the woodwind parts and then the0:18:37percussion and the string parts all completely specified the phrasing you know the the tempo everything is0:18:48completely notated although not totally completely because you see this has been written on and it's been written on by the one person who has some latitude in this kind of an approach which is the0:18:58conductor and of course the conductor's struggled with some things that might not have been said like they have to translate the Italian tempo markings into you know actual beats per minute0:19:08and there's lots of contention about how to do that on the right we have a completely different kind of composition this is my favorite things it's Richard Rogers you know from Sound0:19:17of Music that's a very pretty song but you can see here you know just roughly that very little is specified there's a melody and then some chord changes and0:19:27John Coltrane quite famously did a rendition of this piece which is absolutely gorgeous and if you've never heard it you need to hear it before you0:19:36die yes it's great really really great but you see the difference in how much is specified so we've talked a little bit about constraints already in design0:19:46and again it's the same kind of thing happens here our music composition most compositions are about something and this is true also most dances are about something and0:19:55most plays are about something and most screenplays are about something or a few things so this problem that the composers and and an artists set aside0:20:08for themselves are you are normally pretty simple they try to focus on one or a few ideas and a music that would be melodic or Tamil or rhythmic kinds of0:20:18ideas and they're gonna take this fundamental idea and and and work it work it out resolve it try to see what it's about try to explore what it means0:20:28and when you get to larger scales like when Bartok does these larger compositions you end up with this set of constraints at each level right larger0:20:39works just have more structural components but they're very stratified so he has all kinds of techniques for dealing with harmony in the small and0:20:48form in the large on the flip side now we have the performers space improvisation and the root there you know takes us to not foreseen or not0:20:58provided so it hasn't all been written down in advance right it's not completely specified there's this melody and these changes and they provide0:21:08constraints for a performer who has to provide the variations on the fly the thing I think that's really important here is people think that some people0:21:18think that improvisation is you know some genius just spontaneously emoting it is not that it is not the way it works the the best improvisers practice0:21:29the most and John Coltrane is a great example of somebody who practiced in an amazing amount of time and studied quite extensively and what you end up hearing0:21:40when you hear an improvisation is an application of a lot of knowledge and a tremendous amount of vocabulary it's0:21:49almost like dynamic composition that that's happening in these in these improvisation 's and you can tell like0:21:58now there are a lot of releases that include outtakes so you go and you listen to this thing you and you listen to it the first track and it's like I've been listening to this for years it was amazing he came up with this and0:22:08he listened to the outtakes and there's like five takes with very very similar solos I mean Coltrane was working out this composition he was going to perform0:22:17dynamically but the resources behind it were things that he had prepared he didn't just like make it up as he went along0:22:26so it's this delicate balance of being able to be dynamic but having compositional sensibilities to apply on-the-fly that's what developers need so there's0:22:35another great term in this space called harmony and you know the dictionary definition says a corridor congruity and that makes sense0:22:45you know the degree to which things fit together specifically in music we get this notion of simultaneity so melody is0:22:54about sequentiality right and harmony is about simultaneity when things are sounding at the same time whether it's an instrument that can play a chord or an ensemble where all the instruments0:23:03playing at the same time yields yields a combination of tones there's another notion of harmony which is sort of the rulebook right there's a mathematics to0:23:12harmony which is sort of the something you could study it's an order of science I don't know you can have that argument later but there there is this sort of system there are systems about how0:23:23things fit together that you could study and I will argue here that harmonic sensibility is a critical design skill this is really0:23:32what you need to acquire if you want to make systems that are going to work so I like bar talking Coltrane has examples because both of them were masters of0:23:41harmony and by this I don't mean they were masters of the rulebook they were masters of the the the notion of harmony in fact both of them were students of0:23:51harmonious nests if you look at their look at their careers and what they did you know Bartok you know came from a0:24:00Romantic tradition and was aware of a lot of intellectual exercises going on in composition to try to modernize it break free of the old rules of diatonic0:24:09harmony and go to new rules you know just we're going to get rid of the those constraints and just have differ constraints and and he sort of never really went there with serialism he0:24:19stuck with with with essentially eternal systems but he went beyond the rules he tried to figure out what exactly worked0:24:28and didn't work and he explored for instance this Hungarian folk music which was which was tonal but didn't follow the classical rulebook because it was it0:24:37was folk wasn't you know academic and similarly you see Coltrane doing the exact same thing right on giant steps there's this famous reharmonization0:24:48system that he developed that really broke the rules of the day but were fundamentally about retaining what0:24:57worked about harmony and finding new ways to figure out things that work together so they both essentially developed new systems while while0:25:07retaining a focus on what fits together and I think that artistically they're just tremendous artists because there0:25:16was a lot of intellectual effort I mean if you look at the insides of some of what Coltrane is doing it just seems like the most emotional thing but it has a tremendous amount of intellectual0:25:25stuff going on there it's the same thing with Bartok you can listen some piece it will make you cry and then you go and you look at the score and it's full of like Fibonacci ratios it's like oh my0:25:35god that's really cool all right so what does this have to do with anything that we do right closure programming0:25:44languages tools or anything like that right its closure like a song is it like a symphony is it like these things or0:25:55languages like these things no what are they like we're gonna play the like game they're like instruments so let's look0:26:07at instruments this is what I'm particularly fond of I happen to have one that's just like this instruments are particularly interesting right0:26:17instruments start with something called an excitation right again we see the same notion most instruments are about one thing right whether you're going to0:26:27pluck string cause a reed to vibrate whack on a string whack on the you know a drum skin there's this fundamental excitation0:26:38and the rest of the instrument is completely about it it's about shaping it and conveying it then any particular instruments going to0:26:47provide some sort of human control interface right these are all interfaces they're really good interfaces you should study them they're very0:26:56interesting well it's keys on the piano right or on the saxophone frets on the guitar and pedals on a piano and all instruments have these things most of0:27:05them at least fundamentally address pitch and also volume but they can do they can do other things as well so this is where the pro the performer can0:27:16exercise control and then the rest of the instrument is oriented towards projecting this excitation so there's an0:27:25excitation that you get to shape and the the rest of the instrument is about directing that energy at a particular outcome usually it's about directing the0:27:34sound at the audience but when you try to extract the ideas out of this it's there's a fundamental idea and the rest of the instruments about directing it at0:27:44a good outcome there's another interesting aspect of instruments which is that while there is this initial excitation once you get an instrument in0:27:53play once you get this body that has some air inside of it it has its own modes of vibration so it will tend to vibrate at certain frequencies and0:28:04that's known as the resonance of an instrument and this is sort of the harmony of the physics of instruments right a good instrument designer right is going to try to make an instrument0:28:14whose whose fundamental resonances are compatible with the the excitation so again the harmony sort of comes into0:28:23play when we look at instruments we see that they're incredibly limited right the piano can't even play any in-between notes it's like it only plays exactly0:28:33the notes - which is the keys correspond and a saxophone can't play more than one note at a time except you know with some techniques and even then it can't play0:28:42arbitrary pairs of notes at a time right most instruments are minimal yet in some ways sufficient right so an instrument may have a limited range but if it's an0:28:51instrument designed for western music within that range it probably has all the notes right but there may be other things that are limits like certain transitions between notes or registers0:29:02might be awkward or impossible but there are instruments that don't comply like a blues harmonica doesn't have all the notes it has only one key at a time and0:29:13it's a little bit like our dsls right the thing is that players overcome these things how many people here play piano right so how do you overcome the fact0:29:22that there are no in-between notes what's it what do you do yeah like trills right trills and grace notes and things like that are all techniques piano players and piano composers use to0:29:32deal with the fact that there's no in between notes Coltrane quite famously because he had such dexterity and and speed on the saxophone had this0:29:42technique that was coined the sheets of sand where he could play a scale so fast that he could he could give you the0:29:51effect as if there was harmony even though he can only really play one note at a time so they can fix this stuff by performance you know because otherwise0:30:02what are we gonna do we're gonna submit a patch for pianos let's fix that nasty one at a time you know no one between those problems so why don't why don't we do this why haven't know all these0:30:11things been fixed why isn't the saxophone been fixed you know to play more than one note and and the reason is that no one wants to play a choose a phone no one wants to be to choose a0:30:21phone player all right some people do want to be useful let me say this then no one wants to0:30:30compose for choose a phone right let's imagine you had an orchestra where everyone was sitting in front of one of0:30:39these this is a modular synthesizer if you don't know right it's an electronic instrument where you you you patch together modules that are either just you know0:30:48tone generators or filters or things like that and eventually you can control it with the keyboard or some other sort of source but but what happens if you try0:30:57to compose where the base units are things like this well you have this problem right your fundamental target is complex right each sub component you're0:31:06trying to reuse is really complex you also have this nested design problem right as a composer if you know composers study the instruments they0:31:15study flute and they learn what the range is and what transitions are good and what transitions are bad what's hard and what's easy and they've learned that for the entire orchestra and they can0:31:24look at a flute or think about fluid and know I know what fluid is about and what it can do and therefore I can use that knowledge to build something that uses flute and violin and whatever and make0:31:33it make something that works together but if each piece is complex you end up with this nested design problem you can't look at one of these things and know what it's going to sound like when0:31:42you press the key and it may do something completely different tomorrow and like if you were wondering what would happen if you pulled out one of the wires you know that's a good0:31:51question it's totally fair so it's hard to it's hard to build things out of things like this other interesting things about instruments instruments are0:32:01made for people who can play them isn't that outrageous isn't that scandalous they're made for0:32:11people who can actually play them and that's a problem right because beginners can't play they're not yet players they0:32:20don't know how to do it again I think you know there should be outraged on the internet we should submit patches we should fix like the cello right should cellos auto-tune0:32:32right or maybe they should have red and green lights right it's green when you're in tune it's red when you're not in tune or maybe they shouldn't make any0:32:43sound at all till you get it right wait is that is that how it works is that what we want right no that's not how it0:32:52works look at these kids they're being subjected to cellos there's no this there's nothing helping them here0:33:01although apparently apparently their shoes have been taken away until they get it right but otherwise those you know they're smaller but those are real0:33:10cellos they're hard to play they're awkward they sound terrible they're gonna be out of tune they're just it's gonna be tough for a while for these kids but if they had any of0:33:19those kinds of aids they would never actually learn to play cello they'd never learned to hear themselves and to tune themselves and to listen and and0:33:28and playing a cello is about being able to hear more than anything else and that's true of most instruments so we need players here's where I would rant0:33:39I'm not gonna rant but just as a simple well I'm gonna rant a little bit you know as a simple example look here's a guitar player a harp player double bass0:33:49player all holding up their their blisters imagine if you downloaded a library off the internet and it gave you blisters right the horror right and yet0:34:00every musician right has overcome a barrier to entry similar to this right the thing we have to remember is that0:34:09humans are incredibly capable right in particular we are really really good learners we've just evolved to learn that's what we do the other thing we're0:34:19really good at is teaching right and the thing is that neither of these things are effort free they take time they take effort we should not sell humanity short0:34:30right by trying to solve you know the problem of beginners in our stuff we need to make things for people to use and we need to teach people and trust0:34:40people to be able to learn how to do that because fundamentally we are novices and and the other thing is we're only absolute beginners for a very short0:34:49period of time on the flip side we're beginners forever right we never totally get it we're gonna be learning on an ongoing basis this is just the human condition0:34:59this is not something to be fixed it's not something to submit a patch for right this is how it works right so effort is not a bad thing right these0:35:09are two guys who are experts and yet you know they're still trying right this is not this is not autocomplete right so0:35:20just as we shouldn't target beginners in our designs nor should we try to eliminate all effort it's an anti-pattern it's not going to yield a0:35:29good instrument okay it's okay for there to be effort another interesting thing about instruments is that they're0:35:38usually for one user right this is a tool made for two people to use at the same time that's extremely rare for tools or instruments to be for what more0:35:48than one person to play at the same time and yet we have these complicated T MUX whatever things that people parent now I0:35:57don't really want to diss pairing because while I personally don't understand it I can see analogies for instance rather than to two musicians at the same instrument to two musicians you0:36:06know playing in the same room and that obviously has good effects but you have to wonder you know if this is just a wave you know to keep somebody from0:36:15typing all the time right if we haven't otherwise built in time for design is pairing one way that we're trying to buy it it's a fair question right so what0:36:25should the ratio be between planning and performance right what's what's the ratio of time spent practicing and studying versus performing and recording0:36:34for musicians how many people here are musicians or play instrument I it's usually pretty high amongst people in the software world do you spend more0:36:44time practicing or on stage definitely practicing right if you're hired to perform in an orchestra I mean they don't even book you for even half of your time to be actually sit in the0:36:54orchestra they know you're gonna have to spend time shedding I don't know why as software developers we think we can just show up when we how many people here dedicate 50 percent or0:37:04more of their time to not programming yeah not too many but that's not everybody else practices I mean0:37:14obviously we went to school and study wherever we studied but yeah musicians are not like well I went to school so now I don't need to practice anymore I'm just gonna like go and show up at the orchestra and they're gonna0:37:23let me play because I showed up this is a great quote you know you you have to you have to prepare to be creative and you have to keep preparing to be0:37:32creative or else it's not gonna happen you can't just show up and play but I know you know there's gonna be complaints right we're we're in a whole0:37:42different space right Coltrane couldn't build a website in a day you know I don't know why this has become so important to us is really like a stupid0:37:52thing to be important especially to an entire industry so I'm not really gonna spend a lot of time on that but it's it is a fair question to say you know in what ways is this different right in0:38:01software we seem to have saw I mean like this all these ones and zeros and like so many ways to put them together right it's not as simple as making an instrument it's not wood and metal we0:38:10just have this sort of infinite nature to the to our to our resources you know how does that impact you know the difference in how design works and well0:38:20I just have to show this so obviously as soon as we had the ability to make sound by using electricity right so having the0:38:29electricity drive a speaker and have the speaker vibrate and have sound comes through the room I mean the initial applications were we recorded sound with microphones somehow captured it or transferred it live right and then the0:38:39wire hit a speaker and then it came out on the other side but as soon as that capability was there people started imagining well could we just take out that first part and directly generate0:38:48some electricity that would sound good when it comes out the speaker and from that was born electronic music and a music synthesis and then the theremin is0:38:57one of the earliest synthesizers and it's from I think the 20s and it's quite interesting it has extremely simplistic sound generation capabilities there's a0:39:07little oscillator in there and something that changes the the waveform very slightly and the way you play a theremin is that there are those two things those two pieces of metal are antennae0:39:16and and the one the vertical one is controls pitch and the closer you get to the intent of the higher the pitch and the further away you get the lower the0:39:25pitch and the loop antenna controls the volume and the closer you get to the antenna of the lower the volume and the further away the higher the volume and you play theremin by moving your hand0:39:36through the air you do not touch it so it's an extremely simple instrument but over the years more and more sophisticated electronic music musical0:39:45instruments came about those modular synthesizers I showed before are built out of modules and again now we're starting to see the same kinds of things we see in software now modularity right0:39:55there's modules each module is about something right so there's an oscillator there and the filter and and they have interfaces they connect together and0:40:05it's quite stunning the this is pre digital so the way these things communicate is through control voltage completely analog just dynamic voltage0:40:16variations are what connect them together in your patch voltage from connector to connector and you build systems but of course we also start to start to see the levels right if you0:40:26look at the back of one of these modules there's another piece of design there right there's the circuit and these are analog circuits that um that determine0:40:36what the module does the other thing that's interesting it's a little bit hard to see from this diagram is that each of these knobs has a corresponding jack in other words there's a human0:40:46interface and and a machine interface to the same things and the machine interfaces were were there all the time in fact they were first and then the0:40:56human interface has come we have to remember that as design thing right what's wrong with sequel what's wrong with sequel is there's no machine0:41:05interface to sequel they only designed a human interface to sequel and we've suffered ever since right because we have to send these strings around so you0:41:14won't always have a machine you can always build a human race on top of a machine interface but and the other is often disgusting all right so now we see the things that0:41:25we start to see in in in programming right there's this whole design stack this guy Yves Busan he he's amazing I think he's like a biochemist or something and in his spare time he's a0:41:35C++ programmer and it's his other spare time he actually makes these modules so he designs and builds analog modules and then he of course he builds synthesizers0:41:44and then he builds sounds with those synthesizers by patching things together and then hopefully sometimes he gets to play that little keyboard there and actually make music but there's this0:41:54entire stack of design the design the modules the assembly of the modules into a into a rack things oozes your choices then the patching which is designing the sound that the modules going to make and0:42:04finally maybe some music later so it's interesting to see when that stuff gets captured so this guy happened to help this company make this synthesizer which0:42:14incorporates several of those kinds of modules inside but doesn't have any wires on the surface so a bunch of decisions have been made about how the modules will connect together the0:42:23oscillator will fill the filter and the envelope generator will control this those things were all already decided this same guy made those decisions and they were captured in this device that's0:42:33now a lot closer to the to the programmer in fact you don't see the machine interface at all there's only the human interface and it's more like an instrument that you could just turn0:42:42on and play of course you still have to do some sound design on the on the top the important thing to know is that as you start looking at things with these design stacks you really have to pay0:42:53attention to where you are in the stack right when when this guy is is soldering together one of these modules he's not making music that day right he's not and0:43:05and and this is the problem we have right our problem is you know this is what somebody feels like right this is what somebody feels like when you say you should use emacs they feel like they0:43:16want to make music and you handed them a soldering iron right that's what that's what you just did and why does this happen to us it happens to us because we use the same stuff all the way down for0:43:27us it's code at every layer it's always software right in this in this thing that is like this you do something different when you perform you do something different when you're0:43:36sound design you do something different when you assemble modules and something different still when you you know wire things together behind the scene but for0:43:45for software developers we all have soldering irons we all can like jump to the absolute bottom and and mess around it doesn't actually mean that we're skilled enough to do that0:43:54but it's there and it uses the same stuff and we think well you know I got some solder and some spare time I should you know design a filter and and I think0:44:05it yields to a tremendous amount of distraction and also tremendous amount of expansion you know we change things and enhance things because we can because we're all luthiers you know I'm0:44:15glad I mean it would be cool to know how to build a guitar but you know if I knew how to do that I'd play less guitar because I'd be fiddling around with with with wood and you need to decide what0:44:26you what you want to do so there's a sense of which having a lot of choices which we seem to always seek when I gotta have choices you know don't repress me man it's the0:44:35opposite of enabling us to accomplish things and in fact you'll see time and time again people documenting the fact that constraint is actually a driver0:44:44right when you have fewer choices you make them and you get on with it when you have a ton of choices you sit around and like wonder about things all day so constraint is a driver of creativity0:44:53that's not a new thing I mean people have been saying this forever but this is what they mean this is how this is where it comes from so if I was going to advocate anything it would be you know0:45:03as a as a as a as a community and an industry we spend an unbelievable amount of time catering to ourselves it's just0:45:12ridiculous it really is because we can talking about it you know enhancing things adding stuff and you know you0:45:22know where this goes right it just it creates these monsters right and you know the chances are good that every one0:45:31of these modules is a good idea let's just say it is every one of these things is a good idea but if you if you take every one and you just keep accreting them and adding them together you end up0:45:41with something that you can't play I mean I don't care if you could configure this thing with spring right nobody wants to play this and in fact no0:45:51one can play this this one this one actually plays itself it's in a museum somewhere and it spontaneously makes0:46:00music no one sits it in and tries to play it it's just it plays its off and I mean I'm sure it makes interesting sounds but I don't think it can play my0:46:10favorite things because it doesn't it has no memory it just makes up stuff so I think the thing we need to keep in mind is there are people who can make0:46:19music by waving their hands through the air right they don't need Emacs or anything else they don't need it a million options or deep class0:46:29hierarchies or anything else don't start playing this right now the whole room will be like theremin whatever but it's really cool to listen to this so getting0:46:41back to the topic what is design I would say on one level design is about imagining things right that that doesn't mean like imagining that you have every0:46:50possible option you want to embrace the constraints you because you're gonna use them they're gonna help you the trick is to not let them get you down right the0:46:59key thing the design is saying I have the set of problems I have the set of constraints and I know I can solve that right if you combine the optimism with0:47:10the constraints you're gonna get designs and so you want to imagine a lot you want to take these two things and run with them and try not to like get the first answer but try to get a lot of0:47:19answers and then have them to choose from however the net flipside of design is design is about making decisions right after you've got this spectrum of0:47:30things you think are interesting answers you know maybe you thought of a hundred times as many things as you as you as you need to you need to admit very0:47:39little you want to mostly say no design is about making decisions right in fact the value of a design is in conveying0:47:48those decisions to the next person right when that guy worked on that synthesizer with the company and made those decisions it's to help out the next person the musician that doesn't want to0:47:58solder right they need to be able to trust the decisions the prior person made so they can get on with their level right if you leave every option open and you0:48:09make everything configurable you're not designing you're failing to design you're failing to make decisions and convey them to the next person you're not actually helping them right you're giving them a chooser phone right on the0:48:20Performing side we already saw a little a little bit of this the Performing is preparing its it's practicing right which which we do right we do a lot of0:48:30coding so we get a lot of practice in it but it's also studying and I think the biggest thing in trying to connect you know performing to to developing or0:48:40coding is is this is this nature right Coltrane was great because he had fantastic design sensibilities that he could he could apply right it wasn't all0:48:52about his fingers I mean he had incredible dexterity he did great he had incredible technique and incredible tone that he got from playing a lot but what0:49:02he played was a result of what he studied and the analysis that he did and the thinking that he had done about how music works he if he just had the finger0:49:13speed and everything else he wouldn't have been the great player that he was so to wrap things up take things apart0:49:22take things apart with a knot with a hammer and smashing them but with an eye towards how you're going to put them back together that's what design is about you want to design like Bartok and0:49:33by that I don't mean you know specifying every note for every player that I'm not advocating that I don't I don't believe that's the way software design is or ever should work but to think about the0:49:44way things fit together at every level of the thing you're working on when you're working on large things as he did you have to bring the design sensibilities to the small things and to0:49:54the medium-sized things and to the large things and often apply different techniques and different ways of thinking for each level you know ways that are appropriate you want a code0:50:04like Coltrane right bring those bring that harmonic sensibility bring that design sensibility to to the coding that you do0:50:14into the code that you right I think most fundamentally we want to seek out and build languages and libraries that are like instruments in0:50:24all the ways that I just talked about in particular in the simplicity aspect right all those things I showed you were0:50:33different ways to say instruments are simple in that deep way that's what we want and finally to pursue harmony to0:50:43actually think about how things fit together it's real easy to code write out a problem and then get an answer but if you haven't thought about how things fit together that answer is not going to be0:50:53easy to maintain or easy to change or easy to reuse so you have to think about harmony as you go and then go do it I0:51:04hope you enjoy the rest of the conference Thanks [Applause]0:00:00Design, Composition, and Performance - Rich Hickey
0:00:00 hi thanks for coming I'm very excited about this conference it's always great0:00:09and I'd like to thank the organizers for inviting me today to talk about design composition and performance so we start0:00:18with a legal disclaimer prepared by lawyers we're gonna have some fun with analogies today and the cool thing about analogies0:00:27is there's there's much fun when they're wrong as they are when they're when they're right so design is something we talk about a lot in in software0:00:37development but I think is something that's somewhat beleaguered these days not because people don't do it but0:00:46because I think people are in a hurry and they're trying to get things done and I often get developers asking I'd like to be able to work at the next level and talk about you know the design0:00:56of things before I just code up solutions so what does it mean to design something and as you all know all I do to prepare these talks is go to dictionary.com0:01:05and look stuff up so I looked up design and one of the definitions is this which is really great definitions are always great it says to prepare the plans for a0:01:14work to be executed especially to play on the form and structure of that work and and I think that's that's super important the notion of executed it means that somebody's0:01:24gonna do this we also have a different notion of executing things and software which is that this is going to run which is also interesting there's another definition which is to decide what the0:01:34look of something is and there's nothing wrong with that kind of design I just want to say this talk is not about that at all never when I say design do I have this meaning in mind and of course we go0:01:46to the roots and see that the root is in to mark stuff out and and and that matters so it's really sort of two0:01:55things here we want to make a plan and we want to write something down and of course here's everybody's like oh you know we did this right and we did this0:02:04in the 80s and we had all the stuff and and it was terrible we had these phonebook sized specifications and nothing got done and it was all awful so we already write down code is that0:02:14enough do we still need designs if we this and and the answer is yeah because0:02:23that's not a plan that's just what you did can we generate Docs for implementation same thing right that's not representing a plan to do something0:02:32it's like I already did it and so somebody asked me for a documentation or a design and so I pulled a lever and this came out and of course there's this0:02:42complaint right I don't want to do this these things are big they're top down they're waterfall model etc etc we did0:02:51that already and it's true that happened and they were plans but they weren't good plans and so when I talk about today's is a little bit about what do we want to have0:03:00a design and and when we encounter in the world what do we see so I think that a simple idea behind design is to look at it in terms of taking things apart0:03:10this is the opposite notion we typically have typically people think of about design and they say design is you know making this big involve plan that's gonna solve every issue that the system0:03:20is supposed to address but I think I think that's not what you do when you're trying to get a design that's gonna survive and live over time I think instead what you want to do is break0:03:29things apart in such a way that they can be put back together and that's fundamentally what design is about taking things apart so you can put them back together because obviously taking0:03:38things apart and walking away is not really gonna help the other thing you find in good designs is that they're always about one or very few things0:03:48designs that survive designs that really Foster reuse are about a single thing generally and then you put them together0:03:57so the first thing you do is to take everything apart then you compose them and then you're solving your problem but the first thing is to taking apart and there's nothing about this that's in conflict with iterative methods for0:04:07developing software right this can be an iterative process and and all that happens then is that you get feedback during development to your design so0:04:19what kinds of things would we take apart there's a whole bunch of things that we could take apart and in fact you're constantly finding more things you can take apart the requirements for a system0:04:28you can take apart the order in which things happen who's gonna talk to whom information parts of your system from mechanism0:04:37parts and and you can actually take apart different solutions to assess their merits so let's just look at each of these in turn taking apart requirements this is something we do not0:04:46do often enough somebody says I want a system it does X I need Y you know it's got a doozy we get these feature feature lists and the first thing that we should0:04:55do when we're handed that is to break them break them apart and try to find in the set of requirements in the set of features or desired things or needs the0:05:05actual problems and it's only by doing that that you can start to move forward and say ok I've broken your need into problems that you have and then we can try to make things that solve those0:05:14problems the other thing you're going to do initially with requirements is divide them up the simple way to take them apart is to say these are things I know how to do and these are things I don't know how to do so your knowns from your0:05:24unknown you're going to take apart requirements that are domain side the system must do this to satisfy this business thing from solution side things like we need to run on AWS or something0:05:33like that often you get requirements especially for systems that already exist that are about it's not working0:05:42and everybody's hard that you know how do you fix it's not working and the first thing you have to do when you're trying to fix it's not you know what's not working is to separate out right0:05:51what's the cause of this problem from what's the symptom of this problem all right somebody says my screen is black you're not ready I would say okay I know how to fix black screens and0:06:01start typing because it's not a generic solution to the black screen problem yet and then there are a whole bunch of requirements that are unstated and and0:06:10these need to always be enumerated if if not taking it apart but they need to be in mind at all times unstated requirements are the things that that0:06:19everybody wants the system to avoid like I'd like a system that doesn't keep crashing or use up all the memory or cost too much to run or use too much energy or require a lot of manual effort0:06:29or the users will hate and so the unstated requirements are often a set of things that your software's supposed to not do not cause attributes it's not0:06:39supposed to have so we want those on the table other things just completely different dimension things we can take apart when we do design which is time right you can0:06:48take apart the order of things how would things are going to flow from one to the other you can break system's apart so there's less direct calling you can use queues to do that you can support0:06:57redundant activity with idempotent [Music] approaches commutation is a very important concept that's going to be0:07:06more and more prevalent as we tried to build systems that are highly distributed which says I can make a system order independent by supporting0:07:15operations that are all commutative then I don't care how things come in so it's it's a technique for breaking apart I used to have this order dependency and0:07:24now I don't so now I have to separate things I can talk about independently and transactions are the opposite when you say I do need to know these things going to happen together we can take apart0:07:35place and participants and there's a certain sense in which design is always about this but you know there's this old adage right you just add indirection0:07:45but here we're talking about possibly the whole process of building something right having a design is the thing that lets two teams work independently or people work in two independent languages0:07:55taking apart things is what facilitates the participants the authors as well as the participants for instance the system's by breaking things apart you're0:08:06able to say well run this on this machine here or run this in this tier or we'll put that on the web this one is kind of interesting because I I don't0:08:15see it talked about often enough which is to separate information versus mechanism so there's always information that our system manipulates for instance your system may have the notion of the0:08:26set of users who are logged in that's an idea that enumerated set is a is a piece of information and then you have like0:08:35you use the set class a collection class from your your you know your favorite framework library to put the logged in users and one of the problems I think we0:08:45haven't in software development is we use the same stuff for both of these things but these are two very very different things one is sort of this0:08:54device into which you stick stuff and you can back later it's kind of a little bit of a place and the other is a piece of information which you should really not treat that way at all so pulling these0:09:03things apart telling talking about your system and clearly differentiating the stuff that's information from the stuff that's sort of the mechanics of your0:09:12program is it's quite critical and then finally once we've we think we have an answer we have a potential solution we haven't implemented it yet but we're0:09:21looking at it or maybe we have implemented some of it you want to take those apart to see not just the benefits right those are pretty evident usually but also the trade-offs what part of0:09:31this is not going to work how much is it going to cost to run and does it eventually fit the problem because a lot of times what can happen is you can adopt a solution that is larger than0:09:42your problem and then what do you have you have two problems right yeah you have your problem and now you have this thing that was too big too big for it so0:09:53it's not just about getting answers it's about breaking things apart in a coherent way so I'm a big fan of design I think that we need to do a lot more of it and we need to talk about it more and0:10:02we need to spend more time on it but but I think it's pretty easy to rationalize why why we need it the first is so that we can understand the system right a design is hopefully smaller than the0:10:13code that that implements it and so it's easier to get our head around what it's about the other thing as I was talking about is design is fundamental to coordination0:10:23right if you don't have some plan you can't just send two people off to write you write 1/2 a system you write another half of the system that's the end of the conversation what's gonna happen well0:10:33they're gonna wonder which half of the system they're supposed to write right because there's no plan so there's no way to have coordination and to have multiple groups working on something without a design design also facilitates0:10:45extension and extensibility people are always like oh I want to make something extensible but the easiest way to make something extensible is this breaking it apart thing because when you're broken0:10:54it apart you end up with two separate things you end up with pieces that are meant to connect to other pieces which means there will be connecting points on0:11:04those pieces and therefore when you want to do something new you can make a new extension and it can Lev courage that connecting point because it had to be in place because the things were separate the flipside of that is0:11:14this reuse aspect which is when you broken stuff up into separate pieces that have nice interconnecting points you can pull them out of one context and0:11:23put them in an other context and that's how you get reuse these are not like magical things and they're not attributes of api's necessarily they0:11:32mostly fall out of this decomposition finally testing is greatly facilitated by design ideal testing takes some0:11:41design constraints some specification and turns it into tests as opposed to sort of embodying design inside tests that's inside-out but again that's0:11:51something we have to work more at systems like quick check are interesting because you're basically starting with propositions about your system which reflect a design and saying you write0:12:00the tests computer and finally I think the thing that's often most readily pulled out as an argument against design is I don't have time this is gonna slow0:12:10us down and in fact I think it's it's the opposite in particular there you know those all these adages about you when is it when is it easiest to and least expensive to fix a bug right not out in0:12:21the field right if you've already shipped it it's the most expensive so people like oh we should fix it in you know QA or we should fix it in our code and do you know test-driven design it0:12:31but the thing is you can keep moving back it's most it's easiest to fix your problems in OmniGraffle right you just say ooh that is not gonna work and you0:12:41like move some boxes around and it's fixed it's much cheaper than fixing software but even after you've shipped I think that there's a lot more efficiency in systems that have been designed0:12:51because because you're going to be able to go back to something and usually the answer to your problem in the field is0:13:00I've just insufficiently broken something down and so the solution I'm going to need is just breaking it down more and that's less expensive then I created this giant ball of everything0:13:10and I need to untangle it so I do think in the end it's more efficient so that's design the talk is about design0:13:19composition and performance and so I got to take composition and performance together and one of the beautiful things about dictionaries is0:13:28there's more than one meaning for each word and so there's more than one meaning for composition which of course we think about composing systems out of pieces like I was just describing and we0:13:37think about performance and systems usually as you know how fast did they run but when I think about composition I often think about Bartok and when I0:13:47think about performance I often think about culturing and so these are two musicians now Bartok is a hungarian composer and but he was also a performer0:13:56he was a pianist and taught piano and Coltrane is a famous saxophonist and great performer but was also a composer0:14:06so I'm not trying to pigeonhole these guys but we're gonna use Bartok to stand in for the composer and culturing the standard for the performer and talk0:14:16about two different notions of composition performance and maybe how they might inform software so composition music composition and other kinds of art creation is about0:14:31addressing constraints it's about addressing problems but not real world problems right art doesn't solve real world problems in general and if it does it's more than art it's something that's0:14:40something else so it's quite interesting that the first thing that composers tend to do when they have a blank page they can do whatever they want is make up a0:14:51bunch of problems for themselves they actually create a bunch of self-imposed constraints and that's true of all the other art art forms right in general0:15:01you're gonna see this and composition is designed for performance so we saw that definition of design on the first slide and it's said to be executed and that's0:15:11what composition is you're writing something you're anticipating someone's going to perform it or do it later it's the same thing screenwriters presumably going to act it out choreographers0:15:20presume someone's going to dance it so it's design you're sobbing constrained problems by creating your own constraints and you're designing this something to be executed so it's very0:15:29much a design pile and it's an organizational challenge right you're trying to address these constraints that you've set up for yourself and that's what that's what0:15:38composition is it's quite interesting that when you look at music composition you end up immediately seeing a tremendous variety in in the specificity of compositions and the scale of them0:15:48and and it's it's telling that software source strata straddles these two things the first is you see fully orchestrated0:15:58music right it's fully arranged all the notes are written out for every part and this is typical at larger scale so bigger compositions orchestral compositions and operas and things like0:16:08that tend to have full orchestration and then smaller compositions you might have only a melody written out and the chord0:16:17changes for it for say a song they'll call it a song but we're gonna not talk about words today and and in in those compositions you have a lot more latitude for performers right because0:16:28you're not saying you must play this note at this register on this instrument at this time this loud you just said you know this is the melody and have at it so there's more responsibility for0:16:37performers so there's this whole spectrum I think that when people push back against design they're afraid of this first one right because again back0:16:46in the 80s we had those stuff where people had plans that you would draw pictures and push buttons and it would write programs maybe people still have those plans but they're conducting them0:16:57in secret but I think programmers are like you know don't repress me man I don't want to do this this this big thing but I think that again you're0:17:06going to have the spectrum you're gonna need a lot more writing down especially if you're going to share amongst people and then in the small when you're talking about your own individual effort maybe you don't fully annotate the same0:17:17way so we can see two pictures of this here I don't expect anybody be able to read that but on the left is the concerto for Orchestra it's a Bartok piece and you know this departs all0:17:28right now for the strings and the percussion and the winds and it's all specified although it really isn't though I don't know if you can tell0:17:37there's red markings and some other things on here which were notes taken by who the conductor because the conductor says the or I don't know what to do here0:17:47you didn't tell me exactly exactly exactly what to do so I have to decide what the tempo is or how to balance these two sections against each other but in general that's pretty fully0:17:57specified on the right we have my favorite things as it would appear in like a jazz real book and yeah this is the tune from Rodgers and Hart sound of0:18:11music and and this is all you would get if you're a jazz musician right here are the changes and here's here's the melody and you you move from there so I talked a little bit about constraints and and0:18:22again it's quite interesting to see how this lines up so most compositions are about one or a few things same kind of thing you sort you're setting out you're0:18:32saying what does this piece gonna be you rarely say I'll use these notes for a while and then those notes for a while then those notes for a while and then you know call it done never you never do0:18:42that you see composers come up with little motifs and things that they're going to reuse or transform and sort of riff on as time goes by so there's all0:18:51these ideas that you're going to set up as as boxes in which you're going to work and you have variations of those things that you'll do resolution and0:19:00then the scale the composition really just determines how many of these things you have and maybe how many different levels there are right so a big Bartok0:19:09composition is going to have very very fine-grained constraints about melodic motifs in a very particular part of the the piece and then at higher levels of0:19:18structure deal with you know big form kinds of decisions but again they're self-imposed constraints when we move to0:19:27the performer side of the coin the improvisation side like culturing I think it's quite interesting again0:19:37it's an interesting word it means not foreseen or not provided and not provided means you didn't have you know the answer upfront before you went and did it you're like you weren't handed a0:19:46complete plan before you went and so in the case of a jazz performer you're gonna have melody and changes right and then you're going to go and provide variation0:19:55make something up but I think that people have a tremendous lack of understanding of what goes behind improvisation for instance a lot of0:20:05people think Coltrane is just this genius who was spontaneously emoting I think that improvisation in music is just making stuff up off the top of your head you know it's just amazing it's0:20:15like hacking right just I am so awesome I'm so bright I'm just gonna like make this up but it's been quite interesting to see as we've gone back through the0:20:26archives and have these new releases of old recordings where they put the the alternate takes in there because you'll see Coltrane I mean he had the solo it0:20:36sounds incredibly spontaneous but any listen to the other six versions and you realize that everything that went into the solo that you thought was this amazing one-off he had worked out and he0:20:47was trying them in different orders different justice positions different Cadence's and levels and maybe the order of it you know was was spontaneous but0:20:57there was a tremendous amount of preparation associated with that so so the sense in which improvisation is is is dynamic composition of prepared0:21:07materials a planned material and that to be a great improviser means to make those smaller plans or have those kinds0:21:18of prepared abilities or approaches or sensibilities that you can apply when the time comes in a live situation and0:21:28you have to have a lot of knowledge to do this and a lot of vocabulary to do it it's just not something that you you make up and Coltrane was a genius at0:21:37this preparing you know he he practiced more than anyone in order to seem as if he was making it up most fluently so0:21:48another thing that sort of crosses the lines in composition and performance and in music is this notion of harmony and again we get this nice word for it which0:21:57is a chord or a congruity right how do things line up and again this is lining up notion in the simultaneous imal0:22:07tennety associated with harmony right so we have melody is sequential and harmony is parallel right some music did all this before we had computers and0:22:17so so this is how does things work together at the same time if I played these three notes at the same time what will happen or for Coltrane if I play this note while these chord changes or0:22:26the set of notes what these chord changes is happening what will that be like bar chalk had to imagine you know when the strings are doing this and the and the winds are doing that what will0:22:35what will it sound like all together alright there's also sort of a mathematics of harmony right which is the science behind it or the the way you study the the rules if you will of0:22:46harmony and I'm gonna contend that harmonic sensibility is a super critical design skill this is the thing that you want to nurture in yourself and and it0:22:56may be a little bit hard to see how the mapping works from from music to software but it's fundamentally what a good designer has they know if they make0:23:05this choice in this context that's going to go together and those two things going to work well together and and they know that because of their experience0:23:14and the study that they've done working systems so I think both Bartok and Coltrane are interesting even though they're in completely different genres0:23:26of music and that they were both masters of harmony if nothing else you can say they're the two are similar because they totally mastered harmony and and in fact0:23:37what was interesting about both of them was that they were students of harmonious nassif you will that the thing I think that they were most interested in was what makes things work0:23:47together well Bartok studied obviously the classical tradition but his music was not compliant with those rules and it's0:23:57because he brought a whole bunch of influences in from studies he had done of folk music of Hungary and and what he studied in that music was the sonority0:24:08that was possible in these tunes that didn't follow the classical rules but they still worked and so he pulled out what worked about that and wrote pieces0:24:17that are cards have really recognized as being completely tonal but they are tunnel and there's a there satisfyingly consonant as tonal music is which is quite quite astounding0:24:28similarly Coltrane invented whole new ways of doing reharmonization / / / chord changes that had that same0:24:37sensibility about about harmony so I think that what was cool about both these guys is that they both sort of developed new systems that preserved what was essential about things being0:24:48harmonic or being consonant and and then the other thing that's quite interesting is that on both halves and whether you listen to a Coltrane improvisation or0:24:57the most beautiful engaging piece of Bartok what's behind this is a tremendous amount of intellectual effort and activity I mean you can listen to0:25:06this Bartok peace and be stunned by it just blown away by the emotional content then you go study the score and there's like all these Fibonacci numbers and ratios in it like oh my god this was the0:25:16constraints he set for himself before he wrote this thing that seemed or was so emotionally powerful so there's a lot - a lot to appreciate in both of them but0:25:26what does this have to do with anything that we do in particular what does it have to do with languages and libraries which is really what I want to talk about today languages and libraries is a0:25:38language like closure or any other language it doesn't matter this isn't really about closure is it like a song or languages like songs are they like small compositions well they're like big0:25:47compositions I don't think so I I think that I think that languages and tools to me if you're gonna map this analogy are0:25:57more like instruments so let's talk about instruments it has to be one of my favorites I have that one again0:26:06instruments are sort of their own design problem right instruments start with something called excitation right and0:26:15there's a sense in which most instruments are about one thing right you pluck a string you cause vibration on a reed by blowing on it you strike0:26:25strings with with the amounts of a piano or you hit drums and things like that and what's quite interesting is that0:26:34very few inch or about more than one kind of excitation most instruments about one kind of excitation it's quite rare to see the other then this is combined with0:26:44some sort of control or interface or gold technology on instruments to say and then there's an interface right so this excitation then this is interface for people to go and shape the0:26:54excitation and finally there's an aspect of an instrument which is sort of its fundamental goal in the world which is to take that excitation and direct it at a problem and the problem for most0:27:04instruments is how somebody going to hear this right how do we get the sound across the room so somebody can pick it up and so so instruments are about0:27:13directing the force or energy of the excitation out to out to the audience they're directed and an outcome so it's a little piece of design work associated with an instrument instruments also have0:27:22this other interesting aspect which is resonance right when you design an instrument especially something like a violin or a guitar or anything that has0:27:31a vibrating body to it the body itself is going to interact with the excitation so the excitation the string is going to vibrate or whatever and the body is going to go and say ooh that's I like0:27:40that I'm gonna amplify that and it will amplify some things more than other things so there's a design problem and there's a harmony problem to the physics0:27:49of an instrument to say well you know if I build an instrument whose body resonates at a frequency that's not a harmonic relationship to the vibe to the0:27:59to the strings themselves it's gonna sound awful and it's actually a physics problem to get that harmony right in the wood but instruments have a lot of other0:28:10characteristics and one of them that's quite striking is that instruments are limited they're very limited piano can't play any in-between notes0:28:20it can only play specific notes 12 root of 2 all the way across or maybe you stretch it a little bit but there's no in-between those saxophone can only play0:28:29one note at a time this is awful I mean and these things have been around for hundreds of years I mean they didn't0:28:38have github but I mean somebody should issue a pull request and like fix this right but there's a sense in which0:28:47they're minimal yet sufficient for instance most instruments don't have any missing notes like for whatever range they cover they have all the notes0:28:56that's right well at least we're talking about western instruments in the western scales but they'll tend to have all the notes but not all of them well right0:29:06blues harmonica doesn't have all the notes right there's a kind of musical DSL right it's like you don't need all the notes you're just a business person0:29:15I give you just the blue notes that's all you get and so you know it's just something to fix right there are all kinds of limits not just in the notes0:29:24that can play but the registers that can play and things like that why haven't these all been fixed why can't every instrument do everything and there's a sense in which players can overcome this0:29:33how many people here play piano right so what do you do to deal with the fact that piano can't play the in-between note what do you have you have grace0:29:43notes and trills and mordants and stuff right that give you all that sort of feel around the note thing right John Coltrane famously0:29:52became so adept at the saxophone and he had such physical prowess and and muscle memory and combined it with this0:30:02gargantuan knowledge of harmony that he could play these scales so fast that he can imply not only chords but entire tonality superimpose entire tonalities0:30:12over chord changes by just playing sheets of sound that's what they called it over over over music so it's not necessarily the case that the0:30:22shortcomings of these things need to be fixed in the instrument right there may be there may need to be room for the performer to to do it and there's0:30:31another good reason why we don't fix everything which is that no one wants to play it choose a phone right no one wants to play instrument that like does everything you push here it makes a piano sound and then it makes a drum0:30:40sound and then this happens and that happens so some people do want to play it use0:30:49this is a Keith Emerson sitting in front of a mogh modular synthesizer back in the day and that would that was just wow0:30:58you can make it do anything if you plugged in the wires the right right so so I'll take a step back and say maybe some people do want to play choose the phones but no one I bet0:31:07wants to compose for a choose a phone ensemble right so just imagine that you're sitting in front man Orchestra everybody in the orchestra had one of0:31:16these in front of them right and they put the wires in whatever and you're the conductor you went like this and like when you say go what is going to happen0:31:25right you have no idea you have no idea of what even could possibly happen if you're singing from an auction there's certain category of things that you think might might possibly happen but0:31:35you can kind of get your head around what that might be and so the problem here is that where you try to where you to try to build a bigger system out of0:31:44something with as much so let's say parameterization has these synthesizers you'd end up you're trying to target something that's complex and build something bigger still that's a recipe0:31:55for disaster and there's a sense in which this is just the wrong way to go about things because you've got this design problem that's actually multi-level and it's nested right what0:32:06happens when you say go well it's the sum of what happens for each person what happens for each person well it depends on where they put the wires and what happens you know what determines what happens when you put the wires well each0:32:15module has a different thing that it does it may be a filter or maybe a sound generator or something like that so each there's a level there's a set of levels at which there must be designed0:32:24I must design the modules I must design the sound the patch that cooks them together and then maybe I would try to take on a piece with all of this but unless there was a way to talk about you0:32:34know one of those arrangements and get your head around what it implied you could never build up higher so another0:32:43stunning thing about instruments which is just again it's astounding that the world has continued is that instruments0:32:53are made for people who can play them who can already play them I don't know I hasn't everybody heard of like explain it to me like I'm five or0:33:02whatever we're not supposed to do this anymore let's just make everything for beginners but instrument makers don't do that they don't make anything for beginners they make everything for experienced players instruments are made0:33:12for people who can play them 100% of the time but we have this problem right beginners aren't players yet this is0:33:22this is this is going to cause the world to stop right if you can't have a website with like three buttons on it and and everything that possibly could happen can happen we're done so we0:33:32should fix this right just works technologists we know how to do this so we saw it's a cello right should we make cellos that auto-tune like no matter where you put your finger it's just0:33:41gonna play something good play good note like you're good well just fix that should we have cellos with like red and green lights like if you're playing the0:33:50wrong note you know it's red and you slide around and that's green you're like great I'm good I'm playing the right song right or maybe we should have cellos that don't make any sound at all0:34:00like until you get it right there's nothing and then then you get it so I0:34:10mean do we need to fix this here we go we have a bunch of children young children being subjected to cellos they0:34:19then there's nothing different about these cellos these are regular cellos they're all sitting there in their attitude and it's hurts their hands and it's just awful I think somebody took off took away0:34:30their shoes until they get it right so this is terrible but but it's what happens because what would happen if0:34:39they had any of those other things I just talked about who could ever learn to play cello no one no one would ever learn to play cello there's this great article in the0:34:49current issue of the Atlantic about sort of the trade offs let's say not the pearls but the trade-offs involved in automation it's got a great line in it0:34:58which is that a learning requires inefficiency and it's it's quite important and when I read it and was thinking about this talk I felt like wow0:35:08that's it does go together so we need players I would rant here but I won't but look at this my guitar player with blisters of0:35:18harpists because blisters base by it was blisters right there's this barrier to overcome for every musician imagine right if you downloaded something from0:35:27github and it gave you blisters right the horrors and yet how many people here playing instrument or have at one point0:35:37in their lives yeah a lot of programmers do and for how many people did you just pick it up and it was awesome how many people wished like something could have0:35:46made it more straightforward to get started with them like just made it easy and how many people would have believed after that that they could play later no not at all this is it it's actually0:35:56quite important the level of engagement that's required is quite important so we shouldn't sell you manatee short humans are incredible in particular they're0:36:06incredible learners right one of the things that's really cool is you give a five-year-old or I don't know eight maybe a cello and some decent instruction and they will learn how to0:36:15play cello if they spend enough time doing it in fact humans will pretty much learn how to do anything that they spend enough time doing we're incredibly good at it and we're also really good0:36:25teachers in general so I don't think we need to go to our tools and our instruments and make them oriented towards the first five seconds of people's experience because that's not0:36:35going to serve them well it's especially not going to serve anyone well who wants to achieve any kind of virtuosic ability with the tools right no one we become a0:36:44virtuoso on the cello if they had red and green lights when they started so nine or these two things is effort free but we shouldn't be in a game to try to0:36:54try to eliminate effort because we where we are we are novices right there's a sense in which we're only going to briefly be novices you're only a complete beginner at something for an0:37:03incredibly short period of time and then you're over it it's like we should not optimize for that but on the flip side we're always learners in the matter how much you you know how much time you spend on violin0:37:13who's sister and says I'm done I've completed learning violin I finished it that's awesome I have personally know play violin at all but like I don't think there would be a player on earth no man0:37:22how great they are who would say yeah I finished violin and I moved on to something else we're constantly it's just a human condition to do this so0:37:32things take effort right just like we shouldn't target beginners we shouldn't try to eliminate all effort I mean look at these two guys these two guys are experts right is this the face0:37:43you make when you're IDE autocompletes right does it look like that Oh job I0:37:52doubt you doesn't happen right so your deal life has just been automated away and and I think that's sort of what's interesting is that yeah it sort of0:38:02looks hard and in fact it's probably not hard for either of these two guys but what you're seeing here is a sense of0:38:11the engagement in what they're doing right how engaged you feel on what you're doing when you're programming with IDE that's like doing everything for you you're just like so isolated0:38:22from from what's happening so effort matters another interesting observation that's not really that important to this talk is that instruments and tools are0:38:32usually made for one user at a time like this whole notion of like two guys on one keyboard the program that doesn't0:38:41happen in instruments right now you make ensembles of instruments so I'm doing this and you're doing that we're doing them together in the room it sounds great we do that but dislike two people0:38:50pulling on one tool that almost never happens right so I wonder if you know this pairing thing is just a way to keep0:38:59us from typing all the time to buy one person some time to think right a little bit whoever's not pulling that's got a easy0:39:08easy ride of course this is pretty fast switch back and forth so it begs the question right what ratio of time should we have between planning and performance0:39:18right which is which in programming which one which one is is typing code and it's yellow0:39:28but is that like the way other things work no how about for our Kestrel musician how much time they spend practicing verses at the concert way more time way0:39:40more time and do they do they go and say I practice at college so I'm done practicing no so why do we think that we0:39:50can do this right so we went to college or whatever we learned whatever and then like we just like we're gonna go and we're gonna like do it every day we just do it from here on once school we're done and I think that we do need to0:40:02assess how much time we spend so how many people spend 10% of the time designing 25% 50% so I'm going up it so0:40:12I can know more heads are gonna go up no one's spent 10 percent it's uh it's quite it's quite sad but this is sense in which it's there's a sense in which I0:40:21mean it is sad it is it's actually sad it's not sad as a joke sad it's it's actually sad but the sense there's a0:40:30sense in which it was like all right well this is all so it's so different right Coltrane couldn't build a website in a day I could you know I could do that actually I personally couldn't but I0:40:40know other people who can and that's where another rant would go about how important is that why do we put so much priority on like how fast can a beginner0:40:49do something and how can you like regurgitate a template in a day as none of these are things that we need to do on an ongoing basis to solve problems for the world but it's a fair point that0:41:01you know software is not like instruments it's not way it made out of wood or metal right we have these ones and zeros there's so many combinations there's so many ones and so many zeroes like it just seems so open right so so0:41:13how does this connected so it ends up that there is this connection between instruments and things that are more technological and there are0:41:22technological instruments or electronic instruments this one oh I show a picture because I also have one of those it's called the theremin and you know no0:41:31sooner did we have the ability to turn electrical signals into vibrating loudspeakers by having recorded stuff in order to send the signals through that somebody said I wish we'd have to record stuff I wish I0:41:41could just make up an electrical signal and send it to the speaker let's cut out that performing recording partner let's just do let's just go right for the sound and so electronic music was born0:41:51and this is one of the first electronic instruments where you you play this thing it's two antennae and you you the vertical one controls the pitch the0:42:01closer you get the higher the pitch the further away the lower the pitch and the one that the horizontal one controls the volume and the closer you are the lower the volume and the further away you are0:42:10the higher the volume so you can silence it by touching it and you do that and that's all you got you don't actually touch them at all and the knobs just change the tambour a little bit but0:42:20really not very much it's not really about that this is an incredibly difficult instrument to play it was one of the first ones then things grew up and now we're starting to see things0:42:30more like like we know we understand so these are some of the first electronic instruments that were made or these are the pieces of that instrument you saw before each module does a particular0:42:40thing it might generate sound or generate certain wave shapes or it might be a filter that trims off high frequencies from those shapes or it might generate a low frequency0:42:49oscillation you can use to multiply something else and one of the really cool things about this is not only do you see the first thing you know first or not the first ones but not only do0:42:58you see examples of physical modularity but you also see examples of control so there's these little holes these jacks on the front of these things and they0:43:08actually take or take in or output control voltage it's just voltage you send a voltage in and then depending on the module the voltage might control the0:43:17pitch or the frequency of oscillation or something or the frequency at which the filter kicks in or the wave shape or various things like that and then the0:43:26the knobs are are redundant things that sort of give a human interface to what0:43:35you could have done through control voltages by plugging something into the jack and that there's something incredibly interesting about this and then you see behind these as there's a0:43:44circuit right so we have the layers of of effort but this is a really good lesson here about human versus machine interface these things had a machine interface0:43:53first it was all control voltage and then they put knobs on it so you could patch the control voltages around and build customized things imagine if0:44:02someone had built something sequel without any machine interfaces unix but0:44:11primarily with human interfaces just the knobs like if I gave you a bunch of these models and modules and they just had the knobs and I was like put it together you're gonna be like by doing0:44:21what like putting little remote controllers on the knobs sort of like generating sequel text strings or something why would I want to do that or parsing0:44:30random output from UNIX programs or specifying command-line arguments it's awful right but they they're these hardware guys that are smarter than we are so they built a human and trace on top0:44:42of the machine interface so we're seeing this thing this stacking that occurs right this guy whose name I'm gonna mess up you son is somebody who's who's0:44:54awesome he can work at all the layers of the stack he's I think he's a biochemist or something but in his spare time was not a biochemist he's a C++ programmer and his other spare time he actually0:45:05designs these modules you see behind him a rack of these modules but he designs the modules so he can do the electronics work associated with building a module0:45:14and building like an analog filter for instance or an analog generator and then he obviously can compose them he helps0:45:23people make kits and then you can build them into racks and then you patch them together so then you're at another level of design where you're patching things together and setting the knob positions0:45:32and designing a sound or patch they call them but saying a sound out of the module and then maybe sometimes there's a little keyboard nurse so sometimes he gets to play the keyboard and make music0:45:43with this but this is all these layers associated to what he does and can do he happened to work with this company our0:45:52Toria to produce a an analog synth which is rare these days that used to be all since we're analog like the pictures I was showing you but now they're kind of0:46:01rare everything's become digital they came out with this analog synth and he how them to sign it and he really did design and what's interesting about what he did there was this thing doesn't have wires0:46:10coming out all over the top of it the decisions about it has the same kind of modules inside it but the decisions about how they go together he made he said or he helped them make0:46:19he said we should make this go to that and this is how the fool should work and these were what the parameters should be there are still knobs on the top but a lot of the other stuff has been incorporated in a design that allows0:46:28people to only work at the next level up they do not need to care about what's inside this box and it's quite an important thing because for him right he0:46:39has different days he has days when he's patching stuff together maybe his days when he's playing this thing and that's all fine but days when he's soldering he0:46:48is not making music right and and this is what happens to us right this is what happens to us when we say you should use emacs right it's like somebody wanted to make music and you gave him a soldering0:46:57iron it's like here you go have at it start at the bottom and why does that happen to us and the reason is because for us it's the same stuff all the way0:47:08down for for in that space it's very different right designing an analog filter is look pretty tricky thing just from a mathematics perspective and then0:47:17there's all the you know componentry associated with electronics aspect of it then there's actually being able to solder and put it together on a circuit board and then somebody completely with0:47:28a completely different skill set to go and say I can patch these things together and turn these knobs and listen and understand what the architecture of these things is and make make a sound something else could walk up to that0:47:37whole patch and say I can make I can make a composition with this sound but for us we had the same stuff at all the levels it's code the top levels code the0:47:47middle levels code the bottom of those code we can do it all right we had the same mechanism at every layer so essentially we all do have soldering irons it's like any time you want to you0:47:56can start soldering you could you supposed to be up here doing this you could just start soldering and just because we we we have the soldering iron0:48:05it doesn't mean we're capable of doing things at all layers but we just do because we can't we have got the iron in hand and and I think it leads to a lot of distraction and expansion of scope of0:48:15things it's like I was working on this and I realized if I rewrote the driver I could be you know ten percent faster and now I'm doing something I shouldn't be doing so there's a sense in which having0:48:26so much control over so many parts of stachy gives us this paralysis there's so much that we can do at every point right so what are we going to do and I think that we need to and then of course0:48:36the problem space has some constraints but we need to bring constraints of our own into play we have to do this for ourselves the same way composers do it for themselves or choreographers or0:48:45directors do it for themselves they bring constraints in to help them move forward right this is not a new idea it's a very old idea but it's one we0:48:54have to keep remembering constraint drives creativity when you don't have a lot of choices you're forced to pick an answer and move on you've got choices you could just Mull around about the0:49:03choices all the time so making your own constraints is a way to help help you do that so I think we need to quit fidgeting right and climbing stuff on0:49:12and fiddling around things and tweaking I mean oh my god as an industry we spent an inordinate amount of times time focused on ourselves build tools you0:49:21know automating this and that and like just crazy crazy crazy stuff and talking about it and everything else and we should just be focusing on what we're0:49:30doing because what ends up happening is when you keep fiddling with stuff and when you have no limitations to scope and no constraints right what happens0:49:40right this thing happens and every one of those parts may be a good idea right it's probably all good ideas but if you take every good idea you end up with0:49:50that right and like I don't care if you can figure this thing with spring right it's just it's not playable right no one0:50:00wants to play this and in fact this particular one no one does play it plays itself the the actual patching of it is0:50:09the composition and it's it's got like stochastic elements and it's caused it to generate novelty and it just it plays itself in a museum and0:50:20I mean maybe we want programs like that but maybe we don't so you know in in so we should push back I think you know0:50:30especially in open source projects right this is constant pressure like take my good idea take my good idea take my good idea and they're all good ideas but you know whatever and we need to remember0:50:39right there are people who make music by waving their hands through the air that's it they don't need Emacs or anything else0:50:49they can just do this and I'm telling you if you've ever tried to play the theremin and have it sound like anything other than a siren or a spaceship it's0:50:59brutally difficult to do and and so I mean I don't know if you can see in her face but she's not making a face like the other guys but she is engaged I think the reason why she's not making a0:51:08face is because the pitch changes if you make a face it's that sensitive but at some point go listen to that because it's it's it's beautiful so so what is0:51:20design if we if we take a step back and sort of merge all these things together there's a sense of which design this is imagining right you you're if it's not just regurgitating something that's0:51:30already happened before you've got your facing some set of problems you have to imagine potential solutions right and the first thing you need to do is rush at the constraints you don't want to be0:51:39like don't don't constrain me I'm trying to design it's the opposite of that you're like gimme gimme gimme the constraints I want to know about everything and if you haven't given me0:51:48enough constraints I'm gonna make some make up some because I want this thing to work of course when you're facing all these constraints it seems like negative or can't do this I can't do that it must0:51:57do this in this size and whatever it's like oh you know so you have to be this a sense in which designing is fundamentally an optimistic activity but you have to stay positive in spite of0:52:06all these constraints coming your way you have to stay positive remember people do design that have no constraints and pick constraints in order to get outcomes so that that0:52:16optimism can be borne in the fact that this works and this is a way to make systems that work and you want to imagine a ton of things however actually0:52:27designing is about making decisions which means you try to think up a hundred times as many things as actually use way more things than you use you don't want to think of one thing0:52:36and be like okay let's go do that you know I think of ten things and then say this one is the one we want to do so you want to emit very little you want to be able to say no right because the value0:52:47that you convey in your design is strictly about the decisions you've made right it when when you helped make that synthesizer he made a set of decisions0:52:58are they perfect no and everything you want no do I wish I could patch a wire from here there yeah sometimes I do but you know what I really appreciate the fact that this thing just works and it sounds great0:53:07and I can do the next thing I don't have to fiddle around with the inside of it so if you leave all the options open you're not designing that's not design0:53:16everything everything configurable that's not design that's like do your own thing so performing is preparing its0:53:26planning right you have to practice you have to study and in the end what you want to try to do is develop0:53:35sensibilities that you can apply when you're trying to write code right if writing code is the performing part you have to have patterns techniques knowledge about what works and what0:53:45doesn't to apply to what you're going to do you cannot just make it up as you go so design is taking things apart in0:53:55order to be able to put them back together and that's really all it is every time I encounter something I can boil it back down to that every time I encounter something that I wish my0:54:04design was better I need to do more of this it's over and over and over again it's always this I did not take it apart enough you want to design like Bartok0:54:13that is to say you want to communicate very well you want to be able to work at multiple levels and you want to code like Coltrane you want to take0:54:22preparedness and experience real experience with doing things not experience by doing the same thing over and over again and bring them to bear in0:54:31what it feels like a more improvised thing right I'm encountering a new scenario in a programming project I'm really not making it up I'm really bringing my background in0:54:40play to solve that problem I think you want to find and choose languages and libraries that are like instruments and all the ways that I talked about in0:54:49terms of being simple directed at one thing oriented around people that know how to use them and expressing and0:55:01backing some fundamental excitation or idea those are going to be the most satisfying and in the end pursue harmony in your own designs try to think about0:55:11the nature of harmonious Ness in software right what makes things work together and apply that but thanks very much for listening and I hope you enjoy0:55:20the rest of the conference [Applause]0:00:00Rich Hickey On Clojure
0:00:000:00:12I think I'm here to talk about closure it's a little bit different from the other languages you've seen so I'm gonna try to show you a little bit more of it but I don't have a lot of time so I'm gonna go very quickly it's about three0:00:23years old it is a lisp but if you think you know lisp or hate it or whatever if you could just set any preconceived notions aside I've made a lot of changes0:00:33enhancements and structural differences with lists of the past and in particular closure is not a port of any existing list for the standard lists in0:00:43particular one key difference is that closure is functional and we've heard the word talked about a lot mostly in terms of higher-order functions being you know has two functions and methods0:00:55but really I think the core of functional programming is an emphasis on immutability and on referential transparency on calling functions that0:01:04are free of side effects I wanted a functional language on the JVM for a couple of reasons one I'm a commercial software developer and JVM0:01:13and net are sort of the only two accepted platforms but I want to program at a higher level so closures designed to support the kinds of programs that0:01:22programmers write in Java and my goal is to try to target the application space of Java in particular those commercial0:01:31programs generally involve concurrency these days so that's a prime objective of closures to support concurrency explicitly in the language so you'll see0:01:41the combination of the way it approaches immutability in its data structures with these built-in primitives for concurrency sort of got a complete package for functional programming on0:01:51the JVM closure was also designed for the JVM it's not a port of another language and it was designed to be hosted to interoperate with the host to expose Java embrace Java call Java0:02:02libraries it shares with Java a type system the calling conventions and everything it is a pretty much barrier-free interface to Java so and0:02:13being a lisp it's dynamic and of course now lots of languages are dynamic and have learned many things from list so has the typical things it's interactive0:02:22it has a ripple it sports that kind of devel you can change programs while they run particular has the characteristic of list that makes lists a lisp which is0:02:31that code is data you represent your code as data structures on the compiler and evaluator evaluate data not text so it makes it possible to write programs0:02:40the write programs which is the key to macros and syntax and tactic abstraction closure has a very small core you know0:02:49half a dozen primitives plus some extra primitives for accessing Java which means there's a good chance I could get that part right in addition it makes a0:02:59major change to Lisp I think in lifting off of the data structure which was the console that represented singly linked0:03:09lists it lifts that abstraction off of that data structure and allows the libraries to be built on top of an abstraction and that's made the language0:03:18much more powerful than lisps have been in the past and in particular also allows the power of lists to be extended to Java in addition closure is not0:03:28object-oriented I'd be happy to talk later more about that but it's a it's a feature of closure closure has all the0:03:39basic data types you would expect in particular it has good math that doesn't wrap so arbitrary precision integers0:03:48doubles big decimals ratios those are all represented by their java capital letter types except for ratios that had to add it has strings it has character0:03:57literals it has symbols and keywords which are a little bit different they're used for identifier z' and for keys but they're prime datatypes in closure so0:04:08your composure programs out of these data types those boolean literals Anil unifies with Java null they are the same thing and they represent nothing in closure and district has regex literals0:04:20so these are the atomic data types then it has a set of data structure literals okay so there's a way to write these0:04:29data structures in your program and they mean what they say here okay parenthesized set of things they could be numbers they could be anything0:04:38is a list and lists are singly-linked and they grow at the front whenever I say grow change add remove there's0:04:48quotes because nothing actually changes but they grow at the front they have fast access time to the front linear access time down down the line closure0:04:58also pretty uniquely has immutable vectors which are in square brackets they support very fast indexed access0:05:08and in addition they behave like a ray list and that they grow at the end in constant time there are map literals there in curly braces it's key value key0:05:19value key value key could be anything value could be anything all of the data types are values so the key could be a vector if you want ditto the value0:05:28commas are whitespace put them in if they make it more readable but they're not mandatory a set is in curly braces0:05:37preceded by a hash and that will be unique set of things everything nests that is the syntax from the language0:05:49there's nothing else those data structures are the code there is a text representation that's where I just showed you and there's something called0:05:58a reader which can read that out of a file and turn it into the data structures at that point those data structures get handed to an evaluator which compiles them immediately to Java0:06:10bytecode where they're executed that's if you ask for them to be evaluated you can also use the reader to read data it's a nice format nicer than JSON or XML but in0:06:19particular the syntax of language is not really based around text it's based around these data structures and how they're interpreted and everything else0:06:28is very simple everything that would have been an operator or declaration or anything else is the parenthesized lists where the operator is the first thing in0:06:39addition there are no statements everything is an expression so it's a pretty simple language this is a spell corrector written by Peter Norvig in0:06:49Python and I don't expect to be able to read this my son not too bad but I just wanted to give you an idea of the the syntactic weight of closure Python is an excellent0:06:59target because I think it has one of the lowest syntactic weights of any language given that it uses whitespace space in the way it does so the closure program is the same size it's the same number of0:07:09lines and with the Python program those are the two shortest versions submitted to Norvik he had many people do them in many languages these programs are0:07:18roughly equivalent so it's pretty I'm not really gonna have time to explain all this thing but it has lots of cool0:07:27features list comprehensions pervasive destructuring all kinds of good stuff some syntax related to Java you can0:07:37access statics by namespace slash the name that'll work for methods or will work for data variables there's a syntax0:07:47with dots that allow you to get at members generically a member could be a member of a static thing or a member of an instance and course closure has macros and what you're seeing here0:07:56already is a macro dot dot will take an arbitrary number of things and will put dots in between them as if you wrote that Java so that system dot get properties that get Java version we can0:08:09call new we can also call new like this again though we have macros so instantly we can write code that's a lot shorter0:08:18than Java Java code that's a lot shorter than Java due to a new jframe add a new label with this name pack it and show it0:08:27you know the Java that is that which involves a lot of this not the thing that's not the thing that type of thing and in fact this is the expansion so0:08:38like most lists there's a really tiny core and the rest of the language is written in language in terms of macros and functions the Java is is complete0:08:49and good and wrapper free okay you say import this thing you're using Java there's no wrapper classes nothing in between you and these calls I'll explain0:08:59that a little bit more a little later okay this is sort of closures reason for being the idea here is to allow people0:09:09to do functional program in a way that's roughly as easy as programming in Python or Ruby and it starts by making all of0:09:21those data structures that you saw immutable so you write these literals and your program with these objects but they cannot change like really really can't change in the you know in the all0:09:31their members or final kind of way you couple that with a set of library functions a huge set that manipulate0:09:40these data structures that have no side effects so these functions take arguments they produce new values they don't change things and this really0:09:50makes for better programs your programs are easy to understand easier to test your mock object is kind of a joke that0:09:59object is mocking you for needing it but I think III think that I think that even0:10:10if your second currency side functional program functional programming yields better software but when you put concurrency into the mix I think0:10:19functional programming is a huge huge win for many reasons and of course you read Brian's excellent book and you0:10:28know how many times it's a immutable many many times but it's hard closure makes that easy because it's the default0:10:37it's really functional like let bound locals are immutable I don't even call them variables because it would just be too disappointing there's a function there's a functional0:10:48looping construct which sort of works around the fact that there's no tail calls and of course it does also have higher-order functions and and that kind0:10:57of thing the library this is just a small taste of it for sequences is extremely powerful and it allows you to write programs that when you read them0:11:06they explain what they're doing as opposed to you know imagine the equivalent loop you would have to write in Java and imagine reading that loop to try to determine oh he's splitting this0:11:16up into pieces oh you know he's putting these in between those these functions tell you what they do but there's a couple of things that are interesting about sequences like I said it's an0:11:25abstraction on the list concept of Lisp but because it's based around an interface in Java it means that it can be implemented on many data structures0:11:34not just lists and in fact all of these functions and all of the sequence functions in the library are supported on all of the data structures of closure on java strings Java arrays everything0:11:44Java iterable all Java's collections all of this stuff works on all of that so it's very nice another aspect of raising0:11:54that abstraction off of a concrete data structure means that we can implement sequences lazily that cycle function returns an infinite sequence of numbers0:12:03okay take only uses the first nine but it returns an indefinite amount Clos all of these functions and and the vast0:12:13majority of closures sequence functions are lazy that is to say they do not produce a complete result it's another advantage of not working on a concrete0:12:22data structure is that your intermediate computations do not need to return a complete intermediate data value you can you can concatenate a bunch of0:12:32operations and walk through it and it's effectively like pulling on a pipeline and you're getting one thing at a time you're trading off some ephemeral0:12:41garbage for never needing to have a giant data structure or a set of giant data structures in your program so it's all very pretty oops0:12:53in addition is equivalent set of functionality for maps and sets you define well def just makes a top-level0:13:02variable right we have a little literal map maps are functions of their keys SATs are functions of their keys arrays are functions of their indexes because0:13:11well they really are and so close your lesson be that so you can call a map passing it a key and get the value at that position in addition there's a0:13:21little extra sugar keywords themselves are functions of associative things so we can look up something in a map by just doing0:13:30or that we can access the keys of the map the bowels of the map there's all kinds of functionality associated with maps associate or a so she is the way0:13:41you would associate new keys and value pairs with an existing map and you get a new map M is not changed by that and0:13:53there's some really nice functions right you need to work walk through a bunch of maps and increment counters per key that's it merge with plus a map and a0:14:05set of other maps happens automatically when a build list you merge with conch move up a level working with sets as a0:14:14set of set logic Union intersection move up one more level sets of maps are relations and there's a relational0:14:24algebra for that it's a neat one because it doesn't require rectangles so I said everything0:14:34is immutable which should be scary because immediately people say oh my god the amount of copying must be incredible and it's not viable to do full copying0:14:44so there's this notion of persistent data structures it's an overloaded word like all the overloaded words we have in this case we mean of course persistent data structures or data structures that0:14:53are immutable we're making a new version is cheap and both the new and the old version are available after the operation and both the new and the old version support all the operations of0:15:03the data structure with the same time characteristics as our promised by the data structure so you can mate you can modify an element in a vector in0:15:13constant time and both the new and the old version have access to the elements in constant time so they can't be full0:15:22copies making that copy can take a linear amount of time or that wouldn't be that would match the performance guarantees so of course the trick is to0:15:31use structural sharing but it ends up that vectors and maps are hard persistent data structures lists you see all the time binary trees red black0:15:41trees easy but lie again is usually the best you can do closures persistent data structures are particularly neat the sort of the secret0:15:51sauce of closure I based him around Bagwell's bitmap hash trees but I made them persistent his design was not persistent they're much faster than big0:16:02Ola again they're actually big ol log 32n which is a great example of the difference between you know two things0:16:11I'm in the same shape and one being practical in the other or not the constant factors matter and this is a little bit of what how the representation looks a bitmap hash tree0:16:21divides the bit divides the hash into five bit vectors which are used for 32 way branching yeah Java there are0:16:36interfaces there are Java interfaces for absolutely everything I'm showing you this this library is completely factored into a set of abstractions those abstractions for sequences lists Maps0:16:46vectors a associative reversible it's a nice set of abstractions and you can use it from Java ok we're doing fairly good0:17:00so that's how it works so it's efficient to use these persistent data structures which makes it practical and that was the key problem I had to solve for0:17:10closure because I couldn't convince professional developers to use this if I didn't have hash tables really everybody needs them so now I'll talk a little bit0:17:19about concurrency I'm not really talking about data parallelism like the kind of stuff they were talking about in the fortress talk I have some support for0:17:28that I call you know I have a wrapper around fork/join and do some of that stuff but here I'm really talking about task level parallelism the coarse grained parallelism most applications do0:17:38when they set some threads off to listen to a socket and a couple more to talk to the database and a couple more to deal with the web and this is a big part of the architecture of closure so what do0:17:48we do now right we have direct references to mutable objects this is a Java program and really any object-oriented language that isn't sort of special right so we0:17:57talking Java and Python Ruby they're all the same in this way and that they unify identity in value so we have direct references to mutable things which means0:18:06if we're going to have concurrent operations that might change them we have to use locking in closure we're going to add back those method handles right or those object handles in order0:18:17to get an indirection between a mutable reference and an immutable object it's inspired by SML wrath and then closure0:18:28as concurrency semantics for the references is three different reference types which now means that the language can help it can enforce these concurrency semantics and you end up0:18:38with no locks on the user side so this is what happens now there's gonna be many points in your program they're pointing to a chunk o memory okay and0:18:48that chunk of memory is the identity of the thing so once that's the case they have to take care of all the stuff by themselves right any kind of consistency0:18:58for that chunk of memory they have to arbitrate and there's no language level support for that so enclosure what we do0:19:07is we have an indirection they're going to point to this box that's their identity that box in turn points to an immutable data structure any of the0:19:16things I showed you could be a vector it could be a map it's often a map people will enclosure use those Maps instead of where you would use objects so we separate identity and value to obtain0:19:27the value you have to dereference the values can never change so you can never get an inconsistent value if you've got the value you're safe and you can share0:19:37it easily between threads sure you can this is the real if you ever at Henry0:19:47Baker's paper paper on equality a closure implements ygal he called it where equality flows through until it hits the modifiable thing right and then two modifiable things are only equal if0:19:57they are the same out of file thing so so you could have a member of a structure that was the reference but I0:20:08don't recommend it the one time you do that if you try to sort of control your concurrency maybe lighten up your contention by0:20:17having more references yes I'm gonna this is still an abstract description0:20:27I'm going to show you the three variants of these cells so we haven't seen how to use this yet this is the concept so the concept is we have a reference to a0:20:36thing what does it mean to edit something then okay we're gonna have this immutable thing we just saw a bunch of cool functions I can take one immutable thing and make a new one with some changed value so that's what's0:20:45going to happen a new value is a function of the old value they may share some structure it's all immutable it's all fine so thread-safe no problem never impedes the readers if0:20:55you're reading this well if somebody's creating that again no problem and then we edit okay0:21:04the trick is this swap to say this now points to that that's under the system control there are semantics for that in all cases no munging about with no with0:21:16no control so it's always coordinated there's three semantics I may get to show you to the next dereference we'll see the new value any consumer of either0:21:25value is unaffected by this edit so I've been talking about references generically that box there are three0:21:34kinds they're called refs agents and bars refs allow coordinated change and it's a transactional system agents are0:21:45asynchronous autonomous change you can think a little bit about actor models but the difference between closures agents and actors is you do not require0:21:54message passing to read values from agents bars are thread-local so in all0:22:04cases you have safe way to change things whoo all right wrestling transactions closure has an SDM it's it's my own0:22:13homemade STM I have big differences with the design of a lot of STM's because I think they're designed to allow you to keep doing what you've always done which is work with mutable objects and they're0:22:22just going to turn every mutable field into a tea for whatever I don't actually believe that's gonna work but good luck0:22:31so it's an STM it has what you your usual expectations of transactions in fact it very much maps to what you think of when you think about a database0:22:40transaction I have a atomicity isolation I also have consistency you can hook up a consistency function to any of these refs and it will be run before the state will change you don't pass doesn't0:22:50change I don't have durability because this is an in-memory system right and then it's basically everything happens or nothing does right your usual expectations the big difference with an0:23:00STM is that transactions are speculative they may get run more than once okay they're going to happen they may get to a point they may see somebody else one they'll try again so you have to avoid0:23:12side effects I actually can't help you with that because you can call Java and I can't tell if that's going to be ok or not very easy to use all you do is to0:23:23surround your code with do sync any number of changes to references either directly in that body or in any function that calls will participate in the same transaction if those are in turn in0:23:32their own transactions they join the outer transaction I use multi-version concurrency control which is also a unique part of closures STM it means0:23:42that readers see a consistent snapshot view of the world readers don't have to get restarted to the writers ever and0:23:51there's no read tracking so this is what it looks like to use it will define food to be you're now a reference to this map0:24:03ok we can dereference foo we see what's in there we can grab the value of foo and make a new value we haven't really changed foo we're just playing with its value foo is unchanged this function is0:24:15one of the transactional functions those alter commute and Sat commutes takes a function it takes a reference a function0:24:24in any arguments and we'll run that inside the transaction except I insert transaction so not ok again the system0:24:33is helping you tried to change something outside of a transaction that's not going to happen rat that same thing in a do sink and it works I only have five minutes left so0:24:44I'm not gonna have too much time to talk about this agents do independent state it's sort of a message system where you say send the things you're going to send are going to be functions they'll make0:24:54the change in a thread pool at some point in the future there's no way to coordinate more than one of these changes at the same time they're not the same as actors what's0:25:04cool about it is it looks very similar who is it as an agent with that value we can dereference it the same way we can send foo a function with arguments it0:25:14will change at some point in the future if we immediately went to read it maybe we wouldn't we wouldn't see that change if we wait this basically puts a message in its queue and waits for it to be0:25:23consumed now we know this will have happened we'll see that if nobody else is also talking to fill the job0:25:32integration closure is very good there's a lot of unification closure strings are Java strings the numbers other numbers the collections all implement collection0:25:41my functions and closures are runnable and callable passing to executives that all works perfectly well everything I've shown you all the core abstractions have0:25:50interfaces so they're very easy to consume from Java it's also easy to extend closure from Java or closure as I0:26:00said before the sequence library works on everything you can implement and extend Java interfaces and classes in closure and I have primitives support0:26:10infiltrate in a second from an implementation standpoint I compile dynamically to by code in memory I use Azzam it's great I don't do any ahead of0:26:20time compilation right now every function turns into a class one interesting thing I think about closure is that those function objects actually have a overloaded arity so you can pass0:26:32a function object that has multiple methods with different era T's so it doesn't do any already munging together and it can be very fast you can pass a0:26:42function that might be called with two arguments or three and they'll both be as fast as possible all my signatures take and return objects what's cool is my very attics are built on the0:26:51sequences so you can write a function and pass an infinite set of things and as long as it doesn't try to use them all it will work calls all the function0:27:01calls are straightforward method calls so there's no type system I don't have any thunks I'd open any extra arguments in it's very easy to call from Java you0:27:12can call Java there's only two things are gonna happen it'll have to be a reflective call or it'll be a direct call right there's no wrappers caches thunks none of that stuff0:27:21I support tie pins with the very slightest bit of type int and inference I can make all of your Java calls direct calls so there's no need for reflection0:27:31and I personally don't need invokedynamic because of this it's amazing how few hints you need I do support locals that are primitives0:27:43so I make them into primitives then I inline the math calls to static methods as James said before that's as far as you need to go I make a static method0:27:53called hotspot turns it into primitive math this is the code enclosure this is the speed it's actually really pretty good but you say int int and boom that's0:28:05exactly the same amount of time it takes Java it's exactly the same bytecode after hot spots finished so in your inner loops if you need to do fast math I support primitives and arrays the0:28:15primitives it's exactly the same speed as Java my esteem implementation is very interesting I'd be happy to talk to0:28:25anybody about it who's interested in SDM the two things I would ask for if we're asking for things I'd really love telco0:28:34optimizations it's a big deal it's a big criticism or closure being on the JVM to JVM has no tail calls I've worked around0:28:43it considerably I think between loop recur and the lazy sequences for people that bother to try closure they find the pain of not having Tuggle's it's not that great I still want it and as0:28:53everyone has said so far I would love tagged fix thumbs or anything that makes working with box math fast because I want to have to have people make those0:29:03declarations so to conclude I've been very happy I love the JVM I think it's been a great platform because I directly targeted it0:29:12and because I targeted one five I'm not sitting needing anything I work really well in one five closure is really fast there's ten times as much to closure as0:29:22I've shown you but it does fill a nice niche there is no dynamic functional language on the JVM now there is its closure and I've only been out for a0:29:32little bit less than a year and uptake has been rapid extremely rapid so as0:29:42fast as I can go [Applause]0:00:00Persistent Data Structures and Managed References - Rich Hickey
0:00:00 how many people program in a functional programming language0:00:09okay so halfway preaching to the converted and not in a functional programming language a non-functional program so stole out of that I think0:00:21this will be useful to both audiences in particular if you're not in a functional programming language in fact if you're not in Erlang which I think has a complete story for how they do state all0:00:32the other functional programming languages have you know two aspects they have this is the functional part and then and then you know Haskell has this0:00:42beautiful side where the type system keeps this part pure and then there's the other part which is kind of imperative do this do that and then they0:00:51have a bunch of constructs to make provide facilities for when you need state on that side similarly there are a lot of hybrid functional languages like0:01:00Scala and f-sharp where I think there are questions to be asked about okay here's the pure part what's the story about the other part so what I want to0:01:11do today is to talk about functions and processes and to distinguish the two in fact the core concept in this talk is to try0:01:20to parse out what we mean by identity state and values try to separate those concepts and see how programming with0:01:32values while a really important part of the functional part of your program ends up being a critical part of the non functional part of your program the part that actually has to manage state and0:01:43behave as if things are changing and there are two components to that one is how do you represent composite objects as values a lot of people who are new to0:01:52functional programming wonder about the efficiency and representation issues there and I'll talk about that and finally I'll talk about one approach to0:02:02dealing with state and change in a program the one that closure uses which is compatible with a little bit of philosophy I'm going to start with I'm0:02:12not really going to talk about closure very much how many people were at my talk yesterday okay people who weren't know something about closure okay this is not really a0:02:24closure specific talk there'll be some code later and it shouldn't be too threatening just going to summarize quickly with this one slide what closure is about is a dynamic programming0:02:34language is dynamically typed it's functional in particular its functional in emphasizing immutability not just in you know supporting higher-order0:02:43functions all the data types and closure are immutable it supports concurrency in that it's a0:02:52two-part story one is you have to have good support for immutability and pure functions the other part is you have to have a story for when multiple things are happening at a time and you're going0:03:01to have some perceptible change and closure does in fact that I think it's an important part of a language that you know purports to be functional that have a story about the non functional parts0:03:12closures not particularly object oriented it may be clear after listening to this talk why not because I think as currently implemented a lot of object technologies have big problems when they0:03:23face concurrency and functional programming as I said from a conceptual standpoint nothing about this is really a closure specific so what do we mean by0:03:32functions I think that there's a really easy way so I will function some that you call and that's not what we're talking about here we're talking about a very precise notion of a function which0:03:41is something that you call that takes values as arguments and produces the value as a return when it's given the same arguments always proves the same0:03:50value it doesn't depend on the outside world it doesn't affect the rest of the world so many methods and your classes are not functions by this definition but0:03:59in particular two I want to highlight the fact that functions pure functions have no notion of time time is going to be a critical notion through this talk0:04:11so what is functional programming there are lots of answers to this question and I think people that are into type systems will claim a stronger argument0:04:21for what constitutes functional programming but I'm going to limit the definition here to you know programming that emphasizes program with functions those we want to try to write as much of0:04:31your program you can with pure functions when you do that you get a ton of benefits they've been talked about in other talks it's not really the focus of this talk other than to say even without concurrency0:04:41your program will be easy too easier to understand easier to reason about easier to test more modular and and so forth that all falls out of programming with0:04:51functions to this graded and extent as possible on the other hand when you step back and look at your entire program very few programs on the whole our0:05:00functions you know that take a single input think about it and produce a single output maybe some compilers or theorem provers work that way0:05:09but most real-world programs that I've worked on and I think most real-world programs in the real world don't work that way in particular well even if you0:05:20claimed your program was completely functional if it's going to produce any output it's not because otherwise we're just warm up the machine but even if0:05:30it's mostly functional there's still observable effects of a purely functional program running right it's running on a computer as soon as it's running on a computer it's not math anymore right it's a program running on0:05:41a computer it's consuming memory it's consuming clock cycles it's observably using doing something over time so all programs do things over time but most0:05:55real programs as I say actually have observable behavior that is not just the fact that they're running on a computer but that they're doing things they're interacting with the outside world0:06:04they're talking over sockets they're putting stuff on the screen they're putting things in around the database in particular though we'll use one critical measure about how to define state which0:06:15is if you ask the same question twice and you get different answers then there's state I don't care where you put it you put it in a process you put it in an agent and add them you know in a0:06:25variable doesn't matter in a database if you answer ask the same question twice and get different answers at different times you have state so again the word time just came up again there so I think0:06:38most programs are processes which means we need to talk about the part of your program that can't be purely functional the part that's going to have to produce a different answer at different0:06:47times how do you do that and not make a complete mess out of what you've created with the shiny pure part in particular0:06:57though I want to highlight the fact that this talk is strictly about the notion of state and time in a local context I'm talking about in the same process there0:07:07are a completely different set of requirements and characteristics of distributed programs where you cannot do the same things that you can do in the0:07:16same process so I'm talking only about same process concurrency and state so I0:07:25want to be a little bit more precise about what I mean when I say identity state and value and these kinds of things in particular I want to talk about state and I'll talk about it twice0:07:34one is just a generic statement state is a value of an identity at a time maybe none of that makes sense maybe it sounds0:07:44like a variable from a traditional program which right because I think if you ask somebody who's using traditional programming language do you have state you're like yeah I have some variables0:07:53and I change them and that is not a good sound definition of what constitutes state so it's our variable state do they0:08:04do this job do they manage the value of an identity at over time we could have a variable I we could set it to zero we could set it to 42 we can assign one0:08:14variable to another right is J 42 that depends in a sequential program probably0:08:24kind of sort of in a program that had threads what could go wrong well I didn't say what order these things happened in right or what threads they0:08:34happened in for instance if you said J equals I in a separate thread what bad thing could have happened to you would it see 42 necessarily no definitely not0:08:45because that memory may not have been flushed through to the other threads cache okay it's not volatile necessarily I what else could happen it's bad0:08:56maybe eyes along maybe setting along it's not atomic in your programming language bad so variables are not going0:09:05to be good enough to do the job of managing state all right they're predicated on a single threat of control they actually don't work at all otherwise they're horribly broken by0:09:15concurrency the whole notion of a business this piece of memory is does not work and our programs are built substantially on this whether it's a variable you know sitting on the stack0:09:24or fields in your object same problems ok pieces of memory or insufficient abstractions so we have the problem the0:09:33knob up nuh-nuh animus city of long writes ok that's a problem in a lot of languages there's just not a time I could get half of a number if you look at it from another threat right visibility and memory0:09:44fences have to be accounted for once you have multiple threads of control on a true concurrent box if you have a you know an object and it has a bunch of0:09:55these things collected together the constitutive state now you have the problem of composing you know a composite operation because making it into another valid state requires touching you know several of these these0:10:05variable things which now makes you impose locks or some sort of synchronization to say stay away from me so I can pretend there's only one thread0:10:14of control because that's what my language thought when they wrote it or the language they copied you know thought when they wrote it all of these0:10:25things are examples of the same problem we're having to work around the lack of a model for time ok because there's no point to have having variables if you0:10:34don't have time and just think about that for a second if there's no time notion why would you need a variable if you can't go back to it later and see0:10:43something different and how is it a variable so if we want to be clearer about time which we're not going to be0:10:52in a non physics lecture we're just going to say some what what are some things you think of when you think of time you think of things being before or after other things you think of0:11:01something happening later you think of something happening at the same time two things happening at the same time you think of something happening right now which is sort of a self relative respect0:11:12the time but all these concepts are important in that they're inherently relative right when you think about time you don't it's not a lot about time0:11:22that's hours or you know this particular moments you know with a name on it that's most of our notions of time have to do with relative time the ordering0:11:31between two discrete things what do we mean by values again here's a here's an area where there's just so much0:11:40ambiguity and loose thinking that we can't write correct programs and so we sort of nail this down so the core characteristic of a value is that it's immutable right some values are obvious0:11:51you know numbers we all are comfortable with that concept it's a value but I will contend that until you start thinking about composites of these0:12:00things like numbers as values you're doomed in the future you may not be doomed right now but at some point it's going0:12:09to be a problem for your programs so what went wrong okay we all think 42 is you know indivisible of course we saw if0:12:18we stored it along it may not be depending on our language but but with the idea of 42 we consider to be an atomic concept but we have a big problem in the way we think about composite0:12:28objects some of that falls out of our languages I think you know we have date libraries where you can set the month is a crazy concept it does know there's0:12:39this state and there's that date there's not setting the month of a date and it's another date I think that's there right away you have that problem I set the month of a date and it's0:12:48another date if it's another date you have two dates you don't have one set able date and our class libraries have destroyed our brains in this area0:12:58also the default behavior of our languages you create a new class in most languages and everything is variable and instantly you have this stateful mess that maybe you have to clean up with a0:13:07lot of discipline on your part so I'm going to contend that dates sets Maps everything is a value and should be0:13:16treated like a value and you should separate your concept of value from your concept of change so one more concept0:13:25in the philosophy portion of the talk which is the concept of identity okay this is probably the most nebulous of these things but it's it's an important0:13:34thing what happens in the real world what happens in the real world when we talk about you know today or mom or you know Joe Armstrong is that a a single0:13:45unchanging thing or one way to think about it is we have a logical entity that we associate with different values0:13:56over time okay in other words at any particular moment everything is frozen okay in the next moment we look we see0:14:06something different is that the same thing well if the same if some force is acted on this thing to produce that next thing I consider it to be the same thing you know otherwise they're unrelated0:14:19things two things can pass through the same space they're not the same thing because they're in the same space so a set of values over time where the values are causally related is something we0:14:28need to name these are different values right they may be in different spaces I could walk over here I'm still rich so0:14:37what's happening what's happening is uneasy to understand if you have three notions right there's a state I'm standing right here there's a state I'm0:14:46standing over here right there both values you know if you could stop time for a second nothing about me would be changing and it's me because you know0:14:56I'm using my legs to move myself over here and I'm still talking and you see a set of causal connections between me being here and being there so you say0:15:05that's all register it's not two people you know doing this identities are not the same things as names I just want to make that clear okay I have a mother you0:15:15now have that concept in your head this entity you know this identity but I call her mom and you would call her mrs. Hickey I hope these identities can be0:15:25composite we can talk about the New York Yankees or Americans right no problem those are sets but there are also identities they change over time but at0:15:36any particular point in time they have a value these are the guys who run the Yankees right now any program that's a process needs to have some mechanism for0:15:46identity okay this all goes together so I'll go back and talk about state we have some terms that hopefully mean0:15:55something right now we can say a state is a value of an identity at a time hopefully that makes sense right the identity is the logical thing it's not necessarily place it's not a piece of0:16:05memory okay I'm value is something that never changes okay and time is something that's relative now it's easy to see I0:16:15think why we can't use variables for state in particular that variable and they may not refer to something that's immutable it's already earned that's a problem if you refer if you make a0:16:24variable refer to a variable okay with your building on sand okay the key concept is variable or whatever we're going to do to manage time has got to0:16:35refer to values sets of variables as we traditionally have them can never constitute a value because they're not0:16:44atomically composite okay because we're saying a value of something that's immutable if you could change the parts independently then it's not immutable0:16:53right because there's going to be a moment when one part is halfway there and another part is not there that's not a valid value now you're something's happening in the middle and more0:17:03globally you can say about variables their problem is they have no state transition management okay that's the management of time a coordination model0:17:12for time how do you go from I'm in this state now I'm in that state both states being immutable values so this is the0:17:22sum summary of the philosophy portion a key concept I think is things don't change in place we think that they do0:17:31but they don't the way you can see that this is the case is to incorporate time as a dimension okay what time is it dimension once you have XYZ time guess0:17:43what that's over here if something is happening here this is no different okay things do not change in place time proceeds0:17:53functions of the past create the future okay but both things are values there are a couple of aspects I think to the design of the things are going to show0:18:03you that I think are important when you try to model time in the local context these are things I don't want to give up these are things I know I can achieve by0:18:12brute force in Java and I can't sell my language if I can't achieve them in closure for instance which is co-located0:18:21entities can see each other without cooperation okay there's a lot of messaging models that require cooperation you know if I want to see0:18:30what you're about I have to ask you a question you have to be ready to be asked that question you have to be willing to answer my question but that's not really the way things are when0:18:40you're call located now I know what's happening in the next building but I can see all of you and I could certainly look at the back of your head without asking you permission the other thing I0:18:49think is really important in a local context it's it really should be written off as it impossible in the distributed context is you can do things in a coordinated manner with co-located0:18:59entities in the same process you can say let's all work together and do this alright now as soon as you're distributed you can't do that so the models are going to show you support0:19:09visibility of co-located entities and coordination so let's take a little example a little race Walker pal0:19:18detector ok race walkers they have to walk they can't run they have to walk step step step heel toe and then they can't both feet off the ground at the same time it's a foul and then you get0:19:29you got kicked out of the race so how do we do this while we go and get the left foot position we see it's off the ground we go we get the right foot position we stay off it's off the ground so they're0:19:38running right yeah it sounds funny but you I mean everybody writes programs that do exactly this all the time0:19:47exactly this and you wonder well I mean why white why didn't it work okay we can't work that way okay we can't have0:19:58time and values all munched together where things are changing while we're trying to look at them that doesn't work we can't make decisions we don't make0:20:07decisions as human beings this way snapshots and the ability to consider something as a value at a point in time are critical to perception and decision-making and there is critical in0:20:17programs as they are to us as human beings if you look at our sensory systems they're completely oriented on creating momentary glimpses of a world0:20:28that would otherwise just perceive to be you're completely fungible now how do we achieve this you know programmatically0:20:37well one of the things I think we have to advocate if we want to write programs that can work on multiple cores and benefit from being on multiple courses we can't stop the race we can't stop the0:20:48runner we can say well could you just hang out for a second I wanna see if you're if you're if you're running also we can't expect the runner to cooperate and say could you just tell me if you're0:20:58running but if we could consider the runner to be a value like this guy on the right here it's kind of nice that we can look at him as if it was a value0:21:07there's a point in time that was captured by this photograph right there's a single value I don't have to independently look at the left and right while time proceeds right I've got I've0:21:16got a value in hand it was captured at a point in time the race kept going but I can see that guys got a foul on the right he's got both his feet off the ground that's easy that's the kind of0:21:26easiness you want to have in a logic of your applications you want to be working with values you do not want to be working with things that are running away from you as you're trying to0:21:35examine them so it's not a problem to do this work if we can get the runner's value in a similar way we don't want to0:21:44stop people from conducting sales so we can give them bonuses or do sales reports we need to move to a world in which time can proceed and we can do our0:21:53logic and we don't need to stop everybody so we can do our logic the two things have to be independent so how0:22:02does it work well the first thing is we have to program with values we have to use values to represent not just numbers and even small things like dates but pretty0:22:11much everything collection sets things that you would have modeled those classes should be values I'm not saying that there couldn't be an object-oriented0:22:20system that worked this way I don't know of one that does but you should start looking at your entire object as if it was a value should never be in pieces that you could twiddle independently you0:22:29want a new state of that object you make a new entire new value so then what's the problem with time it becomes a much0:22:38smaller problem all we need to do is get some language constructs or some way to manage the succession of values right0:22:47and identity is going to take on a succession of values over time we just need a way to model that right because we have pure functions we know how to create you know new values from old0:22:56values we only need to Martin to model the time coordination problem what's nice about this is when you separate the two things when you haven't unified0:23:05values with pieces of memory you end up with multiple options for the time semantics you have a bunch of different0:23:14ways to look at it there's message passing and there's transactional but because it's now a separate problem you can take different options you can even have multiple options in the same program so I'm going to say that it's a0:23:25two-pronged approach one part is programmed with values the other part is in this in this example that I'm going to be talking about today enclosures example is a concept called managed0:23:35references which you can think of as they're kind of like variables except it fixes all the problems with variables in other words they're variables that have coordination semantics so they're pretty0:23:44easy to understand they're just variables that aren't broken so there are two parts we're going to talk now about the values and0:23:54one of the things that people cringe at initially if they have a new functional programming languages before is that sounds expensive you know if I have to copy the whole runner every time he moves his foot on0:24:04you know this is no way I'm going to do this and in particular when you start talking about collections and things like that people get extremely paranoid because what they know are sort of the very bad collection classes libraries0:24:15they have which either have no capabilities in this case or some very primitive things you know sometimes there are copy-on-write collection so every time you writes what the entire thing gets copied but there is a0:24:24technology which is not complicated and has a fancy name it's called the persistent data structure that has nothing to do with databases or disk0:24:33but it's a way to efficiently represent a composite object as a value as an immutable thing and to make changes to0:24:42that in an expensive way so change for one of these persistent data structures is really just stealing quotes they don't change it's a function that0:24:51takes an existing instance of the collection or composite and returns another one that has the change in enacted but there's a very particular0:25:01meaning to persistent data structures which is that in order to make these changes the data structure and the change operation have to meet the performance guarantees you expect from0:25:11the collection so if it's a big old log in collection or collection that has you know constant time access or near constant time access those behavioral0:25:21characteristics have to be met by this changing operation it means you can't conduct a change on something that you expect to have login behavior and copy the entire thing because copying the0:25:31entire thing is is linear behavior right so that's the critical thing the other critical aspect of persistent data structures is sometimes you'll see libraries try to cheat okay and they'll0:25:42make the very most recent version this good immutable value but on the way they ruin the old version okay that also is0:25:51not persistent that's another key aspect of the word persistent in a persistent data structure when you make the new version the old version is perfectly fine it's immutable its intact it has0:26:01the same performance guarantees it's not decaying as you produce new values every functional programming language you know tries to cheat this side and eventually says forget this we're going all0:26:11immutable and we're going to pay whatever the performance costs are burkas the logical costs of having old versions decay or have some bizarre behavior either from a multi-threading0:26:21perspective or performance perspective is too high so what I'm going to show you our legitimate persistent collections where the old values have the same performance characteristics and0:26:32the particular example I'm going to show you is the one that's in closure it's derived from some work that Phil Bagwell did on these ideal he called them ideal0:26:42hash trees and they're bitmapped hash trees that have really good performance his versions were not persistent and so what0:26:51I did for closure was I made them persistent the secret to all persistent data structures is that they are trees there you go now you know there are lots0:27:05of different recipes that I think people are very familiar with you know B trees and red black trees and maybe you know or Lang uses some generalized Ballas0:27:14trees I think which are interesting and there's those trees that use randomization techniques for balancing and and other things these are different in particular they're different because0:27:24they are their trees with the I some people call them tries trees the the idea behind that is that you're not going to have a fixed path down to a0:27:35leaf you're going to use only as much of a path as you need to produce a unique leaf position you usually see these things in like string search things or0:27:44maybe I think they're also using like Internet routing tables and stuff like that but here the model is very simple we want at least I wanted for closure something equivalent to a hash table I0:27:54know I can't sell closure to Java programmers if it doesn't have something equivalent to a hash table they don't want to hear about a red-black tree they've know that they know it's okay0:28:03but it's not as good as the hash tables they're used to they need something faster than that and these are the way they the way they work is that you hash0:28:12the value you want to put into the collection you end up with a 32-bit hash you're going to use the first five bits of that hash to see if there's a unique0:28:21position in the first layer of your tree so effectively what's happening is you have a 32 way branching going on in this tree in addition there's some fancy bit0:28:32twiddling going on in each node so that those nodes are sparse they're not fully populated so you're not wasting the space of of not fully populated nodes0:28:41and abusing a combination of population you know bit pop and some algorithms you copy out of what's that hackers factors0:28:52the light by that or you can just use closures in fact closures the vectors which is the same kind of technology it's been ported to factor0:29:02and Scala already which is fine by me so if it's unique in the first five bits we're done we put it in the first level0:29:11of the tree if it's not we're going to look at the next five bits and walk down one more level to the tree until we find some unique position and that we're done we're gonna put the put the value there0:29:20the key thing about this is how deep can this tree get this one's the root so0:29:30down one two three four five six if you had whatever you know four billion things it so if branch is extremely fast0:29:39and you know you can get a million items in depth 3 it's very very shallow so the combination of it being very shallow and using this bit twiddling to walk through0:29:50the sparsely populated nodes in the intermediate levels makes it really fast so that's the representation now we only0:29:59need to talk about how do we make a changed version efficiently and the key there as is true for all of these things is structural sharing all functional0:30:09data structures are essentially recursively defined structurally recursively defined which means that you can make a new version that shares0:30:18substantial structure with the version you just had and that's the key to making efficient copies you're not copying everything you're copying a very little bit and I'll show you in a0:30:27picture in a second how a little bit you use since everything is immutable sharing structure is no problem nothing is going to change about the structure0:30:36that you're sharing which means it's safe for multi-threaded use it's safe for iteration you get none of this you know mutate it while iterating nonsense0:30:46so how do we share structure we use some technology called path copying again this is true for all tree data structures they all work exactly the same way which is if we ignore the0:30:57right-hand path here that's the tree I showed you before it has 15 leaves ok we want to add one under that red outlined0:31:07purple guy at the bottom we I want to add a new node and a 16th guy so what needs to happen is we need to make a copy of that note0:31:16obviously because we're going to be giving him a new child a copy of his parent and finally the route this copy0:31:26gets one additional child and the rest of the structure of the old version was shared so I said level three levels could hold a million items right 32 320:31:38times 36 I get that right that's 32,000 a lot well 3 levels down right 3 levels down from 1 if you count the route it's0:31:474 levels what however it populated this last level was making a new note here it's only ever gonna copy 4 items okay how's0:31:57that old tree looking good still fine we didn't touch it and this is the path we need to copy to make the new one which0:32:06looks like a new tree with this extra item if we're no longer referring to that route it will get garbage collected as will the things that we referring to0:32:16that are no longer referenced it's sort of it's kind of basic but I wanted to show it because a lot of people just are not aware that this is a possible thing this is the kind of data structure I0:32:25think you should be using all the time unless you have some emergency reason and that's why closure works this way all the data structures work like this0:32:34by default you have to you know go through extraordinary efforts to pick something else ok so that's a way to efficiently represent composite objects0:32:43as values we've got one part of the problem solved now we need to talk about coordination methods the conventional way is not really a method conventional0:32:52ways it's your problem ok we saw in the Scalla talk there was a bar it didn't have volatile semantics but it happened0:33:01to be the case that the actor's library in Java conducts some synchronization thing which causes it happens before happens after memory fence effect in0:33:10order to make sure the contents in that bar was valid in another threat you know in your own program you're gonna have that's going to be your worry okay it's0:33:19nice that the actor's library takes care of bars in actors but in bar in your program otherwise are your problem and typically if we're trying to0:33:28do composite objects we have to use locks and everybody knows the problems with locks I think everybody know the problem with locks every know the pain of locks ok locks like they they experts0:33:40can build programs that work with locks but most people don't have the time or energy to do that well and maintaining it is really really difficult it's extremely difficult0:33:50so in closure we're going to just do is just add a level of indirection so directly referring to memory those variables we're going to use in direction and then we're going to add concurrency semantics to these0:34:00references if you watch me talk yesterday I said that but I'll show you some more details today so this quickly is the picture of the current state of the world and a lot of object-oriented0:34:09languages you have a lot of references to the same chunk of memory ok basically it's a free-for-all they don't know that they're going to see consistent object that all the parts are related to each0:34:19other and no one is twiddling with anything unless they can somehow stop the world but the core problem here now that we have lingo for it is this unifies identity and value right the0:34:29only place to put this value for this identity is in the same piece of memory that's a problem we just looked at how to do new values it's great except what0:34:38do we need to do we need to create some new memory right to represent that new value if we say all values of foo have to end up in the same chunk of memory space we can never do a good job so that0:34:52has a lot of problems how do you solve this you just use indirection it's the solution to all computer science problems right one level of indirection and now we have options ok because this0:35:04guy now could be immutable right we've separated the value which is now immutable and the identity which we're going to model with these little boxes0:35:15values never change right if you want to see the current state of an identity you have to dereference that you have to say give me your state what you get out of that is a value that can't change you0:35:26can spend all day looking at it just like you can spend all day looking at the photo to try to see if the runner was foul I want to emphasize if you0:35:36think your object into programming languages encapsulation techniques are a solution to this problem that is not true okay if you have a variable or a field inside your0:35:46object and you write three methods that can change that field that people can call those methods from different threads you haven't encapsulated anything from a concurrency standpoint okay you've just0:35:57spread the problem and hid it behind something so I'm going to call those boxes references we have too many overloaded terms I can't think of any0:36:07new words it's a reference because it's it refers to something else so identities are references that refer to their values but the critical thing is0:36:17in closure these are the only things that you can mutate unless you drop to Java and use Java stuff of course there's still classes and arrays and all that but if you want to follow the closure model you're gonna have these0:36:26references they are the only things that can change and what they do is just manage time okay in other words you can atomically move from one value which is0:36:36immutable to another value which is immutable and each of the reference types provides different semantics for time so what are the characteristics of0:36:46the semantics one of which is one of them is can other people see these changes I'm making is it shared okay because there's one way to manage time0:36:55which is the Star Trek alternate universe model where there's a bad Kirk in one universe time line and a good Kirk and another and they'll never meet0:37:05of course the problem with that is that occasionally they do meet but one way is isolation so we'll see the last model is isolation but in general most of these0:37:16models are around making changes that other parts of your program can see so sharing the second part is synchronicity and here we mean synchronicity in the0:37:26sense of now what now means to the caller in other words from a self relative standpoint is the change I'm asking for gonna happen now or at some0:37:36other time relative to me is it independent and we're gonna call those differences synchronous if it happens now relative to me it's synchronous it doesn't have now relative to me it's0:37:45asynchronous it's a sentence at some other some point in the future can't say exactly when and the final characteristic of these references where you know again you get0:37:55different choices and options is whether or not the change is coordinated okay I can be an independent runner and run all by myself and I'm completely fine but a0:38:05lot of times you need to move something from one collection to another collection right you don't ever want to be in both you don't ever want to be in neither okay that requires coordination0:38:15that's impossible to do with independent autonomous entities you need coordination and it ends up that in the local case you can do coordination0:38:24distributing coordination like this is you know a fool's errand probably but people keep trying I don't think that0:38:33there's ever going to be distributed coherent coordinated change but people are already recognizing the fact that you know if you're willing to delay consistency you can sort of have0:38:45coordination but in the local model it's perfectly possible to get coordinated change otherwise changes autonomous okay I change by myself I don't care what0:38:56you're doing we and no two of us can do something together so now we have these these four these three characteristics closure has four types of references0:39:05that have make different choices in these three areas refs are shared people can see the changes they're synchronous they change right now they change in the transaction which0:39:14means that you can change more than one reference in the same transaction and those changes will be coordinated sort of the hardest problem is that coordinated change problem0:39:23agents are autonomous they'll feel a lot more like actors in an actor model their shared people can see them they're asynchronous so you ask for a change0:39:33it's going to happen at some point in the future but you're going to immediately return and they're autonomous there's no coordinating the activities of agents atoms are shared0:39:44people can see the changes they're synchronous they have right now so that's the difference between them and agents and they're also autonomous you can't change more than one atom in a0:39:53single unit of work and finally closure has something called bars they isolate changes with the you know good Kirk bad Kirk alternate universe model0:40:03is only for any identity there's a unique value in every threat so you can't possibly see the changes in different threats I'm not going to talk too much about that that's kind of a0:40:12special-purpose construct it's arrived from lisps okay so one of the things that's nice about the way these references work is they have a uniform0:40:22state transition model all of them have different functions that change the state that say move from one state to another state and they use different names because they have different0:40:31semantics I know what people get all confused about is this happening asynchronously or synchronously or do any a transaction but the model is always the same you're going to take a call one of the changing functions0:40:41you're going to pass the reference the box and you're gonna say please use this function so you're gonna pass a function maybe with these arguments apply it to the current state of the box and use its0:40:53return value as the new state okay so the function will be passed the current state under some constraints either atomically within a transaction some way0:41:02it will be passed the current state it can calculate a new state again it's a pure function you're passing that new state becomes the new value of the reference you can always enclosures0:41:15references you can always see the current state of a reference by D referencing it in other words that's the local visibility because it's completely free to do and it yields much more0:41:26efficient programs to be able to do that if you have to ask for permission to see collections every time going to see them it doesn't work in the local context in0:41:36addition one of the other shared attributes of these things is that they there's no user locking you don't have any locking to do this work and none of these constructs can deadlock so what0:41:47does it mean to edit something in this new world you're going to have a reference to a value right we can make a new value Alucard on the side we're going to call a function create this new0:41:56value which we intend to become the new state of foo all right the new value is the function of the old it can share structure we just saw that doing this doesn't impede anybody who's0:42:05reading foo right they're completely free to keep reading they don't have to stop while we figure out the new version of foo in addition it's not impeded by people0:42:14reading we don't have to wait for people to stop reading so we start making a new version this is the kind of thing you're going to need for high-throughput concurrency and then0:42:23going to a new state is just an atomic swapping of this box to look at the new value the new immutable value that's always coordinated there's always rules0:42:33for how that happens I just showed you the multiple semantics anytime somebody references this after okay more time words after this happens they'll see the0:42:42new value consumers are unaffected okay if I was looking at the old value I don't get disturbed by this happening I'm just looking at an old value it's like I'm looking at a picture of the0:42:51runner to see I mean I know the race is over that's okay we need to behave that way if you've been programming for so long as I have that you know it's really0:43:00hard to break from I own the world and I stop the world the world goes when I say go and I mean we have to just break from that that's that's the future we have to understand that we're going to be going0:43:10to be working with data that is not necessarily the very latest data that's just the future for us okay so the hard0:43:20references as I said are there the transactional ones closure has a software transactional memory system I almost hate using this term because people like to criticize STM as if it0:43:29was one thing there's a whole bunch of different stands they have radically different characteristics closures is radically different from the other ones but they all share some things which is0:43:40basically a model that feels a lot like a database model you can only change them within a transaction all the changes you make to an entire set of references rafts inside a transaction0:43:49happen together or none of them happen okay that's atomicity you don't see the effects of any other transactions while you're running they don't see your effects so normal things the one unique0:43:59thing about SDM transactions is that they're speculative right you may not win somebody else may win and you will automatically retry up to a certain0:44:08limit which means that your transactions cannot contain side-effects this is the way you do coordination you can't really0:44:18do coordination without some technique like this you can't build a system on independent entities and and do this this kind of work so in practice what you do you just wrap your code with do0:44:27sync which just faint means this is a transaction there are two functions altering commute which worked like I described they take a function a reference in a function and some args and say apply this to the0:44:38reference in the transaction and make the return value of the new state internally closure uses multi-version concurrency control which I also think is a very critical component to doing0:44:47STM in a way that's going to work in the real world a lot of STM designs or you know you just write your app in the terrible way you were with your object-oriented language you know0:44:56banging on fields and STM is magically going to make that better I don't believe in that at all closure system is not designed for that kind of work if you make every part of0:45:06your object a ref it isn't going to work and I'm not going to feel bad for you because I just explained how to do it you make your object of value and atomically switch that value and0:45:15everything is better but you do have this issue of again like people would criticize STM is universally because most STM's do something called read tracking in0:45:25order to you know make sure that nothing bad happened while your transaction was going on they track every read that you do in addition to all the rights that you do I also believe that that is not0:45:34going to work so closure there's no read tracking the way it accomplishes that is with a technique called multi-version concurrency control which is the way Oracle and PostgreSQL work as databases0:45:44where essentially old values can be kept around in order to provide a snapshot of the world for transactions while other transactions that are writing can0:45:53continue then ends up being extremely effective but it falls out of this necessity to be using references to0:46:02values it's got to be cheap for me to keep an old value around for you right I just showed you how it is cheap if you're using persistent data structures all these things go together you don't0:46:12do all this stuff together you don't have an answer to this problem in my opinion but when you when you do this it's really nice so nbcc STM does not do0:46:21retracting so what does it look like in practice we defined food to be a ref that's a transactional box to that map0:46:30okay we can dereference foo and we see what's in there unfortunately the names order changes because their hash maps so they don't guarantee any of water of0:46:40iteration we can go and manipulate the value sighs Fuu we can say give me the map that's inside Fuu and and associate the0:46:49a key with Lucy that returns a new value right nothing about that impact of the reference when I took the value out I made another value okay we can do all kinds of calculations completely outside0:47:00of the transactional system it's still a functional programming language right get the value out and write functional programs so that didn't have any effect on Fuu okay we can go and we can use0:47:10that commute function which actually says you'll take a reference commute a reference with the function associative map the key a and the value Lucy and0:47:19that fails because there's the semantics to those rests which is that you can only do this for reps inside a transaction so you get an error if you0:47:30however you put that same work inside a transaction it succeeds and when the transaction is complete that is the value of foo I don't have a lot of time0:47:41to talk about the implementation details but again don't think that SCM is one thing if you've read one paper on SCM you know nothing about SCM if you've0:47:50read all the papers about SCM you know a little bit more than nothing about us to him none of us know anything about SDM this is still a research topic but I do0:47:59know this this works and it works really well and it makes it easy to write programs that don't use locks I think all the programs that I've written in my career I could have used this anytime I0:48:10needed coordinator change and it would have been fine people can bang on it and try to push the scalability issues and whatnot from a correctness standpoint this is a godsend however inside unlike0:48:23some STM's closures SDM is not spinning optimistic it does use locks uses wait notify it does not churn processes will wait for other processes it's got0:48:32deadlock detection it's got HBase barging this extreme minimum in fact I think what is actually the minimum amount of sharing and the transactional0:48:41system which is one calves which is that for the timestamps you know people have demonstrated you can have more on one calves continuously with eighty threads0:48:50and that's about the limit of scalability but when you actually have some work in your transactions it's no problem you know I've run stuff on an as little box with six of course and that Kaz is not going to0:49:00be the problem as I said there's no Reid tracking it is important that this SEM is designed for coarse-grained orientation it's not one of these snake0:49:10oil STM so you can do what you were doing you have to do this new thing you have to use references to immutable values then you can use my STM it's not0:49:19going to make your old programs good and the breeders don't then get impeded by writers and and and vice versa0:49:28it also supports community I don't really have time to explain right now I do want to show you one other model because it's very different and it's nice and that it's very different yet very much the same we've sort of0:49:38isolated change from values you can take a completely different approach to time okay so in an agent which is another kind of these reference cells each agent0:49:49is completely independent they have their own state and it cannot be coordinated with any other state changes through actions which are essentially0:49:58just ordinary functions that you're going to send to the agent with the function called send or send off that function is going to return immediately so you're gonna send this function and0:50:07some data say you know at some point in the future apply this function to the current value of the agent with these arguments and make the new return value of the function the new state of the0:50:17agent that happens asynchronously on a thread from a thread pool only one action per agent happens at a time so0:50:26agents essentially have sort of an input you know mailbox queue so they also do all their work serially so another promise of the semantics of of an agent0:50:35again as with the other reference types you can just dereference it and see what's in there if you do successive actions two agents inside the same0:50:44action they are held in to the action completes so they can see the new state the agents do coordinate with transactions which is kind of nice so0:50:53one of the problems is you saw no side effects and transaction so you're wondering you know how do I send let somebody know I completed this transaction successfully I need to send them a message or do something0:51:02side-effect II it ends up that if you send an agent action during a transaction that's held until the transaction commits so if the transaction gets retried those0:51:11just don't go out until the transaction actually succeeds so that coordination is a really nice feature these two things work together they're not quite0:51:20actors the difference with an actor model is that's a distributed model you don't have direct access to the state in an actor model because you can't because you can't distribute that since I'm not0:51:30doing distribution I can let you access the state directly which means it's a suitable place to put something that you actually may need to share a lot without0:51:39necessarily serializing activity so what does this look like to use I say def food to be an agent referring to a map i dereference it i see the contents of the0:51:49map I send that reference the same function associate a with Lucy I look at it right away it may not be there yet right some0:52:00amount of time will pass though I can't promise you what and then it will be different okay this is a different way of thinking about things with people who program in Erlang completely you do0:52:09amazing things thinking about things this way right things could be asynchronous you cannot keep probing your computer as if it was you know your old Apple and there was only you and your assembly0:52:19language and you know you were king of the universe things happen at the same time now Adams a very similar story to0:52:29agents right they're independent you can't coordinate change to Adams there's a different name for the state change function it's called swap again it takes an ordinary function of the old state to0:52:38the new state and the change happens synchronously now so that's the difference between Adams and agents happens right now this is a model for a compare and swap ok comparing compare0:52:50and swap or compare and set is a is a primitive that is going to let you look at a piece of compare and set memory and say I wanted to turn into this and it0:52:59will turn into this only if it's no longer l me if it is still back so you look at it you see it's that you want to turn into this if it's still that inside0:53:08atomically it'll say ok I'll make it this the problem with calves like by itself is you usually want to read the value do something with it and then put0:53:17it back and so you get this integral between when you looked at it and when you try to do the casts and of course when you do that and somebody else has done something that can is gonna fail I know what do you do well0:53:28typically a well-written casting where cast is a suitable data structure will have a little spin loop but you're gonna spin in a value into a cast well Adams0:53:37do the spinning for you as a result the function may be called more than once again we're in this world where you should be programming with these side effects free functions because they need0:53:47to be called more than once both in transactions and in atoms so you have to avoid side effects but the value you get out of this is that when you succeed you0:53:56know the function you applied was applied to the value of the function was passed and the result that got put in had no intervening activity occur on that atom that's a powerful construct0:54:06you need to have and look at these that looks like the other ones right to find food to be an atom that refers to that map right the reference it is there we0:54:16swap immediately we get the new value so this is uniform state transition model right that's what refs look like start a transaction commute or alter your raft0:54:27passing a function and some arguments the results of the functions your new value agents same thing except completely different time semantics it happens asynchronously in a thread pool0:54:36sometime later you return immediately atoms happen right now but are independent from the others you need all these things to write a real multi-threaded program especially in the0:54:46local model these are all things that I need to do in my career writing concurrent programs in the local part of the program and I don't think you can do without them so here they are but it's a0:54:56uniform way to go so in summary immutable values are critical for functional programming but is that they're also critical for state right we0:55:07cannot really manage time and state without immutable values if you're gonna let two things change time and ends value you're you can't do anything0:55:17that's reliable persistent data structures let you represent composite objects efficiently immutably once0:55:27you're able to accept this constraint of immutability on your values you have all these options I mean I'm working on a fifth reference type with different slightly different semantics it's easy0:55:37to do because of separated time management from value management and finally I think this is pretty easy to use if you've seen some other models this is a lot like0:55:46variables that work so thank you0:00:00Keynote The Value of Values - Rich Hickey
0:00:00 thanks very much for having me yes so today's talk is going to be the value of values and I like to start by0:00:14pulling the room how many people are in IT or an IT related field this is great the key is to start with an easy0:00:24question after the party so what does that stand for it stands for information technology and0:00:38one of the themes of this talk is going to be keeping in mind what information means and what we're actually trying to accomplish and looking at the tools and0:00:49technologies we're using and seeing if they are actually suitable for accomplishing what we're trying to do so I will start with that keyword because0:00:58the technology part I think is straightforward and we'll look at information and of course we'll start with the definition everybody knows this0:01:08is my stick right now if you're gonna do a talk you just pick a word and then you go look it up in the dictionary and and you're rolling and it's a it's a it's a0:01:18cheap trick but it's actually quite useful because there's a lot of the history of human thought sort of boiled down into into language so if we look at0:01:27the word information it's based on the word inform and inform actually means to convey knowledge via fax to to shape0:01:37your mind or to shape someone else's mind by communicating facts to them that's what it means to inform and the0:01:47key word I think in here again which is going to be a theme of this talk is the word facts because we're gonna try to give more precise meaning to that and see if our information technology0:01:57actually manipulates information because that's what information is information is those facts that are used to inform0:02:06and not anything else not any of the artifacts we use to to represent it so start again with you know what is the0:02:16fact no this is not not the dictionary definition so the fact is is a place where information is stored and there's0:02:28a there's a place for every piece of information and every fact has a set of operations like definitely get and maybe0:02:38set and although maybe if set is doesn't have the right controls for the fact there might be other kinds of operations0:02:47that would control that and then that's essential right that operations control how facts can change and and then we0:02:56want to communicate about facts we all we need to do is convey their locations right how many people are uncomfortable0:03:08right now I am I can't even keep a straight face with this slide are you kidding me this is not right this is very very wrong if if my partners0:03:20through Holloway's in the audience we probably almost had a heart attack well well the slide is up this is not what a fact is and yet a lot about that0:03:31description is similar to what our programs do so let's dig in to the word0:03:40place and a place means place means a particular portion of space and space is another word that's very interesting is going to come up later the the key word0:03:51here is particular and portion and the delimiting nature of this the same this other definition has that same characteristic right an area used for a0:04:02particular purpose you know a specific area and we're really comfortable with the notion of place right because we have two critical places we're0:04:11constantly manipulating with our programs one is memory and the other is you know the disk and they very much Co0:04:20align with place right they are places you know there's only so much memory and their particular justice in memory and there's only so much disk space and there's sectors on the disk and these are all places they're subdivisions of the0:04:30universe you know there's only so much in the universe that's on my harddrive there's only so much of the universe that's in in the memory of my computer0:04:39so I mean I want to look at what we are calling information systems now because we're building information systems and0:04:48in memory we're building the matter of mutable objects but mutable objects are actually abstractions of places right0:04:57there they they don't actually have meaning other than that there are little barricades we've set up in front of memory so that we don't have to directly0:05:07manipulate memory addresses anymore so have this abstraction that is an object that sort of helps us manipulate that place you know without too much0:05:17craziness and a key characteristic here is that objects have methods right they have those operations we talked about before that facts really don't have0:05:26objects critically had them there they're operationally defined and we use them to provide a layer of abstraction0:05:35over the places that our program uses and the same thing happens in storage right we again have you know tables and0:05:44documents and records and these these higher-level notions that fundamentally are born of the desire to again abstract0:05:54away the details of that fact that we're working with the place but the abstraction isn't really a first-class abstraction other than to hide place0:06:03from our programs actually isn't a sound abstraction above that and one of the ways you can tell that it's a place oriented abstraction is this update the0:06:14same notion of going to a particular part of the universe and manipulating it so these are what we were building information systems on right now and I think we may have some difficulty seeing0:06:27how that's correct so I have a new label for this it's called plop and plop is place oriented programming that's what0:06:36most of us have done for most of our careers and most of us continue to do and it's characterized by a very basic operation which is new information0:06:45replaces the old information it's that simple if that's happening you're doing place oriented programming it doesn't0:06:54matter if it's in memory doesn't matter if it's on disk if new information replaces the old you're doing place oriented programming and it doesn't matter if your implementation technology0:07:05is not actually doing that directly so I don't care if you're using MVCC or an append-only database if the logical result is that0:07:17new information replaces the old that's an in-place system even if you know for efficiency it depends on the disk if if the if in the end it can only give you0:07:26the most recent piece of information that is a place oriented system it doesn't matter if it's actually going back to the same disk set sector and there's a very good reason why we were0:07:38doing place oriented programming decades ago right computers the first computers were really tiny they had incredibly limited memories very very small discs0:07:49if they had any discs at all and so we had to do place oriented program there was no ways to get a computer to do anything useful unless we took the tiny0:07:59amount of memory that we had and completely mapped out the role of every place in in in terms of you know we defined our program in terms of the role0:08:09of every place of memory in helping our program accomplish what was supposed to accomplish guys still gave a talk let in the most recent couple of years where he0:08:19had this great anecdote about you know the worst program he had ever written and he held up a card showing what it was and then he described the computer on which I ran which had you know 4,0000:08:28words of memory and how you know there was this map of that memory and this part here was this dispatch table and then there was some code here but sometimes you could cram data into it then there was this other jump table and0:08:39then some data structures and every every program knew exactly where in memory those those portions were and directly use the addresses and that0:08:50that's how you had to do it and then we got bigger memories and we said that we don't really want a program with dresses anymore and so we added some stuff but the basis for the way we0:08:59computed was still that right we're still just trying not to do that but probably not deal with the the hassles of knowing addresses directly you know0:09:08we use some indirection there the the problem is that those constraints the constraints sky steel and the early pioneers of computing faced before him0:09:20they're gone you know computers just in the time I've been using them are a million times more capacious in memory0:09:29and disk then when I started which was after he started but we're still doing place oriented programming I think we0:09:38definitely need to consider why that is so one reason that always gets brought up right away is that there's this efficiency to manipulating places and0:09:49and that's definitely true and I'm not opposed to that I've bashed as many bits as as the next person and and I know how0:10:00much fun that is and I know how fast that can be and and I think that there there still is a role fo