Type system innovation propagation

2021-09-03T17:30:00+00:00

TL;DR🔗</a></h2>

Incorporation of established programming language theory approaches is desired by mainstream language designers.

The fashion in which parametric polymorphism has enabled generics in Java and Go demonstrates this.</li>
Go with generics has the potential to solve the expression problem.</li>
C++ has got it right straight away and work has been done to improve parametric polymorphism to allow for ergonomic higher kinded types (generic types that themselves accept type variables).</li> </ul> </li>
Incorporation of PLT approaches can be unified and applied to many different compilers</li>
Further work is required to further improve expressiveness and ergonomics of languages with type systems.

Most of languages with type systems lack scalable ways to deal with heterogenous data.</li>
Structure-aware features and row polymorphism asks for a wider adoption than just in PureScript.</li>
Lack of efficient structure-aware features algorithms holds back the adoption greatly.</li> </ul> </li> </ul>
Why not settle for naive or simple type systems?🔗</a></h2>
Most language designers agree that type systems should have first-class treatment in programming languages. Almost all the programming languages saw their type systems evolve to incorporate new features. In this post we'll study some of such cases and motivate the need for furthering type system R&D beyond what we have now at our disposal.</p>
To do that, we shall look at the history of two mainstream programming languages (Java and Go) through the lens of generic computing in said languages. In this post, when we talk about generic computing, we mean "ways to program in a type-agnostic way" or "writing a program that doesn't just work on one concrete type, but works on some class of types".</p>
Thus, generic computing is instrumental even to the most basic programming. Data structures (trees, arrays, ...) are foundational to the discipline and intrinsically generic. The challenge then, is to encode them in a type-safe way. A motivational example would be Java's "Hashtable", as seen in version 1.0, dated 7th of January, 1998.</p>
Razor-sharp generic computing 🔗</a></h2>
Consider `get</code> function from Java 1.0:</p>`
`public synchronized Object get(Object key) { </span> HashtableEntry tab[] = table; </span> int hash = key.hashCode(); </span> int index = (hash & 0x7FFFFFFF) % tab.length; </span> for (HashtableEntry e = tab[index] ; e != null ; e = e.next) { </span> if ((e.hash == hash) && e.key.equals(key)) { </span> return e.value; </span> } </span> } </span> return null; </span>} </span></code></pre>`
Considerations for the billion dollar mistake</a> aside, when we talk about type safety of this snippet, we see that, on line three of it, we call method `hashCode()</code> of an instance of class Object</code>. This approach to "generics" asks engineers to have a single point in the closed type hierarchy, which mandates all the necessary methods for the generic applications. This approach is a source of headache for library implementers. Even if we negotiate that using Interfaces is good enough for implementing generic programs (think, get</code> would accept IHashable</code> instead of Object</code>), the problems still exist.</p>`
`Upcasting (also known as generalisation, treatment of a subtype as a supertype) to an interface or an Object would result in the return value of a wider-than-needed type, which would require for downcasting (also known as specialisation, treatment of a supertype as a subtype) later on, throwing away type guarantees and creating a space for errors.</li>`Less significantly, overlapping abstract method names in interfaces without resolving facilities make generic programming via upcasting less scalable.</li> </ul> The pioneering language in the modern type systems engineering, which gave raise to Haskell and Ocaml is called "ML". ML, in mid-seventies, has introduced something called "parametric polymorphism", the idea of which is to let programmers have variables for types themselves in a similar way that programmers have variables for values. Modern Java's Hashtable uses parametric polymorphism and is said to be "polymorphic in key and value types":</p> public class Hashtable<K,V> </span>extends Dictionary<K,V> </span>implements Map<K,V>, Cloneable, Serializable </span></code></pre> Case study: type variables for better polymorphism🔗</a></h2> Generic Java🔗</a></h3> As we discussed, initial approach to generic programming in Java was to use Object, the common super-class for any Java class. Pizza language</a>, made by Odersky (eventually, the creator of Scala) and Wadler (co-designer of Haskell), released one year after Java, was a superset of Java that was a bit more principled and allowed for type variables that would then be "erased" and translated into Object class, automating upcasting and downcasting, thus retaining type safety. It also allows to remove the problem with exponential blow-up of compiled artefacts like the one seen in C++ due to conditional code generation. More on that later.</p> Type erasure is greatly misunderstood and some shortcomings of Java type system is misattributed to it, but it's not without its drawbacks. Most notably, one cannot use type variables in Java in to cast values to that type. I.e. (T)x</code> is not a valid expression if T is type variable. The other drawback of type erasure is that even if a generic data structure or method is parametrised with a primitive type, the overhead of boxing it (turning it into a Java class) will be carried via erasure. Note that none of the drawbacks of type erasure limit type safety, only expressiveness and performance.</p> Wadler et al., after Pizza was released, made a minimum viable formalisation of Java, which was instrumental for eventual inclusion of generics in Java in version 1.5, in 2004.</p> Generic Go🔗</a></h3> Go is infamous for the longest time between the release of an industrial language and getting generics. Importantly, it gave room for what I call void </code> polymorphism. In Go circa 2021, it's interface{}</code> polymorphism and, without going into much details about why it works, we'll present you with real code that makes use of it:</p> func ToBoolE(i interface{}) (bool, error) { </span> i = indirect(i) </span> </span> switch b := i.(type) { </span> case bool: </span> return b, nil </span> case nil: </span> return false, nil </span> case int: </span> if i.(int) != 0 { </span> return true, nil </span> } </span> return false, nil </span> case string: </span> return strconv.ParseBool(i.(string)) </span> default: </span> return false, fmt.Errorf("unable to cast %#v of type %T to bool", i, i) </span> } </span>} </span></code></pre> This is clearly problematic, because usage of interface{}</code> type in programs poisons them with runtime switching over type information, unlifting the failure detection from the realm of static analysis to the realm of dynamic monitoring. Furthermore, a slight change in the acceptable types shall cause a refactoring hell! There would be no way to know, when you extend domain of your interface{}</code> function, which other functions need to have their domain also extended.</p> Similarly to introducing generics to Java, introducing generics to Go included two stages: formalisation and implementation proposal. Given the experience of the team who is behind generics in Go experience in the matter (a lot of it is thanks to having Wadler on board), in case of Go, proper formalisation came first, it was implemented later.</p> Another reason for starting with formalisation first in case of Go, perhaps, is rooted in the fact that adding parametric polymorphism to Go is harder than doing so in Java. Indeed, one of the great features of Go language is that its struct-interface supertyping is open.</p> package s </span> </span>type Nil struct{} </span> </span>func (n Nil)Show() string { </span> return "{}" </span>} </span></code></pre> A structure with a function in a package defined independently can indeed happen to implement an interface defined in another package:</p> package main </span> </span>import ( </span> "fmt" </span> . "doma.dev/s" </span>) </span> </span>type Shower interface { </span> Show() string </span>} </span> </span>func f(a Shower) string { </span> return a.Show() </span>} </span> </span>func main() { </span> var x = Nil{} </span> fmt.Println(f(&x)) </span>} </span></code></pre> Further complication which warranted careful planning for this feature was that the goal was to use code generation (fancy word for which is "monomoprhisation" because poly-morphic things spawn a bunch of mono-morphic things), instead of type erasure, to achieve more versatile generics at the expense of binary size.</p> Finally, a proposal</a> that adds generics with constraints (which programmers can create and use in their code) was implemented. </p> An important outtake from this case study is that the same idea from programming language theory can be applied to completely different compilers following the same methodology!</p> Go and expression problem test🔗</a></h3> Besides, Generic Go, as currently implemented almost</em> passes the expression problem test.</p> The expression problem, essentially, states that without changing the existing source code in modules (except for the integration module) and while preserving type safety, codebase is extendable with:</p> a new type, implementing all existing functions;</li> a new function over all existing types.</li> </ul> The expression problem test is then formulated as follows:</p> Work with expressions for a calculator DSL that builds up arithmetic expressions and then evaluates them (hence the name of "expression problem").</li> Start with an expression type case "constant" which holds a value of some primitive numeric type.</li> Implement a function "evaluate" that takes an expression and returns the corresponding value of the primitive numeric type.</li> Implement "evaluate" for "constant".</li> Encode expression "plus" that denotes adding up two expressions.</li> Extend "evaluate" to work on it without changing other modules.</li> Implement "to string" function for both expressions ("plus" and "constant") without changing other modules.</li> In the integration module, demonstrate that any function is callable over any defined type case.</li> Erase all code for "plus" and "to string".</li> Reimplement "to string" first.</li> Reimplement "plus" second, then extending "evaluate" and "to string".</li> </ul> If generic constraint narrowing would be possible in Generic Go as implemented (it was planned to be possible in the original research), we would have been able to write the following code to solve the expression problem in Go:</p> // package A at time 0 </span>type ExprConst[T any] struct { </span> UnConst T </span>} </span> </span>// Currently impossible because receiver arguments have to have exactly the </span>// same type signature, including specificity of the type parameters, as their </span>// struct declarations. </span>func (e ExprConst[int]) Eval() int { </span> return e.UnConst </span>} </span>// end of package A at time 0 </span> </span>// package E at time 0 </span>type Evaler interface { </span> Eval() int </span>} </span>// end of package E at time 0 </span> </span>// package P at time 1 </span>type ExprPlus[L, R any] struct { </span> Left L </span> Right R </span>} </span> </span>// Currently impossible </span>func (e ExprPlus[Evaler, Evaler]) Eval() int { </span> return e.Left.Eval() + e.Right.Eval() </span>} </span>// end of package P at time 1 </span> </span>// package E at time 2 </span>type Evaler ... </span> </span>type Shower interface { </span> Show() string </span>} </span>// end of package E at time 2 </span> </span>// package A at time 2 </span>type ExprConst... </span> </span>func ...Eval() int... </span> </span>func (e ExprConst[int]) Show() string { </span> return strconv.Itoa(e.Const) </span>} </span>// end of package A at time 2 </span> </span>// package P at time 2 </span>type ExprPlus... </span> </span>func ...Eval() int... </span> </span>func (e ExprPlus[Shower, Shower]) Show() string { </span> return fmt.Sprintf("( %s + %s )", e.Left.Show(), e.Right.Show()) </span>} </span>// end of package P </span> </span>// package main at time 2 </span>type Expr interface { </span> Evaler </span> Shower </span>} </span>func main() { </span> var e Expr = ExprPlus[Expr]{ </span> ExprPlus[Expr]{ </span> ExprConst[Expr]{ 30 }, </span> ExprConst[Expr]{ 11 }, </span> }, </span> ExprConst[Expr]{ 1 } </span> } </span> fmt.Printf("%d = %s", e.Eval(), e.Show()) </span>} </span>// end of package main </span></code></pre> Then, when one would run this, the output would be 42 = ( ( 30 + 11 ) + 1 )</code>.</p> Quoting Robert Griesemer, one of the contributors to the FG paper and one of the main implementers of Generic Go</p> Even though we can type-check that, we don't know to implement it efficiently in the presence of interfaces (which would also have methods with corresponding type parameters).</p> </blockquote> Maybe some day...</p> More evidence of usefulness of R&D in type systems🔗</a></h2> There are many other examples that demonstrate adoption of programming language theory results in mainstream languages. To name a few:</p> Rediscovery of higher kinded types in C++ (something very little type systems allow for natively), and a long process of evolution to make them ergonomic.</li> Design and inclusion of higher kinded types into Scala by Martin Odersky.</li> Allowing for ergonomic higher order functions in C++ and Java</li> Function type treatment in mainstream languages, from Golang to Rust.</li> </ul> There is also an innovation that is on the verge of breaking through into mainstream languages.</p> Structure-aware type systems and row polymorphism🔗</a></h3> As we discussed</a>, type systems, by definition, limit the expressiveness of languages. And yet, they are well worth it as far as budgets are concerned. Let's start this post with exploring a classical expressiveness shortcoming of languages with type systems: the problem of operating on heterogenous data.</p> Imagine we need to store a hierarchy of countries and cities in the same tree. An untyped approach would be simple: make distinct objects for countries, cities, neighbourhoods and then add children</code> field to each, putting necessary objects on lower levels of the hierarchy:</p> let city1 = {"name": "Riga", "longestStreet": "Brivibas"}; </span>let city2 = {"name": "Zagreb", "longestStreet": "Ilica"}; </span>let country1 = {"name": "Latvia", "ownName": "Latvija", "capital": city1}; </span>let country2 = {"name": "Croatia", "ownName": "Hrvatska", "capital": city2}; </span>let city11 = {"name": "Zilupe", "longestStreet": "Brivibas"}; </span>let city22 = {"name": "Split", "longestStreet": "Domovinskog Rata"}; </span>let world = </span> {"name": "Earth", </span> "children": </span> [{...country1, "children": [city1, city11]}, </span> {...country2, "children": [city2, city22]}] </span> }; </span></code></pre> Naively, the same can be achieved by having a tree type, parametrised with a union type that encodes either a City or a Country.</p> data World = World { name :: Text } </span>data Country = Country { name :: Text, capital :: City } </span>data City = City { name :: Text, longestStreet :: Text } </span>data Value = W (World, [Country]) | C (Country, [City]) | T City </span></code></pre> However, quite some problems arise when we want to extend encoding to also capture streets, for instance. Our union type shall change along with type definition for City. This topic is far from being trivial to solve in a polymorphic fashion in typed languages. There is modern research</a> that shows that it's doable by introducing "pattern structures" into structure-aware type systems.</p> Relevant to the issue of heterogenity, solving problems such as capability tracking and diverse effect systems, is row polymorphism. It's another structure-aware approach to polymorphism, which is said to work on types with rows (records), and allows to define functions that are polymorphic in something except for some rows. In our example, a row-polymorphic function over our structure, could perhaps ask for any type for which name :: Text</code> is defined, along with, perhaps, non-zero other rows. It would then accept anything in our heterogenous structure, since everything is named. If it feels to you like this walks like duck typing and quacks like duck typing then yes, you are right. It is exactly a way to formalise duck typing and introduce it into the type systems. It is a common theme, however, that for PLT to be adopted in the industry, systems need to be engineered that implement the theory. But when you introduce one feature to a system, you trade off ease of introduction of other features (this is why we don't have and we will never have a universal language that is good at everything). In case of row polymorphism, the challenge is an efficient representation of records</a>. Gladly, default implementation of PureScript piggy-backs node.js efficiency. We expect row polymorphism to make its way into functional programming languages from already existing implementations in PureScript and an industrial laboratory language Ermine and eventually be adopted in mainstream languages.</p> Notable ommissions🔗</a></h2> It is hard to provide full survey of polymorphism and tangent topics in one little blog post. This is why we had to pick our battles. We have considered, but decided to ommit or mention just briefly, the following subjects (with links to introductory posts about them):</p> Importance of higher-kinded types</a>.</li> Using tagless representations to pass expression problem test</a> (tagless final for intermediate haskellers</a>).</li> Using polymorphic recursion for typing heterogenous data</a>.</li> </ul> Parting words🔗</a></h2> In most mainstream languages, existing facilities to boost expressiveness of type system is sufficient in majority of cases without sacrificing guarantees. If you find yourself needing more, sometimes introducing refactoring loops into your feature implementation process can be wise. In well-typed systems, refactoring is cheap and introducing such loops be detrimental to time to market compared to using untyped approaches. That said, for the sake of accepting many potential architectures that would be possible if type systems were more rich, we need to press on as a community and create compilers that take novel research ideas or ideas from other languages in a continuous struggle to unify those into ergonomc systems. Furthermore, along with regaining expressiveness, this work often is capable to tighten the compile-time guarantees</a>. More about it in the upcoming blog post.</p> All in all, we think that exploration of repeated success of adoption of parametric polymorphism by mainstream languages does good enough job to motivate businesses to look at the proceedings in the field!</p>

Why type systems matter 2021-08-26T16:02:02+00:00 TL;DR🔗</a></h2> Computer programmers expect their language environments to reject bad programs. It's largely done with lightweight formal methods.</li> Runtime monitoring (the thing that tells you that undefined is not a funciton</code>) is an example of such a formal method.</li> A type system is also an example of such a formal method.</li> Type systems target a range of properties deemed as "bad", for which it is guaranteed that programs having those are rejected.</li> This may and does result in "good" programs having "bad" properties being rejected. We say that it means that type systems limit expresiveness of a language.</li> "If it compiles, it works" is a dangerous misconception, but there is a synergy between correctness and passing a type check.</li> Alternative to using type systems, however, is less feasible for adequately-budgeted developments because it moves error detection to later stages of system's lifecycle, resulting in significantly larger costs.</li> Type systems are also great for rapid prototyping, technical validation and deriving specifications from requirements.</li> Type systems allow you to test for only truly run-time faults, which are more often than not related to side-effects.</li> If your organisation can cope with the possible friction that type systems bring, not using type systems is irresponsible.</li> </ul> A fistful of formal methods🔗</a></h2> We want our programming language environments at large to be able to tell well-behaved programs from those that behave poorly. There are several ways to achieve this.</p> Runtime monitoring, which considers things like operations on incompatible objects (a la Python and JavaScript) and underappreciated contract programming based on preconditions and postconditions, as well as invariant checking (a la Eiffel</a> and DLang</a>).</p> </li> Some will remember model-driven engineering with UML modelling (my Vim highlighted "UML" as a non-existent word! It brings me joy). Automatically deriving constraints from such models and rejecting models that are self-contradicting or breaking some constraints. (a la EMFtoCSP</a>).</p> </li> Both digital and analog</a> circuits can be accepted or rejected based on automatically derived finite state machine models and checking for desired properties.</p> </li> Type systems for rejecting classes of poorly-behaved programs statically, during compilation (a la Java, Haskell).</p> </li> </ul> A rather interesting observation is that these discriminators should be reproducible, which calls for underlying formalisms. Furthermore, it's preferred that domain experts (JavaScript programmers, UML architects, embedded systems engineers) can reap benefits from those. That property is called "lightweight" in culture. When we put these considerations together, we see that all of these things, including JavaScript's runtime monitoring, which many people may deem as basic, are lightweight formal methods! Not scary at all.</p> Not all formal methods, however, are made for the same reason and not everything achievable with one can be achieved with another. To illustrate that, consider the following use-case: we build up an array of validation functions and then, at the validation site we call them one by one.</p> 01 | let module1 = { </span>02 | defaultValidators: [ </span>03 | (x) => 2 == x.split(' ').length, </span>04 | ], </span>05 | validate: (input) => (fs) => </span>06 | fs.reduce((acc, f) => acc && f(input), true), </span>07 | }; </span>08 | </span>09 | let module2 = { </span>10 | alsoCapitalised: [ </span>11 | (x) => </span>12 | x.split(' ').reduce( </span>13 | (acc, x) => acc && (/[A-Z]/.test(x[0])), true </span>14 | ) </span>15 | ] + module1.defaultValidators, </span>16 | } </span>17 | </span>18 | let main = { </span>19 | main: (input) => { </span>20 | let validators = module2.alsoCapitalised; </span>21 | if (module1.validate(input)(validators)) { </span>22 | console.log("It's time to open the door"); </span>23 | } </span>24 | } </span>25 | } </span>26 | </span>27 | main.main("Viktor Tsoi"); </span></code></pre> When we run this code, the following error will be reported:</p> Uncaught TypeError: fs.reduce is not a function </span> validate debugger eval code:6 </span> main debugger eval code:21 </span> <anonymous> debugger eval code:27 </span></code></pre> The true place where the error happens is line 15. Getting there from lines 6 and 21 would probably require a little bit of debugging, especially in a real project. Indeed, the error happens due to nonsense operation +</code> over two arrays. Let's fix it:</p> 15 | ].concat(module1.defaultValidators), </span></code></pre> When we run the code again, we get the expected message in the log:</p> It's time to open the door </span></code></pre> Let's compare runtime monitoring with a type system.</p> Here's the code with the same error in Haskell.</p> 01 | {-# LANGUAGE OverloadedStrings #-} </span>02 | import qualified Data.Text as T </span>03 | import Data.Text( Text ) </span>04 | import Data.Char( isUpper ) </span>05 | </span>06 | defaultValidators :: [Text -> Bool] </span>07 | defaultValidators = [\x -> 2 == (length $ T.splitOn " " x)] </span>08 | </span>09 | validate :: Text -> [Text -> Bool] -> Bool </span>10 | validate input fs = foldl (\acc f -> acc && f input) True fs </span>11 | </span>12 | alsoCapitalised :: [Text -> Bool] </span>13 | alsoCapitalised = [\x -> foldl (\acc w -> acc && (isUpper $ T.head w)) </span>14 | True </span>15 | (T.splitOn " " x)] + defaultValidators </span>16 | </span>17 | main :: IO () </span>18 | main = </span>19 | case validate input defaultValidators of </span>20 | True -> putStrLn "It's time to open the door" </span>21 | _ -> putStrLn "Close the door behind me" </span>22 | where </span>23 | input = "Viktor Tsoi" </span></code></pre> When we try to compile it, we're going to get an error that says exactly what is wrong. To be able to apply +</code>, operands had to be classified as numbers via typeclass Num</code>. This typeclass doesn't include lists of validator functions. Note that if we would want to define addition on such values, we would be able to, by providing an appropriate instance of Num</code>. But it's a rather horrible idea, so we'll fix the bug instead.</p> /tmp/hi.hs:13:19: error: </span> No instance for (Num [Text -> Bool]) arising from a use of '+' </span></code></pre> With this error, we quickly can replace line 15 with one using ++</code>, the list concatenation operator:</p> 15 | (T.splitOn " " x)] ++ defaultValidators </span></code></pre> When we run this one, we get the correct result!</p> Static checking allows us to catch many classes of errors like this one early, which saves a lot of money, since—as systems engineering teaches us—the cost of fixing a fault in a system grows exponentially as a function of how far it is from the requirement gathering stage in the lifecycle of a system.</p> Benefits of using type systems🔗</a></h3> Type systems come in different shapes and sizes: different type systems may be geared to eliminate different classes of incorrect programs. However, most type systems—collaterally—eliminate programmers' errors such as incomplete case analyses, mismatched units, et cetera. For example, if we get rid of line 21 in the Haskell listing entirely, in a well-configured GHC (which treats warnings as errors and warns about everything "suspicious"), we'll get the following error, indicating incomplete case analysis:</p> /tmp/hi.hs:19:3: error: [-Wincomplete-patterns, -Werror=incomplete-patterns] </span> Pattern match(es) are non-exhaustive </span> In a case alternative: Patterns not matched: False </span> | </span>19 | case validate input defaultValidators of </span> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... </span></code></pre> Of course, it's impossible to reap this benefit without a certain wit and discipline. I like to call that discipline "tight typing", many Haskellers call it "having as concrete data structures as possible" (it's an elaboration of the mantra "abstract functions, concrete data"). For instance, the DummyTag</code> type you have seen earlier is loose, because it is used in the places of the model accepting types with incompatible terms. Quoting my colleague:</p> Nothing prevents you to subtract height from pressure if both of those are encoded as Float.</p> </blockquote> Type checkers are powerful refactoring tools. Anecdotally, at work, once we had to restructure approximately fifty modules while separating part of those into a library and fusing arguments into structures in another part of those. The whole refactoring was done and released by one person in one working day. It would have been impossible without a type checker to ensure the completeness of said refactoring. Another anecdote comes from my colleague:</p> When I worked for AlphaSheets.com (a startup "acquihired" by Google), I refactored the whole codebase, threading through an App monad instead of IO. I did this for a week in part due to painful and regular rebasing onto the main branch. But when the thing was compiled, the only tests that failed were the ones covering a place that was stubbed with undefined.</p> </blockquote> Notorious for a steep learning curve and perceived feature development slow-down, languages with type systems—granted a certain engineering savviness—can serve as amazing facilitators of rapid prototyping. The goal of prototyping is often to figure out the best specification for the requirements at hand. Expressive type systems often allow encoding relationships between domain entities without writing complex business logic. If one views types as claims that something is possible and values of said types as proofs that it indeed is possible, this approach makes a lot of sense. Of course, mileage can vary and it works better the fewer side-effects there are in a system, but remember that you can simulate side effects via type encodings too! For instance, instead of figuring out a proper way to do cryptography while prototyping, we can encode an interface of a crypto-system and populate it with dummy functions, complying with it:</p> data PKC pass sk pk slip sig sigmsg enc plain cipher = PKC </span> { -- | Initial key derivation function </span> kdf :: pass -> (sk, pk, slip), </span> -- | Rederive with kdf </span> rekdf :: slip -> pass -> (sk, pk), </span> -- | Sign data with key @sk@ and produce a detached @signature@ </span> -- possibly containing @pk@ for verification. </span> sign :: sigmsg -> (sk, pk) -> signature pk, </span> -- | Verifies @signature@ container's validity. </span> verify :: sigmsg -> signature pk -> Bool, </span> -- | Encrypt data of type @plain@ to the key @pk@ and produce @encrypted@ </span> -- containing @cipher@. </span> encrypt :: plain -> pk -> encrypted pk cipher, </span> -- | Decrypts @cipher@, contained in an @encrypted@ container into @plain@. </span> decrypt :: encrypted pk cipher -> sk -> Maybe plain </span> } </span></code></pre> This is a model of public key cryptography. Populating this model with functions would, together with tests that verify correct behaviours and error handling of a cryptosystem, serve as proof of the possibility of a correct implementation of public key cryptography in Haskell. Perhaps more importantly, it would also provide an interactive specification for doing so, perhaps even in another language! Let's give an example of some functions slotting into that model.</p> data DummyTag = DummySK | DummyPK | DummySlip </span> deriving (Show, Eq) </span> </span>-- | Note that we abuse the fact that in dummy implementation </span>-- secret key and public key are both represented with a </span>-- DummyTagged type alias to embed secret key together with </span>-- the message. </span>newtype DummySigned msg key = DummySigned {sig :: (key, (key, msg))} </span> deriving (Show, Eq) </span> </span>-- ... </span> </span>type DummyTagged = (DummyTag, ByteString) </span> </span>-- ... </span> </span>dummyKdf :: ByteString -> IO (Maybe (DummyTagged, DummyTagged, DummyTagged)) </span>dummyKdf pass = pure $ Just ((DummyPK, pass), (DummySK, pass), (DummySlip, pass)) </span> </span>dummyRekdf :: DummyTagged -> ByteString -> Maybe (DummyTagged, DummyTagged) </span>dummyRekdf (DummySlip, x) pass = </span> go (x == pass) </span> where </span> go True = Just ((DummyPK, pass), (DummySK, pass)) </span> go False = Nothing </span>dummyRekdf _ _ = error "DummySlip expected in the 1st argument" </span> </span>-- ... </span> </span>dummySign :: (DummyTagged, DummyTagged) -> msg -> DummySigned msg DummyTagged </span>dummySign (verificationKey@(DummyPK, _), signingKey@(DummySK, _)) blob = </span> DummySigned (verificationKey, (signingKey, blob)) </span>dummySign _ _ = error "The first argument has to be a tuple of DummySK and DummyPK" </span> </span>-- | Note that we're not comparing public key with secret key </span>-- but rather compare embedded bytestrings which match if the keys </span>-- were derived from the same password by kdf </span>dummyVerify :: Eq a => DummySigned a DummyTagged -> a -> Bool </span>dummyVerify (DummySigned ((DummyPK, signedAs), ((DummySK, signedWith), signedWhat))) candidate = </span> (signedAs == signedWith) && (candidate == signedWhat) </span></code></pre> Now to keep prototyping, we just need to instantiate PKC data type with this collection of functions. Later on, when we switch to a real cryptographic system, we will use it to make another value of type PKC. If all the properties of the initial prototype were preserved, it shall serve as proof to the claim that "production public key cryptographic systems exist as modelled". We can and should verify that early, but not too early to slow the prototyping down. Similarly, HTTP client-server interactions can be modelled. After the prototype is completed, one will end up with a runnable, compilable specification for the software they're about to write. What's fairly amazing is that if the company doesn't want to switch from Haskell (or whichever strongly-typed language they were using to model) to another language for the actual product, they can repurpose this prototype for an incremental rewrite into an MVP!</p> Now to the last, but not the least important benefit of type systems! These days, usage of higher-order and anonymous functions is prominent even in more conservative ecosystems like Java's. Type systems greatly assist in reasoning about the code's overall behaviour. In general, type systems are one of many tools for writing self-documenting code. An illustration for those, who (like me) keep forgetting the order of the accumulator and an the iterated value in reducers:</p> Prelude> :t foldl </span>foldl :: Foldable t => (b -> a -> b) -> b -> t a -> b </span></code></pre> Yeah, if the language's ecosystem is tightly integrated with the underlying type system, amazing things are possible, from simple type signature lookups at your fingertips to full-blown type signature search engines like Hoogle</a> and Serokell's hackage-search</a>. Conversely, if the language ecosystem evolved independently from type system, such as TypeScript and Dialyzer extending their respective languages with success typings, likely, self-documenting benefits and improved discoverability will be way less pronounced, if at all.</p> Type systems also enable and encourage writing composable code, meaning that the programmers are nudged towards writing well-structured and modular codebases, no matter what is the unit of modularity: a Java class or a Haskell module. Of course, there are ways to write spaghetti code in any language, but it's harder when the whole ecosystem imposes structure.</p> Drawbacks of type systems🔗</a></h3> Type systems aren't entirely free. Both the users (programmers) and the computer itself have to do extra work to write a program that would be accepted by a language with a type system. Sometimes, that work takes non-trivial amounts of computational time, but in practice, it's seldom more computationally intense than, say, code generation via templates.</p> Also, not quite a drawback, but rather something many people don't understand about type systems. There's an "if it compiles, it works" meme, but it's extremely misleading. Type systems, by their static nature, can't prove the presence of features of a program, only absence. But this yields a couple of actual drawbacks:</p> With type systems, you almost always pick your fights. Languages geared towards eliminating one kind of bad program behaviours won't eliminate another, perhaps, side-stepping it altogether by deferring it to the language runtime.</p> </li> Scrutiny, thus power, of type systems is always in tension with expressiveness, which is understood as the measure of the programs that are well-behaved at runtime, which are rejected by the type checker.</p> </li> Sometimes complex type checking algorithms rely on heuristics, which limit expressiveness in ways, unexpected by the user</a>.</p> </li> </ul> Bottom line🔗</a></h2> We'll end this article with a nice heuristic for architects to think about using type systems. Perform the following thought experiment:</p> Approximate how much would it cost to write, deploy and run a 100%-coverage random testing system, searching for runtime errors (in our JS example, it would be a test making sure that there is no validator crashed by a string and there is no call in the program that crashes it).</p> </li> Approximate how much would it cost to train developers to use a "tight" type system, which would give the same sort of guarantees "for free" and in static fashion (many different kinds of tests are still needed in this approach, but the tests we've described in p.1 are given to us for free by the type checker).</p> </li> </ol> If cost 2 is even comparable to, let alone lower than, cost 1, going for using type systems to refuse incorrect programs is warranted. Quoting "The Toyota Way":</p> Focusing on quality actually reduced cost more than focusing only on cost.</p> </blockquote> Getting things done with shell scripting 2021-04-09T14:25:00+00:00 TL;DR🔗</a></h2> When to resort to shell scripts Portability is important</li> Problem at hand is compact</li> File system interaction</li> Command-line program automation</li> </ul> </li> When to search for an alternative Extensibility is required</li> Coding mission-critical stuff</li> </ul> </li> Shell scripting is dangerous, use shellcheck and limit yourself in idioms used</li> </ul> Computer consoles are the second most important UX improvement in computing, surpassed only by window managers. Consoles went a long way from allowing the computer user to enter programs to be executed by a single-process operating system to converting them into toolboxes. Gladly, most of it was happening in Bell Labs under the supervision of the unstoppable innovator Douglas McIlroy</a>.</p> The Multics "shell"</a>, just like inputs on other terminal-enabled computers of that era, was an instrument to accept a program and execute it on the Multics OS, the predecessor of UNIX. The earliest versions of the Multics shell already had input/output redirection. It wasn't until Douglas McIlroy discovered (not invented)</a> command pipelines, it became a golden standard across all the operating systems.</p> Pipelining and UNIX philosophy is allowing for writing small problem-specific programs that can be later composed. But the biggest reason for the usage of UNIX shell scripts in 2021 is portability. Indeed, every modern system has a UNIX shell readily available. Furthermore, often making a reasonably well-written shell script is "good enough for the job". But how does one do that? Let's explore the answer to this question, assuming that the reader already knows the very basics of shell scripting</a>.</p> Minimal shell scripts in bash🔗</a></h2> Let's get the most important consideration out of the way. Writing secure and reliable shell scripts is almost impossible</a>. The least one can do is to use shellcheck</code></a>, which has integration with VSCode</a>. Furthermore, be very disciplined with user inputs and do your best to quote every argument that has a variable in it.</p> Alright, with that out of the way, let's talk about step-by-step items one has to do to make a reasonable shell script.</p> Setting up your shell environment🔗</a></h3> Before starting programming shells, we need to first determine which shell scripting language will we use. There are three schools of thought about this:</p> Use bash</code> by default. Bash is a reasonable middle ground between having a lot of features and being portable since it is shipped with each of the popular OS.</li> Use sh</code> by default and bash</code> when advanced features are needed. This is a purist approach. It offers the most portability but requires distinguishing between basic features and bash-exclusive features.</li> Use zsh</code>, fish</code> or some other "hipster" shell for everything. I'm mentioning this school of thought for completeness. Since it breaks portability, people who pick this option may just as well code in Python.</li> </ol> As one can guess, we suggest simply using bash</code> for everything. Of course, Apple seems to be deprecating bash</code> as the default shell, but it's not going anywhere from mac os systems. Conversely, Windows 10 has support for reasonable bash</code> integration with its WSL programme. It requires some setup, but these days WSL2 seems to become the default for Windows development.</p> Besides, bash</code> scripts have fine-grained built-in support for lowering the impact of the inevitable bugs. While setting up your shell, we suggest you use the following options</a>:</p> #!/usr/bin/env bash </span>set</span> -euo pipefail </span></code></pre> Optionally add -x</code> for easier debugging.</p> Command line argument processing🔗</a></h3> If you, for some reason, want to use shell to write something huge like a full-blown issue tracker with git backend</a>, you'll need to use getopts</code></a> or make your own despatch system</a>. In this post, however, we shall consider simple argument processing. We heavily advocate for short and concise scripts that do one thing and one thing only, after all.</p> First things first, let's see how to print help:</p> if </span>[[ </span>"$</span>1</span>" </span>== </span>"--help" </span>|| </span>"$</span>1</span>" </span>== </span>"-h" </span>]]</span>; then </span> </span>cat </span><<EOH </span>frgtmv: for each file read from STDIN, forget its filename entirely or amend part of it. </span>... </span>''frgtmv'' will then ''mv'' each of these files to ''</span>\$</span>(date +'%Y%m%d%H%M%S%N')'', preserving the file extension. </span>... </span>EOH </span> </span>exit </span>fi </span></code></pre> Key points:</p> We're using <<EOH / EOH</code> "heredoc" syntax and make sure that we don't indent the lines under it.</li> We're escaping special characters like $</code> with a backslash. If we wouldn't, bash would evaluate internals in a subshell.</li> Don't forget to exit</code> after printing the help!</li> </ol> Now let's use -n</code>, a predicate checking if a variable is set, to prepare the variables needed. Often we want to set up defaults at the beginning of the file:</p> _mode</span>=</span>"forget" </span>_pattern_from</span>=</span>"" </span>_replace_with</span>=</span>"" </span> </span>if </span>[ </span>-n </span>"$</span>1</span>" </span>]</span>; then </span> </span>_mode</span>=</span>"amend" </span> </span>_pattern_from</span>=</span>"$</span>1</span>" </span>fi </span> </span>if </span>[ </span>-n </span>"$</span>2</span>" </span>]</span>; then </span> </span>_replace_with</span>=</span>"$</span>2</span>" </span>fi </span></code></pre> Sometimes you would need to exit if an argument is not supplied. We use -z</code> that checks if a string is not empty for this:</p> # Exit if file or directory is not submitted or not a valid file or directory </span>if </span>[ </span>-z </span>"$</span>1</span>" </span>]</span>; then </span> </span>echo </span>"We really need the first argument" </span> </span>exit</span> 228 </span>fi </span></code></pre> STDIN, pipes and GNU parallel🔗</a></h3> Sometimes you need to receive input from STDIN through a pipe or user input. It's done using read -r</code>:</p> while </span>read </span>-r</span> _x</span>; do </span> </span>mv</span> -v </span>"$</span>_x</span>" "$(</span>date</span> +'%Y%m%d%H%M%S%N').${</span>_x</span>#*</span>.}" </span>done </span></code></pre> If you care about performance more than about portability, use cat -</code> to pass your STDIN to GNU parallel, following this pattern:</p> function </span>forget</span>() { </span> </span>mv</span> -v </span>"$</span>2</span>" "$(</span>date</span> +'%Y%m%d%H%M%S%N').$</span>1</span>.${</span>2</span>#*</span>.}" </span>} </span>export </span>-f </span>forget </span># (A) </span> </span>if </span>[[ </span>$</span>_mode </span>== </span>"forget" </span>]]</span>; then </span> </span>cat</span> - </span>| </span>parallel</span> forget {%} {} </span># (B) </span>fi </span></code></pre> In (A)</code>, the parallel payload is implemented as a bash function. Notice the export</code> statement! We suggest writing payloads that receive two variables: the parallel job ID ({%}</code>) and the currently read out item from the STDIN stream ({}</code>). The payload is called from (B)</code>.</p> Here are some interesting parallel</code> techniques:</p> --keep-order</code> to guarantee that the order of the input will be kept. Requires several file handles per input, which may turn out to be a bottle-neck.</li> find . -print0 | parallel -0 f {}</code> to work in null-terminated mode.</li> parallel 'echo "{%}:{1}:{2}";' ::: 1 2 ::: a b c</code> will prarallelise a Cartesian product of input sets {1,2} × {a,b,c}.</li> </ul> Bash parameter expansion🔗</a></h3> Confusing many, if not all new shell users, "parameter expansion</a>" has a veil of mystery around it. In our humble opinion, there are several causes for this effect:</p> Parameter expansion is a blanket name that unites accessing values and substituting values.</li> Within these use-cases, there is a myriad of conditional behaviours and they are decided based on the kind of parameter.</li> There isn't much expansion going on. Word "expanded" is simply an arcane way to say "reduced to a value".</li> </ol> Let's start from the beginning. A parameter in bash is either a variable (like $HOME</code>), a positional "argument" parameter (like $1</code>), or a special parameter (like $@</code>).</p> "Expansion" is the process of reduction</a> of parameters to values. Variables expand in the way one would expect from variables. Special parameters</a>, however, can have context-dependent expansions. Expansions have special syntaxes to tack on additional computations like string substitution, length calculation, etc.</p> Let's use the following variable as an example: x="a.b.c"</code>. Here is a list of the most often used parameter expansions, according to us:</p> Stripping. Use-case: get a file extension or remove a file extension. ${x%.*} ≡ a.b</code></li> ${x%%.*} ≡ a</code></li> ${x#*.} ≡ b.c</code></li> ${x##*.} ≡ c</code></li> </ul> </li> String replacement. ${x/./\!} ≡ a!b.c</code></li> ${x//./\!} ≡ a!b!c</code></li> </ul> </li> Array enumeration with IFS</code> and [@]</code>. IFS="."; for v in ${x[@]}; do echo -n "($v)"; done ≡ (a)(b)(c)</code></li> </ul> </li> </ol> You can also construct arrays with a "compound assignment":</p> x</span>=</span>(a b c) </span>for</span> v </span>in </span>${</span>x[</span>@</span>]</span>}</span>; do </span> </span>echo </span>-n </span>"($</span>v</span>)" </span>done </span></code></pre> IFS manipulation is not normally needed. If you're changing the splitting context, you should see if there is another way. You might be falling victim to the XY problem</a>.</p> This tutorial should provide good-enough techniques to quickly and effectively implement shell scripts that do what you want them to. Quote your variables, fail fast, make backups to recover from destructive changes, don't use too many "advanced features" since they are error-prone, and good luck!</p> We leave you with some shell scripts we wrote that push the boundaries of what shell should be used to:</p> A chess clock made in bash</a></li> A script I use daily: spawn or reattach to given project's tmux</code> session</a></li> Auto-detect canonical path to the script and eval modules residing in include/</code> directory</a></li> Terminate</a> process tree</a>, printing debug output</a></li> </ul> If there will be community interest, we will take some time to cover extensible shell scripting in Haskell with Turtle. As usual, reach out to us on Twitter</a> or in the comments on Dev.to</a> or Medium</a> mirrors.</p> Parser combinators in Rust 2021-03-30T17:20:00+00:00 TL;DR🔗</a></h2> Don't use regular expressions for parsing</li> Parser combinators are a way to construct composable computations with higher-order functions Examples: many1(digit1)</code></li> alt((tag("hello"), tag("sveiki")))</code></li> pair(description, preceded(space0, tags))</code> </li> </ul> </li> </ul> </li> Parser combinators are easy to use to get results quickly</li> They are sufficient for 99% of pragmatic uses, falling short only if your library's sole purpose is parsing</li> </ul> Role of parsing in computing🔗</a></h2> Data processing is a pillar of computing. To run an algorithm, one must first build up some data structures in memory. The way to populate data structures is to get some raw data and load it into memory. Data scientists work with raw data, clean it and create well-formatted data sets. Programming language designers tokenise source code files and then parse those into abstract syntax trees. Web-scraper author navigates scraped HTML and extracts values of interest.</p> Informally, each of these steps can be called "parsing". This post talks about how to do complete, composable and correct parsing in anger</em>. What do we mean by this?</p> Parsing in anger</em> considers the problem of data transformation pragmatically. A theoretically optimal solution is not required. Instead, the goal is to write a correct parser as quickly as possible.</li> Composable parsing</em> means that the resulting parser may consist of "smaller" components. It can itself be later on used as a component in "bigger" parsers.</li> Complete parsing</em> means that the input shall be consumed entirely. If the input can have any deviations or errors, its author shall encode them in the resulting parser.</li> </ul> So how do we achieve it? Let's first talk about how to not</em> do it.</p> Forget about regular expressions🔗</a></h2> Thanks to the popularity of now perished Perl programming language, a whole generation of computer programmers was making futile attempts to parse non-regular languages with regular expressions</a>. Regular expressions are no more than encodings of finite-state automata.</p> Items over arrows are characters of {0, 1} alphabet. Circles are states, q1 is "accepting state". Arrows denote state transitions.</em></p> Non-deterministic finite-state automata can rather elegantly accept many non-trivial languages. Classical example is that no regular expression exists that accepts strings of form "ab", "aabb", "aaabbb", ... Equivalently, one can't solve the matching parentheses problem with a regular expression. The simplest stack machine is needed for that.</p> Stack automaton can be in several states at once. A state with no transitions "fizzles" on any input. (@\*</code> matches character '(</code>' with any stack state. ε@ε</code> matches instantaneously as the automaton gets to state p, but only if the stack is empty. The best introductory book for those interested in formal languages</a>.</em></p> Thus, regular expressions are nowhere close to providing enough facilities to work with context-free grammars. But they may be sufficiently powerful to clean data or extract some values, so why are we saying you shouldn't use them ever? Practicality reasons!</p> Let's take an example from some Regex Cookbook post (medium-paywalled link</a>). This way we know it's an actual approach used in the industry. Here is one of the regular expressions author offers:</p> ^(((h..ps?|f.p):\/\/)?(?:([\w\-\.])+(\[?\.\]?)([\w]){2,4}|(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\[?\.\]?){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)))*([\w\/+=%&_\.~?\-]*)$ </span></code></pre> Many can superficially understand what is going on here. This regex seemingly has something to do with links, but even when we resort to automated explanation</a>, things don't get much clearer. Well, according to the author, this regex is supposed to detect "defanged" URLs. Now let's see all ways in which it and any other sufficiently large regular expression fail.</p> It is wrong: it doesn't match https://ctflearn.com/</code> (notice zero-width spaces).</li> It requires external tokenising, so no plug-and-play: it doesn't match ␣https://ctflearn.com/</code> (notice leading space).</li> External tokenisation is specific to this expression: it doesn't match https://ctflearn.com,</code> (notice trailing comma).</li> It's impossible to fix it: matching optional characters around each printable character would turn it from a large and poorly readable piece of code into a huge and completely unreadable one. Your brain wouldn't even be able to guess h..ps</code> and f.p</code> bits.</li> It can't be used to extract values. Regexps don't "parse data into data structures". Rather they accept or decline strings. Thus, additional post-processing is required to make use of their output.</li> </ol> Regular expressions have intrinsic problems. To us, it means that only short expressions should be used. The author uses them exclusively with grep</code>, find</code>, and vim</code>.</p> These days, gladly, a better parsing methodology becomes mainstream with working libraries in all the popular languages. As you can guess from the title, it's called "parser combinators".</p> Step-by-step guide to composable parsing🔗</a></h2> In the spirit of our previous blogs, let's solve some practical task. Consider you have to write an interactive TODO application, the pinnacle of practicality. It specifies the following commands:</p> add ${some word}* ${some #hashtag}*</code> (appends item ID)</li> done ${some item ID}</code> (marks entry at item ID as resolved)</li> search ${some word or some #hashtag}+</code> (searches across entries, returns a list of matching item IDs)</li> </ul> Let's first define how will we represent parsed data, omitting the boring bits:</p> pub </span>enum </span>Entry </span>{ </span> Done (Index), </span> Add (Description, </span>Vec</span><Tag>), </span> Search (SearchParams), </span>} </span></code></pre> Now let's use the nom</code> library to enjoy expressive and declarative parsing. It has or used to have macro API and function API. Since in v5 of the library macro API was very glitchy, we shall use function API, which we have tested with v6.</p> We will be parsing the commands line by line. Begin with declaring</em> the top-level parse for a line and meet your first parser combinator: alt</code>.</p> pub </span>fn </span>command</span>(</span>input</span>: </span>&</span>str</span>) </span>-> IResult<&</span>str</span>, Entry> </span>{ </span>/* A */ </span> </span>alt</span>((done, add, search))(input) </span>/* B */ </span>} </span></code></pre> In (A)</code> is declared that our function command</code> is a parser. IResult</code></a> captures parsed type (in our case, str&</code>) and output data structure (in our case, Entry</code>).</p> In (B)</code> we combine three parsers: add</code>, done</code>, and search</code> with nom::branch::alt</code></a> combinator. It attempts to apply each of these parsers starting from the leftmost until one succeeds.</p> Now, let's have a look at the simplest parser out of the three:</p> fn </span>done</span>(</span>input</span>: </span>&</span>str</span>) </span>-> IResult<&</span>str</span>, Entry> </span>{ </span> </span>let </span>(rest, value) </span>= </span>preceded</span>( </span>/* A */ </span> </span>pair</span>(</span>tag</span>(</span>"done"</span>), ws), </span>/* B */ </span> </span>many1</span>(digit1) </span>/* C */ </span> )(input)</span>?</span>; </span> </span>Ok</span>(( </span> rest, </span> </span>Entry</span>::</span>Done( </span>/* D */ </span> </span>Index</span>::</span>new( </span>vec_to_u64</span>(value) ) </span> ) </span> )) </span>} </span></code></pre> The first combinator we see straight away is preceded</code>. It forgets parse (B)</code> and keeps only the output of (C)</code>. (B)</code> still will consume input, however! Generally speaking, it combines</em> two computations into a composition that runs both of them, returning what the second one returns. It is not the same as just running them in a sequence because here we build up a computation</em>, but we will run it later on!</p> Interestingly, if we were writing Haskell we wouldn't find "preceded" combinator in our parser library</a>. The reason is that what we described in the previous paragraph is called "right applicative arrow", or, as was coined during Ben Clifford's wonderful talk</a> "right sparrow</em>":</p> λ</span>> :</span>t </span>(*>) </span>(*>) </span>:: </span>Applicative</span> f </span>=></span> f a </span>-></span> f b </span>-></span> f b </span></code></pre> The other two combinators are pretty self-explanatory. pair</code> combines parsers into a sequence, with the ws</code> parser being a parser that consumes single whitespace. Here is a naive definition of ws</code>: one_of(" \t")</code>. many1</code> repeats a digit1</code> parse at least one time to succeed. digit1</code> is implemented in nom</code> itself.</p> Now let's solidify the understanding of how to make sure that our parsers can be used by others.</p> We have already discussed that to achieve that, we need to return IResult</code>. Now it's time to remember that it's still a "Result" type, so its constructors are still Err</code> and Ok</code>:</p> Err</code> variant of Result is constructed via the ?</code> modifier, that passes any potential error arising in parse (A)</code> through.</li> Ok</code> variant of Result is constructed in (D)</code> by transforming many1</code> output (which is a vector of digits) into an unsigned 64-bit integer. It's done with vec_to_u64</code> function, which is omitted for brevity.</li> </ul> The shape of Ok</code> value for IResult<in, out></code> is Ok((rest: in, value: out))</code>. Here rest</code> is the remaining input to be parsed, and value</code> is the output result of the parser. You can see that preceded</code> parse in (A)</code> followed the very same pattern.</p> Here are more advanced parsers, that should solidify your intuition about how to use parser combinators in anger:</p> fn </span>add</span>(</span>input</span>: </span>&</span>str</span>) </span>-> IResult<&</span>str</span>, Entry> </span>{ </span> </span>let </span>(rest, (d, ts)) </span>= </span>preceded</span>( </span>/* B */ </span> </span>pair</span>(</span>tag</span>(</span>"add"</span>), ws), </span> </span>pair</span>(description, </span>preceded</span>(space0, tags)) </span>/* A */ </span> )(input)</span>?</span>; </span> </span>Ok</span>( ( </span> rest, </span> </span>Entry</span>::</span>Add( </span>Description</span>::</span>new(</span>&</span>d), ts ) </span> ) ) </span>} </span> </span>fn </span>search</span>(</span>input</span>: </span>&</span>str</span>) </span>-> IResult<&</span>str</span>, Entry> </span>{ </span> </span>let </span>(rest, mash) </span>= </span>preceded</span>( </span> </span>pair</span>(</span>tag</span>(</span>"search"</span>), ws), </span> </span>separated_list</span>( </span> </span>tag</span>(</span>" "</span>), </span> </span>alt</span>((tag_contents, search_word)) </span>/* C */ </span> ) </span> )(input)</span>?</span>; </span> </span>Ok</span>((rest, </span>mash_to_entry</span>(mash))) </span>} </span> </span>fn </span>mash_to_entry</span>(</span>mash</span>: </span>Vec</span><SearchWordOrTag>) </span>-> Entry </span>/* D */ </span>{ </span>/* ... */ </span>} </span></code></pre> Parsing with combinators is so self-descriptive, it's hard to find things that need to be clarified, but here are a couple of highlights:</p> Repeat preceded</code> to focus on the data you need to parse out, see (A)</code> and binding in (B)</code>.</li> Sometimes, you have to parse heterogeneous lists of things. The best way to do that in our experience is to create a separate data type to enclose this heterogeneity (SearchWordOrTag</code>, in our case) and then use separated_list</code> parser over alt</code> of options, like in (C)</code>. Finally, when you have a vector of matches, you can fold it into a neater data structure as needed by using a conversion function (see (D)</code>).</li> </ul> This should be enough guidance for you to start getting comfortable with this amazing combinator-based parsing methodology. Here are some parting thoughts:</p> Pay close attention to whitespaces, which can be a little tricky, especially since we're not aware of the automatic tokenisation option in nom.</li> Take a look at choosing a combinator</a> documentation page for the version of nom you are using (NB! entries in this table are pointing to macro versions of combinators rather than function versions).</li> If you so choose, you can check out code truly written in anger</a>, which inspired the snippets in this blog post. The code is authored by Chris Höppner and Jonn Mostovoy.</li> </ul> If parsing is not your product or the main purpose of your library, odds are, parser combinators shall be sufficiently expressive and sufficiently performant for your tasks. We hope you liked this post and happy parsing!</p> If you have any questions, you can reach out to Jonn</a> and Pola</a> directly. Start the conversation in the comments of the mirrors of this article on Dev.to</a> and Medium</a>.</p> Pattern matching in Rust and other imperative languages 2021-03-18T07:28:17+00:00 TL;DR🔗</a></h2> Rust is an imperative language that has the most pattern-related language facilities Has both shallow destructuring and deep destructuring</li> if let</code> matching form can be utilised to alleviate Rust's lack multiple-head functions</li> </ul> </li> Python has the weakest support for pattern-related facilities but it's about to change Language support for pattern-matching included in alpha (Edit thanks to reddit)).</li> Has packing/unpacking operators</li> </ul> </li> JavaScript has a lot of pattern-related language features Position-based destructuring for arrays and key-based for objects</li> Rest parameters, supporting destructuring</li> Shallow-copy spread operator</li> With support from Microsoft, Facebook and NPM, proper pattern-matching in JS is inevitable</li> </ul> </li> C++ has powerful libraries for pattern matching. Language support is likely in C++23</li> </ul> All the time, ideas and approaches sift into the world of conventional programming languages world from the programming language theory research and functional programming world. Even Excel has lambdas now!</p> In this post, we shall cover pattern matching in various imperative programming languages. We shall help you adopt pattern matching techniques to boost the expressiveness and conciseness of your code.</p> An example from a C++ evolution proposal.</em></p> Pattern matching in Rust🔗</a></h2> Rust has the most advanced and well-designed pattern system among all imperative languages. Part of it, of course, can be attributed to the fact that developers of Rust had the luxury of building a language from the ground up. But most significantly, it stems from the rigour and culture of design and development.</p> Pattern matching facilities in Rust language are almost as rich as in its older functional brother Haskell. To learn about them along with us, first, consider the following task (inspired by a real-life use-case):</p> Explore a non-strictly-structured JSON object where keys are species and values are sets of animals of these species.</p> If an animal's coat is fur or feathers, it's Cute</code>, otherwise it's Weird</code>. If a species is "aye-aye", it's Endangered</code>. There may be new criteria discovered later on that change categorisation of a particular animal or species.</p> Categorise animals with distinct name</code>s found in the given data set!</p> </blockquote> So let's start with encoding the categories:</p> #[derive(Hash, Debug, PartialEq, Eq, PartialOrd, Ord)] </span>/* A */</span> </span>pub </span>enum </span>Category </span>{</span> </span> Cute,</span> </span> Weird,</span> </span> Endangered,</span> </span>}</span> </span></code></pre> (A)</code> makes sure that Rust will order values from top to bottom, so that Cute < Weird < Endangered</code>. This ordering will be important later on.</p> Now to encode the rules from the task. Since our JSON is unstructured, we can't rely on any property existing, so we can't safely unwrap</code> or reliably coerce JSON to some data Rust data structure:</p> fn </span>cat_species</span>(</span>v</span>: </span>&</span>str</span>) </span>-> Category </span>{</span> </span> </span>match</span> v </span>{</span> </span> </span>"aye-aye" </span>=> </span>Category</span>::</span>Endangered, </span>/* A */</span> </span> </span>_ => </span>Category</span>::</span>Cute, </span>/* B */</span> </span> </span>}</span> </span>}</span> </span></code></pre> Our first match</code>! How exciting! This match is equivalent to switching over contents of variable v</code>, of course. However, it offers more flexibility later on. With the power of destructuring, we can match complex structures, not just single variables.</p> (A)</code> shows how to match a literal value, (B)</code> shows the "catch-all" clause. This pattern match reads species named "aye-aye" is endangered, other species are cute</em>.</p> Now let's have a look at how to write something more interesting:</p> fn </span>cat_animal_first_attempt</span>(</span>v</span>: </span>&</span>Value) </span>-> Category </span>{</span> </span> </span>match</span> v[</span>"coat"</span>]</span>.</span>as_str</span>() </span>{</span> </span> </span>Some</span>(</span>"fur"</span>) </span>| </span>Some</span>(</span>"feathers"</span>) </span>=> </span>Category</span>::</span>Cute,</span> </span> </span>_ => </span>Category</span>::</span>Weird,</span> </span> </span>}</span> </span>}</span> </span></code></pre> The rule of cuteness is satisfied, no unwrapping used. There are also no explicit checks</em> if the value has Some contents or it has None! This listing confidently states: animals with a fur coat or with a feather coat are cute, others are weird</em>.</p> But is this implementation good enough? One can check by considering a rule getting added, just as requirements warned us:</p> Animals that have the albino mutation are Endangered</code>. Otherwise, previous rules apply.</p> </blockquote> fn </span>cat_animal_first_attempt_1</span>(</span>v</span>: </span>&</span>Value) </span>-> Category </span>{</span> </span> </span>let</span> cat </span>= match</span> v[</span>"coat"</span>]</span>.</span>as_str</span>() </span>{ </span>/* A */</span> </span> </span>Some</span>(</span>"fur"</span>) </span>| </span>Some</span>(</span>"feathers"</span>) </span>=> </span>Category</span>::</span>Cute, </span>/* B */</span> </span> </span>_ => </span>Category</span>::</span>Weird,</span> </span> </span>}</span> </span> </span>match</span> v[</span>"mutation"</span>]</span>.</span>as_str</span>() </span>{</span> </span> </span>Some</span>(</span>"albino"</span>) </span>=> </span>Category</span>::</span>Endangered,</span> </span> </span>_ =></span> cat</span> </span> </span>}</span> </span>}</span> </span></code></pre> The snippet became bulky and boilerplate-y... We now have to thread some variable like in (A)</code>. We have to remember not to short-circuit computation in (B)</code> by adding a return</code> by accident. In case an additional rule pops up, we will need to decide between mutable cat</code> or versioned.</p> So is this it? Pattern matching collapses the moment we need to capture some heterogeneous set of matches? Not quite. Let us introduce if let</code> statement, made just for this sort of challenge:</p> fn </span>cat_animal</span>(</span>v</span>: </span>&</span>Value) </span>-> Category </span>{</span> </span> </span>if </span>let </span>Some</span>(</span>"albino"</span>) </span>=</span> v[</span>"mutation"</span>]</span>.</span>as_str</span>() </span>{</span> </span> </span>Category</span>::</span>Endangered</span> </span> </span>} </span>else if </span>let </span>Some</span>(</span>"fur"</span>)</span> </span> </span>| </span>Some</span>(</span>"feathers"</span>)</span> </span> </span>=</span> v[</span>"coat"</span>]</span>.</span>as_str</span>() </span>{</span> </span> </span>Category</span>::</span>Cute</span> </span> </span>} </span>else </span>{</span> </span> </span>Category</span>::</span>Weird</span> </span> </span>}</span> </span>}</span> </span></code></pre> Now that's more like it. But wait, what does it mean? As with other pattern matches, left hand side is a pattern (for instance, Some("albino")</code>) and right hand side is value (for instance, v["mutation"].as_str()</code>). A branch under if</code> will get executed when and only when the LHS pattern shall match the RHS value.</p> Pattern matching with if let</code> syntax makes us start with the most specific clause and fall through to less specific clauses in an unambiguous order, taking away excessive liberty and thus making the code less error-prone.</p> Putting it all together🔗</a></h2> pub </span>fn </span>categorise</span>(</span> </span> </span>data</span>: HashMap<</span>String</span>, </span>Vec</span><Value>>,</span> </span>) </span>-> HashMap<Category, </span>Vec</span><</span>String</span>>> </span>{</span> </span> </span>let </span>mut</span> retval </span>= </span>HashMap</span>::</span>new();</span> </span> </span>for </span>(species, animals) </span>in</span> data </span>{</span> </span> </span>for</span> animal </span>in</span> animals </span>{</span> </span> </span> </span>if </span>let </span>Some</span>(name) </span>= </span>(animal[</span>"name"</span>]</span>.</span>as_str</span>()) </span>{ </span>/* A */</span> </span> retval</span> </span> </span>.</span>entry</span>(</span>max</span>(</span>cat_species</span>(species</span>.</span>as_str</span>()),</span> </span> </span>cat_animal</span>(</span>&</span>animal))) </span>/* B */</span> </span> </span>.</span>or_insert</span>(</span>Vec</span>::</span>new()) </span>/* C */</span> </span> </span>.</span>push</span>(name</span>.</span>to_string</span>())</span> </span> </span>}</span> </span> </span> </span>}</span> </span> </span>}</span> </span> retval</span> </span>}</span> </span></code></pre> Now that we have categorisation functions, we can proceed to categorise our data set. If (A)</code> if let</code> match fails (current animal has no name supplied), we'll move to the next iteration. Not all the patterns have to have the catch-all arm.</p> Otherwise, the variable name</code> will store the current animal's name and we will chain some functions from a handy HashMap</code> API. In (B)</code> we use the Ord</code> instance of Category</code> enum to determine the highest priority category between species-based categorisation and per-animal categorisation with std::cmp::max</code> function.</p> Then HashMap</code>'s entry</code> returns the reference to the value under the category. If there is None, or_insert</code> in (C)</code> inserts an empty vector and returns a reference to it. Finally, we can push the name of the current animal to this vector, and it will appear in our mapping!</p> We hope that this guide provides a reasonable introduction to pattern matching in Rust. See the full code of the example module on sourcehut</a>.</p> Let's finish the post with some information about pattern-related features of other popular imperative languages.</p> Patterns in modern JavaScript🔗</a></h2> const </span>foldAndDump </span>= </span>(</span>path</span>, </span>xs</span>, </span>...</span>cutoffs</span>) </span>=> </span>{</span> </span> </span>// snip</span> </span> </span>for </span>(</span>c </span>of </span>cutoffs</span>) {</span> </span> </span>//snap</span> </span> }</span> </span>}</span> </span></code></pre> An old feature of ECMAScript, the JS standard called "rest parameters" ...cutoffs</code> will match arguments of a function beyond the second into an array called cutoffs</em>.</p> var </span>rs </span>= </span>[];</span> </span>for </span>(</span>let </span>[</span>printing</span>, </span>info</span>] of</span> </span> </span>Object</span>.</span>entries</span>(</span>allPrintingsJson</span>[</span>'data'</span>]))</span> </span>{</span> </span> </span>rs</span>.</span>push</span>({ </span>...</span>info</span>, </span>"_pv_set"</span>: </span>printing </span>});</span> </span>}</span> </span></code></pre> When the ellipsis isn't in the argument list</em>, it means that we're dealing with a newer feature called "spread syntax". ...info</code> means "include info</code> object as is". Analogously, spread syntax can spread an enumerable object across arguments of a function call:</p> const </span>xs </span>= </span>[</span>1</span>,</span>2</span>,</span>3</span>];</span> </span>console</span>.</span>log</span>(</span>sum</span>(</span>...</span>xs</span>));</span> </span></code></pre> Finally, there is unpacking, which is a pretty standard feature by now:</p> > [a,b] = [1,2]</span> </span>[ 1, 2 ]</span> </span>> {x,y} = {y: a, x: b}</span> </span>{ y: 1, x: 2 }</span> </span>> {k,l} = {y: a, x: b}</span> </span>{ y: 1, x: 2 }</span> </span>> [a,b,x,y,k,l]</span> </span>[ 1, 2, 2, 1, undefined, undefined ]</span> </span></code></pre> Packing and unpacking in Python🔗</a></h2> In modern Python, any iterable is unpackable:</p> >>> a, *b, c = {'hello': 'world', 4: 2, 'rest': True, False: False}</span> </span>>>> a, b, c</span> </span>('hello', [4, 'rest'], False)</span> </span></code></pre> *</code> is analogous to JS's ellipsis (...</code>) operator. It can collect some "the rest of the values", but it can also work as a spread for iterables:</p> >>> print(*[1, 2, 3])</span> </span>1 2 3</span> </span></code></pre> Conversely, in spirit of Python, there's a special case operator called "dictionary unpacking operator". It works very similar to spread operator:</p> >>> print({'x': True, **{'y': False}, **{'x': False, 'z': True}})</span> </span>{'x': False, 'y': False, 'z': True}</span> </span></code></pre> Rightmost spread precedes.</p> Pack your bags: we're going pattern matching🔗</a></h2> Every single language that is in active development is looking to adopt more and more features from functional languages, and pattern matching is no difference.</p> We'll conclude this post with a list of languages that will adopt proper pattern matching, ranked by degree of our certainty in adoption.</p> Pattern matching in Python🔗</a></h3> There were different proposals throughout the history of Python, but PEP 634</a> got implemented</li> Alpha version of Python with "structural pattern matching" is available since March 1st</a></li> </ul> Pattern matching in C++🔗</a></h3> Pattern matching as seen in this evolution document</a> is likely to land in C++23</li> While you wait, there's always a library or two that does a reasonable job</a> mimicking the new standard</li> </ul> Pattern matching in JavaScript🔗</a></h3> Tied for the first place in "the most likely to adopt proper pattern matching", JavaScript's standard called "ECMAScript", has this proposal</a> backed by Microsoft, Facebook and NPM.</li> The proposal is thoroughly reviewed and was moved to "stage 1", which puts the theoretical release of this feature in the 2023-2025 range.</li> You can check our maths by inspecting git log</code>s in completed proposals repository</a>.</li> </ul> The idea of pattern matching is to have a code execution branch based on patterns, instead of conditions. Instead of trying to encode properties of values</em> necessary for a code branch to get executed, programmers who use pattern-matching encode how should values look like</em> for it to happen. Thus, in imperative languages, pattern matching promises more expressive and declarative code compared to predicate statements such as if</code> and case</code>, bar some corner cases.</p> It might be a subtle difference, but once you get it, you add a very powerful way of expression to your arsenal.</p> We find that understanding these concepts is akin to the understanding of declarative vs imperative programming paradigms. To those interested in the philosophy of the matter, we suggest finding a cosy evening to curl up with a cup of steaming drink and watch Kevlin Henney's "declarative thinking, declarative practice" talk:</p> </iframe> Kevlin Henney: Declarative Thinking, Declarative Practice. ACCU 2016. Non-tracking YouTube embed.</em></p> Erlang and GNU poke are (better than) hex editors 2020-12-17T03:31:19+00:00 TL;DR🔗</a></h2> Simple binary patching and experimentation can be done with wxHexEditor</code></a>.</li> Erlang and Elixir are the only general-purpose programming languages that allow for binary pattern-matching, and should be used for functional and declarative binary editing.</li> GNU poke is a domain-specific language for "poking at binaries". Different binary units (bits, bytes, uints, ints, structs) are supported natively and a genius mapping operator removes a lot of boilerplate when it comes to reading data into semantic structures and writing data out to memory or files.</li> </ul> There are different ways to structure data within a file. In the age of web, the most prominent one is JSON, followed by middle-aged XML and configuration-oriented YAML. Human-readable ways to structure data, however, are just one side of the spectrum. The other side is pure binary data mapped to some data structures to be computer-readable. Features like bit fields in C</a> enable humans to define such mappings, but they also serve as a way to decode said structures. Wiki example, when compiled and disassembled</a> doesn't mention BoxProps</code> at all. Only when we add a function like the one below is included in the source code, would compiler spare some instructions to figure out where to get desired bits from.</p> bool </span>getShowBorder</span>(BoxProps </span>x</span>) </span>{</span> </span> </span>return</span> x</span>.</span>show_border</span>;</span> </span>}</span> </span></code></pre> Packing bits into structures warrants a whole separate post, but it's worth mentioning that bitfields is a controversial feature of C99 and many C developers suggest not using it in favor of using bitmaps and manually written packing / unpacking functions. One of the reasons for it is that it's not defined how the compiler packs bitfields, which means in practice, that some compilers reorder under-octet bitfields for the sake of optimisations.</p> Traditionally, non-human readable data is explored using hexdump</code> or, less commonly xxd</code></a> to print as binary instead of hex. Finally, there is od</code> which defaults to decimal representation. But what if we need to edit it slightly for experimentation</a>, error correction</a> or binary carving</a>? There is no single traditional tool for binary editing, but there's a range of tools, collectively called hex editors</em>.</p> Shortcomings of hex editors🔗</a></h2> Sadly, most of readily available hex editors have two major shortcomings:</p> They only work on bytes</li> They rarely handle huge binary files in a responsive way</li> Some of hex editors have stability issues, i.e. they crash a lot</li> </ol> The worst offenders for crashes are ghex and bless, with latter barely working at all. Which is funny, because these two editors are frequently suggested as the best out there.</p> Due to these circumstances, it's wise to stick to wxHexEditor</code></a> for very basic inspection/interaction and neglect hex editors altogether when it comes to advanced processing.</p> Hands on!🔗</a></h2> Let's say we want to extract 7-bit ASCII from this file</a>. To do this we need to take the last bit of every byte and shove it in the back of this binary. Here is a single-take attempt to do it with a hex editor:</p> </p> Even if one knows what to do, it's less than trivial to not screw it up, especially while trying to save time on insertions of zeros. However, a hex editor is a great instrument to quickly validate that the method works and move on to more automated tools.</p> Binary processing with Erlang🔗</a></h2> My favorite tool to do binary processing with is Erlang. It has a minimalist Prolog-like syntax, but most importantly, native support to work with binaries</a>, bitstrings</a>, including binary pattern matching (see "Examples" subsection)</a>.</p> To do with Erlang what we failed to do with hex editor, we will write the following program, following our hex editor attempt:</p> send_guest_bits_home</span>(</span> </span> <<</span>A</span>:</span>7</span>, </span>G</span>:</span>1</span>, </span>Rest</span>/</span>binary</span>>>, </span>% (E)</span> </span> {</span>Hosts</span>, </span>Guests</span>} </span>% (A)</span> </span>) -></span> </span> </span>send_guest_bits_home</span>(</span> </span> </span>Rest</span>,</span> </span> { <<</span>Hosts</span>/</span>bitstring</span>, </span>0</span>:</span>1</span>, </span>A</span>:</span>7</span>>> </span>% (B)</span> </span> , </span>case </span>Guests </span>of </span>% (F)</span> </span> <<</span>_</span>/</span>binary</span>>> -></span> </span> <<</span>Guests</span>/</span>binary</span>, </span>0</span>:</span>1</span>, </span>G</span>:</span>1</span>>>; </span>% (C)</span> </span> <<</span>_</span>/</span>bitstring</span>>> -></span> </span> <<</span>Guests</span>/</span>bitstring</span>, </span>G</span>:</span>1</span>>> </span>% (D)</span> </span> </span>end </span>});</span> </span></code></pre> We call seven-bit clusters "Hosts", and the appended bits, that we need to send to the back "Guests".</p> Here we use standard recursion with accumulator, which in our case is a tuple of Hosts</code> and Guests</code>. We bind current values of accumulator in (A)</code> and update them in (B)</code> and (C)</code>/(D)</code>.</p> In (E)</code>, we match first byte of challenge with variables A</code> (7 bits), and G</code> (1 bit), and the rest of challenge gets matched with variable Rest</code>. As you can see, in this context X:N</code> syntax means "match N</code> bits of some binary value with variable X</code>". As opposed to a similar syntax you can see in (B)</code> and (C)</code>, where syntax 0:1</code> means "span decimal value 0</code> over 1</code> bits". We specify that Rest</code> variable is binary</code>, not bitstring</code>, to make sure it's byte-aligned. If it was bitstring</code>, it would mean that it may be any amount of bits large, even seven or one.</p> Hosts</code> update happens in (B)</code> is that we take seven bits, pad it with a single 0</code> on the left and shove into the end of the first element of accumulator.</p> Case statement in (F)</code> checks if Guests</code> accumulator variable is byte-aligned (its bit size is divisible by eight) or not. If it is, next iteration of Guests</code> accumulator will be padded with a 0-bit, as seen in (C)</code>. If it isn't we just append the "guest bit" to Guests</code> accumulator verbatim, as seen in (D)</code>.</p> send_guest_bits_home</span>(</span> </span> <<>>, </span>% (A)</span> </span> {</span>Hosts</span>, </span>Guests</span>}</span> </span>) -></span> </span> <<</span>Hosts</span>/</span>bitstring</span>, </span>Guests</span>/</span>bitstring</span>>>.</span> </span></code></pre> Finally, in (A)</code>, when the remaining set of bits is empty, we return concatenation of the two accumulator variables.</p> solve</span>() -></span> </span> {</span>ok</span>, </span>Chal</span>} </span>=</span> </span> </span>file</span>:</span>read_file</span>(</span> </span> </span>"LastBitOfEveryByteIsAGuest.ASCII"</span> </span> ),</span> </span> </span>Out </span>=</span> </span> </span>send_guest_bits_home</span>(</span> </span> </span>Chal</span>, {<<>>, <<>>}</span> </span> ),</span> </span> </span>file</span>:</span>write_file</span>(</span> </span> </span>"LastBitOfEveryByteIsAGuest.txt"</span>,</span> </span> </span>Out</span> </span> ).</span> </span> </span></code></pre> When we run it, we get the hidden message!</p> $ erl</span> </span>Erlang/OTP 22 [erts-10.6.4] [source] [64-bit] ...</span> </span> </span>Eshell V10.6.4 (abort with ^G)</span> </span>1> c(solve).</span> </span>solve.erl:2: Warning: export_all flag enabled ...</span> </span>{ok,solve}</span> </span>2> solve:solve().</span> </span>ok</span> </span>3></span> </span>BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded</span> </span> (v)ersion (k)ill (D)b-tables (d)istribution</span> </span>a</span> </span>Thu Mar 04 01:27:26:530089000 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ cat LastBitOfEveryByteIsAGuest.txt</span> </span>doma{w4rm357_w3lcom3___t0___th15___bl0g}</span> </span></code></pre> At the time of publishing of this post I thought that there is no way to express binary literals in Erlang, like in Python 0b101 == 5</code>. However, as Reddit user Gwaerondor pointed out</a>, it's possible with Radix#</code>-syntax: 2#101 == 5</code>.</p> GNU poke🔗</a></h2> In 2017, thirty one years after Erlang's initial prototype, a tool whose sole purpose is working with structured binary data was conceived. It's called GNU poke</a></em>, and its v1 got released just several days ago</a>. Let's see how to use it for the same transformation:</p> type I = struct {</span> </span> uint<7> host;</span> </span> uint<1> guest;</span> </span>};</span> </span></code></pre> One of the biggest powers of GNU poke is that you can define a structure and map it to some data in one swift motion with "map" operator @</code>:</p> $ ./poke/poke LastBitOfEveryByteIsAGuest.ASCII</span> </span> * snip *</span> </span>(poke) I[] @ 0#B</span> </span>[I {host=(uint<7>) 100,guest=(uint<1>) 1},</span> </span> ...,</span> </span> I {host=(uint<7>) 95,guest=(uint<1>) 1}]</span> </span></code></pre> I[] @ 0#B</code> means "map the contents of currently loaded IO space onto array of I</code>s". Now let's see how to write the solve script.</p> fun solve = (I[] xs) bit: {</span> </span> var fh =</span> </span> open("hosts.out", IOS_F_READ | IOS_F_WRITE); /* (D) */</span> </span> var fg =</span> </span> open("guests.out", IOS_F_READ | IOS_F_WRITE); /* (E) */</span> </span> var bitsWrote = 0;</span> </span> var bytesWrote = 0;</span> </span> for (i in xs) {</span> </span> if (bitsWrote % 8 == 0) {</span> </span> bit[] @ fg : bitsWrote#b =</span> </span> [(0 as bit), i.guest]; /* (B) */</span> </span> bitsWrote += 2;</span> </span> } else {</span> </span> bit[] @ fg : bitsWrote#b =</span> </span> [i.guest]; /* (C) */</span> </span> bitsWrote += 1;</span> </span> }</span> </span> byte[] @ fh : bytesWrote#B =</span> </span> [ (0 as bit):::i.host ]; /* (A) */</span> </span> bytesWrote += 1;</span> </span> }</span> </span> close(fh);</span> </span> close(fg);</span> </span> return 0;</span> </span>}</span> </span></code></pre> An interesting part of this function is (A)</code>, it uses binary concatenation operator :::</code>. It can concatenate heterogenous binary data types like a bit and a 7-bit blob.</p> Then, (B)</code> and (C)</code> write out guest bits, following the same logic as in Erlang implementation. This is a pretty dense construction though, so let's read it together. bit[] @ fg : bitsWrote#b</code> reads "map an array of bits to fg</code> with a bit</em> offset of bitsWrote</code>". Now on the right hand side there is a value that will get mapped, which in (B)</code> is two bits: 0</code>-padding and the guest bit and in (C)</code> just the guest bit.</p> Now the sadder bit is that (D)</code> and (E)</code>: these files need to exist and contain enough data to map the outputs of the program, otherwise when ran, one'll get EOF exception</em>. Here's how to run this solution:</p> Fri Mar 05 23:10:22:029970500 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ echo -n "00000000000000000000000000000000000000000000000" > hosts.out</span> </span>Fri Mar 05 23:10:36:841098700 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ echo -n "00000000000000000000000000000000000000000000000" > guests.out</span> </span>Fri Mar 05 23:10:40:652833300 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ ./poke/poke LastBitOfEveryByteIsAGuest.ASCII</span> </span> _____</span> </span> ---' __\_______</span> </span> ______) GNU poke 1.0</span> </span> __)</span> </span> __)</span> </span> ---._______)</span> </span> </span>Copyright (C) 2019-2021 The poke authors.</span> </span>License GPLv3+: GNU GPL version 3 or later ...</span> </span>This is free software: you are free to change and redistribute it.</span> </span>There is NO WARRANTY, to the extent permitted by law.</span> </span> </span>Powered by Jitter 0.9.258.</span> </span>Perpetrated by Jose E. Marchesi.</span> </span> </span>For help, type ".help".</span> </span>Type ".exit" to leave the program.</span> </span>(poke) .load solve.poke</span> </span>(poke) var xs = I[] @ 0#B</span> </span>(poke) solve(xs)</span> </span>(uint<1>) 0</span> </span>(poke)</span> </span>Fri Mar 05 23:11:25:337643300 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ cat hosts.out</span> </span>doma{w4rm357_w3lcom3___t0___th15___000000000000</span> </span>Fri Mar 05 23:11:30:513672000 sweater@conflagrate ~/gnu/poke-1.0/build</span> </span>$ cat guests.out</span> </span>bl0g}000000000000000000000000000000000000000000</span> </span></code></pre> Picking the right tool at the right time🔗</a></h1> In conclusion, it seems like it's wise to use</p> hex editors allow for quick manual exploration,</li> erlang allows to follow functional style better,</li> GNU poke is imperative and procedure-oriented, but gives more flexibility in literal representations</li> allows for less boilerplate due to genius mapping operator</li> </ul> </li> </ul> Of course, you can use all three at different stages of your work with binaries.</p> Good luck with binary exploration and happy hacking!</p>