I got my hands on a cute little CTF artefact, a file named AnExtraBitForEveryByte.

Its hexdump didn't look too promising (to me, since I'm not too experienced in CTFs):

My first hypothesis was that to get ASCII from it, I should either drop last bit or first bit of every byte. I lost a metric ton of time trying to use hex editors to do anything meaningful, and my verdict is the following:

hex editors are just interactive hex viewers.

Furthermore, many of them were crashing during usage, causing even more rage. Sadly I didn't document the worst offenders, but I remember bless hex editor outright not launching, wxhex crashing, Gnome being just bad.

Gladly, that wasn't my first rodeo when it comes to working with binary data. I've implemented a full Bitcoin blockchain parser from scratch in 2013 and I did it... (you can guess it if you know me at all)... using BEAM languages.

So I figured I'd spin trusty Erlang virtual machine and let it do hex editting.

Erlang bitstrings are very flexible, but 99.9999999% of real world usecases will be covered by

• X:BitSize syntax. Which as expression (RHS) means “take X” and span it over “BitSize” bits, and as a pattern (LHS) means “match BitSize bits to X, if X is value” or “bind BitSize bits to the X if X is identifier”.
• Bys/binary syntax. Which as expression inserts bytes into a binary variable, and as a pattern is an equivalent of binding the tail of a list.
• Bis/bitstring syntax. This is a somewhat overlooked syntax that does the same thing as /binary except it has a per-bit granularity instead of per-byte.
• <<...>> syntax. To use three syntaxes above, you must enclose statements in double-triangle-brackets. For example Pad5 = <<0:5>>.

Don't worry if that explanation was too dense, let's go step-by-step and have a look at how to use all of it in practice.

Let's begin with simply looping over bytes. As the syntax summary above suggests, we'll need to use X:8 syntax and Bys/binary syntax. Download the binary sample: wget https://ctf.cdn.doma.dev/AnExtraBitForEveryByte, open a text editor and create a file called “hello_binary.erl” with the following contents:

Alright, now let's compile and run it, all through Erlang's interactive shell:

How wonderful! Our idbyte function is indeed an identity function for byte-aligned binaries.

Now, in the same interactive session, let's see what will happen if we give it just seven bits:

As expected, it didn't work. To work with pure bits, we need to use bitstrings, as is shown in interaction number 4.

With that knowledge, we can attempt to write our function that drops last bit of every byte.

Add the following functions to your file:

Let's compile and run!

That's a little bit cryptic. What that means is that file:write_file returns {error, badarg} instead of ok.

Returning {error, ErrorTag} is a standard practice for Erlang (which is a pattern that might be familiar to Go programmers).

If we think a little bit, we'll realise that it's logical. Thanks to bit_size/1 call we know that our binary is 266 bits big. Since remainder of division of 266 by 8 is not 0, we can't write this binary into a byte-aligned file.

Let's fix this issue by padding resulting binary with zeroes. Add the following functions to your file:

Let's run it again, and then hexdump -C it in hopes to get the flag.

Welp, we didn't get the flag, but at least we have most of the tools needed to implement binary stream editing with Erlang! And we can validate that our padding with zeroes worked by observing that 0x40 = 64, which is exactly one with six zeroes in binary.

With a simple query we can learn, that the most popular 7-bit format is the 7-bit ASCII. Now all that is left is to obtain a 7-bit ASCII table and try decoding.

By far the fastest way to do it is to pipe HTML containing the table through w3m pager, getting clean, tab-separated content, and then write a VIM macro to transform it into a data structure.

Sadly, I didn't record myself doing that, but you'd better believe that it took me well under five minutes.

I chose to python as the target language, don't really know why, must be its reputation as the language of choice for doing hacky write-only stuff in post-Perl world. Anyway, I ended up with the following code to attempt 7-bit ASCII decoding:

Indeed, when we run the script, we get a valid UUID, which must be the flag:

That's it for now. If you have any questions or feedback on the article, shoot me a toot on Mastodon. Speaking of Mastodon! Registration is still open on our server. Make sure to grab an account both for fun and since we plan exclusive offers for active members of Doma community!

Sat 21 Nov 09:40:36 GMT 2020

What is this doma thing all about? Here is the gist.

TL;DR

Doma is a prinicpled product-oriented company. Its products are privacy-enabled and open-source, deployments are subscription-based. At the time of writing, doma is making their first full-stack product. It will result in rapid prototyping SDK to make it easier for businesses to reinvest into themselves. Doma is available for consulation for FP and data science.

Principles

• Short production cycles: we aim to have exactly one mythical person-month for full development. It is then followed by another two weeks for measuring usefulness.
• No useless work: Even early adopters of our stuff will have to commit some sort of resource to using our products and tools. The default approach would be a subscription for a nominal price or unsupported self-hosting.
• Go where the users are: we won't make users come to us. Instead, we will do our best to fit their preferences and workflows.

• Data storage ethics: We are not selling or disclosing data, metadata or run “opt-out” targetted marketing. And we never will.
• Responsible deployments on less-private platforms: there is a world on the other side of the garage doors. Whilst deploying on popular platforms, we commit to posting noticable messaging and promoting free and non-opressive alternatives to said platforms.
• Warrant canaries: Each of our products that stores user data on its servers shall have warrant canaries.

• Build rapid prototyping toolchain: after we're done slow-rolling a full-stack project currently in development, we will have a reliable way to build typed, performant products, targetting every platform.
• Prove its feasibility: build another product, without spending more than a person-month on it.
• Turn toolchain and processes into a software kit: my current main company has the linear business model problem, which is typical for a consultancy. To address that issue, we are actively trying (and somewhat failing) to enter product market. It seems reasonable to me that many businesses that want to get better profit asymptotics would want to self-incubate products. Our proven kit should allow them to have bounded risks at least as far as software development and marketing are concerned.
• Consultancy: when me and my partner aren't doing stuff on our two jobs (that's two jobs per each person, not two jobs in total), we're ready to consult on the topics of
  • functional programming
  • haskell
  • erlang
  • elixir
  • rust
  • maximising utilisation of success and structural typings
  • data science
  • forecasting
  • statistical analysis
  • data visualisation
  • machine learning

• Ready to scale: we plan to keep on making product prototypes until we get validation from a broader community. When that happens, we are going to put resources into the expansion.
• Conventional marketing but ethical: once we have a validated product, we will commence conventional marketing rounds, perhaps using a contractor. We will be careful to craft the messaging in such way that the person who gets targetted by our marketing campaigns knows that we aren't happy to give our money to platforms that are conventionally used to run marketing campaigns on.