~ req, an HTTP scripting language

Posted on Sat 26 Feb 2022 to Programming

Programming languages are always something that have fascinated me, how they're designed, how they're implemented, and how they're used. Whether they're a DSL (domain specific language) or more of a generic programming language. A programming language is always something I had wanted to take a stab at creating, even if it ended up being terrible, or being of no true utility, but only for the sake of learning. Well, over the Christmas break I decided to occupy my time on developing a language, one that was small and simple, designed for a specific use case that I had encountered but hadn't found a solution for. The language that I ended up developing was req, a language designed only for HTTP scripting.

Overview

What do I mean when I say HTTP scripting? Perhaps an example would be best to demonstrate, followed by an explanation,

Token = env "GH_TOKEN";
Headers = (
    Authorizationn: "Bearer $(Token)",
);

Resp = GET "https://api.github.com/user" $Headers -> send;

if $Resp.StatusCode == 200 {
    User = decode json $Resp.Body;
    writeln _ "Hello $(User["login"])";
}

Above is what req looks like. It looks like your typical language, however it offers first-class support for making HTTP requests and working with their responses. The language makes use of builtin commands to handle the sending of requests, the encoding/decoding of data, and the reading/writing of data. These commands also return values that can be stored in variables. The output of one command can be sent as the input of another command via the -> operator. There is no support for user defined commands.

That's what the language looks like, and this is how it is run,

$ req user.req

the above example makes use of the GH_TOKEN environment variable, so if we wanted it to actually function we would need to make sure that was set before invocation,

$ GH_TOKEN=<token> req user.req

So, from a brief overview you can see that there is some familiarity with other languages out there. I call this a scripting language, as opposed to a programming language, as it is interpreted, and extremely limited in scope and capabilities.

req can also be used via the REPL, which is accessed simply by invoking the binary and passing no arguments to it. This can be used as a scratchpad to plan out what you want your scripts to do, or as a means of exploring an HTTP service and its endpoints,

$ req
req devel a5ddbe7 Sat Jan 29 11:34:38 2022 +0000
> Resp = GET "https://httpbin.org/json" -> send
> writeln _ $Resp
HTTP/2.0 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 429
Content-Type: application/json
Date: Sat, 26 Feb 2022 14:14:56 GMT
Server: gunicorn/19.9.0

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

Why

Now, why did I want to create yet another scripting language. Two reasons.

The first was for the sake of learning. Some time ago I had received copies of the books Writing An Interpreter In Go and Writing A Compiler In Go. I had worked through the first book at the start of 2020, and enjoyed what I had learned, wanted to put what I had learned to practice. At the time however, I couldn't think of a fun or interesting language that I would have wanted to develop, so I shelved the prospect of it for some time, which brings me to my second reason...

I think most developers have to interact with an HTTP service at some point during their day job, in a way which would require some form of scripting. Perhaps you're trying to debug an API, so you pull open a terminal and fire off a few curl requests, and see what response comes back. Or maybe you want to scrape a site for its information. Either way, you're doing something that involves some tinkering.

I have been there too. And it is this scenario that made me wonder if there was a tool out there that allowed for easily working with HTTP requests and their responses in a programmatic way. Sure, you could use curl and shell scripting, and wrangle the data through sed, awk, and jq to get the data you need, but this approach can be fragile. On the other hand you could use a full fledged programming language. This way, you would have more control, but it can be perhaps a bit too verbose at times if all you want to do is send off an HTTP request.

This is was prompted my development on req. A high-level scripting language that allows you to easily send out HTTP requests, and work with their responses. The main benefit of the language, in my opinion, is that it tries to make working with HTTP requests as semantic as possible, take the following,

Resp = GET "https://httpbin.org/json" -> send;

here we want to send a GET request to the https://httpbin.org/json endpoint. Writing this out either in a script or the REPL can feel more natural than what you might otherwise write when using curl, for example. Then let's say we want to decode the response data into JSON,

Data = decode json $Resp.Body;

again, it's like we're describing what we want to do with the response. This is what I wanted to achieve with this language, keep it limited in scope, and hopefully offer some utility in the realm of HTTP scripting.

Conclusion

What I've covered in this post is a simple overview of the language, and the reasons behind it's implementation. I haven't gone into my justifications as to how/why the language was designed the way it was, but that could perhaps be another post down the line. If what I've shown so far interests you, then feel free to take a look at the code for it on GitHub: https://github.com/andrewpillar/req and feel free to spool through the documentation there too, to get a sense of the language.