Retrospective on the S3 Bridge for iRODS

I am rapidly approaching the end of my time working on the s3 bridge, and I sit here with a pile of problems and a number of successes. On one hand, I have a reasonably performant project with a relatively low overhead, that happens to work fairly well on the limited operations that it set out to support, on the other I have done very little to prepare for publishing or users actually using the thing. So, let’s see what worked and what didn’t.

What Worked

1. Boost ASIO and Boost Beast were excellent to work with, their support for C++ coroutines really bridged the pain points that I had run into previously.
2. Boost Url is interesting and useful(and absolutely necessary to work with boost beast imho)
3. Nlohmann json is an excellent library
4. fmt is a pleasure to use.
5. CMake is
6. Separating the S3 actions into their own files and functions worked well.

Let’s look at what we got from the more recent ASIO support for coroutines. The core of the connection accepting machinery is contained in this short function.

asio::awaitable<void> listener()
{
    auto executor = co_await this_coro::executor;
    asio::ip::tcp::acceptor acceptor(executor, {asio::ip::tcp::v4(), port});
    for (;;) {
        asio::ip::tcp::socket socket = co_await acceptor.async_accept(boost::asio::use_awaitable);
        asio::co_spawn(executor, handle_request(std::move(socket)), asio::detached);
    }
}

Were it not for the compiler magic of co_await and its friend co_spawn, this would be a nest of callback oriented code, and while that’s perfectly fine, I guess, it’s not a very pleasant experience to write(I suppose that I didn’t have this reaction to next.js back before that got async support, so it must just be the overhead of writing a type to manage the context for this). But with C++ coroutines, all that is handled automatically, and it’s wonderful honestly. With a bit more machinery in the standard library, C++ coroutines have the opportunity to bring a lot of benefit to the language in the future.

Boost URL is new enough that it was added officially to Boost during the period I was working on this project, and while I’m sure that it was in the works for quite a while, it is definitely not coming a moment too soon.

What didn’t

I had a “my machine” first mentality that has reared its ugly side, and if I had been building docker images and toolchains to start with I would have a much more reasonable view of what the future will look like with this piece of software. Bolting this stuff on at the end is frustrating and nearly derailed the entire project up to this point(oof, ouch, my poor local build environment).

vcpkg was the source of a number of mysterious problems at the end, and I’m still not exactly sure what caused it to suddenly start shorting out where it did, this may just be a result of the idiosyncratic way that iRODS handles its dependencies and externals, but it might also just point to vcpkg still not quite being good enough for the purpose it seeks to fulfill(I’m sure that the development team has a number of ideas to bring the ring closer to being closed, even if C and C++ compilation and dependency management is likely to be a source of trouble for quite a while in the future).

I think the best way to prevent this from happening is to ensure that I have it working that way to begin with. Local compilation is nice, but if I can’t build it on a docker container then what hope do other people have of building it? People who aren’t the author of a program tend to be far less tolerant of difficulty building it than the author is, just like they are less tolerant of debugging problems.