Webstack overhaul
- Dominant language
- No language data
- Stars
- 7
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Hey anyone who happens to still read things around here :)
I'm opening this issue to discuss some of my recent efforts in a new (still unregistered) package [HTTP.jl](https://github.com/JuliaWeb/HTTP.jl).
The julia webstack has long been the neglected child of the julia ecosystem, hobbling along with just enough attention to still be considered useable packages, but not much else. To be fair, the webstack originally started out as a HackerSchool project that thankfully provided basic functionality, but which unfortunately has atrophied as the original authors have moved on to bigger and better things. We found an unlikely hero in @malmaud when he stepped up and fixed some glaring usability issues with Requests.jl, as well as creating the MbedTLS.jl package to fill the glaring TLS need, but otherwise, the webstack as a whole has received very little attention over the last 3-4 years.
I have some personal and professional interests around building web services in Julia, but which require a solid stack. As I researched and dug into our webstack code, I originally was just aiming to make some improvements to HttpServer, but the more I looked around, the more I realized a complete overhaul was needed. In particular, most of the webstack code used outdated idioms from 3-4 years ago and was failing to utilize some of the great new languages features that have been released recently. There were also glaring issues of rampant code duplication due to the micro-separation of functionality between HttpCommon, HttpParser, URIParser, Requests, HttpServer; as well as a broad lack of testing in most areas.
The approach I've taken to the webstack overhaul has involved:
* Research: in particular, reading through the actual HTTP 1.1 spec to derive necessary functionality and generate tests, as well as other language http implementations. I became particularly interested in the implementation in [go](https://golang.org/pkg/net/http/), due to the friendly licensing, modern-ness (i.e. implemented in recent years), and general web focus of the language
* Webstack consolidation: most other languages/libraries don't separate webstack functionality as granular as it is in julia; the most separation you generally see is between client vs. server. While the intent is right, there just aren't that many uses of having HttpCommon or HttpParser exist as stand-alone packages. In particular, this has led to a lot of code duplication in the webstack, as well as quite a few cases of "code squatting", i.e. code living in a repo that actually extends/is relevant for a different package entirely, not to mention the problem users have in figuring out where to request functionality or submit issues. In general, I think it's made the webstack less maintainable and harder to develop on/with having everything separated out. My first efforts in overhauling were to combine the existing webstack packages into a single git repo, making sure to preserve the git histories of each.
* Design from the top down: I tried to take what most high-level client/server implementations look like in other libraries/languages and come up with a Julia-version that makes the most sense. Starting at the highest-levels and working down helps drive the right abstractions and fuels testing.
* Overall webstack quality: one of my main goals in this endeavor is to make the webstack code "Base worthy"; meaning it's well-tested, well-documented, and as many warts as possible are excised, so that, as a package, it could be ready to be included in Base. Obviously there are a lot of other design decisions for including something like this in Base (or rather, the Julia [standard library](https://github.com/JuliaLang/julia/issues/5155)), and that's not ultimately my goal, just that it get to and remain in a state that it **could**.
State of progress:
* I know, I know, I probably should have posted this earlier, but my initial research/code exploration kind of snow-balled into the overhaul and I ended up having the time/motivation to do it, so I just started.
* The overhaul is about 2/3 of the way to a v0.1, IMO. The client-side has been totally revamped and I'm working on wrapping up a few remaining "robustness" features and filling in tests and docs. Server-side has been through a cycle or two of redesign, but there's still some core design left to do around utilizing Julia's built-in parallelism and how that affects the core server definition and event loop.
* In terms of dependencies, I'm aiming for one, julia or shared library otherwise. HTTP.jl will depend on MbedTLS.jl, but won't be hard-coded to it. MbedTLS provides an implementation of the TransportLayerSecurity.jl interface, that other TLS packages could do as well, and swapping out MbedTLS with another TLS package would be a matter of changing [a single line](https://github.com/JuliaWeb/HTTP.jl/blob/fbb6472774fd0e5d127299dddbbea7cae7bb8289/src/HTTP.jl#L7). Currently, we're relying on the http-parser library from joyent/node, but the actual library code isn't more than 1K LOC and I have plans to do a pure reimplementation in Julia (shouldn't take more than a week or two); this will leave us with the single dependency of a TLS package.
* All of that said, I'd say the package as a whole is 2-3 months from being registered and tagged for v0.1.
I'd like to welcome anyone and everyone to provide feedback to the discussion here about what they think and ideas for moving the webstack forward. Also feel free to take HTTP.jl for a spin, it should be fairly functional already. I'd of course welcome all feedback and help designing, implementing, and testing the package from as many people as possible. Happy to open up a JuliaWeb gitter if we want a more fast-paced place to discuss if there's interest, otherwise, we can use github issues and the Julia discourse forum to discuss.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.