Merge branch 'main' into wolfssl-testing

.github/workflows/build.yaml

@@ -0,0 +1,28 @@
+name: build
+run-name: Build libcurl.js
+on: [push]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: download repo
+        uses: actions/checkout@v4
+        with:
+          submodules: 'true'
+      
+      - name: install deps
+        run: | 
+          sudo apt-get update
+          sudo apt-get install -y make cmake emscripten autoconf automake libtool pkg-config wget xxd python3-selenium python3-websockets
+      
+      - name: run build
+        working-directory: ./client
+        run: ./build.sh all
+
+      - name: upload img
+        uses: actions/upload-artifact@v4
+        with:
+          name: libcurl
+          path: client/out/*
+          compression-level: 9

README.md

@@ -6,6 +6,10 @@ This is an experimental port of [libcurl](https://curl.se/libcurl/) to WebAssemb
 - Fetch compatible API
 - End to end encryption between the browser and the destination server
 - Support for up to TLS 1.3
+- Support for tunneling HTTP/2 connections 
+- Support for proxying WebSockets
+- Bypass CORS restrictions
+- Low latency via multiplexing and reusing open connections
 
 ## Building:
 You can build this project by running the following commands:
@@ -14,25 +18,40 @@ git clone https://github.com/ading2210/libcurl.js --recursive
 cd libcurl.js/client
 ./build.sh
 ```
-Make sure you have emscripten, git, and the various C build tools installed. The build script will generate `client/out/libcurl.js` as well as `client/out/libcurl_module.mjs`, which is an ES6 module.
+Make sure you have emscripten, git, and the various C build tools installed. The only OS supported for building libcurl.js is Linux. On Debian-based systems, you can run the following command to install all the dependencies:
+```
+sudo apt install make cmake emscripten autoconf automake libtool pkg-config wget xxd jq
+```
+
+The build script will generate `client/out/libcurl.js` as well as `client/out/libcurl.mjs`, which is an ES6 module. You can supply the following arguments to the build script to control the build:
+- `release` - Use all optimizations.
+- `single_file` - Include the WASM binary in the outputted JS using base64. 
+- `all` - Build twice, once normally, and once as a single file.
 
 ## Javascript API:
 
 ### Importing the Library:
-To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` a directory of your choice. Then you can simply link to it using a script tag and you will be able to use libcurl.js in your projects. Deferring the script load is recommended because the JS file is too large to download immediately.
+To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` and `client/out/libcurl.wasm` to a directory of your choice. After the script is loaded, call `libcurl.load_wasm`, specifying the url of the `libcurl.wasm` file.
 
 ```html
-<script defer src="./out/libcurl.js"></script>
+<script defer src="./out/libcurl.js" onload="libcurl.load_wasm('/out/libcurl.wasm');"></script>
+```
+
+Alternatively, prebuilt versions can be found on NPM and jsDelivr. You can use the following URLs to load libcurl.js from a third party CDN.
+```
+https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.js
+https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.wasm
 ```
 
 To know when libcurl.js has finished loading, you can use the `libcurl_load` DOM event. 
 ```js
 document.addEventListener("libcurl_load", ()=>{
+  libcurl.set_websocket(`wss://${location.hostname}/ws/`);
   console.log("libcurl.js ready!");
 });
 ```
 
-Once loaded, there will be a `window.libcurl` object which includes all the API functions.
+Once loaded, there will be a `window.libcurl` object which includes all the API functions. The `libcurl.ready` property can also be used to know if the WASM has loaded.
 
 ### Making HTTP Requests:
 To perform HTTP requests, use `libcurl.fetch`, which takes the same arguments as the browser's regular `fetch` function. Like the standard Fetch API, `libcurl.fetch` will also return a `Response` object.
@@ -41,11 +60,44 @@ let r = await libcurl.fetch("https://ading.dev");
 console.log(await r.text());
 ```
 
+Most of the standard Fetch API's features are supported, with the exception of:
+- CORS enforcement
+- `FormData` or `URLSearchParams` as the request body
+- Sending credentials/cookies automatically
+- Caching
+
+### Creating WebSocket Connections:
+To use WebSockets, create a `libcurl.WebSocket` object, which works identically to the regular [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) object.
+```js
+let ws = new libcurl.WebSocket("wss://echo.websocket.org");
+ws.addEventListener("open", () => {
+  console.log("ws connected!");
+  ws.send("hello".repeat(128));
+});
+ws.addEventListener("message", (event) => {
+  console.log(event.data);
+});
+```
+
 ### Changing the Websocket URL:
 You can change the URL of the websocket proxy by using `libcurl.set_websocket`.
 ```js
 libcurl.set_websocket("ws://localhost:6001/");
 ```
+If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash.
+
+### Getting Libcurl's Output:
+If you want more information about a connection, you can pass the `_libcurl_verbose` argument to the `libcurl.fetch` function.
+```js
+await libcurl.fetch("https://example.com", {_libcurl_verbose: 1});
+```
+By default this will print the output to the browser console, but you can set `libcurl.stdout` and `libcurl.stderr` to intercept these messages. This callback will be executed on every line of text that libcurl outputs.
+```js
+libcurl.stderr = (text) => {document.body.innerHTML += text};
+```
+
+### Getting Version Info:
+You can get version information from the `libcurl.version` object. This object will also contain the versions of all the C libraries that libcurl.js uses. `libcurl.version.lib` returns the version of libcurl.js itself. 
 
 ## Proxy Server:
 The proxy server consists of a standard [Wisp](https://github.com/MercuryWorkshop/wisp-protocol) server, allowing multiple TCP connections to share the same websocket.
@@ -53,18 +105,18 @@ The proxy server consists of a standard [Wisp](https://github.com/MercuryWorksho
 To host the proxy server, run the following commands:
 ```
 git clone https://github.com/ading2210/libcurl.js --recursive
-cd libcurl.js/server
+cd libcurl.js
-./run.sh
+server/run.sh --static=./client
 ```
 
-You can use the `HOST` and `PORT` environment variables to control the hostname and port that the proxy server listens on.
+For a full list of server arguments, see the [wisp-server-python documentation](https://github.com/MercuryWorkshop/wisp-server-python).
 
 ## Copyright:
 This project is licensed under the GNU AGPL v3.
 
 ### Copyright Notice:
 ```
-ading2210/libcurl.js - A port of libcurl to WASM
+ading2210/libcurl.js - A port of libcurl to WASM for use in the browser.
 Copyright (C) 2023 ading2210
 
 This program is free software: you can redistribute it and/or modify

client/build.sh

@@ -2,53 +2,93 @@
 
 set -e
 
-INCLUDE_DIR="build/curl-wasm/include/"
+#path definitions
-LIB_DIR="build/curl-wasm/lib/"
+OUT_DIR="${OUT_DIR:=out}"
-OUT_FILE="out/libcurl.js"
+BUILD_DIR="build"
-ES6_FILE="out/libcurl_module.mjs"
+C_DIR="libcurl"
-MODULE_FILE="out/emscripten_compiled.js"
 FRAGMENTS_DIR="fragments"
-WRAPPER_SOURCE="main.js"
+JAVSCRIPT_DIR="javascript"
 WISP_CLIENT="wisp_client"
 
-EXPORTED_FUNCS="_init_curl,_start_request,_tick_request,_active_requests"
+INCLUDE_DIR="$BUILD_DIR/curl-wasm/include/"
-RUNTIME_METHODS="addFunction,removeFunction,allocate,ALLOC_NORMAL"
+LIB_DIR="$BUILD_DIR/curl-wasm/lib/"
-COMPILER_OPTIONS="-o $MODULE_FILE -lcurl -lwolfssl -lcjson -lz -lbrotlidec -lbrotlicommon -I $INCLUDE_DIR -L $LIB_DIR"
+OUT_FILE="$OUT_DIR/libcurl.js"
-EMSCRIPTEN_OPTIONS="-lwebsocket.js -sASSERTIONS=1 -sALLOW_TABLE_GROWTH -sALLOW_MEMORY_GROWTH -sEXPORTED_FUNCTIONS=$EXPORTED_FUNCS -sEXPORTED_RUNTIME_METHODS=$RUNTIME_METHODS"
+ES6_FILE="$OUT_DIR/libcurl.mjs"
+MODULE_FILE="$OUT_DIR/emscripten_compiled.js"
+COMPILED_FILE="$OUT_DIR/emscripten_compiled.wasm"
+WASM_FILE="$OUT_DIR/libcurl.wasm"
 
-if [ "$1" = "release" ]; then
+#read exported functions
+EXPORTED_FUNCS=""
+for func in $(cat exported_funcs.txt); do
+  EXPORTED_FUNCS="$EXPORTED_FUNCS,_$func"
+done
+EXPORTED_FUNCS="${EXPORTED_FUNCS:1}"
+
+#compile options
+RUNTIME_METHODS="addFunction,removeFunction,allocate,ALLOC_NORMAL"
+COMPILER_OPTIONS="-o $MODULE_FILE -lcurl -lwolfssl -lcjson -lz -lbrotlidec -lbrotlicommon -lnghttp2 -I $INCLUDE_DIR -L $LIB_DIR"
+EMSCRIPTEN_OPTIONS="-lwebsocket.js -sENVIRONMENT=worker,web -sASSERTIONS=1 -sLLD_REPORT_UNDEFINED -sALLOW_TABLE_GROWTH -sALLOW_MEMORY_GROWTH -sEXPORTED_FUNCTIONS=$EXPORTED_FUNCS -sEXPORTED_RUNTIME_METHODS=$RUNTIME_METHODS"
+
+#clean output dir
+rm -rf $OUT_DIR
+mkdir -p $OUT_DIR
+
+if [[ "$*" == *"all"* ]]; then
+  mkdir -p $OUT_DIR/release
+  mkdir -p $OUT_DIR/single_file
+  OUT_DIR=$OUT_DIR/release ./build.sh release
+  OUT_DIR=$OUT_DIR/single_file ./build.sh release single_file
+  mv $OUT_DIR/release/* $OUT_DIR
+  mv $OUT_DIR/single_file/* $OUT_DIR
+  rm -rf $OUT_DIR/release
+  rm -rf $OUT_DIR/single_file
+  exit 0
+fi
+
+if [[ "$*" == *"release"* ]]; then
   COMPILER_OPTIONS="-Oz -flto $COMPILER_OPTIONS"
-  EMSCRIPTEN_OPTIONS="-sSINGLE_FILE $EMSCRIPTEN_OPTIONS"
+  echo "note: building with release optimizations"
 else
   COMPILER_OPTIONS="$COMPILER_OPTIONS --profiling -g"
 fi
 
+if [[ "$*" == *"single_file"* ]]; then
+  EMSCRIPTEN_OPTIONS="-sSINGLE_FILE $EMSCRIPTEN_OPTIONS"
+  OUT_FILE="$OUT_DIR/libcurl_full.js"
+  ES6_FILE="$OUT_DIR/libcurl_full.mjs"
+  echo "note: building as a single js file"
+fi
+
 #ensure deps are compiled
 tools/all_deps.sh
 tools/generate_cert.sh
 
-#clean output dir
+#compile the main c file
-rm -rf out
+COMPILE_CMD="emcc $C_DIR/*.c $COMPILER_OPTIONS $EMSCRIPTEN_OPTIONS"
-mkdir -p out
-
-#compile the main c file - but only if the source has been modified
-COMPILE_CMD="emcc main.c $COMPILER_OPTIONS $EMSCRIPTEN_OPTIONS"
 echo $COMPILE_CMD
 $COMPILE_CMD
+mv $COMPILED_FILE $WASM_FILE || true
 
 #merge compiled emscripten module and wrapper code
-cp $WRAPPER_SOURCE $OUT_FILE
+cp $JAVSCRIPT_DIR/main.js $OUT_FILE
 sed -i "/__emscripten_output__/r $MODULE_FILE" $OUT_FILE
 rm $MODULE_FILE
 
-#add wisp libraries
+#add version number and copyright notice
+VERSION=$(cat package.json | jq -r '.version')
+sed -i "s/__library_version__/$VERSION/" $OUT_FILE
+
+#add extra libraries
 sed -i "/__extra_libraries__/r $WISP_CLIENT/polyfill.js" $OUT_FILE
 sed -i "/__extra_libraries__/r $WISP_CLIENT/wisp.js" $OUT_FILE
-sed -i "/__extra_libraries__/r ./messages.js" $OUT_FILE
+sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/messages.js" $OUT_FILE
+sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/websocket.js" $OUT_FILE
+sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/copyright.js" $OUT_FILE
 
 #apply patches
-python3 patcher.py $FRAGMENTS_DIR $OUT_FILE
+python3 tools/patch_js.py $FRAGMENTS_DIR $OUT_FILE
 
 #generate es6 module
 cp $OUT_FILE $ES6_FILE
-sed -i 's/window.libcurl/export const libcurl/' $ES6_FILE
+sed -i 's/const libcurl = /export const libcurl = /' $ES6_FILE

client/exported_funcs.txt

@@ -0,0 +1,19 @@
+init_curl
+start_request
+tick_request
+active_requests
+
+get_version
+
+recv_from_websocket
+send_to_websocket
+close_websocket
+cleanup_websocket
+get_result_size
+get_result_buffer
+get_result_code
+get_result_closed
+get_result_bytes_left
+get_result_is_text
+
+free

client/fragments/load_later.js

@@ -0,0 +1,16 @@
+/* REPLACE
+var asm ?= ?createWasm\(\);
+*/
+if (isDataURI(wasmBinaryFile)) var asm = createWasm();
+else var asm = null;
+
+/* REPLACE
+var wasmExports ?= ?createWasm\(\);
+*/
+if (isDataURI(wasmBinaryFile)) var wasmExports = createWasm();
+else var wasmExports = null;
+
+/* REPLACE
+run\(\);\n\n
+*/
+if (isDataURI(wasmBinaryFile)) run();

client/fragments/silence_socket.js

@@ -4,6 +4,6 @@ err\("__syscall_getsockname " ?\+ ?fd\);
 
 
 /* INSERT
-function _emscripten_console_error\(str\) {
+function _emscripten_console_error\(str\) ?{
 */
 if (UTF8ToString(str).endsWith("__syscall_setsockopt\\n")) return;

client/index.html

@@ -3,11 +3,12 @@
   <head>
     <link rel="icon" href="data:;base64,=">
 
-    <script defer src="./out/libcurl.js"></script>
+    <script defer src="./out/libcurl.js" onload="libcurl.load_wasm('/out/libcurl.wasm');"></script>
     <script>
       document.addEventListener("libcurl_load", ()=>{
-        console.log("libcurl.js ready!");
+        libcurl.set_websocket(location.href.replace("http", "ws"));
-      })
+        console.log(`loaded libcurl.js v${libcurl.version.lib}`);
+      });
     </script>
   </head>
   <body>

client/javascript/copyright.js

@@ -0,0 +1,15 @@
+const copyright_notice = `ading2210/libcurl.js - A port of libcurl to WASM for use in the browser.
+Copyright (C) 2023 ading2210
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.`;

client/javascript/main.js

@@ -0,0 +1,323 @@
+/*
+ading2210/libcurl.js - A port of libcurl to WASM for the browser.
+Copyright (C) 2023 ading2210
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+*/
+
+//everything is wrapped in a function to prevent emscripten from polluting the global scope
+const libcurl = (function() {
+
+//emscripten compiled code is inserted here
+/* __emscripten_output__ */
+
+//extra client code goes here
+/* __extra_libraries__ */
+
+var websocket_url = null;
+var event_loop = null;
+var active_requests = 0;
+var wasm_ready = false;
+var version_dict = null;
+const libcurl_version = "__library_version__";
+
+function check_loaded(check_websocket) {
+  if (!wasm_ready) {
+    throw new Error("wasm not loaded yet, please call libcurl.load_wasm first");
+  }
+  if (!websocket_url && check_websocket) {
+    throw new Error("websocket proxy url not set, please call libcurl.set_websocket");
+  }
+}
+
+//a case insensitive dictionary for request headers
+class HeadersDict {
+  constructor(obj) {
+    for (let key in obj) {
+      this[key] = obj[key];
+    }
+    return new Proxy(this, this);
+  }
+  get(target, prop) {
+    let keys = Object.keys(this);
+    for (let key of keys) {
+      if (key.toLowerCase() === prop.toLowerCase()) {
+        return this[key];
+      }
+    }
+  }
+  set(target, prop, value) {
+    let keys = Object.keys(this);
+    for (let key of keys) {
+      if (key.toLowerCase() === prop.toLowerCase()) {
+        this[key] = value;
+      }
+    }
+    this[prop] = value;
+    return true;
+  }
+}
+
+function allocate_str(str) {
+  return allocate(intArrayFromString(str), ALLOC_NORMAL);
+}
+
+function allocate_array(array) {
+  return allocate(array, ALLOC_NORMAL);
+}
+
+//low level interface with c code
+function perform_request(url, params, js_data_callback, js_end_callback, body=null) {
+  let params_str = JSON.stringify(params);
+  let end_callback_ptr;
+  let data_callback_ptr;
+  let url_ptr = allocate_str(url);
+  let params_ptr = allocate_str(params_str);
+
+  let body_ptr = null;
+  let body_length = 0;
+  if (body) { //assume body is an int8array
+    body_ptr = allocate_array(body);
+    body_length = body.length;
+  }
+
+  let end_callback = (error, response_json_ptr) => {
+    let response_json = UTF8ToString(response_json_ptr);
+    let response_info = JSON.parse(response_json);
+
+    Module.removeFunction(end_callback_ptr);
+    Module.removeFunction(data_callback_ptr);
+    if (body_ptr) _free(body_ptr);
+    _free(url_ptr);
+    _free(response_json_ptr);
+    
+    if (error != 0) console.error("request failed with error code " + error);
+    active_requests --;
+    js_end_callback(error, response_info);
+  }
+
+  let data_callback = (chunk_ptr, chunk_size) => {
+    let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size);
+    let chunk = new Uint8Array(data);
+    js_data_callback(chunk);
+  }
+
+  end_callback_ptr = Module.addFunction(end_callback, "vii");
+  data_callback_ptr = Module.addFunction(data_callback, "vii");
+  let http_handle = _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, body_ptr, body_length);
+  _free(params_ptr);
+  
+  active_requests ++;
+  _tick_request();
+  if (!event_loop) {
+    event_loop = setInterval(() => {
+      if (_active_requests() || active_requests) {
+        _tick_request();
+      }
+      else {
+        clearInterval(event_loop);
+        event_loop = null;
+      }
+    }, 0);
+  }
+
+  return http_handle;
+}
+
+function merge_arrays(arrays) {
+  let total_len = arrays.reduce((acc, val) => acc + val.length, 0);
+  let new_array = new Uint8Array(total_len);
+  let offset = 0;
+  for (let array of arrays) {
+    new_array.set(array, offset);
+    offset += array.length;
+  }
+  return new_array;
+}
+
+function create_response(response_data, response_info) {
+  response_info.ok = response_info.status >= 200 && response_info.status < 300;
+  response_info.statusText = status_messages[response_info.status] || "";
+  if (response_info.status === 204 || response_info.status === 205) {
+    response_data = null;
+  }
+
+  //construct base response object
+  let response_obj = new Response(response_data, response_info);
+  for (let key in response_info) {
+    if (key == "headers") continue;
+    Object.defineProperty(response_obj, key, {
+      writable: false,
+      value: response_info[key]
+    });
+  }
+
+  //create headers object
+  Object.defineProperty(response_obj, "headers", {
+    writable: false,
+    value: new Headers()
+  });
+  for (let header_name in response_info.headers) {
+    let header_value = response_info.headers[header_name];
+    response_obj.headers.append(header_name, header_value);
+  }
+  
+  return response_obj;
+}
+
+async function parse_body(data) {
+  let data_array = null;
+  if (typeof data === "string") {
+    data_array = new TextEncoder().encode(data);
+  }
+
+  else if (data instanceof Blob) {
+    let array_buffer = await data.arrayBuffer();
+    data_array = new Uint8Array(array_buffer);
+  }
+
+  //any typedarray
+  else if (data instanceof ArrayBuffer) {
+    //dataview objects
+    if (ArrayBuffer.isView(data) && data instanceof DataView) {
+      data_array = new Uint8Array(data.buffer);
+    }
+    //regular typed arrays
+    else if (ArrayBuffer.isView(data)) {
+      data_array = Uint8Array.from(data);
+    }
+    //regular arraybuffers
+    else {
+      data_array = new Uint8Array(data);
+    }
+  }
+
+  else if (data instanceof ReadableStream) {
+    let chunks = [];
+    for await (let chunk of data) {
+      chunks.push(chunk);
+    }
+    data_array = merge_arrays(chunks);
+  }
+
+  else {
+    throw "invalid data type to be sent";
+  }
+  return data_array;
+}
+
+async function create_options(params) {
+  let body = null;
+  if (params.body) {
+    body = await parse_body(params.body);
+    params.body = true;
+  }
+
+  if (!params.headers) params.headers = {};
+  params.headers = new HeadersDict(params.headers);
+
+  if (params.referer) {
+    params.headers["Referer"] = params.referer;
+  }
+  if (!params.headers["User-Agent"]) {
+    params.headers["User-Agent"] = navigator.userAgent;
+  }
+
+  return body;
+}
+
+//wrap perform_request in a promise
+function perform_request_async(url, params, body) {
+  return new Promise((resolve, reject) => {
+    let chunks = [];
+    let data_callback = (new_data) => {
+      chunks.push(new_data);
+    };
+    
+    let finish_callback = (error, response_info) => {
+      if (error != 0) {
+        reject("libcurl.js encountered an error: " + error);
+        return;
+      }
+      let response_data = merge_arrays(chunks);
+      chunks = null;
+      let response_obj = create_response(response_data, response_info);
+      resolve(response_obj);
+    }
+    perform_request(url, params, data_callback, finish_callback, body);
+  });
+}
+
+async function libcurl_fetch(url, params={}) {
+  check_loaded(true);
+  let body = await create_options(params);
+  return await perform_request_async(url, params, body);
+}
+
+function set_websocket_url(url) {
+  websocket_url = url;
+  if (!Module.websocket) {
+    document.addEventListener("libcurl_load", () => {
+      set_websocket_url(url);
+    });
+  }
+  else Module.websocket.url = url;
+}
+
+function get_version() {
+  if (!wasm_ready) return null;
+  if (version_dict) return version_dict;
+
+  let version_ptr = _get_version();
+  let version_str = UTF8ToString(version_ptr);
+  _free(version_ptr);
+  version_dict = JSON.parse(version_str);
+  version_dict.lib = libcurl_version;
+  return version_dict;
+}
+
+function main() {
+  wasm_ready = true;
+  _init_curl();
+  set_websocket_url(websocket_url);
+
+  let load_event = new Event("libcurl_load");
+  document.dispatchEvent(load_event);
+}
+
+function load_wasm(url) {
+  wasmBinaryFile = url;
+  createWasm();
+  run();
+}
+
+Module.onRuntimeInitialized = main;
+return {
+  fetch: libcurl_fetch,
+  set_websocket: set_websocket_url,
+  load_wasm: load_wasm,
+  wisp: _wisp_connections,
+  WebSocket: CurlWebSocket,
+  
+  get copyright() {return copyright_notice},
+  get version() {return get_version()},
+  get ready() {return wasm_ready},
+
+  get stdout() {return out},
+  set stdout(callback) {out = callback},
+  get stderr() {return err},
+  set stderr(callback) {err = callback}
+}
+
+})()

client/javascript/websocket.js

@@ -0,0 +1,201 @@
+//class for custom websocket
+
+class CurlWebSocket extends EventTarget {
+  constructor(url, protocols=[], websocket_debug=false) {
+    super();
+    check_loaded(true);
+    if (!url.startsWith("wss://") && !url.startsWith("ws://")) {
+      throw new SyntaxError("invalid url");
+    }
+
+    this.url = url;
+    this.protocols = protocols;
+    this.binaryType = "blob";
+    this.recv_buffer = [];
+    this.websocket_debug = websocket_debug;
+
+    //legacy event handlers
+    this.onopen = () => {};
+    this.onerror = () => {};
+    this.onmessage = () => {};
+    this.onclose = () => {};
+
+    this.CONNECTING = 0;
+    this.OPEN = 1;
+    this.CLOSING = 2;
+    this.CLOSED = 3;
+
+    this.connect();
+  }
+
+  connect() {
+    this.status = this.CONNECTING;
+    let data_callback = () => {};
+    let finish_callback = (error, response_info) => {
+      this.finish_callback(error, response_info);
+    }
+    let options = {};
+    if (this.protocols) {
+      options.headers = {
+        "Sec-Websocket-Protocol": this.protocols.join(", "),
+      };
+    }
+    if (this.websocket_debug) {
+      options._libcurl_verbose = 1;
+    }
+    this.http_handle = perform_request(this.url, options, data_callback, finish_callback, null);
+    this.recv_loop();
+  }
+
+  recv() {
+    let buffer_size = 64*1024;
+    let result_ptr = _recv_from_websocket(this.http_handle, buffer_size);
+    let data_ptr = _get_result_buffer(result_ptr);
+    let result_code = _get_result_code(result_ptr);
+
+    if (result_code == 0) { //CURLE_OK - data recieved 
+      if (_get_result_closed(result_ptr)) {
+        //this.pass_buffer();
+        this.close_callback();
+        return;
+      }
+
+      let data_size = _get_result_size(result_ptr);
+      let data_heap = Module.HEAPU8.subarray(data_ptr, data_ptr + data_size);
+      let data = new Uint8Array(data_heap);
+      
+      this.recv_buffer.push(data);
+      if (data_size !== buffer_size && !_get_result_bytes_left(result_ptr)) { //message finished
+        let full_data = merge_arrays(this.recv_buffer);
+        let is_text = _get_result_is_text(result_ptr)
+        this.recv_buffer = [];
+        this.recv_callback(full_data, is_text);
+      }
+    }
+
+    if (result_code == 52) { //CURLE_GOT_NOTHING - socket closed
+      this.close_callback();
+    }
+    
+    _free(data_ptr);
+    _free(result_ptr);
+  }
+
+  recv_loop() {
+    this.event_loop = setInterval(() => {
+      this.recv();
+    }, 1);
+  }
+
+  recv_callback(data, is_text=false) {
+    let converted;
+    if (is_text) {
+      converted = new TextDecoder().decode(data);
+    }
+    else {
+      if (this.binaryType == "blob") {
+        converted = new Blob(data);
+      }
+      else if (this.binaryType == "arraybuffer") {
+        converted = data.buffer;
+      }
+      else {
+        throw "invalid binaryType string";
+      }  
+    }
+
+    let msg_event = new MessageEvent("message", {data: converted});
+    this.onmessage(msg_event);
+    this.dispatchEvent(msg_event);
+  }
+
+  close_callback(error=false) {
+    if (this.status == this.CLOSED) return;
+    this.status = this.CLOSED;
+
+    clearInterval(this.event_loop);
+    _cleanup_websocket();
+
+    if (error) {
+      let error_event = new Event("error");
+      this.dispatchEvent(error_event);
+      this.onerror(error_event);
+    }
+    else {
+      let close_event = new CloseEvent("close");
+      this.dispatchEvent(close_event);
+      this.onclose(close_event);
+    }
+  }
+
+  finish_callback(error, response_info) {
+    this.status = this.OPEN;
+    if (error != 0) this.close_callback(true);
+    let open_event = new Event("open");
+    this.onopen(open_event);
+    this.dispatchEvent(open_event);
+  }
+
+  send(data) {
+    let is_text = false;
+    if (this.status === this.CONNECTING) {
+      throw new DOMException("ws not ready yet");
+    }
+    if (this.status === this.CLOSED) {
+      return;
+    }
+
+    let data_array;
+    if (typeof data === "string") {
+      data_array = new TextEncoder().encode(data);
+      is_text = true;
+    }
+    else if (data instanceof Blob) {
+      data.arrayBuffer().then(array_buffer => {
+        data_array = new Uint8Array(array_buffer);
+        this.send(data_array);
+      });
+      return;
+    }
+    //any typedarray
+    else if (data instanceof ArrayBuffer) {
+      //dataview objects
+      if (ArrayBuffer.isView(data) && data instanceof DataView) {
+        data_array = new Uint8Array(data.buffer);
+      }
+      //regular arraybuffers
+      else {
+        data_array = new Uint8Array(data);
+      }
+    }
+    //regular typed arrays
+    else if (ArrayBuffer.isView(data)) {
+      data_array = Uint8Array.from(data);
+    }
+    else {
+      throw "invalid data type to be sent";
+    }
+
+    let data_ptr = allocate_array(data_array);
+    let data_len = data.length;
+    _send_to_websocket(this.http_handle, data_ptr, data_len, is_text);
+    _free(data_ptr);
+  }
+
+  close() {
+    _close_websocket(this.http_handle);
+  }
+
+  get readyState() {
+    return this.status;
+  }
+  get bufferedAmount() {
+    return 0;
+  }
+  get protocol() {
+    return "";
+  }
+  get extensions() {
+    return "";
+  }
+}

client/libcurl/main.c

@@ -10,28 +10,19 @@
 #include "cacert.h"
 #include "curl/multi.h"
 
-typedef void(*DataCallback)(char* chunk_ptr, int chunk_size);
+#include "util.h"
-typedef void(*EndCallback)(int error, char* response_json);
+#include "types.h"
+
 void finish_request(CURLMsg *curl_msg);
 
 #define ERROR_REDIRECT_DISALLOWED -1
 
 CURLM *multi_handle;
 int request_active = 0;
+struct curl_blob cacert_blob;
 
-struct RequestInfo {
+size_t write_function(void *data, size_t size, size_t nmemb, DataCallback data_callback) {
-  int abort_on_redirect;
+  size_t real_size = size * nmemb;
-  struct CURLMsg *curl_msg;
-  struct curl_slist* headers_list;
-  EndCallback end_callback;
-};
-
-int starts_with(const char *a, const char *b) {
-  return strncmp(a, b, strlen(b)) == 0;
-}
-
-int write_function(void *data, size_t size, size_t nmemb, DataCallback data_callback) {
-  long real_size = size * nmemb;
   char* chunk = malloc(real_size);
   memcpy(chunk, data, real_size);
   data_callback(chunk, real_size);
@@ -57,13 +48,13 @@ void tick_request() {
   }
 }
 
-void start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, const char* body, int body_length) {
+CURL* start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, const char* body, int body_length) {
   CURL *http_handle = curl_easy_init();  
   int abort_on_redirect = 0;
+  int prevent_cleanup = 0;
  
   curl_easy_setopt(http_handle, CURLOPT_URL, url);
-  curl_easy_setopt(http_handle, CURLOPT_CAINFO, "/cacert.pem");
+  curl_easy_setopt(http_handle, CURLOPT_CAINFO_BLOB , cacert_blob);
-  curl_easy_setopt(http_handle, CURLOPT_CAPATH, "/cacert.pem");
 
   //callbacks to pass the response data back to js
   curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, &write_function);
@@ -72,10 +63,12 @@ void start_request(const char* url, const char* json_params, DataCallback data_c
   //some default options
   curl_easy_setopt(http_handle, CURLOPT_FOLLOWLOCATION, 1);
   curl_easy_setopt(http_handle, CURLOPT_ACCEPT_ENCODING, "");
+  curl_easy_setopt(http_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
 
   //if url is a websocket, tell curl that we should handle the connection manually
   if (starts_with(url, "wss://") || starts_with(url, "ws://")) {
     curl_easy_setopt(http_handle, CURLOPT_CONNECT_ONLY, 2L);
+    prevent_cleanup = 1;
   }
 
   //parse json options
@@ -134,9 +127,12 @@ void start_request(const char* url, const char* json_params, DataCallback data_c
   request_info->curl_msg = NULL;
   request_info->headers_list = headers_list;
   request_info->end_callback = end_callback;
+  request_info->prevent_cleanup = prevent_cleanup;
   curl_easy_setopt(http_handle, CURLOPT_PRIVATE, request_info);
   
   curl_multi_add_handle(multi_handle, http_handle);
+
+  return http_handle;
 }
 
 void finish_request(CURLMsg *curl_msg) {
@@ -184,23 +180,22 @@ void finish_request(CURLMsg *curl_msg) {
   
   //clean up curl
   curl_slist_free_all(request_info->headers_list);
+  (*request_info->end_callback)(error, response_json_str);
+  if (request_info->prevent_cleanup) {
+    return;
+  }
   curl_multi_remove_handle(multi_handle, http_handle);
   curl_easy_cleanup(http_handle);
-  (*request_info->end_callback)(error, response_json_str);
   free(request_info);
 }
 
-char* copy_bytes(const char* ptr, const int size) {
-  char* new_ptr = malloc(size);
-  memcpy(new_ptr, ptr, size);
-  return new_ptr;
-}
-
 void init_curl() {
   curl_global_init(CURL_GLOBAL_DEFAULT);
   multi_handle = curl_multi_init();
+  curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 50L);
+  curl_multi_setopt(multi_handle, CURLMOPT_MAXCONNECTS, 40L);
   
-  FILE *file = fopen("/cacert.pem", "wb");
+  cacert_blob.data = _cacert_pem;
-  fwrite(_cacert_pem, 1, _cacert_pem_len, file);
+  cacert_blob.len = _cacert_pem_len;
-  fclose(file);
+  cacert_blob.flags = CURL_BLOB_NOCOPY;
 }

client/libcurl/types.h

@@ -0,0 +1,19 @@
+typedef void(*DataCallback)(char* chunk_ptr, int chunk_size);
+typedef void(*EndCallback)(int error, char* response_json);
+
+struct RequestInfo {
+  int abort_on_redirect;
+  int prevent_cleanup;
+  struct CURLMsg *curl_msg;
+  struct curl_slist* headers_list;
+  EndCallback end_callback;
+};
+
+struct WSResult {
+  int res;
+  int buffer_size;
+  int closed;
+  int bytes_left;
+  int is_text;
+  char* buffer;
+};

client/libcurl/util.c

@@ -0,0 +1,37 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "curl/curl.h"
+#include "cjson/cJSON.h"
+
+int starts_with(const char *a, const char *b) {
+  return strncmp(a, b, strlen(b)) == 0;
+}
+
+char* get_version() {
+  struct curl_version_info_data *version_info = curl_version_info(CURLVERSION_NOW);
+  cJSON* version_json = cJSON_CreateObject();
+
+  cJSON* protocols_array = cJSON_CreateArray();
+  const char *const *protocols = version_info->protocols;
+  for (; *protocols != NULL; protocols++) {
+    cJSON* protocol_item = cJSON_CreateString(*protocols);
+    cJSON_AddItemToArray(protocols_array, protocol_item);
+  }
+  
+  cJSON* curl_version_item = cJSON_CreateString(version_info->version);
+  cJSON* ssl_version_item = cJSON_CreateString(version_info->ssl_version);
+  cJSON* brotli_version_item = cJSON_CreateString(version_info->brotli_version);
+  cJSON* nghttp2_version_item = cJSON_CreateString(version_info->nghttp2_version);
+  
+  cJSON_AddItemToObject(version_json, "curl", curl_version_item);
+  cJSON_AddItemToObject(version_json, "ssl", ssl_version_item);
+  cJSON_AddItemToObject(version_json, "brotli", brotli_version_item);
+  cJSON_AddItemToObject(version_json, "nghttp2", nghttp2_version_item);
+  cJSON_AddItemToObject(version_json, "protocols", protocols_array);
+
+  char* version_json_str = cJSON_Print(version_json);
+  cJSON_Delete(version_json);
+  return version_json_str;
+}

client/libcurl/util.h

@@ -0,0 +1 @@
+int starts_with(const char *a, const char *b);

client/libcurl/websocket.c

@@ -0,0 +1,66 @@
+#include <stdlib.h>
+
+#include "curl/curl.h"
+#include "curl/websockets.h"
+
+#include "types.h"
+
+extern CURLM* multi_handle;
+
+struct WSResult* recv_from_websocket(CURL* http_handle, int buffer_size) {
+  const struct curl_ws_frame* ws_meta;
+  char* buffer = malloc(buffer_size);
+  size_t result_len;
+  CURLcode res = curl_ws_recv(http_handle, buffer, buffer_size, &result_len, &ws_meta);
+
+  struct WSResult* result = malloc(sizeof(struct WSResult));
+  result->buffer_size = result_len;
+  result->buffer = buffer;
+  result->res = (int) res;
+  result->closed = (ws_meta->flags & CURLWS_CLOSE);
+  result->is_text = (ws_meta->flags & CURLWS_TEXT);
+  result->bytes_left = ws_meta->bytesleft;
+  return result;
+}
+
+int send_to_websocket(CURL* http_handle, const char* data, int data_len, int is_text) {
+  size_t sent;
+  unsigned int flags = CURLWS_BINARY;
+  if (is_text) flags = CURLWS_TEXT;
+  CURLcode res = curl_ws_send(http_handle, data, data_len, &sent, 0, flags);
+  return (int) res;
+}
+
+void close_websocket(CURL* http_handle) {
+  size_t sent;
+  curl_ws_send(http_handle, "", 0, &sent, 0, CURLWS_CLOSE);
+}
+
+//clean up the http handle associated with the websocket, since the main loop can't do this automatically
+void cleanup_websocket(CURL* http_handle) {
+  struct RequestInfo *request_info;
+  curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info);
+
+  curl_multi_remove_handle(multi_handle, http_handle);
+  curl_easy_cleanup(http_handle);
+  free(request_info);
+}
+
+int get_result_size (const struct WSResult* result) {
+  return result->buffer_size;
+}
+char* get_result_buffer (const struct WSResult* result) {
+  return result->buffer;
+}
+int get_result_code (const struct WSResult* result) {
+  return result->res;
+}
+int get_result_closed (const struct WSResult* result) {
+  return result->closed;
+}
+int get_result_bytes_left (const struct WSResult* result) {
+  return result->bytes_left;
+}
+int get_result_is_text (const struct WSResult* result) {
+  return result->is_text;
+}

client/main.js

@@ -1,196 +0,0 @@
-//everything is wrapped in a function to prevent emscripten from polluting the global scope
-window.libcurl = (function() {
-
-//emscripten compiled code is inserted here
-/* __emscripten_output__ */
-
-//extra client code goes here
-/* __extra_libraries__ */
-
-const websocket_url = `wss://${location.hostname}/ws/`;
-var event_loop = null;
-
-//a case insensitive dictionary for request headers
-class Headers {
-  constructor(obj) {
-    for (let key in obj) {
-      this[key] = obj[key];
-    }
-    return new Proxy(this, this);
-  }
-  get(target, prop) {
-    let keys = Object.keys(this);
-    for (let key of keys) {
-      if (key.toLowerCase() === prop.toLowerCase()) {
-        return this[key];
-      }
-    }
-  }
-  set(target, prop, value) {
-    let keys = Object.keys(this);
-    for (let key of keys) {
-      if (key.toLowerCase() === prop.toLowerCase()) {
-        this[key] = value;
-      }
-    }
-    this[prop] = value;
-    return true;
-  }
-}
-
-function is_str(obj) {
-  return typeof obj === 'string' || obj instanceof String;
-}
-
-function allocate_str(str) {
-  return allocate(intArrayFromString(str), ALLOC_NORMAL);
-}
-
-function allocate_array(array) {
-  return allocate(array, ALLOC_NORMAL);
-}
-
-//low level interface with c code
-function perform_request(url, params, js_data_callback, js_end_callback, body=null) {
-  let params_str = JSON.stringify(params);
-  let end_callback_ptr;
-  let data_callback_ptr;
-  let url_ptr = allocate_str(url);
-  let params_ptr = allocate_str(params_str);
-
-  let body_ptr = null;
-  let body_length = 0;
-  if (body) { //assume body is an int8array
-    body_ptr = allocate_array(body);
-    body_length = body.length;
-  }
-
-  let end_callback = (error, response_json_ptr) => {
-    let response_json = UTF8ToString(response_json_ptr);
-    let response_info = JSON.parse(response_json);
-
-    Module.removeFunction(end_callback_ptr);
-    Module.removeFunction(data_callback_ptr);
-    if (body_ptr) _free(body_ptr);
-    _free(url_ptr);
-    _free(response_json_ptr);
-    
-    js_end_callback(error, response_info);
-  }
-
-  let data_callback = (chunk_ptr, chunk_size) => {
-    let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size);
-    let chunk = new Uint8Array(data);
-    js_data_callback(chunk);
-  }
-
-  end_callback_ptr = Module.addFunction(end_callback, "vii");
-  data_callback_ptr = Module.addFunction(data_callback, "vii");
-  _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, body_ptr, body_length);
-  _free(params_ptr);
-  
-  _tick_request();
-  if (!event_loop) {
-    event_loop = setInterval(() => {
-      _tick_request();
-      if (!_active_requests()) {
-        clearInterval(event_loop);
-        event_loop = null;
-      }
-    }, 0);
-  }
-}
-
-function merge_arrays(arrays) {
-  let total_len = arrays.reduce((acc, val) => acc + val.length, 0);
-  let new_array = new Uint8Array(total_len);
-  let offset = 0;
-  for (let array of arrays) {
-    new_array.set(array, offset);
-    offset += array.length;
-  }
-  return new_array;
-}
-
-function create_response(response_data, response_info) {
-  delete response_info.error;
-  response_info.ok = response_info.status >= 200 && response_info.status < 300;
-  response_info.statusText = status_messages[response_info.status] || "";
-
-  let response_obj = new Response(response_data, response_info);
-  for (let key in response_info) {
-    Object.defineProperty(response_obj, key, {
-      writable: false,
-      value: response_info[key]
-    });
-  }
-  return response_obj;
-}
-
-function create_options(params) {
-  let body = null;
-  if (params.body) {
-    if (is_str(params.body)) {
-      body = new TextEncoder().encode(params.body);
-    }
-    else {
-      body = Uint8Array.from(params);
-    }
-    params.body = true;
-  }
-
-  if (!params.headers) params.headers = {};
-  params.headers = new Headers(params.headers);
-
-  if (params.referer) {
-    params.headers["Referer"] = params.referer;
-  }
-  if (!params.headers["User-Agent"]) {
-    params.headers["User-Agent"] = navigator.userAgent;
-  }
-
-  return body;
-}
-
-function libcurl_fetch(url, params={}) {
-  let body = create_options(params);
-
-  return new Promise((resolve, reject) => {
-    let chunks = [];
-    let data_callback = (new_data) => {
-      chunks.push(new_data);
-    };
-    
-    let finish_callback = (error, response_info) => {
-      if (error != 0) {
-        reject("libcurl.js encountered an error: " + error + "\n" + response_info.error);
-        return;
-      }
-      let response_data = merge_arrays(chunks);
-      let response_obj = create_response(response_data, response_info);
-      resolve(response_obj);
-    }
-    perform_request(url, params, data_callback, finish_callback, body);
-  })
-}
-
-function set_websocket_url(url) {
-  Module.websocket.url = url;
-}
-
-function main() {
-  console.log("emscripten module loaded");
-  _init_curl();
-  set_websocket_url(websocket_url);
-
-  let load_event = new Event("libcurl_load");
-  document.dispatchEvent(load_event);
-}
-
-Module.onRuntimeInitialized = main;
-return {
-  fetch: libcurl_fetch,
-  set_websocket: set_websocket_url,
-}
-
-})()

client/package.json

@@ -0,0 +1,19 @@
+{
+  "name": "libcurl.js",
+  "version": "0.3.7",
+  "description": "An experimental port of libcurl to WebAssembly for use in the browser.",
+  "main": "libcurl.mjs",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/ading2210/libcurl.js.git"
+  },
+  "author": "ading2210",
+  "license": "AGPL-3.0-or-later",
+  "bugs": {
+    "url": "https://github.com/ading2210/libcurl.js/issues"
+  },
+  "homepage": "https://github.com/ading2210/libcurl.js"
+}

client/publish.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+#publish libcurl.js as an npm package
+
+./build.sh all
+
+cp package.json out
+cp ../README.md out
+cp ../LICENSE out
+
+cd out
+npm publish

client/tests/index.html

@@ -0,0 +1,40 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <link rel="icon" href="data:;base64,=">
+
+    <script defer src="/out/libcurl.js" onload="libcurl.load_wasm('/out/libcurl.wasm');"></script>
+    <script>
+      function create_flag(result) {
+        let element = document.createElement("div");
+        element.setAttribute("result", result);
+        element.className = "flag";
+        document.body.append(element);
+      }
+
+      function assert(condition, message) {
+        if (!condition) {
+          throw new Error(message || "Assertion failed");
+        }
+      }
+
+    
+      document.addEventListener("libcurl_load", async ()=>{
+        try {
+          libcurl.set_websocket(`ws://localhost:6001/ws/`);
+          let r = await fetch("/tests/scripts/" + location.hash.substring(1));
+          eval(await r.text());
+          await test();
+          create_flag("success");
+        }
+        catch (e) {
+          console.error(e.stack || e);
+          create_flag("error");
+        }
+      });
+    </script>
+  </head>
+  <body>
+    <p>libcurl.js unit test runner</p>
+  </body>
+</html>

client/tests/run.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+set -e
+
+trap "exit" INT TERM
+trap "kill 0" EXIT
+../server/run.sh --static=$(pwd) >/dev/null &
+
+echo -n "waiting for wisp server to start"
+i=0
+until $(curl --output /dev/null --silent --head "http://localhost:6001/"); do
+  if [ "$i" = "30" ]; then
+    echo -e "\ntests failed. wisp server failed to start"
+    exit 1
+  fi
+
+  echo -n "."
+  i=$(($i+1))
+  sleep 1
+done
+echo 
+
+
+sleep 1
+echo "wisp server ready, running tests"
+python3 tests/run_tests.py

client/tests/run_tests.py

@@ -0,0 +1,53 @@
+import unittest
+
+from selenium import webdriver
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+class JSTest(unittest.TestCase):
+  def setUp(self):
+    options = webdriver.ChromeOptions()
+    options.add_argument("start-maximized")
+    options.add_argument("enable-automation")
+    options.add_argument("--headless")
+    options.add_argument("--no-sandbox")
+    options.add_argument("--disable-dev-shm-usage")
+    options.add_argument("--disable-browser-side-navigation")
+    options.add_argument("--disable-gpu")
+    options.set_capability("goog:loggingPrefs", {"browser": "ALL"})
+
+    self.browser = webdriver.Chrome(options=options)
+
+  def tearDown(self):
+    self.browser.quit()
+  
+  def run_test(self, script):
+    self.browser.get(f"http://localhost:6001/tests/#{script}")
+    wait = WebDriverWait(self.browser, 20)
+    result = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.flag'))).get_attribute("result")
+
+    if result != "success":
+      for entry in self.browser.get_log("browser"):
+        print(f"{entry['level']}: {entry['message']}")
+
+    assert result == "success"
+  
+  def test_fetch_once(self):
+    self.run_test("fetch_once.js")
+
+  def test_fetch_multiple(self):
+    self.run_test("fetch_multiple.js")
+
+  def test_fetch_parallel(self):
+    self.run_test("fetch_parallel.js")
+  
+  def test_websocket(self):
+    self.run_test("test_websocket.js")
+  
+  def test_redirect_out(self):
+    self.run_test("redirect_out.js")
+
+if __name__ == "__main__":
+  unittest.main()

client/tests/scripts/fetch_multiple.js

@@ -0,0 +1,7 @@
+async function test() {
+  await libcurl.fetch("https://example.com/");
+  for (let i=0; i<20; i++) {
+    let r = await libcurl.fetch("https://example.com/");
+    assert(r.status === 200, "wrong status");  
+  }
+}

client/tests/scripts/fetch_once.js

@@ -0,0 +1,4 @@
+async function test() {
+  let r = await libcurl.fetch("https://example.com/");
+  assert(r.status === 200, "wrong status");
+}

client/tests/scripts/fetch_parallel.js

@@ -0,0 +1,8 @@
+async function test() {
+  await libcurl.fetch("https://www.example.com/");
+  let promises = [];
+  for (let i=0; i<10; i++) {
+      promises.push(libcurl.fetch("https://www.example.com/"))
+  }
+  await Promise.all(promises);
+}

client/tests/scripts/redirect_out.js

@@ -0,0 +1,11 @@
+async function test() {
+  let output = [];
+  function out_callback(text) {
+    output.push(text);
+  }
+  libcurl.stdout = out_callback;
+  libcurl.stderr = out_callback;
+  await libcurl.fetch("https://example.com/", {_libcurl_verbose: 1});
+  console.log(output);
+  assert(output[0] === "* Host example.com:443 was resolved.", "unexpected output in stderr");
+}

client/tests/scripts/test_websocket.js

@@ -0,0 +1,24 @@
+function test() {
+  let message_len = 128*1024;
+
+  return new Promise((resolve, reject) => {
+    let ws = new libcurl.WebSocket("wss://echo.websocket.org");
+    ws.addEventListener("open", () => {
+      ws.send("hello".repeat(message_len));
+    });
+    
+    let messages = 0;
+    ws.addEventListener("message", (event) => {
+      messages += 1;
+      if (messages >= 2) {
+        if (event.data !== "hello".repeat(message_len)) reject("unexpected response");
+        if (messages >= 11) resolve();
+        ws.send("hello".repeat(message_len));
+      }
+    });
+
+    ws.addEventListener("error", () => {
+      reject("ws error occurred");
+    });
+  })
+}

client/tools/all_deps.sh

@@ -2,6 +2,7 @@
 
 #build all deps
 
+set -e
 mkdir -p build
 
 WOLFSSL_PREFIX=$(realpath build/wolfssl-wasm)
@@ -9,6 +10,7 @@ CJSON_PREFIX=$(realpath build/cjson-wasm)
 CURL_PREFIX=$(realpath build/curl-wasm)
 ZLIB_PREFIX=$(realpath build/zlib-wasm)
 BROTLI_PREFIX=$(realpath build/brotli-wasm)
+NGHTTP2_PREFIX=$(realpath build/nghttp2-wasm)
 
 if [ ! -d $WOLFSSL_PREFIX ]; then
   tools/openssl.sh
@@ -22,6 +24,9 @@ fi
 if [ ! -d $BROTLI_PREFIX ]; then
   tools/brotli.sh
 fi
+if [ ! -d $NGHTTP2_PREFIX ]; then
+  tools/nghttp2.sh
+fi
 if [ ! -d $CURL_PREFIX ]; then
   tools/curl.sh
 fi
@@ -29,4 +34,5 @@ fi
 cp -r $WOLFSSL_PREFIX/* $CURL_PREFIX
 cp -r $CJSON_PREFIX/* $CURL_PREFIX
 cp -r $ZLIB_PREFIX/* $CURL_PREFIX
-cp -r $BROTLI_PREFIX/* $CURL_PREFIX
+cp -r $BROTLI_PREFIX/* $CURL_PREFIX
+cp -r $NGHTTP2_PREFIX/* $CURL_PREFIX

client/tools/brotli.sh

@@ -10,7 +10,7 @@ PREFIX=$(realpath build/brotli-wasm)
 
 cd build
 rm -rf brotli
-git clone -b master --depth=1 https://github.com/google/brotli
+git clone -b v1.1.0 --depth=1 https://github.com/google/brotli
 cd brotli
 
 emcmake cmake . -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./installed

client/tools/cjson.sh

@@ -11,7 +11,7 @@ mkdir -p $PREFIX
 
 cd build
 rm -rf cjson
-git clone -b master --depth=1 https://github.com/DaveGamble/cJSON cjson
+git clone -b v1.7.17 --depth=1 https://github.com/DaveGamble/cJSON cjson
 cd cjson
 
 sed -i 's/-fstack-protector-strong//' Makefile

client/tools/curl.sh

@@ -10,14 +10,15 @@ PREFIX=$(realpath build/curl-wasm)
 WOLFSSL_PREFIX=$(realpath build/wolfssl-wasm)
 ZLIB_PREFIX=$(realpath build/zlib-wasm)
 BROTLI_PREFIX=$(realpath build/brotli-wasm)
+NGHTTP2_PREFIX=$(realpath build/nghttp2-wasm)
 
 cd build
 rm -rf curl
-git clone -b master --depth=1 https://github.com/curl/curl
+git clone -b curl-8_6_0 --depth=1 https://github.com/curl/curl
 cd curl
 
 autoreconf -fi
-emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX
+emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX --with-nghttp2=$NGHTTP2_PREFIX
 emmake make -j$CORE_COUNT CFLAGS="-Oz -pthread" LIBS="-lbrotlicommon"
 
 rm -rf $PREFIX

client/tools/nghttp2.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+#compile nghttp2 for use with emscripten
+
+set -x
+set -e
+
+CORE_COUNT=$(nproc --all)
+PREFIX=$(realpath build/nghttp2-wasm)
+
+cd build
+rm -rf nghttp2
+git clone -b v1.59.0 --depth=1 https://github.com/nghttp2/nghttp2
+cd nghttp2
+
+rm -rf $PREFIX
+mkdir -p $PREFIX
+
+autoreconf -fi
+emconfigure ./configure --host i686-linux --enable-static --disable-shared --enable-lib-only --prefix=$PREFIX
+emmake make -j$CORE_COUNT
+make install
+
+cd ../../

client/tools/patch_js.py

@@ -16,6 +16,9 @@ for fragment_file in fragments_path.iterdir():
   matches = re.findall(match_regex, fragment_text, re.S)
 
   for mode, patch_regex, patch_text, _ in matches:
+    fragment_matches = re.findall(patch_regex, target_text)
+    if not fragment_matches:
+      print(f"warning: regex did not match anything for '{patch_regex}'");
     if mode == "DELETE":
       target_text = re.sub(patch_regex, "", target_text)
     elif mode == "REPLACE":

client/tools/zlib.sh

@@ -10,7 +10,7 @@ PREFIX=$(realpath build/zlib-wasm)
 
 cd build
 rm -rf zlib
-git clone -b master --depth=1 https://github.com/madler/zlib
+git clone -b v1.3.1 --depth=1 https://github.com/madler/zlib
 cd zlib
 
 emconfigure ./configure --static

client/wisp_client

@@ -1 +1 @@
-Subproject commit 0a80885090b6247f42bc07cc85b441d8d719f551
+Subproject commit 51ad95a6d912ec404c20284f0cded40c0b5c4e62

server/run.sh

@@ -4,14 +4,17 @@
 
 set -e
 
-cd wisp_server
+SCRIPT_PATH=$(realpath $0)
-if [ ! -d ".venv" ]; then
+BASE_PATH=$(dirname $SCRIPT_PATH)
-  python3 -m venv .venv
+SERVER_PATH="$BASE_PATH/wisp_server"
+
+if [ ! -d "$SERVER_PATH.venv" ]; then
+  python3 -m venv $SERVER_PATH/.venv
 fi
-source .venv/bin/activate
+source $SERVER_PATH/.venv/bin/activate
 
 if ! python3 -c "import websockets" 2> /dev/null; then
-  pip3 install -r requirements.txt
+  pip3 install -r $SERVER_PATH/requirements.txt
 fi
 
-python3 main.py
+python3 $SERVER_PATH/main.py "$@"

server/wisp_server

@@ -1 +1 @@
-Subproject commit 874734623e4dfc4652b34a1bc61e1e35ca86dee8
+Subproject commit 3b0d432e89cff7eaa850ba8605b180189e237f1b