diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml new file mode 100644 index 0000000..51d7a27 --- /dev/null +++ b/.github/workflows/build.yaml @@ -0,0 +1,28 @@ +name: build +run-name: Build libcurl.js +on: [push] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: download repo + uses: actions/checkout@v4 + with: + submodules: 'true' + + - name: install deps + run: | + sudo apt-get update + sudo apt-get install -y make cmake emscripten autoconf automake libtool pkg-config wget xxd python3-selenium python3-websockets + + - name: run build + working-directory: ./client + run: ./build.sh all + + - name: upload img + uses: actions/upload-artifact@v4 + with: + name: libcurl + path: client/out/* + compression-level: 9 \ No newline at end of file diff --git a/README.md b/README.md index 5cbddb6..e9aa0d0 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,10 @@ This is an experimental port of [libcurl](https://curl.se/libcurl/) to WebAssemb - Fetch compatible API - End to end encryption between the browser and the destination server - Support for up to TLS 1.3 +- Support for tunneling HTTP/2 connections +- Support for proxying WebSockets +- Bypass CORS restrictions +- Low latency via multiplexing and reusing open connections ## Building: You can build this project by running the following commands: @@ -14,25 +18,40 @@ git clone https://github.com/ading2210/libcurl.js --recursive cd libcurl.js/client ./build.sh ``` -Make sure you have emscripten, git, and the various C build tools installed. The build script will generate `client/out/libcurl.js` as well as `client/out/libcurl_module.mjs`, which is an ES6 module. +Make sure you have emscripten, git, and the various C build tools installed. The only OS supported for building libcurl.js is Linux. On Debian-based systems, you can run the following command to install all the dependencies: +``` +sudo apt install make cmake emscripten autoconf automake libtool pkg-config wget xxd jq +``` + +The build script will generate `client/out/libcurl.js` as well as `client/out/libcurl.mjs`, which is an ES6 module. You can supply the following arguments to the build script to control the build: +- `release` - Use all optimizations. +- `single_file` - Include the WASM binary in the outputted JS using base64. +- `all` - Build twice, once normally, and once as a single file. ## Javascript API: ### Importing the Library: -To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` a directory of your choice. Then you can simply link to it using a script tag and you will be able to use libcurl.js in your projects. Deferring the script load is recommended because the JS file is too large to download immediately. +To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` and `client/out/libcurl.wasm` to a directory of your choice. After the script is loaded, call `libcurl.load_wasm`, specifying the url of the `libcurl.wasm` file. ```html - + +``` + +Alternatively, prebuilt versions can be found on NPM and jsDelivr. You can use the following URLs to load libcurl.js from a third party CDN. +``` +https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.js +https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.wasm ``` To know when libcurl.js has finished loading, you can use the `libcurl_load` DOM event. ```js document.addEventListener("libcurl_load", ()=>{ + libcurl.set_websocket(`wss://${location.hostname}/ws/`); console.log("libcurl.js ready!"); }); ``` -Once loaded, there will be a `window.libcurl` object which includes all the API functions. +Once loaded, there will be a `window.libcurl` object which includes all the API functions. The `libcurl.ready` property can also be used to know if the WASM has loaded. ### Making HTTP Requests: To perform HTTP requests, use `libcurl.fetch`, which takes the same arguments as the browser's regular `fetch` function. Like the standard Fetch API, `libcurl.fetch` will also return a `Response` object. @@ -41,11 +60,44 @@ let r = await libcurl.fetch("https://ading.dev"); console.log(await r.text()); ``` +Most of the standard Fetch API's features are supported, with the exception of: +- CORS enforcement +- `FormData` or `URLSearchParams` as the request body +- Sending credentials/cookies automatically +- Caching + +### Creating WebSocket Connections: +To use WebSockets, create a `libcurl.WebSocket` object, which works identically to the regular [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) object. +```js +let ws = new libcurl.WebSocket("wss://echo.websocket.org"); +ws.addEventListener("open", () => { + console.log("ws connected!"); + ws.send("hello".repeat(128)); +}); +ws.addEventListener("message", (event) => { + console.log(event.data); +}); +``` + ### Changing the Websocket URL: You can change the URL of the websocket proxy by using `libcurl.set_websocket`. ```js libcurl.set_websocket("ws://localhost:6001/"); ``` +If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash. + +### Getting Libcurl's Output: +If you want more information about a connection, you can pass the `_libcurl_verbose` argument to the `libcurl.fetch` function. +```js +await libcurl.fetch("https://example.com", {_libcurl_verbose: 1}); +``` +By default this will print the output to the browser console, but you can set `libcurl.stdout` and `libcurl.stderr` to intercept these messages. This callback will be executed on every line of text that libcurl outputs. +```js +libcurl.stderr = (text) => {document.body.innerHTML += text}; +``` + +### Getting Version Info: +You can get version information from the `libcurl.version` object. This object will also contain the versions of all the C libraries that libcurl.js uses. `libcurl.version.lib` returns the version of libcurl.js itself. ## Proxy Server: The proxy server consists of a standard [Wisp](https://github.com/MercuryWorkshop/wisp-protocol) server, allowing multiple TCP connections to share the same websocket. @@ -53,18 +105,18 @@ The proxy server consists of a standard [Wisp](https://github.com/MercuryWorksho To host the proxy server, run the following commands: ``` git clone https://github.com/ading2210/libcurl.js --recursive -cd libcurl.js/server -./run.sh +cd libcurl.js +server/run.sh --static=./client ``` -You can use the `HOST` and `PORT` environment variables to control the hostname and port that the proxy server listens on. +For a full list of server arguments, see the [wisp-server-python documentation](https://github.com/MercuryWorkshop/wisp-server-python). ## Copyright: This project is licensed under the GNU AGPL v3. ### Copyright Notice: ``` -ading2210/libcurl.js - A port of libcurl to WASM +ading2210/libcurl.js - A port of libcurl to WASM for use in the browser. Copyright (C) 2023 ading2210 This program is free software: you can redistribute it and/or modify diff --git a/client/build.sh b/client/build.sh index 9caabfd..570c734 100755 --- a/client/build.sh +++ b/client/build.sh @@ -2,53 +2,93 @@ set -e -INCLUDE_DIR="build/curl-wasm/include/" -LIB_DIR="build/curl-wasm/lib/" -OUT_FILE="out/libcurl.js" -ES6_FILE="out/libcurl_module.mjs" -MODULE_FILE="out/emscripten_compiled.js" +#path definitions +OUT_DIR="${OUT_DIR:=out}" +BUILD_DIR="build" +C_DIR="libcurl" FRAGMENTS_DIR="fragments" -WRAPPER_SOURCE="main.js" +JAVSCRIPT_DIR="javascript" WISP_CLIENT="wisp_client" -EXPORTED_FUNCS="_init_curl,_start_request,_tick_request,_active_requests" -RUNTIME_METHODS="addFunction,removeFunction,allocate,ALLOC_NORMAL" -COMPILER_OPTIONS="-o $MODULE_FILE -lcurl -lwolfssl -lcjson -lz -lbrotlidec -lbrotlicommon -I $INCLUDE_DIR -L $LIB_DIR" -EMSCRIPTEN_OPTIONS="-lwebsocket.js -sASSERTIONS=1 -sALLOW_TABLE_GROWTH -sALLOW_MEMORY_GROWTH -sEXPORTED_FUNCTIONS=$EXPORTED_FUNCS -sEXPORTED_RUNTIME_METHODS=$RUNTIME_METHODS" +INCLUDE_DIR="$BUILD_DIR/curl-wasm/include/" +LIB_DIR="$BUILD_DIR/curl-wasm/lib/" +OUT_FILE="$OUT_DIR/libcurl.js" +ES6_FILE="$OUT_DIR/libcurl.mjs" +MODULE_FILE="$OUT_DIR/emscripten_compiled.js" +COMPILED_FILE="$OUT_DIR/emscripten_compiled.wasm" +WASM_FILE="$OUT_DIR/libcurl.wasm" -if [ "$1" = "release" ]; then +#read exported functions +EXPORTED_FUNCS="" +for func in $(cat exported_funcs.txt); do + EXPORTED_FUNCS="$EXPORTED_FUNCS,_$func" +done +EXPORTED_FUNCS="${EXPORTED_FUNCS:1}" + +#compile options +RUNTIME_METHODS="addFunction,removeFunction,allocate,ALLOC_NORMAL" +COMPILER_OPTIONS="-o $MODULE_FILE -lcurl -lwolfssl -lcjson -lz -lbrotlidec -lbrotlicommon -lnghttp2 -I $INCLUDE_DIR -L $LIB_DIR" +EMSCRIPTEN_OPTIONS="-lwebsocket.js -sENVIRONMENT=worker,web -sASSERTIONS=1 -sLLD_REPORT_UNDEFINED -sALLOW_TABLE_GROWTH -sALLOW_MEMORY_GROWTH -sEXPORTED_FUNCTIONS=$EXPORTED_FUNCS -sEXPORTED_RUNTIME_METHODS=$RUNTIME_METHODS" + +#clean output dir +rm -rf $OUT_DIR +mkdir -p $OUT_DIR + +if [[ "$*" == *"all"* ]]; then + mkdir -p $OUT_DIR/release + mkdir -p $OUT_DIR/single_file + OUT_DIR=$OUT_DIR/release ./build.sh release + OUT_DIR=$OUT_DIR/single_file ./build.sh release single_file + mv $OUT_DIR/release/* $OUT_DIR + mv $OUT_DIR/single_file/* $OUT_DIR + rm -rf $OUT_DIR/release + rm -rf $OUT_DIR/single_file + exit 0 +fi + +if [[ "$*" == *"release"* ]]; then COMPILER_OPTIONS="-Oz -flto $COMPILER_OPTIONS" - EMSCRIPTEN_OPTIONS="-sSINGLE_FILE $EMSCRIPTEN_OPTIONS" + echo "note: building with release optimizations" else COMPILER_OPTIONS="$COMPILER_OPTIONS --profiling -g" fi +if [[ "$*" == *"single_file"* ]]; then + EMSCRIPTEN_OPTIONS="-sSINGLE_FILE $EMSCRIPTEN_OPTIONS" + OUT_FILE="$OUT_DIR/libcurl_full.js" + ES6_FILE="$OUT_DIR/libcurl_full.mjs" + echo "note: building as a single js file" +fi + #ensure deps are compiled tools/all_deps.sh tools/generate_cert.sh -#clean output dir -rm -rf out -mkdir -p out - -#compile the main c file - but only if the source has been modified -COMPILE_CMD="emcc main.c $COMPILER_OPTIONS $EMSCRIPTEN_OPTIONS" +#compile the main c file +COMPILE_CMD="emcc $C_DIR/*.c $COMPILER_OPTIONS $EMSCRIPTEN_OPTIONS" echo $COMPILE_CMD $COMPILE_CMD +mv $COMPILED_FILE $WASM_FILE || true #merge compiled emscripten module and wrapper code -cp $WRAPPER_SOURCE $OUT_FILE +cp $JAVSCRIPT_DIR/main.js $OUT_FILE sed -i "/__emscripten_output__/r $MODULE_FILE" $OUT_FILE rm $MODULE_FILE -#add wisp libraries +#add version number and copyright notice +VERSION=$(cat package.json | jq -r '.version') +sed -i "s/__library_version__/$VERSION/" $OUT_FILE + +#add extra libraries sed -i "/__extra_libraries__/r $WISP_CLIENT/polyfill.js" $OUT_FILE sed -i "/__extra_libraries__/r $WISP_CLIENT/wisp.js" $OUT_FILE -sed -i "/__extra_libraries__/r ./messages.js" $OUT_FILE +sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/messages.js" $OUT_FILE +sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/websocket.js" $OUT_FILE +sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/copyright.js" $OUT_FILE #apply patches -python3 patcher.py $FRAGMENTS_DIR $OUT_FILE +python3 tools/patch_js.py $FRAGMENTS_DIR $OUT_FILE #generate es6 module cp $OUT_FILE $ES6_FILE -sed -i 's/window.libcurl/export const libcurl/' $ES6_FILE \ No newline at end of file +sed -i 's/const libcurl = /export const libcurl = /' $ES6_FILE \ No newline at end of file diff --git a/client/exported_funcs.txt b/client/exported_funcs.txt new file mode 100644 index 0000000..f437e28 --- /dev/null +++ b/client/exported_funcs.txt @@ -0,0 +1,19 @@ +init_curl +start_request +tick_request +active_requests + +get_version + +recv_from_websocket +send_to_websocket +close_websocket +cleanup_websocket +get_result_size +get_result_buffer +get_result_code +get_result_closed +get_result_bytes_left +get_result_is_text + +free \ No newline at end of file diff --git a/client/fragments/load_later.js b/client/fragments/load_later.js new file mode 100644 index 0000000..2fb25b4 --- /dev/null +++ b/client/fragments/load_later.js @@ -0,0 +1,16 @@ +/* REPLACE +var asm ?= ?createWasm\(\); +*/ +if (isDataURI(wasmBinaryFile)) var asm = createWasm(); +else var asm = null; + +/* REPLACE +var wasmExports ?= ?createWasm\(\); +*/ +if (isDataURI(wasmBinaryFile)) var wasmExports = createWasm(); +else var wasmExports = null; + +/* REPLACE +run\(\);\n\n +*/ +if (isDataURI(wasmBinaryFile)) run(); \ No newline at end of file diff --git a/client/fragments/silence_socket.js b/client/fragments/silence_socket.js index 2178f6a..de7bf00 100644 --- a/client/fragments/silence_socket.js +++ b/client/fragments/silence_socket.js @@ -4,6 +4,6 @@ err\("__syscall_getsockname " ?\+ ?fd\); /* INSERT -function _emscripten_console_error\(str\) { +function _emscripten_console_error\(str\) ?{ */ if (UTF8ToString(str).endsWith("__syscall_setsockopt\\n")) return; \ No newline at end of file diff --git a/client/index.html b/client/index.html index ca21b6a..8555276 100644 --- a/client/index.html +++ b/client/index.html @@ -3,11 +3,12 @@ - + diff --git a/client/javascript/copyright.js b/client/javascript/copyright.js new file mode 100644 index 0000000..9ad6c3e --- /dev/null +++ b/client/javascript/copyright.js @@ -0,0 +1,15 @@ +const copyright_notice = `ading2210/libcurl.js - A port of libcurl to WASM for use in the browser. +Copyright (C) 2023 ading2210 + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see .`; \ No newline at end of file diff --git a/client/javascript/main.js b/client/javascript/main.js new file mode 100644 index 0000000..fc8b6f8 --- /dev/null +++ b/client/javascript/main.js @@ -0,0 +1,323 @@ +/* +ading2210/libcurl.js - A port of libcurl to WASM for the browser. +Copyright (C) 2023 ading2210 + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU Affero General Public License for more details. + +You should have received a copy of the GNU Affero General Public License +along with this program. If not, see . +*/ + +//everything is wrapped in a function to prevent emscripten from polluting the global scope +const libcurl = (function() { + +//emscripten compiled code is inserted here +/* __emscripten_output__ */ + +//extra client code goes here +/* __extra_libraries__ */ + +var websocket_url = null; +var event_loop = null; +var active_requests = 0; +var wasm_ready = false; +var version_dict = null; +const libcurl_version = "__library_version__"; + +function check_loaded(check_websocket) { + if (!wasm_ready) { + throw new Error("wasm not loaded yet, please call libcurl.load_wasm first"); + } + if (!websocket_url && check_websocket) { + throw new Error("websocket proxy url not set, please call libcurl.set_websocket"); + } +} + +//a case insensitive dictionary for request headers +class HeadersDict { + constructor(obj) { + for (let key in obj) { + this[key] = obj[key]; + } + return new Proxy(this, this); + } + get(target, prop) { + let keys = Object.keys(this); + for (let key of keys) { + if (key.toLowerCase() === prop.toLowerCase()) { + return this[key]; + } + } + } + set(target, prop, value) { + let keys = Object.keys(this); + for (let key of keys) { + if (key.toLowerCase() === prop.toLowerCase()) { + this[key] = value; + } + } + this[prop] = value; + return true; + } +} + +function allocate_str(str) { + return allocate(intArrayFromString(str), ALLOC_NORMAL); +} + +function allocate_array(array) { + return allocate(array, ALLOC_NORMAL); +} + +//low level interface with c code +function perform_request(url, params, js_data_callback, js_end_callback, body=null) { + let params_str = JSON.stringify(params); + let end_callback_ptr; + let data_callback_ptr; + let url_ptr = allocate_str(url); + let params_ptr = allocate_str(params_str); + + let body_ptr = null; + let body_length = 0; + if (body) { //assume body is an int8array + body_ptr = allocate_array(body); + body_length = body.length; + } + + let end_callback = (error, response_json_ptr) => { + let response_json = UTF8ToString(response_json_ptr); + let response_info = JSON.parse(response_json); + + Module.removeFunction(end_callback_ptr); + Module.removeFunction(data_callback_ptr); + if (body_ptr) _free(body_ptr); + _free(url_ptr); + _free(response_json_ptr); + + if (error != 0) console.error("request failed with error code " + error); + active_requests --; + js_end_callback(error, response_info); + } + + let data_callback = (chunk_ptr, chunk_size) => { + let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size); + let chunk = new Uint8Array(data); + js_data_callback(chunk); + } + + end_callback_ptr = Module.addFunction(end_callback, "vii"); + data_callback_ptr = Module.addFunction(data_callback, "vii"); + let http_handle = _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, body_ptr, body_length); + _free(params_ptr); + + active_requests ++; + _tick_request(); + if (!event_loop) { + event_loop = setInterval(() => { + if (_active_requests() || active_requests) { + _tick_request(); + } + else { + clearInterval(event_loop); + event_loop = null; + } + }, 0); + } + + return http_handle; +} + +function merge_arrays(arrays) { + let total_len = arrays.reduce((acc, val) => acc + val.length, 0); + let new_array = new Uint8Array(total_len); + let offset = 0; + for (let array of arrays) { + new_array.set(array, offset); + offset += array.length; + } + return new_array; +} + +function create_response(response_data, response_info) { + response_info.ok = response_info.status >= 200 && response_info.status < 300; + response_info.statusText = status_messages[response_info.status] || ""; + if (response_info.status === 204 || response_info.status === 205) { + response_data = null; + } + + //construct base response object + let response_obj = new Response(response_data, response_info); + for (let key in response_info) { + if (key == "headers") continue; + Object.defineProperty(response_obj, key, { + writable: false, + value: response_info[key] + }); + } + + //create headers object + Object.defineProperty(response_obj, "headers", { + writable: false, + value: new Headers() + }); + for (let header_name in response_info.headers) { + let header_value = response_info.headers[header_name]; + response_obj.headers.append(header_name, header_value); + } + + return response_obj; +} + +async function parse_body(data) { + let data_array = null; + if (typeof data === "string") { + data_array = new TextEncoder().encode(data); + } + + else if (data instanceof Blob) { + let array_buffer = await data.arrayBuffer(); + data_array = new Uint8Array(array_buffer); + } + + //any typedarray + else if (data instanceof ArrayBuffer) { + //dataview objects + if (ArrayBuffer.isView(data) && data instanceof DataView) { + data_array = new Uint8Array(data.buffer); + } + //regular typed arrays + else if (ArrayBuffer.isView(data)) { + data_array = Uint8Array.from(data); + } + //regular arraybuffers + else { + data_array = new Uint8Array(data); + } + } + + else if (data instanceof ReadableStream) { + let chunks = []; + for await (let chunk of data) { + chunks.push(chunk); + } + data_array = merge_arrays(chunks); + } + + else { + throw "invalid data type to be sent"; + } + return data_array; +} + +async function create_options(params) { + let body = null; + if (params.body) { + body = await parse_body(params.body); + params.body = true; + } + + if (!params.headers) params.headers = {}; + params.headers = new HeadersDict(params.headers); + + if (params.referer) { + params.headers["Referer"] = params.referer; + } + if (!params.headers["User-Agent"]) { + params.headers["User-Agent"] = navigator.userAgent; + } + + return body; +} + +//wrap perform_request in a promise +function perform_request_async(url, params, body) { + return new Promise((resolve, reject) => { + let chunks = []; + let data_callback = (new_data) => { + chunks.push(new_data); + }; + + let finish_callback = (error, response_info) => { + if (error != 0) { + reject("libcurl.js encountered an error: " + error); + return; + } + let response_data = merge_arrays(chunks); + chunks = null; + let response_obj = create_response(response_data, response_info); + resolve(response_obj); + } + perform_request(url, params, data_callback, finish_callback, body); + }); +} + +async function libcurl_fetch(url, params={}) { + check_loaded(true); + let body = await create_options(params); + return await perform_request_async(url, params, body); +} + +function set_websocket_url(url) { + websocket_url = url; + if (!Module.websocket) { + document.addEventListener("libcurl_load", () => { + set_websocket_url(url); + }); + } + else Module.websocket.url = url; +} + +function get_version() { + if (!wasm_ready) return null; + if (version_dict) return version_dict; + + let version_ptr = _get_version(); + let version_str = UTF8ToString(version_ptr); + _free(version_ptr); + version_dict = JSON.parse(version_str); + version_dict.lib = libcurl_version; + return version_dict; +} + +function main() { + wasm_ready = true; + _init_curl(); + set_websocket_url(websocket_url); + + let load_event = new Event("libcurl_load"); + document.dispatchEvent(load_event); +} + +function load_wasm(url) { + wasmBinaryFile = url; + createWasm(); + run(); +} + +Module.onRuntimeInitialized = main; +return { + fetch: libcurl_fetch, + set_websocket: set_websocket_url, + load_wasm: load_wasm, + wisp: _wisp_connections, + WebSocket: CurlWebSocket, + + get copyright() {return copyright_notice}, + get version() {return get_version()}, + get ready() {return wasm_ready}, + + get stdout() {return out}, + set stdout(callback) {out = callback}, + get stderr() {return err}, + set stderr(callback) {err = callback} +} + +})() \ No newline at end of file diff --git a/client/messages.js b/client/javascript/messages.js similarity index 100% rename from client/messages.js rename to client/javascript/messages.js diff --git a/client/javascript/websocket.js b/client/javascript/websocket.js new file mode 100644 index 0000000..23af368 --- /dev/null +++ b/client/javascript/websocket.js @@ -0,0 +1,201 @@ +//class for custom websocket + +class CurlWebSocket extends EventTarget { + constructor(url, protocols=[], websocket_debug=false) { + super(); + check_loaded(true); + if (!url.startsWith("wss://") && !url.startsWith("ws://")) { + throw new SyntaxError("invalid url"); + } + + this.url = url; + this.protocols = protocols; + this.binaryType = "blob"; + this.recv_buffer = []; + this.websocket_debug = websocket_debug; + + //legacy event handlers + this.onopen = () => {}; + this.onerror = () => {}; + this.onmessage = () => {}; + this.onclose = () => {}; + + this.CONNECTING = 0; + this.OPEN = 1; + this.CLOSING = 2; + this.CLOSED = 3; + + this.connect(); + } + + connect() { + this.status = this.CONNECTING; + let data_callback = () => {}; + let finish_callback = (error, response_info) => { + this.finish_callback(error, response_info); + } + let options = {}; + if (this.protocols) { + options.headers = { + "Sec-Websocket-Protocol": this.protocols.join(", "), + }; + } + if (this.websocket_debug) { + options._libcurl_verbose = 1; + } + this.http_handle = perform_request(this.url, options, data_callback, finish_callback, null); + this.recv_loop(); + } + + recv() { + let buffer_size = 64*1024; + let result_ptr = _recv_from_websocket(this.http_handle, buffer_size); + let data_ptr = _get_result_buffer(result_ptr); + let result_code = _get_result_code(result_ptr); + + if (result_code == 0) { //CURLE_OK - data recieved + if (_get_result_closed(result_ptr)) { + //this.pass_buffer(); + this.close_callback(); + return; + } + + let data_size = _get_result_size(result_ptr); + let data_heap = Module.HEAPU8.subarray(data_ptr, data_ptr + data_size); + let data = new Uint8Array(data_heap); + + this.recv_buffer.push(data); + if (data_size !== buffer_size && !_get_result_bytes_left(result_ptr)) { //message finished + let full_data = merge_arrays(this.recv_buffer); + let is_text = _get_result_is_text(result_ptr) + this.recv_buffer = []; + this.recv_callback(full_data, is_text); + } + } + + if (result_code == 52) { //CURLE_GOT_NOTHING - socket closed + this.close_callback(); + } + + _free(data_ptr); + _free(result_ptr); + } + + recv_loop() { + this.event_loop = setInterval(() => { + this.recv(); + }, 1); + } + + recv_callback(data, is_text=false) { + let converted; + if (is_text) { + converted = new TextDecoder().decode(data); + } + else { + if (this.binaryType == "blob") { + converted = new Blob(data); + } + else if (this.binaryType == "arraybuffer") { + converted = data.buffer; + } + else { + throw "invalid binaryType string"; + } + } + + let msg_event = new MessageEvent("message", {data: converted}); + this.onmessage(msg_event); + this.dispatchEvent(msg_event); + } + + close_callback(error=false) { + if (this.status == this.CLOSED) return; + this.status = this.CLOSED; + + clearInterval(this.event_loop); + _cleanup_websocket(); + + if (error) { + let error_event = new Event("error"); + this.dispatchEvent(error_event); + this.onerror(error_event); + } + else { + let close_event = new CloseEvent("close"); + this.dispatchEvent(close_event); + this.onclose(close_event); + } + } + + finish_callback(error, response_info) { + this.status = this.OPEN; + if (error != 0) this.close_callback(true); + let open_event = new Event("open"); + this.onopen(open_event); + this.dispatchEvent(open_event); + } + + send(data) { + let is_text = false; + if (this.status === this.CONNECTING) { + throw new DOMException("ws not ready yet"); + } + if (this.status === this.CLOSED) { + return; + } + + let data_array; + if (typeof data === "string") { + data_array = new TextEncoder().encode(data); + is_text = true; + } + else if (data instanceof Blob) { + data.arrayBuffer().then(array_buffer => { + data_array = new Uint8Array(array_buffer); + this.send(data_array); + }); + return; + } + //any typedarray + else if (data instanceof ArrayBuffer) { + //dataview objects + if (ArrayBuffer.isView(data) && data instanceof DataView) { + data_array = new Uint8Array(data.buffer); + } + //regular arraybuffers + else { + data_array = new Uint8Array(data); + } + } + //regular typed arrays + else if (ArrayBuffer.isView(data)) { + data_array = Uint8Array.from(data); + } + else { + throw "invalid data type to be sent"; + } + + let data_ptr = allocate_array(data_array); + let data_len = data.length; + _send_to_websocket(this.http_handle, data_ptr, data_len, is_text); + _free(data_ptr); + } + + close() { + _close_websocket(this.http_handle); + } + + get readyState() { + return this.status; + } + get bufferedAmount() { + return 0; + } + get protocol() { + return ""; + } + get extensions() { + return ""; + } +} \ No newline at end of file diff --git a/client/main.c b/client/libcurl/main.c similarity index 84% rename from client/main.c rename to client/libcurl/main.c index be843ef..4f5578a 100644 --- a/client/main.c +++ b/client/libcurl/main.c @@ -10,28 +10,19 @@ #include "cacert.h" #include "curl/multi.h" -typedef void(*DataCallback)(char* chunk_ptr, int chunk_size); -typedef void(*EndCallback)(int error, char* response_json); +#include "util.h" +#include "types.h" + void finish_request(CURLMsg *curl_msg); #define ERROR_REDIRECT_DISALLOWED -1 CURLM *multi_handle; int request_active = 0; +struct curl_blob cacert_blob; -struct RequestInfo { - int abort_on_redirect; - struct CURLMsg *curl_msg; - struct curl_slist* headers_list; - EndCallback end_callback; -}; - -int starts_with(const char *a, const char *b) { - return strncmp(a, b, strlen(b)) == 0; -} - -int write_function(void *data, size_t size, size_t nmemb, DataCallback data_callback) { - long real_size = size * nmemb; +size_t write_function(void *data, size_t size, size_t nmemb, DataCallback data_callback) { + size_t real_size = size * nmemb; char* chunk = malloc(real_size); memcpy(chunk, data, real_size); data_callback(chunk, real_size); @@ -57,13 +48,13 @@ void tick_request() { } } -void start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, const char* body, int body_length) { +CURL* start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, const char* body, int body_length) { CURL *http_handle = curl_easy_init(); int abort_on_redirect = 0; + int prevent_cleanup = 0; curl_easy_setopt(http_handle, CURLOPT_URL, url); - curl_easy_setopt(http_handle, CURLOPT_CAINFO, "/cacert.pem"); - curl_easy_setopt(http_handle, CURLOPT_CAPATH, "/cacert.pem"); + curl_easy_setopt(http_handle, CURLOPT_CAINFO_BLOB , cacert_blob); //callbacks to pass the response data back to js curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, &write_function); @@ -72,10 +63,12 @@ void start_request(const char* url, const char* json_params, DataCallback data_c //some default options curl_easy_setopt(http_handle, CURLOPT_FOLLOWLOCATION, 1); curl_easy_setopt(http_handle, CURLOPT_ACCEPT_ENCODING, ""); + curl_easy_setopt(http_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0); //if url is a websocket, tell curl that we should handle the connection manually if (starts_with(url, "wss://") || starts_with(url, "ws://")) { curl_easy_setopt(http_handle, CURLOPT_CONNECT_ONLY, 2L); + prevent_cleanup = 1; } //parse json options @@ -134,9 +127,12 @@ void start_request(const char* url, const char* json_params, DataCallback data_c request_info->curl_msg = NULL; request_info->headers_list = headers_list; request_info->end_callback = end_callback; + request_info->prevent_cleanup = prevent_cleanup; curl_easy_setopt(http_handle, CURLOPT_PRIVATE, request_info); curl_multi_add_handle(multi_handle, http_handle); + + return http_handle; } void finish_request(CURLMsg *curl_msg) { @@ -184,23 +180,22 @@ void finish_request(CURLMsg *curl_msg) { //clean up curl curl_slist_free_all(request_info->headers_list); + (*request_info->end_callback)(error, response_json_str); + if (request_info->prevent_cleanup) { + return; + } curl_multi_remove_handle(multi_handle, http_handle); curl_easy_cleanup(http_handle); - (*request_info->end_callback)(error, response_json_str); free(request_info); } -char* copy_bytes(const char* ptr, const int size) { - char* new_ptr = malloc(size); - memcpy(new_ptr, ptr, size); - return new_ptr; -} - void init_curl() { curl_global_init(CURL_GLOBAL_DEFAULT); multi_handle = curl_multi_init(); + curl_multi_setopt(multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 50L); + curl_multi_setopt(multi_handle, CURLMOPT_MAXCONNECTS, 40L); - FILE *file = fopen("/cacert.pem", "wb"); - fwrite(_cacert_pem, 1, _cacert_pem_len, file); - fclose(file); + cacert_blob.data = _cacert_pem; + cacert_blob.len = _cacert_pem_len; + cacert_blob.flags = CURL_BLOB_NOCOPY; } \ No newline at end of file diff --git a/client/libcurl/types.h b/client/libcurl/types.h new file mode 100644 index 0000000..84fb83a --- /dev/null +++ b/client/libcurl/types.h @@ -0,0 +1,19 @@ +typedef void(*DataCallback)(char* chunk_ptr, int chunk_size); +typedef void(*EndCallback)(int error, char* response_json); + +struct RequestInfo { + int abort_on_redirect; + int prevent_cleanup; + struct CURLMsg *curl_msg; + struct curl_slist* headers_list; + EndCallback end_callback; +}; + +struct WSResult { + int res; + int buffer_size; + int closed; + int bytes_left; + int is_text; + char* buffer; +}; \ No newline at end of file diff --git a/client/libcurl/util.c b/client/libcurl/util.c new file mode 100644 index 0000000..170d3ce --- /dev/null +++ b/client/libcurl/util.c @@ -0,0 +1,37 @@ +#include +#include +#include + +#include "curl/curl.h" +#include "cjson/cJSON.h" + +int starts_with(const char *a, const char *b) { + return strncmp(a, b, strlen(b)) == 0; +} + +char* get_version() { + struct curl_version_info_data *version_info = curl_version_info(CURLVERSION_NOW); + cJSON* version_json = cJSON_CreateObject(); + + cJSON* protocols_array = cJSON_CreateArray(); + const char *const *protocols = version_info->protocols; + for (; *protocols != NULL; protocols++) { + cJSON* protocol_item = cJSON_CreateString(*protocols); + cJSON_AddItemToArray(protocols_array, protocol_item); + } + + cJSON* curl_version_item = cJSON_CreateString(version_info->version); + cJSON* ssl_version_item = cJSON_CreateString(version_info->ssl_version); + cJSON* brotli_version_item = cJSON_CreateString(version_info->brotli_version); + cJSON* nghttp2_version_item = cJSON_CreateString(version_info->nghttp2_version); + + cJSON_AddItemToObject(version_json, "curl", curl_version_item); + cJSON_AddItemToObject(version_json, "ssl", ssl_version_item); + cJSON_AddItemToObject(version_json, "brotli", brotli_version_item); + cJSON_AddItemToObject(version_json, "nghttp2", nghttp2_version_item); + cJSON_AddItemToObject(version_json, "protocols", protocols_array); + + char* version_json_str = cJSON_Print(version_json); + cJSON_Delete(version_json); + return version_json_str; +} \ No newline at end of file diff --git a/client/libcurl/util.h b/client/libcurl/util.h new file mode 100644 index 0000000..fdb4982 --- /dev/null +++ b/client/libcurl/util.h @@ -0,0 +1 @@ +int starts_with(const char *a, const char *b); \ No newline at end of file diff --git a/client/libcurl/websocket.c b/client/libcurl/websocket.c new file mode 100644 index 0000000..4c7e5fd --- /dev/null +++ b/client/libcurl/websocket.c @@ -0,0 +1,66 @@ +#include + +#include "curl/curl.h" +#include "curl/websockets.h" + +#include "types.h" + +extern CURLM* multi_handle; + +struct WSResult* recv_from_websocket(CURL* http_handle, int buffer_size) { + const struct curl_ws_frame* ws_meta; + char* buffer = malloc(buffer_size); + size_t result_len; + CURLcode res = curl_ws_recv(http_handle, buffer, buffer_size, &result_len, &ws_meta); + + struct WSResult* result = malloc(sizeof(struct WSResult)); + result->buffer_size = result_len; + result->buffer = buffer; + result->res = (int) res; + result->closed = (ws_meta->flags & CURLWS_CLOSE); + result->is_text = (ws_meta->flags & CURLWS_TEXT); + result->bytes_left = ws_meta->bytesleft; + return result; +} + +int send_to_websocket(CURL* http_handle, const char* data, int data_len, int is_text) { + size_t sent; + unsigned int flags = CURLWS_BINARY; + if (is_text) flags = CURLWS_TEXT; + CURLcode res = curl_ws_send(http_handle, data, data_len, &sent, 0, flags); + return (int) res; +} + +void close_websocket(CURL* http_handle) { + size_t sent; + curl_ws_send(http_handle, "", 0, &sent, 0, CURLWS_CLOSE); +} + +//clean up the http handle associated with the websocket, since the main loop can't do this automatically +void cleanup_websocket(CURL* http_handle) { + struct RequestInfo *request_info; + curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info); + + curl_multi_remove_handle(multi_handle, http_handle); + curl_easy_cleanup(http_handle); + free(request_info); +} + +int get_result_size (const struct WSResult* result) { + return result->buffer_size; +} +char* get_result_buffer (const struct WSResult* result) { + return result->buffer; +} +int get_result_code (const struct WSResult* result) { + return result->res; +} +int get_result_closed (const struct WSResult* result) { + return result->closed; +} +int get_result_bytes_left (const struct WSResult* result) { + return result->bytes_left; +} +int get_result_is_text (const struct WSResult* result) { + return result->is_text; +} \ No newline at end of file diff --git a/client/main.js b/client/main.js deleted file mode 100644 index d732dcf..0000000 --- a/client/main.js +++ /dev/null @@ -1,196 +0,0 @@ -//everything is wrapped in a function to prevent emscripten from polluting the global scope -window.libcurl = (function() { - -//emscripten compiled code is inserted here -/* __emscripten_output__ */ - -//extra client code goes here -/* __extra_libraries__ */ - -const websocket_url = `wss://${location.hostname}/ws/`; -var event_loop = null; - -//a case insensitive dictionary for request headers -class Headers { - constructor(obj) { - for (let key in obj) { - this[key] = obj[key]; - } - return new Proxy(this, this); - } - get(target, prop) { - let keys = Object.keys(this); - for (let key of keys) { - if (key.toLowerCase() === prop.toLowerCase()) { - return this[key]; - } - } - } - set(target, prop, value) { - let keys = Object.keys(this); - for (let key of keys) { - if (key.toLowerCase() === prop.toLowerCase()) { - this[key] = value; - } - } - this[prop] = value; - return true; - } -} - -function is_str(obj) { - return typeof obj === 'string' || obj instanceof String; -} - -function allocate_str(str) { - return allocate(intArrayFromString(str), ALLOC_NORMAL); -} - -function allocate_array(array) { - return allocate(array, ALLOC_NORMAL); -} - -//low level interface with c code -function perform_request(url, params, js_data_callback, js_end_callback, body=null) { - let params_str = JSON.stringify(params); - let end_callback_ptr; - let data_callback_ptr; - let url_ptr = allocate_str(url); - let params_ptr = allocate_str(params_str); - - let body_ptr = null; - let body_length = 0; - if (body) { //assume body is an int8array - body_ptr = allocate_array(body); - body_length = body.length; - } - - let end_callback = (error, response_json_ptr) => { - let response_json = UTF8ToString(response_json_ptr); - let response_info = JSON.parse(response_json); - - Module.removeFunction(end_callback_ptr); - Module.removeFunction(data_callback_ptr); - if (body_ptr) _free(body_ptr); - _free(url_ptr); - _free(response_json_ptr); - - js_end_callback(error, response_info); - } - - let data_callback = (chunk_ptr, chunk_size) => { - let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size); - let chunk = new Uint8Array(data); - js_data_callback(chunk); - } - - end_callback_ptr = Module.addFunction(end_callback, "vii"); - data_callback_ptr = Module.addFunction(data_callback, "vii"); - _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, body_ptr, body_length); - _free(params_ptr); - - _tick_request(); - if (!event_loop) { - event_loop = setInterval(() => { - _tick_request(); - if (!_active_requests()) { - clearInterval(event_loop); - event_loop = null; - } - }, 0); - } -} - -function merge_arrays(arrays) { - let total_len = arrays.reduce((acc, val) => acc + val.length, 0); - let new_array = new Uint8Array(total_len); - let offset = 0; - for (let array of arrays) { - new_array.set(array, offset); - offset += array.length; - } - return new_array; -} - -function create_response(response_data, response_info) { - delete response_info.error; - response_info.ok = response_info.status >= 200 && response_info.status < 300; - response_info.statusText = status_messages[response_info.status] || ""; - - let response_obj = new Response(response_data, response_info); - for (let key in response_info) { - Object.defineProperty(response_obj, key, { - writable: false, - value: response_info[key] - }); - } - return response_obj; -} - -function create_options(params) { - let body = null; - if (params.body) { - if (is_str(params.body)) { - body = new TextEncoder().encode(params.body); - } - else { - body = Uint8Array.from(params); - } - params.body = true; - } - - if (!params.headers) params.headers = {}; - params.headers = new Headers(params.headers); - - if (params.referer) { - params.headers["Referer"] = params.referer; - } - if (!params.headers["User-Agent"]) { - params.headers["User-Agent"] = navigator.userAgent; - } - - return body; -} - -function libcurl_fetch(url, params={}) { - let body = create_options(params); - - return new Promise((resolve, reject) => { - let chunks = []; - let data_callback = (new_data) => { - chunks.push(new_data); - }; - - let finish_callback = (error, response_info) => { - if (error != 0) { - reject("libcurl.js encountered an error: " + error + "\n" + response_info.error); - return; - } - let response_data = merge_arrays(chunks); - let response_obj = create_response(response_data, response_info); - resolve(response_obj); - } - perform_request(url, params, data_callback, finish_callback, body); - }) -} - -function set_websocket_url(url) { - Module.websocket.url = url; -} - -function main() { - console.log("emscripten module loaded"); - _init_curl(); - set_websocket_url(websocket_url); - - let load_event = new Event("libcurl_load"); - document.dispatchEvent(load_event); -} - -Module.onRuntimeInitialized = main; -return { - fetch: libcurl_fetch, - set_websocket: set_websocket_url, -} - -})() \ No newline at end of file diff --git a/client/package.json b/client/package.json new file mode 100644 index 0000000..d85ef44 --- /dev/null +++ b/client/package.json @@ -0,0 +1,19 @@ +{ + "name": "libcurl.js", + "version": "0.3.7", + "description": "An experimental port of libcurl to WebAssembly for use in the browser.", + "main": "libcurl.mjs", + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/ading2210/libcurl.js.git" + }, + "author": "ading2210", + "license": "AGPL-3.0-or-later", + "bugs": { + "url": "https://github.com/ading2210/libcurl.js/issues" + }, + "homepage": "https://github.com/ading2210/libcurl.js" +} diff --git a/client/publish.sh b/client/publish.sh new file mode 100755 index 0000000..2a055e0 --- /dev/null +++ b/client/publish.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +#publish libcurl.js as an npm package + +./build.sh all + +cp package.json out +cp ../README.md out +cp ../LICENSE out + +cd out +npm publish \ No newline at end of file diff --git a/client/tests/index.html b/client/tests/index.html new file mode 100644 index 0000000..c0c317b --- /dev/null +++ b/client/tests/index.html @@ -0,0 +1,40 @@ + + + + + + + + + +

libcurl.js unit test runner

+ + \ No newline at end of file diff --git a/client/tests/run.sh b/client/tests/run.sh new file mode 100755 index 0000000..62b74b6 --- /dev/null +++ b/client/tests/run.sh @@ -0,0 +1,26 @@ +#!/bin/bash + +set -e + +trap "exit" INT TERM +trap "kill 0" EXIT +../server/run.sh --static=$(pwd) >/dev/null & + +echo -n "waiting for wisp server to start" +i=0 +until $(curl --output /dev/null --silent --head "http://localhost:6001/"); do + if [ "$i" = "30" ]; then + echo -e "\ntests failed. wisp server failed to start" + exit 1 + fi + + echo -n "." + i=$(($i+1)) + sleep 1 +done +echo + + +sleep 1 +echo "wisp server ready, running tests" +python3 tests/run_tests.py diff --git a/client/tests/run_tests.py b/client/tests/run_tests.py new file mode 100644 index 0000000..5b14bf5 --- /dev/null +++ b/client/tests/run_tests.py @@ -0,0 +1,53 @@ +import unittest + +from selenium import webdriver +from selenium.webdriver.common.desired_capabilities import DesiredCapabilities +from selenium.webdriver.common.by import By +from selenium.webdriver.support.ui import WebDriverWait +from selenium.webdriver.support import expected_conditions as EC + +class JSTest(unittest.TestCase): + def setUp(self): + options = webdriver.ChromeOptions() + options.add_argument("start-maximized") + options.add_argument("enable-automation") + options.add_argument("--headless") + options.add_argument("--no-sandbox") + options.add_argument("--disable-dev-shm-usage") + options.add_argument("--disable-browser-side-navigation") + options.add_argument("--disable-gpu") + options.set_capability("goog:loggingPrefs", {"browser": "ALL"}) + + self.browser = webdriver.Chrome(options=options) + + def tearDown(self): + self.browser.quit() + + def run_test(self, script): + self.browser.get(f"http://localhost:6001/tests/#{script}") + wait = WebDriverWait(self.browser, 20) + result = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.flag'))).get_attribute("result") + + if result != "success": + for entry in self.browser.get_log("browser"): + print(f"{entry['level']}: {entry['message']}") + + assert result == "success" + + def test_fetch_once(self): + self.run_test("fetch_once.js") + + def test_fetch_multiple(self): + self.run_test("fetch_multiple.js") + + def test_fetch_parallel(self): + self.run_test("fetch_parallel.js") + + def test_websocket(self): + self.run_test("test_websocket.js") + + def test_redirect_out(self): + self.run_test("redirect_out.js") + +if __name__ == "__main__": + unittest.main() \ No newline at end of file diff --git a/client/tests/scripts/fetch_multiple.js b/client/tests/scripts/fetch_multiple.js new file mode 100644 index 0000000..fe7cb86 --- /dev/null +++ b/client/tests/scripts/fetch_multiple.js @@ -0,0 +1,7 @@ +async function test() { + await libcurl.fetch("https://example.com/"); + for (let i=0; i<20; i++) { + let r = await libcurl.fetch("https://example.com/"); + assert(r.status === 200, "wrong status"); + } +} \ No newline at end of file diff --git a/client/tests/scripts/fetch_once.js b/client/tests/scripts/fetch_once.js new file mode 100644 index 0000000..e94fefa --- /dev/null +++ b/client/tests/scripts/fetch_once.js @@ -0,0 +1,4 @@ +async function test() { + let r = await libcurl.fetch("https://example.com/"); + assert(r.status === 200, "wrong status"); +} \ No newline at end of file diff --git a/client/tests/scripts/fetch_parallel.js b/client/tests/scripts/fetch_parallel.js new file mode 100644 index 0000000..98a083a --- /dev/null +++ b/client/tests/scripts/fetch_parallel.js @@ -0,0 +1,8 @@ +async function test() { + await libcurl.fetch("https://www.example.com/"); + let promises = []; + for (let i=0; i<10; i++) { + promises.push(libcurl.fetch("https://www.example.com/")) + } + await Promise.all(promises); +} diff --git a/client/tests/scripts/redirect_out.js b/client/tests/scripts/redirect_out.js new file mode 100644 index 0000000..34d2aab --- /dev/null +++ b/client/tests/scripts/redirect_out.js @@ -0,0 +1,11 @@ +async function test() { + let output = []; + function out_callback(text) { + output.push(text); + } + libcurl.stdout = out_callback; + libcurl.stderr = out_callback; + await libcurl.fetch("https://example.com/", {_libcurl_verbose: 1}); + console.log(output); + assert(output[0] === "* Host example.com:443 was resolved.", "unexpected output in stderr"); +} \ No newline at end of file diff --git a/client/tests/scripts/test_websocket.js b/client/tests/scripts/test_websocket.js new file mode 100644 index 0000000..4a10cf3 --- /dev/null +++ b/client/tests/scripts/test_websocket.js @@ -0,0 +1,24 @@ +function test() { + let message_len = 128*1024; + + return new Promise((resolve, reject) => { + let ws = new libcurl.WebSocket("wss://echo.websocket.org"); + ws.addEventListener("open", () => { + ws.send("hello".repeat(message_len)); + }); + + let messages = 0; + ws.addEventListener("message", (event) => { + messages += 1; + if (messages >= 2) { + if (event.data !== "hello".repeat(message_len)) reject("unexpected response"); + if (messages >= 11) resolve(); + ws.send("hello".repeat(message_len)); + } + }); + + ws.addEventListener("error", () => { + reject("ws error occurred"); + }); + }) +} \ No newline at end of file diff --git a/client/tools/all_deps.sh b/client/tools/all_deps.sh index 20fbeb5..3df44b1 100755 --- a/client/tools/all_deps.sh +++ b/client/tools/all_deps.sh @@ -2,6 +2,7 @@ #build all deps +set -e mkdir -p build WOLFSSL_PREFIX=$(realpath build/wolfssl-wasm) @@ -9,6 +10,7 @@ CJSON_PREFIX=$(realpath build/cjson-wasm) CURL_PREFIX=$(realpath build/curl-wasm) ZLIB_PREFIX=$(realpath build/zlib-wasm) BROTLI_PREFIX=$(realpath build/brotli-wasm) +NGHTTP2_PREFIX=$(realpath build/nghttp2-wasm) if [ ! -d $WOLFSSL_PREFIX ]; then tools/openssl.sh @@ -22,6 +24,9 @@ fi if [ ! -d $BROTLI_PREFIX ]; then tools/brotli.sh fi +if [ ! -d $NGHTTP2_PREFIX ]; then + tools/nghttp2.sh +fi if [ ! -d $CURL_PREFIX ]; then tools/curl.sh fi @@ -29,4 +34,5 @@ fi cp -r $WOLFSSL_PREFIX/* $CURL_PREFIX cp -r $CJSON_PREFIX/* $CURL_PREFIX cp -r $ZLIB_PREFIX/* $CURL_PREFIX -cp -r $BROTLI_PREFIX/* $CURL_PREFIX \ No newline at end of file +cp -r $BROTLI_PREFIX/* $CURL_PREFIX +cp -r $NGHTTP2_PREFIX/* $CURL_PREFIX \ No newline at end of file diff --git a/client/tools/brotli.sh b/client/tools/brotli.sh index 1210f4b..c9cba4a 100755 --- a/client/tools/brotli.sh +++ b/client/tools/brotli.sh @@ -10,7 +10,7 @@ PREFIX=$(realpath build/brotli-wasm) cd build rm -rf brotli -git clone -b master --depth=1 https://github.com/google/brotli +git clone -b v1.1.0 --depth=1 https://github.com/google/brotli cd brotli emcmake cmake . -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./installed diff --git a/client/tools/cjson.sh b/client/tools/cjson.sh index acf7290..5cdfc1e 100755 --- a/client/tools/cjson.sh +++ b/client/tools/cjson.sh @@ -11,7 +11,7 @@ mkdir -p $PREFIX cd build rm -rf cjson -git clone -b master --depth=1 https://github.com/DaveGamble/cJSON cjson +git clone -b v1.7.17 --depth=1 https://github.com/DaveGamble/cJSON cjson cd cjson sed -i 's/-fstack-protector-strong//' Makefile diff --git a/client/tools/curl.sh b/client/tools/curl.sh index 148fae2..4d52f71 100755 --- a/client/tools/curl.sh +++ b/client/tools/curl.sh @@ -10,14 +10,15 @@ PREFIX=$(realpath build/curl-wasm) WOLFSSL_PREFIX=$(realpath build/wolfssl-wasm) ZLIB_PREFIX=$(realpath build/zlib-wasm) BROTLI_PREFIX=$(realpath build/brotli-wasm) +NGHTTP2_PREFIX=$(realpath build/nghttp2-wasm) cd build rm -rf curl -git clone -b master --depth=1 https://github.com/curl/curl +git clone -b curl-8_6_0 --depth=1 https://github.com/curl/curl cd curl autoreconf -fi -emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX +emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX --with-nghttp2=$NGHTTP2_PREFIX emmake make -j$CORE_COUNT CFLAGS="-Oz -pthread" LIBS="-lbrotlicommon" rm -rf $PREFIX diff --git a/client/tools/nghttp2.sh b/client/tools/nghttp2.sh new file mode 100755 index 0000000..58a5aba --- /dev/null +++ b/client/tools/nghttp2.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +#compile nghttp2 for use with emscripten + +set -x +set -e + +CORE_COUNT=$(nproc --all) +PREFIX=$(realpath build/nghttp2-wasm) + +cd build +rm -rf nghttp2 +git clone -b v1.59.0 --depth=1 https://github.com/nghttp2/nghttp2 +cd nghttp2 + +rm -rf $PREFIX +mkdir -p $PREFIX + +autoreconf -fi +emconfigure ./configure --host i686-linux --enable-static --disable-shared --enable-lib-only --prefix=$PREFIX +emmake make -j$CORE_COUNT +make install + +cd ../../ \ No newline at end of file diff --git a/client/patcher.py b/client/tools/patch_js.py similarity index 83% rename from client/patcher.py rename to client/tools/patch_js.py index bf06c6d..65efcc5 100644 --- a/client/patcher.py +++ b/client/tools/patch_js.py @@ -16,6 +16,9 @@ for fragment_file in fragments_path.iterdir(): matches = re.findall(match_regex, fragment_text, re.S) for mode, patch_regex, patch_text, _ in matches: + fragment_matches = re.findall(patch_regex, target_text) + if not fragment_matches: + print(f"warning: regex did not match anything for '{patch_regex}'"); if mode == "DELETE": target_text = re.sub(patch_regex, "", target_text) elif mode == "REPLACE": diff --git a/client/tools/zlib.sh b/client/tools/zlib.sh index 624246e..10c7bd1 100755 --- a/client/tools/zlib.sh +++ b/client/tools/zlib.sh @@ -10,7 +10,7 @@ PREFIX=$(realpath build/zlib-wasm) cd build rm -rf zlib -git clone -b master --depth=1 https://github.com/madler/zlib +git clone -b v1.3.1 --depth=1 https://github.com/madler/zlib cd zlib emconfigure ./configure --static diff --git a/client/wisp_client b/client/wisp_client index 0a80885..51ad95a 160000 --- a/client/wisp_client +++ b/client/wisp_client @@ -1 +1 @@ -Subproject commit 0a80885090b6247f42bc07cc85b441d8d719f551 +Subproject commit 51ad95a6d912ec404c20284f0cded40c0b5c4e62 diff --git a/server/run.sh b/server/run.sh index 4f3cfeb..94d0ae1 100755 --- a/server/run.sh +++ b/server/run.sh @@ -4,14 +4,17 @@ set -e -cd wisp_server -if [ ! -d ".venv" ]; then - python3 -m venv .venv +SCRIPT_PATH=$(realpath $0) +BASE_PATH=$(dirname $SCRIPT_PATH) +SERVER_PATH="$BASE_PATH/wisp_server" + +if [ ! -d "$SERVER_PATH.venv" ]; then + python3 -m venv $SERVER_PATH/.venv fi -source .venv/bin/activate +source $SERVER_PATH/.venv/bin/activate if ! python3 -c "import websockets" 2> /dev/null; then - pip3 install -r requirements.txt + pip3 install -r $SERVER_PATH/requirements.txt fi -python3 main.py \ No newline at end of file +python3 $SERVER_PATH/main.py "$@" \ No newline at end of file diff --git a/server/wisp_server b/server/wisp_server index 8747346..3b0d432 160000 --- a/server/wisp_server +++ b/server/wisp_server @@ -1 +1 @@ -Subproject commit 874734623e4dfc4652b34a1bc61e1e35ca86dee8 +Subproject commit 3b0d432e89cff7eaa850ba8605b180189e237f1b