mercuryworkshop-mirror/libcurl.js
1
0
Fork 0
mirror of https://github.com/ading2210/libcurl.js.git synced 2025-05-13 22:40:01 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into wolfssl-testing

Browse source
This commit is contained in:
ading2210 2024-03-19 13:54:38 -04:00
commit 8ae1907f50
25 changed files with 922 additions and 322 deletions
Download patch file Download diff file Expand all files Collapse all files

79
CHANGELOG.md Normal file
View file

@ -0,0 +1,79 @@
# Libcurl.js Changelog:
## v0.5.3 (3/9/24):
- Update Wisp client and server, which improves error handling
- Expose the wisp-client-js version in the API
## v0.5.2 (3/10/24):
- Fix a bug with error handling
## v0.5.1 (3/10/24):
- Added support for aborting requests
## v0.5.0 (3/9/24):
- Added support for streaming HTTP responses via a readable stream
- Improve compatibility for older browsers
- Support for all types of fetch request bodies
## v0.4.2 (3/7/24):
- Expose a function to get error strings
## v0.4.1 (3/7/24):
- Fix handling of duplicate HTTP headers
## v0.4.0 (3/7/24):
- Add TLS socket support
- Add function to get the CA cert bundle
- Re-add wsproxy support
- Add custom network transport support
- Split WebSocket API into simple `libcurl.CurlWebSocket` and `libcurl.WebSocket` polyfill
- Refactor WebSocket API code
## v0.3.9 (3/3/24):
- Fix running libcurl.js inside a web worker
## v0.3.8 (2/28/24):
- Update Wisp client and server
- Expose Wisp client API functions
## v0.3.7 (2/27/24):
- Pin C library versions to stable
- Load the CA certs directly from memory instead of from the Emscripten virtual filesystem
## v0.3.6 (2/26/24):
- Fix ES6 module syntax
## v0.3.4 (2/24/24):
- Limit max TCP connections to 50
## v0.3.3 (2/4/24):
- Fix a memory leak with WebSockets
## v0.3.2 (2/4/24):
- Fix handling of 204 and 205 response codes
- Add verbose option to WebSocket API
- Fix conversion of request payloads
## v0.3.1 (2/3/24):
- Add a copyright notice to the JS bundle
## v0.3.0 (2/3/24):
- Add API to get the libcurl.js version and C library versions
- Add checks to ensure that the Websocket proxy URL has been set
## v0.2.0 (2/2/24):
- Add an option to redirect the verbose curl output.
- Use separate callbacks for stdout and stderr.
## v0.1.2 (2/1/23):
- Fix bundling the WASM into a single file
- Add unit tests
## v0.1.1 (1/29/23):
- Don't set a default websocket proxy URL
## v0.1.0 (1/28/23):
- Initial release on NPM
- Add Github Actions for automatic builds
- Add WebSocket support
- Add Fetch API support

135
README.md
View file

@ -2,14 +2,39 @@
This is an experimental port of [libcurl](https://curl.se/libcurl/) to WebAssembly for use in the browser. It provides an interface compatible with the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), allowing you to proxy HTTPS requests from the browser with full TLS encryption. Unlike previous implementations, the proxy server cannot read the contents of your requests. This is an experimental port of [libcurl](https://curl.se/libcurl/) to WebAssembly for use in the browser. It provides an interface compatible with the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), allowing you to proxy HTTPS requests from the browser with full TLS encryption. Unlike previous implementations, the proxy server cannot read the contents of your requests.
## Table of Contents:
- [Features](#features)
- [Building](#building)
- [Javascript API](#javascript-api)
* [Importing the Library](#importing-the-library)
* [Making HTTP Requests](#making-http-requests)
* [Creating WebSocket Connections](#creating-websocket-connections)
* [Using TLS Sockets](#using-tls-sockets)
* [Changing the Network Transport](#changing-the-network-transport)
* [Changing the Websocket Proxy URL](#changing-the-websocket-proxy-url)
* [Getting Libcurl's Output](#getting-libcurl-s-output)
* [Getting Error Strings](#getting-error-strings)
* [Getting Version Info](#getting-version-info)
* [Getting the CA Certificates Bundle](#getting-the-ca-certificates-bundle)
- [Proxy Server](#proxy-server)
- [Project Structure](#project-structure)
- [Copyright](#copyright)
* [Copyright Notice](#copyright-notice)
<small>Table of contents generated with [markdown-toc](http://ecotrust-canada.github.io/markdown-toc/).</small>
## Features: ## Features:
- Fetch compatible API - Fetch compatible API
- End to end encryption between the browser and the destination server - End to end encryption between the browser and the destination server
- Support for up to TLS 1.3 - Support for up to TLS 1.3
- Support for tunneling HTTP/2 connections - Support for tunneling HTTP/2 connections
- Support for proxying WebSockets - Support for proxying WebSockets
- Bypass CORS restrictions - Bypass CORS restrictions without compromising on privacy
- Low latency via multiplexing and reusing open connections - Low latency via multiplexing and reusing open connections
- Use raw TLS sockets in the browser
- Custom network transport support
- Works inside web workers without needing special permissions or headers
- Works in all major browsers (Chromium >= 64, Firefox >= 65, Safari >= 14)
## Building: ## Building:
You can build this project by running the following commands: You can build this project by running the following commands:
@ -31,7 +56,7 @@ The build script will generate `client/out/libcurl.js` as well as `client/out/li
## Javascript API: ## Javascript API:
### Importing the Library: ### Importing the Library:
To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` and `client/out/libcurl.wasm` to a directory of your choice. After the script is loaded, call `libcurl.load_wasm`, specifying the url of the `libcurl.wasm` file. To import the library, follow the build instructions in the previous section, and copy `client/out/libcurl.js` and `client/out/libcurl.wasm` to a directory of your choice. After the script is loaded, call `libcurl.load_wasm`, specifying the url of the `libcurl.wasm` file. You do not need to call `libcurl.load_wasm` if you use the `libcurl_full.js` file, as the WASM will be bundled into the JS file.
```html ```html
<script defer src="./out/libcurl.js" onload="libcurl.load_wasm('/out/libcurl.wasm');"></script> <script defer src="./out/libcurl.js" onload="libcurl.load_wasm('/out/libcurl.wasm');"></script>
@ -51,6 +76,13 @@ document.addEventListener("libcurl_load", ()=>{
}); });
``` ```
You may also use the, the `libcurl.onload` callback, which can be useful for running libcurl.js inside a web worker.
```js
libcurl.onload = () => {
console.log("libcurl.js ready!");
}
```
Once loaded, there will be a `window.libcurl` object which includes all the API functions. The `libcurl.ready` property can also be used to know if the WASM has loaded. Once loaded, there will be a `window.libcurl` object which includes all the API functions. The `libcurl.ready` property can also be used to know if the WASM has loaded.
### Making HTTP Requests: ### Making HTTP Requests:
@ -62,12 +94,43 @@ console.log(await r.text());
Most of the standard Fetch API's features are supported, with the exception of: Most of the standard Fetch API's features are supported, with the exception of:
- CORS enforcement - CORS enforcement
- `FormData` or `URLSearchParams` as the request body
- Sending credentials/cookies automatically - Sending credentials/cookies automatically
- Caching - Caching
The response may contain multiple HTTP headers with the same name, which the `Headers` object isn't able to properly represent. If this matters to you, use `response.raw_headers`, which is an array of key value pairs, instead of `response.headers`. There is support for streaming the response body using a `ReadableStream`, as well as canceling requests using an `AbortSignal`.
Also note that there is a hard limit of 50 active TCP connections due to emscripten limitations.
### Creating WebSocket Connections: ### Creating WebSocket Connections:
To use WebSockets, create a `libcurl.WebSocket` object, which works identically to the regular [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) object. To use WebSockets, create a `libcurl.CurlWebSocket` object, which takes the following arguments:
- `url` - The Websocket URL.
- `protocols` - A optional list of websocket subprotocols, as an array of strings.
- `options` - An optional object with extra settings to pass to curl.
The valid WebSocket options are:
- `headers` - HTTP request headers for the websocket handshake.
- `verbose` - A boolean flag that toggles the verbose libcurl output. This verbose output will be passed to the function defined in `libcurl.stderr`, which is `console.warn` by default.
The following callbacks are available:
- `CurlWebSocket.onopen` - Called when the websocket is successfully connected.
- `CurlWebSocket.onmessage` - Called when a websocket message is received from the server. The data is passed to the first argument of the function, and it will be either a `Uint8Array` or a string, depending on the type of message.
- `CurlWebSocket.onclose` - Called when the websocket is cleanly closed with no error.
- `CurlWebSocket.onerror` - Called when the websocket encounters an unexpected error. The [error code](https://curl.se/libcurl/c/libcurl-errors.html) is passed to the first argument of the function.
The `CurlWebSocket.send` function can be used to send data to the websocket. The only argument is the data that is to be sent, which must be either a string or a `Uint8Array`.
```js
let ws = new libcurl.CurlWebSocket("wss://echo.websocket.org", [], {verbose: 1});
ws.onopen = () => {
console.log("ws connected!");
ws.send("hello".repeat(100));
};
ws.onmessage = (data) => {
console.log(data);
};
```
You can also use the `libcurl.WebSocket` object, which works identically to the regular [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) object. It uses the same arguments as the simpler `CurlWebSocket` API.
```js ```js
let ws = new libcurl.WebSocket("wss://echo.websocket.org"); let ws = new libcurl.WebSocket("wss://echo.websocket.org");
ws.addEventListener("open", () => { ws.addEventListener("open", () => {
@ -79,7 +142,38 @@ ws.addEventListener("message", (event) => {
}); });
``` ```
### Changing the Websocket URL: ### Using TLS Sockets:
Raw TLS sockets can be created with the `libcurl.TLSSocket` class, which takes the following arguments:
- `host` - The hostname to connect to.
- `port` - The TCP port to connect to.
- `options` - An optional object with extra settings to pass to curl.
The valid TLS socket options are:
- `verbose` - A boolean flag that toggles the verbose libcurl output.
The callbacks work similarly to the `libcurl.CurlWebSocket` object, with the main difference being that the `onmessage` callback always returns a `Uint8Array`.
The `TLSSocket.send` function can be used to send data to the socket. The only argument is the data that is to be sent, which must be a `Uint8Array`.
```js
let socket = new libcurl.TLSSocket("ading.dev", 443, {verbose: 1});
socket.onopen = () => {
console.log("socket connected!");
let str = "GET / HTTP/1.1\r\nHost: ading.dev\r\nConnection: close\r\n\r\n";
socket.send(new TextEncoder().encode(str));
};
socket.onmessage = (data) => {
console.log(new TextDecoder().decode(data));
};
```
### Changing the Network Transport:
You can change the underlying network transport by setting `libcurl.transport`. The following values are accepted:
- `"wisp"` - Use the [Wisp protocol](https://github.com/MercuryWorkshop/wisp-protocol).
- `"wsproxy"` - Use the wsproxy protocol, where a new websocket is created for each TCP connection.
- Any custom class - Use a custom network protocol. If you pass in custom code here, it must be roughly conformant with the standard `WebSocket` API. The URL that is passed into this fake websocket always looks like `"wss://example.com/ws/ading.dev:443"`, where `wss://example.com/ws/` is the proxy server URL, and `ading.dev:443` is the destination server.
### Changing the Websocket Proxy URL:
You can change the URL of the websocket proxy by using `libcurl.set_websocket`. You can change the URL of the websocket proxy by using `libcurl.set_websocket`.
```js ```js
libcurl.set_websocket("ws://localhost:6001/"); libcurl.set_websocket("ws://localhost:6001/");
@ -87,18 +181,36 @@ libcurl.set_websocket("ws://localhost:6001/");
If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash. If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash.
### Getting Libcurl's Output: ### Getting Libcurl's Output:
If you want more information about a connection, you can pass the `_libcurl_verbose` argument to the `libcurl.fetch` function. If you want more information about a connection, you can pass the `_libcurl_verbose` argument to the `libcurl.fetch` function. These are the same messages that you would see if you ran `curl -v` on the command line.
```js ```js
await libcurl.fetch("https://example.com", {_libcurl_verbose: 1}); await libcurl.fetch("https://example.com", {_libcurl_verbose: 1});
``` ```
By default this will print the output to the browser console, but you can set `libcurl.stdout` and `libcurl.stderr` to intercept these messages. This callback will be executed on every line of text that libcurl outputs. By default this will print the output to the browser console, but you can set `libcurl.stdout` and `libcurl.stderr` to intercept these messages. This callback will be executed on every line of text that libcurl outputs.
```js ```js
libcurl.stderr = (text) => {document.body.innerHTML += text}; libcurl.stderr = (text) => {document.body.innerHTML += text};
``` ```
Libcurl.js will also output some error messages to the browser console. You can intercept these messages by setting the `libcurl.logger` callback, which takes two arguments:
- `type` - The type of message. This will be one of the following: `"log"`, `"warn"`, `"error"`
- `text` - The text that is to be logged.
This may be useful if you are running libcurl.js inside a web worker and do not have access to the regular console API.
### Getting Error Strings:
Libcurl.js reports errors based on the [error codes](https://curl.se/libcurl/c/libcurl-errors.html) defined by the libcurl C library. The `libcurl.get_error_string` function can be used to get an error string from an error code.
```js
console.log(libcurl.get_error_string(56));
//"Failure when receiving data from the peer"
```
### Getting Version Info: ### Getting Version Info:
You can get version information from the `libcurl.version` object. This object will also contain the versions of all the C libraries that libcurl.js uses. `libcurl.version.lib` returns the version of libcurl.js itself. You can get version information from the `libcurl.version` object. This object will also contain the versions of all the C libraries that libcurl.js uses. `libcurl.version.lib` returns the version of libcurl.js itself.
### Getting the CA Certificates Bundle:
You can get the CA cert bundle that libcurl uses by calling `libcurl.get_cacert`. The function will return a string with the certificates in PEM format. The cert bundle comes from the [official curl website](https://curl.se/docs/caextract.html), which is extracted from the Mozilla Firefox source code.
## Proxy Server: ## Proxy Server:
The proxy server consists of a standard [Wisp](https://github.com/MercuryWorkshop/wisp-protocol) server, allowing multiple TCP connections to share the same websocket. The proxy server consists of a standard [Wisp](https://github.com/MercuryWorkshop/wisp-protocol) server, allowing multiple TCP connections to share the same websocket.
@ -111,6 +223,17 @@ server/run.sh --static=./client
For a full list of server arguments, see the [wisp-server-python documentation](https://github.com/MercuryWorkshop/wisp-server-python). For a full list of server arguments, see the [wisp-server-python documentation](https://github.com/MercuryWorkshop/wisp-server-python).
## Project Structure:
- `client` - Contains all the client-side code.
- `fragments` - Various patches for the JS that emscripten produces. The script which does the patching can be found at `client/tools/patch_js.py`.
- `javascript` - All the code for the Javascript API, and for interfacing with the compiled C code.
- `libcurl` - The C code that interfaces with the libcurl library and gets compiled by emscripten.
- `tests` - Unit tests and the scripts for running them.
- `tools` - Helper shell scripts for the build process, and for compiling the various C libraries.
- `wisp_client` - A submodule for the Wisp client library.
- `server` - Contains all the server-side code for running the websocket proxy server.
- `wisp_sever` - A submodule for the Python Wisp server.
## Copyright: ## Copyright:
This project is licensed under the GNU AGPL v3. This project is licensed under the GNU AGPL v3.

9
client/build.sh
View file

@ -78,13 +78,20 @@ rm $MODULE_FILE
#add version number and copyright notice #add version number and copyright notice
VERSION=$(cat package.json | jq -r '.version') VERSION=$(cat package.json | jq -r '.version')
sed -i "s/__library_version__/$VERSION/" $OUT_FILE sed -i "s/__library_version__/$VERSION/" $OUT_FILE
WISP_VERSION=$(cat $WISP_CLIENT/package.json | jq -r '.version')
sed -i "s/__wisp_version__/$WISP_VERSION/" $OUT_FILE
#add extra libraries #add extra libraries
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/copyright.js" $OUT_FILE
sed -i "/__extra_libraries__/r $WISP_CLIENT/polyfill.js" $OUT_FILE sed -i "/__extra_libraries__/r $WISP_CLIENT/polyfill.js" $OUT_FILE
sed -i "/__extra_libraries__/r $WISP_CLIENT/wisp.js" $OUT_FILE sed -i "/__extra_libraries__/r $WISP_CLIENT/wisp.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/messages.js" $OUT_FILE sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/messages.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/tls_socket.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/websocket.js" $OUT_FILE sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/websocket.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/copyright.js" $OUT_FILE sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/ws_polyfill.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/util.js" $OUT_FILE
sed -i "/__extra_libraries__/r $JAVSCRIPT_DIR/logger.js" $OUT_FILE
#apply patches #apply patches
python3 tools/patch_js.py $FRAGMENTS_DIR $OUT_FILE python3 tools/patch_js.py $FRAGMENTS_DIR $OUT_FILE

7
client/exported_funcs.txt
View file

@ -4,11 +4,13 @@ tick_request
active_requests active_requests
get_version get_version
get_cacert
get_error_str
recv_from_websocket recv_from_websocket
send_to_websocket send_to_websocket
close_websocket close_websocket
cleanup_websocket cleanup_handle
get_result_size get_result_size
get_result_buffer get_result_buffer
get_result_code get_result_code
@ -16,4 +18,7 @@ get_result_closed
get_result_bytes_left get_result_bytes_left
get_result_is_text get_result_is_text
recv_from_socket
send_to_socket
free free

12
client/fragments/wisp_support.js
View file

@ -1,4 +1,14 @@
/* REPLACE /* REPLACE
new WebSocketConstructor new WebSocketConstructor
*/ */
new WispWebSocket new ((() => {
if (api.transport === "wisp") {
return WispWebSocket;
}
else if (api.transport === "wsproxy") {
return WebSocket;
}
else { //custom transports
return api.transport;
}
})())

23
client/javascript/copyright.js
View file

@ -1,15 +1,10 @@
const copyright_notice = `ading2210/libcurl.js - A port of libcurl to WASM for use in the browser. const copyright_notice = `libcurl.js is licensed under the GNU AGPL v3. You can find the license text and source code at the project's git repository: https://github.com/ading2210/libcurl.js
Copyright (C) 2023 ading2210
This program is free software: you can redistribute it and/or modify Several C libraries are used, and their licenses are listed below:
it under the terms of the GNU Affero General Public License as published by - libcurl: curl License (https://curl.se/docs/copyright.html)
the Free Software Foundation, either version 3 of the License, or - openssl: Apache License 2.0 (https://github.com/openssl/openssl/blob/master/LICENSE.txt)
(at your option) any later version. - cjson: MIT License (https://github.com/DaveGamble/cJSON/blob/master/LICENSE)
- zlib: zlib License (https://www.zlib.net/zlib_license.html)
This program is distributed in the hope that it will be useful, - brotli: MIT License (https://github.com/google/brotli/blob/master/LICENSE)
but WITHOUT ANY WARRANTY; without even the implied warranty of - nghttp2: MIT License (https://github.com/nghttp2/nghttp2/blob/master/COPYING)
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the `;
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.`;

18
client/javascript/logger.js Normal file
View file

@ -0,0 +1,18 @@
function logger(type, text) {
if (type === "log")
console.log(text);
else if (type === "warn")
console.warn(text);
else if (type === "error")
console.error(text);
}
function log_msg(text) {
logger("log", text);
}
function warn_msg(text) {
logger("warn", text);
}
function error_msg(text) {
logger("error", text);
}

261
client/javascript/main.js
View file

@ -30,7 +30,9 @@ var event_loop = null;
var active_requests = 0; var active_requests = 0;
var wasm_ready = false; var wasm_ready = false;
var version_dict = null; var version_dict = null;
var api = null;
const libcurl_version = "__library_version__"; const libcurl_version = "__library_version__";
const wisp_version = "__wisp_version__";
function check_loaded(check_websocket) { function check_loaded(check_websocket) {
if (!wasm_ready) { if (!wasm_ready) {
@ -41,47 +43,12 @@ function check_loaded(check_websocket) {
} }
} }
//a case insensitive dictionary for request headers
class HeadersDict {
constructor(obj) {
for (let key in obj) {
this[key] = obj[key];
}
return new Proxy(this, this);
}
get(target, prop) {
let keys = Object.keys(this);
for (let key of keys) {
if (key.toLowerCase() === prop.toLowerCase()) {
return this[key];
}
}
}
set(target, prop, value) {
let keys = Object.keys(this);
for (let key of keys) {
if (key.toLowerCase() === prop.toLowerCase()) {
this[key] = value;
}
}
this[prop] = value;
return true;
}
}
function allocate_str(str) {
return allocate(intArrayFromString(str), ALLOC_NORMAL);
}
function allocate_array(array) {
return allocate(array, ALLOC_NORMAL);
}
//low level interface with c code //low level interface with c code
function perform_request(url, params, js_data_callback, js_end_callback, body=null) { function perform_request(url, params, js_data_callback, js_end_callback, js_headers_callback, body=null) {
let params_str = JSON.stringify(params); let params_str = JSON.stringify(params);
let end_callback_ptr; let end_callback_ptr;
let data_callback_ptr; let data_callback_ptr;
let headers_callback_ptr;
let url_ptr = allocate_str(url); let url_ptr = allocate_str(url);
let params_ptr = allocate_str(params_str); let params_ptr = allocate_str(params_str);
@ -92,30 +59,40 @@ function perform_request(url, params, js_data_callback, js_end_callback, body=nu
body_length = body.length; body_length = body.length;
} }
let end_callback = (error, response_json_ptr) => { function end_callback(error) {
let response_json = UTF8ToString(response_json_ptr);
let response_info = JSON.parse(response_json);
Module.removeFunction(end_callback_ptr); Module.removeFunction(end_callback_ptr);
Module.removeFunction(data_callback_ptr); Module.removeFunction(data_callback_ptr);
if (body_ptr) _free(body_ptr); Module.removeFunction(headers_callback_ptr);
_free(url_ptr);
_free(response_json_ptr);
if (error != 0) console.error("request failed with error code " + error);
active_requests --; active_requests --;
js_end_callback(error, response_info); js_end_callback(error);
} }
let data_callback = (chunk_ptr, chunk_size) => { function data_callback(chunk_ptr, chunk_size) {
let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size); let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size);
let chunk = new Uint8Array(data); let chunk = new Uint8Array(data);
js_data_callback(chunk); js_data_callback(chunk);
} }
end_callback_ptr = Module.addFunction(end_callback, "vii"); function headers_callback(response_json_ptr) {
let response_json = UTF8ToString(response_json_ptr);
let response_info = JSON.parse(response_json);
if (body_ptr) _free(body_ptr);
_free(url_ptr);
_free(response_json_ptr);
//if the response status is 0, an error occurred,
//but we don't know what it is yet
if (response_info.status !== 0) {
js_headers_callback(response_info);
}
}
end_callback_ptr = Module.addFunction(end_callback, "vi");
headers_callback_ptr = Module.addFunction(headers_callback, "vi");
data_callback_ptr = Module.addFunction(data_callback, "vii"); data_callback_ptr = Module.addFunction(data_callback, "vii");
let http_handle = _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, body_ptr, body_length); let http_handle = _start_request(url_ptr, params_ptr, data_callback_ptr, end_callback_ptr, headers_callback_ptr, body_ptr, body_length);
_free(params_ptr); _free(params_ptr);
active_requests ++; active_requests ++;
@ -135,17 +112,6 @@ function perform_request(url, params, js_data_callback, js_end_callback, body=nu
return http_handle; return http_handle;
} }
function merge_arrays(arrays) {
let total_len = arrays.reduce((acc, val) => acc + val.length, 0);
let new_array = new Uint8Array(total_len);
let offset = 0;
for (let array of arrays) {
new_array.set(array, offset);
offset += array.length;
}
return new_array;
}
function create_response(response_data, response_info) { function create_response(response_data, response_info) {
response_info.ok = response_info.status >= 200 && response_info.status < 300; response_info.ok = response_info.status >= 200 && response_info.status < 300;
response_info.statusText = status_messages[response_info.status] || ""; response_info.statusText = status_messages[response_info.status] || "";
@ -168,71 +134,42 @@ function create_response(response_data, response_info) {
writable: false, writable: false,
value: new Headers() value: new Headers()
}); });
for (let header_name in response_info.headers) { Object.defineProperty(response_obj, "raw_headers", {
let header_value = response_info.headers[header_name]; writable: false,
value: response_info.headers
});
for (let [header_name, header_value] of response_info.headers) {
response_obj.headers.append(header_name, header_value); response_obj.headers.append(header_name, header_value);
} }
return response_obj; return response_obj;
} }
async function parse_body(data) {
let data_array = null;
if (typeof data === "string") {
data_array = new TextEncoder().encode(data);
}
else if (data instanceof Blob) {
let array_buffer = await data.arrayBuffer();
data_array = new Uint8Array(array_buffer);
}
//any typedarray
else if (data instanceof ArrayBuffer) {
//dataview objects
if (ArrayBuffer.isView(data) && data instanceof DataView) {
data_array = new Uint8Array(data.buffer);
}
//regular typed arrays
else if (ArrayBuffer.isView(data)) {
data_array = Uint8Array.from(data);
}
//regular arraybuffers
else {
data_array = new Uint8Array(data);
}
}
else if (data instanceof ReadableStream) {
let chunks = [];
for await (let chunk of data) {
chunks.push(chunk);
}
data_array = merge_arrays(chunks);
}
else {
throw "invalid data type to be sent";
}
return data_array;
}
async function create_options(params) { async function create_options(params) {
let body = null; let body = null;
if (params.body) { let request_obj = new Request("/", params);
body = await parse_body(params.body); let array_buffer = await request_obj.arrayBuffer();
params.body = true; if (array_buffer.byteLength > 0) {
body = new Uint8Array(array_buffer);
} }
if (!params.headers) params.headers = {}; let headers = params.headers || {};
params.headers = new HeadersDict(params.headers); if (params.headers instanceof Headers) {
for(let [key, value] of headers) {
headers[key] = value;
}
}
params.headers = new HeadersDict(headers);
if (params.referer) { if (params.referrer) {
params.headers["Referer"] = params.referer; params.headers["Referer"] = params.referrer;
} }
if (!params.headers["User-Agent"]) { if (!params.headers["User-Agent"]) {
params.headers["User-Agent"] = navigator.userAgent; params.headers["User-Agent"] = navigator.userAgent;
} }
if (body) {
params.headers["Content-Type"] = request_obj.headers.get("Content-Type");
}
return body; return body;
} }
@ -240,22 +177,63 @@ async function create_options(params) {
//wrap perform_request in a promise //wrap perform_request in a promise
function perform_request_async(url, params, body) { function perform_request_async(url, params, body) {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
let chunks = []; let stream_controller;
let data_callback = (new_data) => { let http_handle;
chunks.push(new_data); let response_obj;
}; let aborted = false;
let finish_callback = (error, response_info) => { //handle abort signals
if (error != 0) { if (params.signal instanceof AbortSignal) {
reject("libcurl.js encountered an error: " + error); params.signal.addEventListener("abort", () => {
return; if (aborted) return;
aborted = true;
_cleanup_handle(http_handle);
if (!response_obj) {
reject(new DOMException("The operation was aborted."));
} }
let response_data = merge_arrays(chunks); else {
chunks = null; stream_controller.error("The operation was aborted.");
let response_obj = create_response(response_data, response_info); }
});
}
let stream = new ReadableStream({
start(controller) {
stream_controller = controller;
}
});
function data_callback(new_data) {
try {
stream_controller.enqueue(new_data);
}
catch (e) {
//the readable stream has been closed elsewhere, so cancel the request
if (e instanceof TypeError) {
_cleanup_handle(http_handle);
}
else {
throw e;
}
}
}
function headers_callback(response_info) {
response_obj = create_response(stream, response_info);
resolve(response_obj); resolve(response_obj);
} }
perform_request(url, params, data_callback, finish_callback, body); function finish_callback(error) {
if (error != 0) {
error_msg(`Request "${url}" failed with error code ${error}: ${get_error_str(error)}`);
reject(`Request failed with error code ${error}: ${get_error_str(error)}`);
return;
}
try {
stream_controller.close();
} //this will only fail if the stream is already errored or closed, which isn't a problem
catch {}
}
http_handle = perform_request(url, params, data_callback, finish_callback, headers_callback, body);
}); });
} }
@ -267,12 +245,9 @@ async function libcurl_fetch(url, params={}) {
function set_websocket_url(url) { function set_websocket_url(url) {
websocket_url = url; websocket_url = url;
if (!Module.websocket) { if (Module.websocket) {
document.addEventListener("libcurl_load", () => { Module.websocket.url = url;
set_websocket_url(url);
});
} }
else Module.websocket.url = url;
} }
function get_version() { function get_version() {
@ -284,17 +259,25 @@ function get_version() {
_free(version_ptr); _free(version_ptr);
version_dict = JSON.parse(version_str); version_dict = JSON.parse(version_str);
version_dict.lib = libcurl_version; version_dict.lib = libcurl_version;
version_dict.wisp = wisp_version;
return version_dict; return version_dict;
} }
function get_cacert() {
return UTF8ToString(_get_cacert());
}
function main() { function main() {
wasm_ready = true; wasm_ready = true;
_init_curl(); _init_curl();
set_websocket_url(websocket_url); set_websocket_url(websocket_url);
if (ENVIRONMENT_IS_WEB) {
let load_event = new Event("libcurl_load"); let load_event = new Event("libcurl_load");
document.dispatchEvent(load_event); document.dispatchEvent(load_event);
} }
api.onload();
}
function load_wasm(url) { function load_wasm(url) {
wasmBinaryFile = url; wasmBinaryFile = url;
@ -303,21 +286,35 @@ function load_wasm(url) {
} }
Module.onRuntimeInitialized = main; Module.onRuntimeInitialized = main;
return { api = {
fetch: libcurl_fetch, fetch: libcurl_fetch,
set_websocket: set_websocket_url, set_websocket: set_websocket_url,
load_wasm: load_wasm, load_wasm: load_wasm,
wisp: _wisp_connections, WebSocket: FakeWebSocket,
WebSocket: CurlWebSocket, CurlWebSocket: CurlWebSocket,
TLSSocket: TLSSocket,
get_cacert: get_cacert,
get_error_string: get_error_str,
wisp_connections: _wisp_connections,
WispConnection: WispConnection,
transport: "wisp",
get copyright() {return copyright_notice}, get copyright() {return copyright_notice},
get version() {return get_version()}, get version() {return get_version()},
get ready() {return wasm_ready}, get ready() {return wasm_ready},
get websocket_url() {return websocket_url},
get stdout() {return out}, get stdout() {return out},
set stdout(callback) {out = callback}, set stdout(callback) {out = callback},
get stderr() {return err}, get stderr() {return err},
set stderr(callback) {err = callback} set stderr(callback) {err = callback},
} get logger() {return logger},
set logger(func) {logger = func},
onload() {}
};
return api;
})() })()

104
client/javascript/tls_socket.js Normal file
View file

@ -0,0 +1,104 @@
//currently broken
class TLSSocket {
constructor(hostname, port, options={}) {
check_loaded(true);
this.hostname = hostname;
this.port = port;
this.url = `https://${hostname}:${port}`;
this.options = options;
this.onopen = () => {};
this.onerror = () => {};
this.onmessage = () => {};
this.onclose = () => {};
this.connected = false;
this.event_loop = null;
this.connect();
}
connect() {
let response_info;
let data_callback = () => {};
let headers_callback = (info) => {
response_info = info;
}
let finish_callback = (error) => {
if (error === 0) {
this.connected = true;
this.event_loop = setInterval(() => {
let data = this.recv();
if (data != null) this.onmessage(data);
}, 0);
this.onopen();
}
else {
this.cleanup(error);
}
}
let request_options = {
_connect_only: 1,
}
if (this.options.verbose) {
request_options._libcurl_verbose = 1;
}
this.http_handle = perform_request(this.url, request_options, data_callback, finish_callback, headers_callback, null);
}
recv() {
let buffer_size = 64*1024;
let result_ptr = _recv_from_socket(this.http_handle, buffer_size);
let data_ptr = _get_result_buffer(result_ptr);
let result_code = _get_result_code(result_ptr);
let result_closed = _get_result_closed(result_ptr);
if (result_code === 0 && !result_closed) { //CURLE_OK - data received
let data_size = _get_result_size(result_ptr);
let data_heap = Module.HEAPU8.subarray(data_ptr, data_ptr + data_size);
let data = new Uint8Array(data_heap);
this.onmessage(data)
}
else if (result_code === 0 && result_closed) {
this.cleanup();
}
else if (result_code != 81) {
this.cleanup(result_code);
}
_free(data_ptr);
_free(result_ptr);
}
send(data_array) {
if (!this.connected) return;
let data_ptr = allocate_array(data_array);
let data_len = data_array.length;
_send_to_socket(this.http_handle, data_ptr, data_len);
_free(data_ptr);
}
cleanup(error=false) {
if (this.http_handle) _cleanup_handle(this.http_handle);
else return;
clearInterval(this.event_loop);
this.connected = false;
if (error) {
this.onerror(error);
}
else {
this.onclose();
}
}
close() {
this.cleanup();
}
}

74
client/javascript/util.js Normal file
View file

@ -0,0 +1,74 @@
//a case insensitive dictionary for request headers
class HeadersDict {
constructor(obj) {
for (let key in obj) {
this[key] = obj[key];
}
return new Proxy(this, this);
}
get(target, prop) {
let keys = Object.keys(this);
for (let key of keys) {
if (key.toLowerCase() === prop.toLowerCase()) {
return this[key];
}
}
}
set(target, prop, value) {
let keys = Object.keys(this);
for (let key of keys) {
if (key.toLowerCase() === prop.toLowerCase()) {
this[key] = value;
}
}
this[prop] = value;
return true;
}
}
function allocate_str(str) {
return allocate(intArrayFromString(str), ALLOC_NORMAL);
}
function allocate_array(array) {
return allocate(array, ALLOC_NORMAL);
}
function get_error_str(error_code) {
let error_ptr = _get_error_str(error_code);
return UTF8ToString(error_ptr);
}
function merge_arrays(arrays) {
let total_len = arrays.reduce((acc, val) => acc + val.length, 0);
let new_array = new Uint8Array(total_len);
let offset = 0;
for (let array of arrays) {
new_array.set(array, offset);
offset += array.length;
}
return new_array;
}
//convert various data types to a uint8array (blobs excluded)
function data_to_array(data) {
//data already in correct type
if (data instanceof Uint8Array) {
return data;
}
else if (typeof data === "string") {
return new TextEncoder().encode(data);
}
else if (data instanceof ArrayBuffer) {
return new Uint8Array(data);
}
//dataview objects or any other typedarray
else if (ArrayBuffer.isView(data)) {
return new Uint8Array(data.buffer);
}
throw "invalid data type to be sent";
}

194
client/javascript/websocket.js
View file

@ -1,8 +1,5 @@
//class for custom websocket class CurlWebSocket {
constructor(url, protocols=[], options={}) {
class CurlWebSocket extends EventTarget {
constructor(url, protocols=[], websocket_debug=false) {
super();
check_loaded(true); check_loaded(true);
if (!url.startsWith("wss://") && !url.startsWith("ws://")) { if (!url.startsWith("wss://") && !url.startsWith("ws://")) {
throw new SyntaxError("invalid url"); throw new SyntaxError("invalid url");
@ -10,41 +7,49 @@ class CurlWebSocket extends EventTarget {
this.url = url; this.url = url;
this.protocols = protocols; this.protocols = protocols;
this.binaryType = "blob"; this.options = options;
this.recv_buffer = [];
this.websocket_debug = websocket_debug;
//legacy event handlers
this.onopen = () => {}; this.onopen = () => {};
this.onerror = () => {}; this.onerror = () => {};
this.onmessage = () => {}; this.onmessage = () => {};
this.onclose = () => {}; this.onclose = () => {};
this.CONNECTING = 0; this.connected = false;
this.OPEN = 1; this.event_loop = null;
this.CLOSING = 2; this.recv_buffer = [];
this.CLOSED = 3;
this.connect(); this.connect();
} }
connect() { connect() {
this.status = this.CONNECTING; let response_info;
let data_callback = () => {}; let data_callback = () => {};
let finish_callback = (error, response_info) => { let headers_callback = (info) => {
this.finish_callback(error, response_info); response_info = info;
} }
let options = {}; let finish_callback = (error) => {
if (this.protocols) { if (error === 0) {
options.headers = { this.connected = true;
"Sec-Websocket-Protocol": this.protocols.join(", "), this.event_loop = setInterval(() => {
let data = this.recv();
if (data !== null) this.onmessage(data);
}, 0);
this.onopen();
}
else {
this.cleanup(error);
}
}
let request_options = {
headers: this.options.headers || {}
}; };
if (this.protocols) {
request_options.headers["Sec-Websocket-Protocol"] = this.protocols.join(", ");
} }
if (this.websocket_debug) { if (this.options.verbose) {
options._libcurl_verbose = 1; request_options._libcurl_verbose = 1;
} }
this.http_handle = perform_request(this.url, options, data_callback, finish_callback, null); this.http_handle = perform_request(this.url, request_options, data_callback, finish_callback, headers_callback, null);
this.recv_loop();
} }
recv() { recv() {
@ -52,12 +57,16 @@ class CurlWebSocket extends EventTarget {
let result_ptr = _recv_from_websocket(this.http_handle, buffer_size); let result_ptr = _recv_from_websocket(this.http_handle, buffer_size);
let data_ptr = _get_result_buffer(result_ptr); let data_ptr = _get_result_buffer(result_ptr);
let result_code = _get_result_code(result_ptr); let result_code = _get_result_code(result_ptr);
let result_closed = _get_result_closed(result_ptr);
let returned_data = null;
if (result_code == 0) { //CURLE_OK - data recieved //CURLE_OK - data received
if (result_code === 0 && !result_closed) {
if (_get_result_closed(result_ptr)) { if (_get_result_closed(result_ptr)) {
//this.pass_buffer(); _free(data_ptr);
this.close_callback(); _free(result_ptr);
return; this.cleanup();
return returned_data;
} }
let data_size = _get_result_size(result_ptr); let data_size = _get_result_size(result_ptr);
@ -69,133 +78,60 @@ class CurlWebSocket extends EventTarget {
let full_data = merge_arrays(this.recv_buffer); let full_data = merge_arrays(this.recv_buffer);
let is_text = _get_result_is_text(result_ptr) let is_text = _get_result_is_text(result_ptr)
this.recv_buffer = []; this.recv_buffer = [];
this.recv_callback(full_data, is_text); if (is_text) {
returned_data = new TextDecoder().decode(full_data);
}
else {
returned_data = full_data;
}
} }
} }
if (result_code == 52) { //CURLE_GOT_NOTHING - socket closed // websocket was cleanly closed by the server
this.close_callback(); else if (result_code === 0 && result_closed) {
this.cleanup();
}
//code is not CURLE_AGAIN - an error must have occurred
else if (result_code !== 81) {
this.cleanup(result_code);
} }
_free(data_ptr); _free(data_ptr);
_free(result_ptr); _free(result_ptr);
return returned_data;
} }
recv_loop() { cleanup(error=0) {
this.event_loop = setInterval(() => { if (this.http_handle) _cleanup_handle(this.http_handle);
this.recv(); else return;
}, 1);
}
recv_callback(data, is_text=false) {
let converted;
if (is_text) {
converted = new TextDecoder().decode(data);
}
else {
if (this.binaryType == "blob") {
converted = new Blob(data);
}
else if (this.binaryType == "arraybuffer") {
converted = data.buffer;
}
else {
throw "invalid binaryType string";
}
}
let msg_event = new MessageEvent("message", {data: converted});
this.onmessage(msg_event);
this.dispatchEvent(msg_event);
}
close_callback(error=false) {
if (this.status == this.CLOSED) return;
this.status = this.CLOSED;
clearInterval(this.event_loop); clearInterval(this.event_loop);
_cleanup_websocket(); this.connected = false;
if (error) { if (error) {
let error_event = new Event("error"); error_msg(`Websocket "${this.url}" encountered error code ${error}: ${get_error_str(error)}`);
this.dispatchEvent(error_event); this.onerror(error);
this.onerror(error_event);
} }
else { else {
let close_event = new CloseEvent("close"); this.onclose();
this.dispatchEvent(close_event);
this.onclose(close_event);
} }
} }
finish_callback(error, response_info) {
this.status = this.OPEN;
if (error != 0) this.close_callback(true);
let open_event = new Event("open");
this.onopen(open_event);
this.dispatchEvent(open_event);
}
send(data) { send(data) {
let is_text = false; let is_text = typeof data === "string";
if (this.status === this.CONNECTING) { if (!this.connected) return;
throw new DOMException("ws not ready yet");
}
if (this.status === this.CLOSED) {
return;
}
let data_array; if (is_text) {
if (typeof data === "string") { data = new TextEncoder().encode(data);
data_array = new TextEncoder().encode(data);
is_text = true;
} }
else if (data instanceof Blob) { let data_ptr = allocate_array(data);
data.arrayBuffer().then(array_buffer => {
data_array = new Uint8Array(array_buffer);
this.send(data_array);
});
return;
}
//any typedarray
else if (data instanceof ArrayBuffer) {
//dataview objects
if (ArrayBuffer.isView(data) && data instanceof DataView) {
data_array = new Uint8Array(data.buffer);
}
//regular arraybuffers
else {
data_array = new Uint8Array(data);
}
}
//regular typed arrays
else if (ArrayBuffer.isView(data)) {
data_array = Uint8Array.from(data);
}
else {
throw "invalid data type to be sent";
}
let data_ptr = allocate_array(data_array);
let data_len = data.length; let data_len = data.length;
_send_to_websocket(this.http_handle, data_ptr, data_len, is_text); _send_to_websocket(this.http_handle, data_ptr, data_len, is_text);
_free(data_ptr); _free(data_ptr);
} }
close() { close() {
_close_websocket(this.http_handle); this.cleanup();
}
get readyState() {
return this.status;
}
get bufferedAmount() {
return 0;
}
get protocol() {
return "";
}
get extensions() {
return "";
} }
} }

114
client/javascript/ws_polyfill.js Normal file
View file

@ -0,0 +1,114 @@
//class for websocket polyfill
class FakeWebSocket extends EventTarget {
constructor(url, protocols=[], options={}) {
super();
this.url = url;
this.protocols = protocols;
this.options = options;
this.binaryType = "blob";
//legacy event handlers
this.onopen = () => {};
this.onerror = () => {};
this.onmessage = () => {};
this.onclose = () => {};
this.CONNECTING = 0;
this.OPEN = 1;
this.CLOSING = 2;
this.CLOSED = 3;
this.status = this.CONNECTING;
this.socket = null;
this.connect();
}
connect() {
this.socket = new CurlWebSocket(this.url, this.protocols, this.options);
this.socket.onopen = () => {
this.status = this.OPEN;
let open_event = new Event("open");
this.onopen(open_event);
this.dispatchEvent(open_event);
}
this.socket.onclose = () => {
this.status = this.CLOSED;
let close_event = new CloseEvent("close");
this.dispatchEvent(close_event);
this.onclose(close_event);
};
this.socket.onerror = (error) => {
this.status = this.CLOSED;
let error_event = new Event("error");
this.dispatchEvent(error_event);
this.onerror(error_event);
}
this.socket.onmessage = (data) => {
let converted;
if (typeof data === "string") {
converted = data;
}
else { //binary frame received as uint8array
if (this.binaryType == "blob") {
converted = new Blob(data);
}
else if (this.binaryType == "arraybuffer") {
converted = data.buffer;
}
else {
throw "invalid binaryType string";
}
}
let msg_event = new MessageEvent("message", {data: converted});
this.onmessage(msg_event);
this.dispatchEvent(msg_event);
}
}
send(data) {
if (this.status === this.CONNECTING) {
throw new DOMException("websocket not ready yet");
}
if (this.status === this.CLOSED) {
return;
}
if (data instanceof Blob) {
(async () => {
let array_buffer = await data.arrayBuffer();
this.socket.send(new Uint8Array(array_buffer));
})();
}
else if (typeof data === "string") {
this.socket.send(data);
}
else {
this.socket.send(data_to_array(data));
}
}
close() {
this.status = this.CLOSING;
this.socket.close();
}
get readyState() {
return this.status;
}
get bufferedAmount() {
return 0;
}
get protocol() {
return this.protocols[0] || "";
}
get extensions() {
return "";
}
}

91
client/libcurl/main.c
View file

@ -7,13 +7,14 @@
#include "curl/easy.h" #include "curl/easy.h"
#include "curl/header.h" #include "curl/header.h"
#include "cjson/cJSON.h" #include "cjson/cJSON.h"
#include "cacert.h"
#include "curl/multi.h" #include "curl/multi.h"
#include "cacert.h"
#include "util.h" #include "util.h"
#include "types.h" #include "types.h"
void finish_request(CURLMsg *curl_msg); void finish_request(CURLMsg *curl_msg);
void forward_headers(struct RequestInfo *request_info);
#define ERROR_REDIRECT_DISALLOWED -1 #define ERROR_REDIRECT_DISALLOWED -1
@ -21,11 +22,15 @@ CURLM *multi_handle;
int request_active = 0; int request_active = 0;
struct curl_blob cacert_blob; struct curl_blob cacert_blob;
size_t write_function(void *data, size_t size, size_t nmemb, DataCallback data_callback) { size_t write_function(void *data, size_t size, size_t nmemb, struct RequestInfo *request_info) {
if (!request_info->headers_received) {
forward_headers(request_info);
}
size_t real_size = size * nmemb; size_t real_size = size * nmemb;
char* chunk = malloc(real_size); char* chunk = malloc(real_size);
memcpy(chunk, data, real_size); memcpy(chunk, data, real_size);
data_callback(chunk, real_size); (*request_info->data_callback)(chunk, real_size);
free(chunk); free(chunk);
return real_size; return real_size;
} }
@ -48,7 +53,7 @@ void tick_request() {
} }
} }
CURL* start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, const char* body, int body_length) { CURL* start_request(const char* url, const char* json_params, DataCallback data_callback, EndCallback end_callback, HeadersCallback headers_callback, const char* body, int body_length) {
CURL *http_handle = curl_easy_init(); CURL *http_handle = curl_easy_init();
int abort_on_redirect = 0; int abort_on_redirect = 0;
int prevent_cleanup = 0; int prevent_cleanup = 0;
@ -56,10 +61,6 @@ CURL* start_request(const char* url, const char* json_params, DataCallback data_
curl_easy_setopt(http_handle, CURLOPT_URL, url); curl_easy_setopt(http_handle, CURLOPT_URL, url);
curl_easy_setopt(http_handle, CURLOPT_CAINFO_BLOB , cacert_blob); curl_easy_setopt(http_handle, CURLOPT_CAINFO_BLOB , cacert_blob);
//callbacks to pass the response data back to js
curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, &write_function);
curl_easy_setopt(http_handle, CURLOPT_WRITEDATA, data_callback);
//some default options //some default options
curl_easy_setopt(http_handle, CURLOPT_FOLLOWLOCATION, 1); curl_easy_setopt(http_handle, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(http_handle, CURLOPT_ACCEPT_ENCODING, ""); curl_easy_setopt(http_handle, CURLOPT_ACCEPT_ENCODING, "");
@ -83,6 +84,13 @@ CURL* start_request(const char* url, const char* json_params, DataCallback data_
curl_easy_setopt(http_handle, CURLOPT_VERBOSE, 1L); curl_easy_setopt(http_handle, CURLOPT_VERBOSE, 1L);
} }
if (strcmp(key, "_connect_only") == 0) {
curl_easy_setopt(http_handle, CURLOPT_CONNECT_ONLY, 1L);
curl_easy_setopt(http_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_easy_setopt(http_handle, CURLOPT_SSL_ENABLE_ALPN, 0L);
prevent_cleanup = 1;
}
if (strcmp(key, "method") == 0 && cJSON_IsString(item)) { if (strcmp(key, "method") == 0 && cJSON_IsString(item)) {
curl_easy_setopt(http_handle, CURLOPT_CUSTOMREQUEST, item->valuestring); curl_easy_setopt(http_handle, CURLOPT_CUSTOMREQUEST, item->valuestring);
} }
@ -122,36 +130,39 @@ CURL* start_request(const char* url, const char* json_params, DataCallback data_
curl_easy_setopt(http_handle, CURLOPT_POSTFIELDSIZE, body_length); curl_easy_setopt(http_handle, CURLOPT_POSTFIELDSIZE, body_length);
} }
//create request metadata struct
struct RequestInfo *request_info = malloc(sizeof(struct RequestInfo)); struct RequestInfo *request_info = malloc(sizeof(struct RequestInfo));
request_info->http_handle = http_handle;
request_info->abort_on_redirect = abort_on_redirect; request_info->abort_on_redirect = abort_on_redirect;
request_info->curl_msg = NULL; request_info->curl_msg = NULL;
request_info->headers_list = headers_list; request_info->headers_list = headers_list;
request_info->end_callback = end_callback;
request_info->prevent_cleanup = prevent_cleanup; request_info->prevent_cleanup = prevent_cleanup;
request_info->headers_received = 0;
request_info->end_callback = end_callback;
request_info->data_callback = data_callback;
request_info->headers_callback = headers_callback;
//callbacks to pass the response data back to js
curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, &write_function);
curl_easy_setopt(http_handle, CURLOPT_WRITEDATA, data_callback);
curl_easy_setopt(http_handle, CURLOPT_PRIVATE, request_info); curl_easy_setopt(http_handle, CURLOPT_PRIVATE, request_info);
curl_easy_setopt(http_handle, CURLOPT_WRITEDATA, request_info);
curl_multi_add_handle(multi_handle, http_handle); curl_multi_add_handle(multi_handle, http_handle);
return http_handle; return http_handle;
} }
void finish_request(CURLMsg *curl_msg) { void forward_headers(struct RequestInfo *request_info) {
//get initial request info from the http handle request_info->headers_received = 1;
struct RequestInfo *request_info; CURL *http_handle = request_info->http_handle;
CURL *http_handle = curl_msg->easy_handle;
curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info);
int error = (int) curl_msg->data.result;
long response_code;
curl_easy_getinfo(http_handle, CURLINFO_RESPONSE_CODE, &response_code);
if (request_info->abort_on_redirect && response_code / 100 == 3) {
error = ERROR_REDIRECT_DISALLOWED;
}
//create new json object with response info //create new json object with response info
cJSON* response_json = cJSON_CreateObject(); cJSON* response_json = cJSON_CreateObject();
long response_code;
curl_easy_getinfo(http_handle, CURLINFO_RESPONSE_CODE, &response_code);
cJSON* status_item = cJSON_CreateNumber(response_code); cJSON* status_item = cJSON_CreateNumber(response_code);
cJSON_AddItemToObject(response_json, "status", status_item); cJSON_AddItemToObject(response_json, "status", status_item);
@ -160,12 +171,16 @@ void finish_request(CURLMsg *curl_msg) {
cJSON* url_item = cJSON_CreateString(response_url); cJSON* url_item = cJSON_CreateString(response_url);
cJSON_AddItemToObject(response_json, "url", url_item); cJSON_AddItemToObject(response_json, "url", url_item);
cJSON* headers_item = cJSON_CreateObject(); cJSON* headers_item = cJSON_CreateArray();
struct curl_header *prev_header = NULL; struct curl_header *prev_header = NULL;
struct curl_header *header = NULL; struct curl_header *header = NULL;
while ((header = curl_easy_nextheader(http_handle, CURLH_HEADER, -1, prev_header))) { while ((header = curl_easy_nextheader(http_handle, CURLH_HEADER, -1, prev_header))) {
cJSON* header_entry = cJSON_CreateString(header->value); cJSON* header_key_entry = cJSON_CreateString(header->name);
cJSON_AddItemToObject(headers_item, header->name, header_entry); cJSON* header_value_entry = cJSON_CreateString(header->value);
cJSON* header_pair_item = cJSON_CreateArray();
cJSON_AddItemToArray(header_pair_item, header_key_entry);
cJSON_AddItemToArray(header_pair_item, header_value_entry);
cJSON_AddItemToArray(headers_item, header_pair_item);
prev_header = header; prev_header = header;
} }
cJSON_AddItemToObject(response_json, "headers", headers_item); cJSON_AddItemToObject(response_json, "headers", headers_item);
@ -178,9 +193,29 @@ void finish_request(CURLMsg *curl_msg) {
char* response_json_str = cJSON_Print(response_json); char* response_json_str = cJSON_Print(response_json);
cJSON_Delete(response_json); cJSON_Delete(response_json);
(*request_info->headers_callback)(response_json_str);
}
void finish_request(CURLMsg *curl_msg) {
//get initial request info from the http handle
struct RequestInfo *request_info;
CURL *http_handle = curl_msg->easy_handle;
curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info);
if (!request_info->headers_received) {
forward_headers(request_info);
}
int error = (int) curl_msg->data.result;
long response_code;
curl_easy_getinfo(http_handle, CURLINFO_RESPONSE_CODE, &response_code);
if (request_info->abort_on_redirect && response_code / 100 == 3) {
error = ERROR_REDIRECT_DISALLOWED;
}
//clean up curl //clean up curl
curl_slist_free_all(request_info->headers_list); curl_slist_free_all(request_info->headers_list);
(*request_info->end_callback)(error, response_json_str); (*request_info->end_callback)(error);
if (request_info->prevent_cleanup) { if (request_info->prevent_cleanup) {
return; return;
} }
@ -189,6 +224,10 @@ void finish_request(CURLMsg *curl_msg) {
free(request_info); free(request_info);
} }
unsigned char* get_cacert() {
return _cacert_pem;
}
void init_curl() { void init_curl() {
curl_global_init(CURL_GLOBAL_DEFAULT); curl_global_init(CURL_GLOBAL_DEFAULT);
multi_handle = curl_multi_init(); multi_handle = curl_multi_init();

24
client/libcurl/tls_socket.c Normal file
View file

@ -0,0 +1,24 @@
#include <stdlib.h>
#include "curl/curl.h"
#include "curl/easy.h"
#include "types.h"
struct WSResult* recv_from_socket(CURL* http_handle, int buffer_size) {
size_t nread;
char* buffer = malloc(buffer_size);
CURLcode res = curl_easy_recv(http_handle, buffer, buffer_size, &nread);
struct WSResult* result = malloc(sizeof(struct WSResult));
result->buffer_size = nread;
result->buffer = buffer;
result->res = (int) res;
result->closed = (nread == 0);
return result;
}
int send_to_socket(CURL* http_handle, const char* data, int data_len) {
size_t sent;
CURLcode res = curl_easy_send(http_handle, data, data_len, &sent);
return (int) res;
}

9
client/libcurl/types.h
View file

@ -1,12 +1,19 @@
#include "curl/curl.h"
typedef void(*DataCallback)(char* chunk_ptr, int chunk_size); typedef void(*DataCallback)(char* chunk_ptr, int chunk_size);
typedef void(*EndCallback)(int error, char* response_json); typedef void(*EndCallback)(int error);
typedef void(*HeadersCallback)(char* response_json);
struct RequestInfo { struct RequestInfo {
CURL* http_handle;
int abort_on_redirect; int abort_on_redirect;
int prevent_cleanup; int prevent_cleanup;
int headers_received;
struct CURLMsg *curl_msg; struct CURLMsg *curl_msg;
struct curl_slist* headers_list; struct curl_slist* headers_list;
DataCallback data_callback;
EndCallback end_callback; EndCallback end_callback;
HeadersCallback headers_callback;
}; };
struct WSResult { struct WSResult {

4
client/libcurl/util.c
View file

@ -35,3 +35,7 @@ char* get_version() {
cJSON_Delete(version_json); cJSON_Delete(version_json);
return version_json_str; return version_json_str;
} }
const char* get_error_str(CURLcode error_code) {
return curl_easy_strerror(error_code);
}

2
client/libcurl/websocket.c
View file

@ -37,7 +37,7 @@ void close_websocket(CURL* http_handle) {
} }
//clean up the http handle associated with the websocket, since the main loop can't do this automatically //clean up the http handle associated with the websocket, since the main loop can't do this automatically
void cleanup_websocket(CURL* http_handle) { void cleanup_handle(CURL* http_handle) {
struct RequestInfo *request_info; struct RequestInfo *request_info;
curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info); curl_easy_getinfo(http_handle, CURLINFO_PRIVATE, &request_info);

2
client/package.json
View file

@ -1,6 +1,6 @@
{ {
"name": "libcurl.js", "name": "libcurl.js",
"version": "0.3.7", "version": "0.5.3",
"description": "An experimental port of libcurl to WebAssembly for use in the browser.", "description": "An experimental port of libcurl to WebAssembly for use in the browser.",
"main": "libcurl.mjs", "main": "libcurl.mjs",
"scripts": { "scripts": {

1
client/publish.sh
View file

@ -7,6 +7,7 @@
cp package.json out cp package.json out
cp ../README.md out cp ../README.md out
cp ../LICENSE out cp ../LICENSE out
cp ../CHANGELOG.md out
cd out cd out
npm publish npm publish

3
client/tests/run_tests.py
View file

@ -49,5 +49,8 @@ class JSTest(unittest.TestCase):
def test_redirect_out(self): def test_redirect_out(self):
self.run_test("redirect_out.js") self.run_test("redirect_out.js")
def test_tls_socket(self):
self.run_test("test_tls_socket.js")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()

27
client/tests/scripts/test_tls_socket.js Normal file
View file

@ -0,0 +1,27 @@
function test() {
return new Promise((resolve, reject) => {
let socket = new libcurl.TLSSocket("cloudflare.com", 443, {verbose: 1});
socket.onopen = () => {
let str = "GET /cdn-cgi/trace HTTP/1.1\r\nHost: cloudflare.com\r\nConnection: close\r\n\r\n";
socket.send(new TextEncoder().encode(str));
};
socket.onmessage = (data) => {
let text = new TextDecoder().decode(data);
if (!text.includes("tls=TLSv1.3")) {
reject("cloudflare reported tls version mismatch");
return;
}
if (!text.includes("HTTP/1.1 200 OK")) {
reject("cloudflare reported http error");
return;
}
resolve();
};
socket.onerror = (error) => {
reject("socket error occurred " + error);
}
});
}

2
client/tools/curl.sh
View file

@ -19,7 +19,7 @@ cd curl
autoreconf -fi autoreconf -fi
emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX --with-nghttp2=$NGHTTP2_PREFIX emconfigure ./configure --host i686-linux --disable-shared --disable-threaded-resolver --without-libpsl --disable-netrc --disable-ipv6 --disable-tftp --disable-ntlm-wb --enable-websockets --with-wolfssl=$WOLFSSL_PREFIX --with-zlib=$ZLIB_PREFIX --with-brotli=$BROTLI_PREFIX --with-nghttp2=$NGHTTP2_PREFIX
emmake make -j$CORE_COUNT CFLAGS="-Oz -pthread" LIBS="-lbrotlicommon" emmake make -j$CORE_COUNT CFLAGS="-Oz" LIBS="-lbrotlicommon"
rm -rf $PREFIX rm -rf $PREFIX
mkdir -p $PREFIX/include mkdir -p $PREFIX/include

2
client/wisp_client

@ -1 +1 @@
Subproject commit 51ad95a6d912ec404c20284f0cded40c0b5c4e62 Subproject commit f63a01d33b850f6f45a059d1f6db3c45cbe47b1e

33
client/worker.html Normal file
View file

@ -0,0 +1,33 @@
<!DOCTYPE html>
<html>
<head>
<link rel="icon" href="data:;base64,=">
<script id="worker1" type="javascript/worker">
importScripts(`${location.origin}/out/libcurl.js`);
async function main() {
libcurl.set_websocket(`${location.origin.replace("http", "ws")}/`);
self.postMessage("loaded libcurl.js v" + libcurl.version.lib);
let r = await libcurl.fetch("https://ifconfig.me/all", {_libcurl_verbose: 1});
self.postMessage(await r.text());
}
libcurl.onload = main;
libcurl.stdout = self.postMessage;
libcurl.stderr = self.postMessage;
libcurl.load_wasm(`${location.origin}/out/libcurl.wasm`);
</script>
<script>
var blob = new Blob([
document.querySelector('#worker1').textContent
], {type: "text/javascript"});
var worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = function(e) {
console.log("Received: " + e.data);
}
</script>
</head>
<body>
<p>emscripten tests</p>
</body>
</html>

2
server/wisp_server

@ -1 +1 @@
Subproject commit 3b0d432e89cff7eaa850ba8605b180189e237f1b Subproject commit b18670f742ec2ff8efaee6074c36dfb568f20ba2