Merge branch 'main' into reclaimtls

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Libcurl.js Changelog:
 
+## v0.6.19 (11/1/24):
+- Improve download speeds even more by reducing memory copies 
+- Fix wrong mime type being reported in response blob
+
+## v0.6.18 (10/29/24):
+- Improve download speeds by increasing libcurl's internal buffer size and listening directly for websocket messages
+
+## v0.6.17 (10/24/24):
+- Enable a per host connection limit. This defaults to 6, which is the same as most web browsers.
+- Use the latest WolfSSL again and compile with workarounds
+- Upgrade to curl 8.10.1
+
 ## v0.6.16 (10/2/24):
 - Fix a bug with `Headers` objects and `Request` objects not being properly handled when passed into `libcurl.fetch`
 - Errors thrown are now `Error` or `TypeError` objects instead of plain strings

README.md

@@ -86,8 +86,8 @@ If you are using the single file version (`libcurl_full.js`), the `libcurl.load_
 
 Alternatively, prebuilt versions can be found on NPM and jsDelivr. You can use the [following URLs](https://cdn.jsdelivr.net/npm/libcurl.js@latest/) to load libcurl.js from a third party CDN.
 ```
-https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.js
+https://cdn.jsdelivr.net/npm/libcurl.js@0.6.7/libcurl.js
-https://cdn.jsdelivr.net/npm/libcurl.js@latest/libcurl.wasm
+https://cdn.jsdelivr.net/npm/libcurl.js@0.6.7/libcurl.wasm
 ```
 
 To know when libcurl.js has finished loading, you can use the `libcurl_load` DOM event. The `libcurl_abort` event will trigger if the Emscripten runtime gets aborted due to a critical error. The `libcurl.events` object contains an `EventTarget` where these events will also be emitted. 
@@ -131,9 +131,9 @@ Most of the standard Fetch API's features are supported, with the exception of:
 
 Sending cookies is supported, but they will not be automatically sent unless you create a new HTTP session, which is covered in the next section.
 
-The response may contain multiple HTTP headers with the same name, which the `Headers` object isn't able to properly represent. If this matters to you, use `response.raw_headers`, which is an array of key value pairs, instead of `response.headers`. There is support for streaming the response body using a `ReadableStream`, as well as canceling requests using an `AbortSignal`. All requests made using this method share the same connection pool, which has a limit of 50 active TCP connections.
+The response may contain multiple HTTP headers with the same name, which the `Headers` object isn't able to properly represent. If this matters to you, use `response.raw_headers`, which is an array of key value pairs, instead of `response.headers`. There is support for streaming the response body using a `ReadableStream`, as well as canceling requests using an `AbortSignal`. All requests made using this method share the same connection pool, which has a limit of 50 active TCP connections (configurable if you use a separate HTTP session).
 
-The `proxy` option may be used to specify the URL of a `socks5h`, `socks4a`, or `http` proxy server. For example `proxy: "socks5h://127.0.0.0:1080"` will set the proxy server for just the current request.
+The `proxy` option may be used to specify the URL of a `socks5h`, `socks4a`, or `http` proxy server. For example `proxy: "socks5h://127.0.0.0:1080"` will set the proxy server for just the current request. This proxy option is separate from the global websocket proxy.
 
 ### Creating New HTTP Sessions:
 To create new sessions for HTTP requests, use the `libcurl.HTTPSession` class. The constructor for this class takes the following arguments:
@@ -146,7 +146,7 @@ The valid HTTP session settings are:
 
 Each HTTP session has the following methods available:
 - `fetch` - Identical to the `libcurl.fetch` function but only creates connections in this session.
-- `set_connections` - Set the connection limits. This takes two arguments, the first being the limit for the connection cache, and the second being the max number of active connections.
+- `set_connections` - Set the connection limits. This takes three arguments: the first [is the hard limit of active connections](https://curl.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html) (default 60), the second is [limit for the connection cache](https://curl.se/libcurl/c/CURLMOPT_MAXCONNECTS.html) (default 50), and the third is the [max connections per host](https://curl.se/libcurl/c/CURLMOPT_MAX_HOST_CONNECTIONS.html) (default 6).
 - `export_cookies` - Export any cookies which were recorded in the session. This will return an empty string if cookies are disabled or no cookies have been set yet. 
 - `close` - Close all connections and clean up the session. You must call this after you are done using the session, otherwise it will leak memory. 
 
@@ -246,6 +246,8 @@ libcurl.set_websocket("ws://localhost:6001/");
 ```
 If the websocket proxy URL is not set and one of the other API functions is called, an error will be thrown. Note that this URL must end with a trailing slash.
 
+Changing the websocket proxy URL will not close any existing connections.
+
 ### Getting Libcurl's Output:
 If you want more information about a connection, you can pass the `_libcurl_verbose` argument to the `libcurl.fetch` function. These are the same messages that you would see if you ran `curl -v` on the command line.
 ```js
@@ -278,7 +280,7 @@ You can get version information from the `libcurl.version` object. This object w
 You can get the CA cert bundle that libcurl uses by calling `libcurl.get_cacert`. The function will return a string with the certificates in PEM format. The cert bundle comes from the [official curl website](https://curl.se/docs/caextract.html), which is extracted from the Mozilla Firefox source code. 
 
 ### Using the Wisp Client
-The `libcurl.wisp` object exposes all of the APIs from [wisp-client-js](https://github.com/MercuryWorkshop/wisp-client-js). This API is not guarenteed to be reliable. 
+The `libcurl.wisp` object exposes all of the APIs from [wisp-client-js](https://github.com/MercuryWorkshop/wisp-client-js). This API is not guaranteed to be stable. 
 
 ## Proxy Server:
 The proxy server consists of a standard [Wisp](https://github.com/MercuryWorkshop/wisp-protocol) server, allowing multiple TCP connections to share the same websocket.
@@ -309,7 +311,7 @@ This project is licensed under the GNU AGPL v3.
 ### Copyright Notice:
 ```
 ading2210/libcurl.js - A port of libcurl to WASM for use in the browser.
-Copyright (C) 2023 ading2210
+Copyright (C) 2024 ading2210
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU Affero General Public License as published by

client/javascript/http.js

@@ -4,7 +4,7 @@ class HTTPSession extends CurlSession {
     this.options = options;
     this.base_url = undefined;
 
-    this.set_connections(50, 40);
+    this.set_connections(50, 40, 6);
     this.import_cookies();
   }
 
@@ -181,6 +181,17 @@ class HTTPSession extends CurlSession {
     for (let [header_name, header_value] of response_info.headers) {
       response_obj.headers.append(header_name, header_value);
     }
+
+    //hack to fix invalid blob type
+    let response_proto = Object.getPrototypeOf(response_obj);
+    response_obj.blob = async () => {
+      let blob = await response_proto.blob.call(response_obj);
+      let mime_type = blob.type.split(";")[0].trim();
+      Object.defineProperty(blob, "type", {
+        value: mime_type
+      });
+      return blob;
+    }
     
     return response_obj;
   }

client/javascript/main.js

@@ -1,6 +1,6 @@
 /*
 ading2210/libcurl.js - A port of libcurl to WASM for the browser.
-Copyright (C) 2023 ading2210
+Copyright (C) 2024 ading2210
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU Affero General Public License as published by

client/javascript/session.js

@@ -28,9 +28,9 @@ class CurlSession {
     }
   }
 
-  set_connections(connections_limit, cache_limit) {
+  set_connections(connections_limit, cache_limit, host_conn_limit=0) {
     this.assert_ready();
-    _session_set_options(this.session_ptr, connections_limit, cache_limit);
+    _session_set_options(this.session_ptr, connections_limit, cache_limit, host_conn_limit);
   }
 
   end_callback(request_id, error) {
@@ -41,8 +41,7 @@ class CurlSession {
 
   data_callback(request_id, chunk_ptr, chunk_size) {
     let data = Module.HEAPU8.subarray(chunk_ptr, chunk_ptr + chunk_size);
-    let chunk = new Uint8Array(data);
+    this.request_callbacks[request_id].data(data);
-    this.request_callbacks[request_id].data(chunk);
   }
 
   headers_callback(request_id, chunk_ptr, chunk_size) {

client/libcurl/request.c

@@ -40,6 +40,7 @@ CURL* create_request(const char* url, int request_id, DataCallback data_callback
 
   curl_easy_setopt(http_handle, CURLOPT_PRIVATE, request_info);
   curl_easy_setopt(http_handle, CURLOPT_URL, url);
+  curl_easy_setopt(http_handle, CURLOPT_BUFFERSIZE, 512*1024);
 
   //callbacks to pass the response data back to js
   curl_easy_setopt(http_handle, CURLOPT_WRITEFUNCTION, &write_function);

client/libcurl/session.c

@@ -27,9 +27,10 @@ void session_perform(struct SessionInfo *session) {
   }
 }
 
-void session_set_options(struct SessionInfo *session, int connections_limit, int cache_limit) {
+void session_set_options(struct SessionInfo *session, int connections_limit, int cache_limit, int host_conn_limit) {
   curl_multi_setopt(session->multi_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, connections_limit);
   curl_multi_setopt(session->multi_handle, CURLMOPT_MAXCONNECTS, cache_limit);
+  curl_multi_setopt(session->multi_handle, CURLMOPT_MAX_HOST_CONNECTIONS, host_conn_limit);
 }
 
 void session_add_request(struct SessionInfo *session, CURL* http_handle) {

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libcurl.js-reclaimtls",
-  "version": "0.6.16-reclaimtls",
+  "version": "0.6.19-reclaimtls",
   "description": "A port of libcurl to WebAssembly, for proxying HTTPS requests from the browser with full TLS encryption",
   "main": "libcurl.mjs",
   "exports": {

server/wisp_server

@@ -1 +1 @@
-Subproject commit 78f872c8e5b180e26ec09cc126e841e12ac75c36
+Subproject commit f931adf9fea476fefffb1f0c3789cafda58d54ba