Commit Graph

17654 Commits

Author SHA1 Message Date
Timothy Jaeryang Baek b7489bbc6c refac 2026-07-26 23:16:58 -04:00
Timothy Jaeryang Baek ed663f16ec refac 2026-07-26 23:09:22 -04:00
Timothy Jaeryang Baek 95d590b360 refac 2026-07-26 23:03:32 -04:00
Timothy Jaeryang Baek 8b206de48e refac 2026-07-26 22:59:23 -04:00
Timothy Jaeryang Baek bf35f64a7f refac 2026-07-26 22:45:11 -04:00
Timothy Jaeryang Baek fc4906c9e9 refac 2026-07-26 22:32:17 -04:00
Timothy Jaeryang Baek aadab2f480 refac 2026-07-26 22:32:06 -04:00
Timothy Jaeryang Baek 846ba80a9d refac 2026-07-26 22:32:03 -04:00
Timothy Jaeryang Baek d94d36ad72 refac 2026-07-26 22:16:31 -04:00
Timothy Jaeryang Baek d14fddf254 refac 2026-07-26 21:55:13 -04:00
Timothy Jaeryang Baek 0cbf337679 refac 2026-07-26 21:54:06 -04:00
Timothy Jaeryang Baek 85c47fb467 refac 2026-07-26 21:51:35 -04:00
Timothy Jaeryang Baek 1de36d600f refac 2026-07-26 21:35:56 -04:00
Timothy Jaeryang Baek 71c4da8c06 refac 2026-07-26 21:12:14 -04:00
Timothy Jaeryang Baek f867825bf3 refac 2026-07-26 21:08:49 -04:00
Timothy Jaeryang Baek b45c020f68 refac 2026-07-26 21:08:44 -04:00
Timothy Jaeryang Baek d2936c880c refac 2026-07-26 21:07:27 -04:00
Timothy Jaeryang Baek d484a2a99e refac 2026-07-26 21:07:20 -04:00
Timothy Jaeryang Baek bab71ed08b refac 2026-07-26 21:06:18 -04:00
Timothy Jaeryang Baek db5c092299 refac 2026-07-26 21:02:31 -04:00
Timothy Jaeryang Baek f798d05586 refac 2026-07-26 19:34:41 -04:00
Timothy Jaeryang Baek 94a60b0457 refac 2026-07-26 19:10:41 -04:00
Classic298 54f06d8c53 perf: build chat responses without deep-copying the blob through model_dump (#27388)
Chat search built each result row with ChatTitleIdResponse(**chat.model_dump(), ...), which recursively copies the entire chat blob per row only for the constructor to ignore everything except id, title and timestamps: a 60-row search page deep-copied up to 60 full conversations. The folder listing, archived and export endpoints and every single-chat response did the same dump-and-revalidate dance via ChatResponse(**chat.model_dump()).

Search rows are now built from the five fields the response actually has (the snippet helper receives the blob by reference as before), and all 18 ChatResponse constructions use ChatResponse.model_validate(chat, from_attributes=True), which reads the fields off the already-validated ChatModel without copying the blob.

Benchmark (~500 KB chat blob):

| metric | before | after |
| --- | --- | --- |
| search result row | 0.05 ms | 0.003 ms |
| ChatResponse construction | 0.05 ms | 0.003 ms |
| per search page (60 rows) | 3 ms | 0.2 ms |

Beyond CPU, each converted row also stops materializing a second full copy of the conversation in memory while the page is being built.

Functionally verified: both construction styles produce identical model_dump() output for ChatResponse (including defaulted fields absent on ChatModel) and for search rows including the snippet.
2026-07-26 18:57:52 -04:00
Classic298 4f93c3e36c fix: authorize before cancelling tasks in the chat delete endpoint (#27006)
DELETE /api/v1/chats/{id} called stop_item_tasks(id) before checking the
caller's chat.delete permission or ownership of the target chat. An
authenticated user who knew another user's chat id could therefore cancel that
chat's in-flight generation (streaming response, title or tag generation) even
though the deletion was then rejected. The chat id is discoverable through
legitimate read-only access to a shared chat or folder.

Reorder the handler to authorize first (admin, or owner holding chat.delete) and
only then cancel tasks and delete, matching the dedicated task-stop endpoint.
Legitimate deletions are unchanged; an unauthorized caller now returns 404 or 401
before any cancellation. The duplicated tag-cleanup and event-publish blocks are
merged.

Co-authored-by: GabrielGomesAL <193945687+GabrielGomesAL@users.noreply.github.com>
2026-07-26 18:57:35 -04:00
Classic298 1ac8ef7853 fix: gate the remaining text contrast failures behind High Contrast Mode (#27558)
Completes the contrast set after #27555, #27554 and the gray-500 branch, which between them cover text-gray-400, text-gray-500, dark:text-gray-600 and placeholders. This is everything still under 4.5:1 after those.

The grey scale in src/tailwind.css is achromatic oklch(L 0 0), so relative luminance is exactly L³. What is left:

- text-gray-300 dark:text-gray-700, the lightest muted pair, at 1.58:1 in light and 2.14:1 in dark. Used for inactive tab labels across the admin, workspace and playground layouts, breadcrumb separators and empty state hints. It resolves to gray-600 in light and gray-400 in dark.
- text-gray-400/70 on the embedded chat history dropdown icon, 2.07:1 against the 3:1 that WCAG 1.4.11 requires of icons.
- Hover states that land lighter than the new resting colour. Once the resting state is gray-600, an element hovering to gray-500 gets less readable on interaction rather than more, so hover and group-hover targets of gray-500 resolve to gray-800. Sidebar/Section.svelte and the citation modal links are the sites this affects.
- The autocompletion ghost text in src/app.css, hardcoded #a0a0a0, 2.65:1 in light. The dark canvas already passes.
- The shimmer used for loading text, a #b4b4b4 gradient clipped to the glyphs at 2.10:1 in light. There is no solid colour to raise, so with the setting on it renders as flat gray-700 text instead.

Everything above is gated on the existing High Contrast Mode setting and changes nothing when it is off. No markup is touched, so this is src/app.css only.

Deliberately left alone: disabled: variants, since WCAG 1.4.3 exempts inactive components; the FileNav breadcrumb ancestors, which are non-clickable; decorative folder icons; and text-gray-100, dark:text-gray-800 and dark:text-gray-900, which are inverse text on filled buttons and already high contrast against their own backgrounds. The ad-hoc dark:text-gray-800 pairs in ChannelModal.svelte and automations/+layout.svelte stay as they are; dark:text-gray-800 doubles as the inverse text on the white buttons in Message.svelte, ResponseMessage.svelte and UserMessage.svelte, so it cannot be remapped in CSS without breaking those.

Not fixed here, and a genuine follow-up: .hljs-comment in src/app.css is #616161, roughly 3:1 on the dark code background. It sits outside the Tailwind grey scale and needs a highlight.js theme override rather than a utility remap.

Verified in a browser against Tailwind's emitted rules and layer order: with the setting on, the lightest pair resolves to gray-600 in light and gray-400 in dark, the hover and group-hover targets to gray-800, ghost text to gray-600 and the shimmer to solid gray-700, while inverse button text and every dark hover variant stay where they are; with the setting off nothing changes in either theme.
2026-07-26 18:57:00 -04:00
Classic298 5b035ea52b fix: let Select announce its selected value and open state (WCAG 2.5.3, 4.1.2) (#27492)
On latest `dev`, the `Select` trigger sets `aria-label={placeholder}`. In the accessible name computation `aria-label` is evaluated before the element's contents, so on a button that renders visible text it **replaces** that text instead of adding to it.

The trigger's content is `selectedLabel`, which resolves to the selected item's label and only falls back to `placeholder` when nothing is selected. So a control visually reading "Week" is exposed to assistive technology as "Select view", permanently, no matter what is selected. The dropdown items expose no `aria-selected` either, the current one is marked with a check icon only, so there is no path by which a screen reader user can find out what the control is set to.

Breaks WCAG 2.5.3 Label in Name (Level A), because the accessible name does not contain the visible label, so voice control cannot target the control by what it says on screen. Also 4.1.2 Name, Role, Value (Level A), because the value is never exposed.

Fix: drop the overriding `aria-label` so the name is computed from the visible text, and expose `aria-expanded` so the open state is conveyed. All 7 call sites plus the `DropdownOptions` wrapper override `slot="trigger"` and every one of them renders `selectedLabel` (or `placeholder`) as text, so no trigger is left unnamed. The two call sites that pass no `placeholder` were already emitting an empty `aria-label`, which is skipped by the name computation, so they are unaffected.

`aria-haspopup` is deliberately not added: the popup is a plain `DropdownMenu` of buttons with no `listbox` role, so claiming one would misdescribe it.

`placeholder` is still used, it drives the `selectedLabel` fallback and `TagSelector` renders it directly.

Severity: Serious. Affects every custom select in Admin Settings, Workspace and the calendar and automations pages.

### Contributor License Agreement

<!--
🚨 DO NOT DELETE THE TEXT BELOW 🚨
Keep the "Contributor License Agreement" confirmation text intact.
Deleting it will trigger the CLA-Bot to INVALIDATE your PR.

Your PR will NOT be reviewed or merged until you check the box below confirming that you have read and agree to the terms of the CLA.
-->

- [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms.

> [!NOTE]
> Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
2026-07-26 18:56:49 -04:00
Classic298 4650f64c1e fix: make admin user table sortable by keyboard (WCAG 2.1.1, 4.1.2) (#27501)
On latest `dev`, the five sortable column headers in the admin Users table are click handling `<th>` elements:

```svelte
<th scope="col" class="px-2.5 py-1.5 font-normal cursor-pointer select-none" on:click={() => setSortKey('name')}>
```

A `<th>` is not interactive. There is no `<button>`, no `tabindex`, no `role` and no key handler, so **sorting the user list is impossible without a mouse**. The sort direction is also conveyed only by an 8×8 pixel chevron, with no programmatic state, so assistive technology cannot report which column is sorted or in which direction.

Breaks WCAG 2.1.1 Keyboard (Level A) and 4.1.2 Name, Role, Value (Level A).

Fix: move the click handler onto a real `<button>` inside the header, which brings native focus, Enter and Space activation and the correct role, and add `aria-sort` to the `<th>`, which already carries `scope="col"` and therefore the implicit `columnheader` role. Only the active column reports a direction, since `orderBy` is a single value; the non sortable actions column deliberately gets no `aria-sort` at all rather than `none`, so it is not advertised as sortable.

The cell padding moves from the `<th>` onto the button so the whole header stays clickable. Left on the `<th>`, the padding ring would have become a dead zone, shrinking the hit target and flipping the cursor at an invisible boundary inside the header.

`cursor-pointer` is dropped from the `<th>` because `src/tailwind.css` already applies it to every `button`.

The repeated `aria-sort` ternary is extracted to a small `sortState` helper rather than pasted five times.

The same mouse only `<th on:click>` pattern still exists in the Analytics, Evaluations and Groups tables and is not touched here.

Severity: Serious. A core admin function is unreachable without a pointing device.

### Contributor License Agreement

<!--
🚨 DO NOT DELETE THE TEXT BELOW 🚨
Keep the "Contributor License Agreement" confirmation text intact.
Deleting it will trigger the CLA-Bot to INVALIDATE your PR.

Your PR will NOT be reviewed or merged until you check the box below confirming that you have read and agree to the terms of the CLA.
-->

- [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms.

> [!NOTE]
> Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
2026-07-26 18:56:38 -04:00
Classic298 c055203f29 fix: refresh OAuth session before the id_token expires (#27520)
`_normalize_token_expiry()` derived the session expiry from the access token alone, and that value is what `oauth_session.expires_at` stores and what both `get_oauth_token()` implementations check to decide whether to refresh five minutes ahead. Providers that issue a shorter-lived id_token than access token (Microsoft Entra ID: roughly 60 minutes against 75) therefore left a window where the session still looked valid while the id_token had already expired, so pipes and tools reading `__oauth_token__["id_token"]` forwarded a dead JWT and downstream services rejected it with 401.

The stored expiry is now capped at the id_token's `exp` claim whenever that JWT expires first, which moves the refresh ahead of the earliest expiring token in the set. This is applied in the single function every session write already passes through, so it covers both the SSO manager and the MCP client manager on their callback and refresh paths alike. Sessions without an id_token, with an opaque one, or with no `exp` claim are unaffected.

Fixes #27066
2026-07-26 18:56:18 -04:00
Classic298 99da2324e3 fix: preserve chunk order when assembling multi-chunk transcriptions (#27417)
When an audio file is split into multiple chunks for transcription, transcribe() collected the per-chunk results with asyncio.as_completed(), which yields results in completion order rather than submission order. Whenever a later chunk finished transcribing before an earlier one, the assembled transcript was scrambled, for example the second half of a recording appearing before the first, and the stored file content plus everything downstream (file preview, full-context retrieval) read out of chronological order. This change awaits the chunk tasks with asyncio.gather() instead, which runs them just as concurrently but returns the results in the order the tasks were created, i.e. chunk_paths order. The existing error handling and chunk cleanup are unchanged: an HTTPException from a chunk is re-raised as is and any other error is wrapped in a 500. Fixes #27143
2026-07-26 18:55:52 -04:00
Classic298 30be10f968 feat: prevent duplicate auth form submissions while one is pending (#27416)
* feat: prevent duplicate auth form submissions while one is pending

When a sign in, sign up or LDAP request is slow, the auth form can be submitted again and every extra click or Enter press starts another concurrent authentication request. The form never tracked a pending state, so submitHandler dispatched a new API call on every submit event. This adds a submitting flag that makes submitHandler ignore re-entrant submits, disables both submit buttons with a dimmed style while a request is in flight and resets the flag in a finally block so the form recovers after a failed attempt. Guarding submitHandler covers button clicks and Enter key submits for sign in, sign up and LDAP alike since all of them flow through the single form submit handler.
Fixes #27264

* feat: show a spinner while an auth request is pending

Disabling the submit button stops a second submission but gives no positive sign that the first one is still running, so on a slow identity provider the form looks unresponsive rather than busy. Both submit buttons now render the existing Spinner next to their label while submitting is set, following the same in-button pattern used by the workspace editors.
2026-07-26 18:55:41 -04:00
Classic298 381149ea5e fix: persist filter outlet() changes to structured message output (#27414)
When a filter's outlet() modified the structured assistant output in place, the change was shown immediately but lost after reload. outlet_filter_handler built its outlet payload with a shallow reference to the message's output list from messages_map, so the filter mutated the stored baseline itself and the subsequent output comparison compared the object against itself, never detecting a change and never persisting it. The same aliasing corrupted originalContent for messages whose text lives only in output. Deepcopy the output when building the outlet payload so messages_map stays a pristine pre-filter baseline and the existing change detection persists outlet-modified output through the existing upsert path. Fixes #27017.
2026-07-26 18:55:21 -04:00
Classic298 69f8be4cf9 fix: show download preparation toast and prevent duplicate zip jobs (#27421)
* fix: show download preparation toast and prevent duplicate zip jobs

Downloading a file or folder from the file navigator gave no feedback while the server prepared the response, which can take 30 seconds or more for large folders that are zipped server-side. Users assumed the click did nothing and pressed Download again, starting additional zip jobs on the server. Both downloadFile and bulkDownload, the two functions every download control funnels through, now show a persistent "Preparing download..." loading toast while the request is in flight and dismiss it once the download starts or fails. A shared downloading flag makes repeated clicks no-ops until the current download finishes, so a single click starts exactly one server-side job. Fixes #27055

* fix: report terminal download failures instead of dismissing the toast

A failed download dismissed the preparation toast without saying anything, which reads as the download silently disappearing. Both download paths now report the failure. The two helpers also declare a nullable return but could still reject once the response body started streaming, so an interrupted transfer escaped as an unhandled rejection and left the same silent dismissal. They now return null in that case, which also stops an interrupted preview from leaving its spinner running.
2026-07-26 18:55:09 -04:00
Timothy Jaeryang Baek bef63a2ae9 refac 2026-07-26 18:54:17 -04:00
Timothy Jaeryang Baek df94268e89 refac 2026-07-26 18:54:07 -04:00
crustopher-lgtm 5efe0951d5 feat: add OpenSERP self-hosted web search backend (#27437)
Add self-hosted OpenSERP as a web search engine option. OpenSERP
provides browser-rendered search across Google, Bing, Yandex, Baidu,
DuckDuckGo, and Ecosia with no API keys required.

- New module: retrieval/web/openserp.py (async, uses aiohttp session pool)
- Config: OPENSERP_BASE_URL env var (defaults to http://localhost:7070)
- Routing: search_web() dispatch for 'openserp' engine
- Follows existing patterns (searxng, brave)

Co-authored-by: crustopher-lgtm <crustopher-lgtm@users.noreply.github.com>
2026-07-26 18:52:08 -04:00
Timothy Jaeryang Baek 42ea8a5a2f refac 2026-07-26 18:50:22 -04:00
G30 bda49ccdb6 fix(ui): close sidebar on mobile when opening Calendar from user menu (#26979)
Every other navigation entry in the user menu (Settings, Admin Panel,
Archived Chats, Workspace, Notes, Automations, Playground, Sign Out)
collapses the sidebar on mobile after navigating, but the Calendar entry
was missing this handling, leaving the sidebar open over the Calendar
page on mobile. Add the same mobile guard used by the sibling entries.
2026-07-26 18:47:12 -04:00
Timothy Jaeryang Baek b81627b2c9 refac 2026-07-26 18:46:39 -04:00
G30 11d72c1ce2 fix(ui): replace history entry on redirect so browser back navigation works (#27478) 2026-07-26 18:44:23 -04:00
G30 79695a1d14 fix: persist modelIdx so duplicate side-by-side models don't collapse on reload (#26980)
When the same model is selected multiple times in a side-by-side chat,
each response is created with a distinct modelIdx (0,1,2,3) that
identifies its column. The backend now owns message persistence, but it
built the assistant placeholders without modelIdx, so the field was
never saved. On reload MultiResponseMessages groups responses by
modelIdx and falls back to grouping by model id when modelIdx is
missing; with duplicate models that fallback lumps every response into
each column, so all columns render the first response (and show a bogus
'1/N' pager).

Send modelIdx with each message_ids entry from the frontend and persist
it on the assistant placeholders in both the new-chat and existing-chat
paths. The message_ids list is now forwarded for every send (not just
multi-model ones) so single-column regenerations in a duplicate-model
chat also keep their column identity across reloads.
2026-07-26 18:44:03 -04:00
G30 86bf927d08 fix(ui): stack automation modal footer on mobile (#27027)
The automation create/edit modal footer laid the schedule/model dropdowns
and the Cancel/Create actions in a single justify-between row. On narrow
modals the fixed-width actions left too little room for the dropdowns,
which wrapped to two stacked lines with Cancel squeezed in the middle.

Stack the footer vertically on mobile (dropdowns row, then a right-aligned
actions row) and restore the horizontal layout at sm and up.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 18:43:45 -04:00
G30 db92ef292f fix: unarchive chats moved into folders and refresh sidebar folders after menu moves (#27485) 2026-07-26 18:43:34 -04:00
G30 71f8b6d5b4 feat: add a master OAuth / OIDC enable toggle in Authentication settings (#26988)
The OAuth / OIDC section in Admin Settings > Authentication had no
enable/disable switch, unlike the LDAP section above it. Add one that
persists via the existing Save flow and actually gates OAuth sign-in,
mirroring how the LDAP toggle works.

- config: new ENABLE_OAUTH persistent config ('oauth.enable'), defaulting
  to True so existing deployments with a provider configured keep working.
- oauth: expose ENABLE_OAUTH via the OAuth runtime config and reject the
  login and callback handlers with 404 when it is disabled.
- /api/config: report no OAuth providers when disabled so the login page
  hides the OAuth buttons (and cannot auto-redirect), without clearing the
  admin's provider configuration.
- auths: expose ENABLE_OAUTH through the admin OAuth config get/update
  endpoints (OAuthConfigForm + OAUTH_CONFIG_KEYS).
- Authentication.svelte: bind the OAuth / OIDC header Switch to the
  persisted oauthConfig.ENABLE_OAUTH and collapse the section when off,
  matching the LDAP header (size, weight, alignment).
2026-07-26 18:43:21 -04:00
Classic298 18ca19044c fix: gate muted text contrast fix behind High Contrast Mode (#27554)
Follow-up to #27495, reopened as a high contrast mode change. Builds on the `high-contrast` class landed in #27555.

Muted UI text is written as text-gray-400 dark:text-gray-600. The grey scale in src/tailwind.css is achromatic oklch(L 0 0), so relative luminance is exactly L³: text-gray-400 is 2.07:1 on white and dark:text-gray-600 is 3.12:1 on #171717. The pair is effectively inverted, and both halves fail the 4.5:1 required by WCAG 1.4.3, with the light value also failing the 3:1 required of icons under 1.4.11. This is the text used for settings and admin section headings, field descriptions, sidebar labels, timestamps and counters, all at 10px to 12px, so the large-text exemption does not apply.

Rather than rewriting the class literal at 251 sites, the remap is two CSS rules that only apply when the existing High Contrast Mode setting is on, so the default theme is untouched:

- text-gray-400 resolves to gray-600 (5.75:1) in light mode
- dark:text-gray-600 resolves to gray-400 (8.65:1) in dark mode

dark:text-gray-500 already passes at 6.46:1 and is left alone.

The rules live in `@layer utilities` and use `:where()` to stay at low specificity: they outrank the base utility but lose to `hover:` and `dark:hover:` variants, so hover feedback keeps working. Verified in a browser against Tailwind's emitted rules and layer order: with the setting on, resting text resolves to gray-600 in light and gray-400 in dark, hover still resolves to its own value, and with the setting off nothing changes in either theme.

One component change is required alongside it. In admin/Settings/Audio.svelte the help text puts links inside the muted block via `[&_a]:text-gray-600`; once the surrounding prose resolves to gray-600 the link becomes the same colour as the text it sits in, and it has no resting underline, which would be a new WCAG 1.4.1 failure. The link moves to gray-900. This is the only place in the codebase where a link colour is nested inside muted text.

Letting the variants win has one edge: a few elements hover to a grey lighter than their new resting colour, so hovering would have lowered contrast instead of raising it. A light-mode hover landing on gray-500 now resolves to gray-800, which keeps the hover darker than the gray-600 resting state. Sidebar/Section.svelte is the site this branch would otherwise break.

Not covered here: text-gray-500 dark:text-gray-400 (2.77:1 in light), which is a separate branch.
2026-07-26 18:37:49 -04:00
Timothy Jaeryang Baek 453b9fb029 refac 2026-07-26 18:36:49 -04:00
Classic298 50afbc5319 fix: allow setting model order via MODEL_ORDER_LIST env var (#27420)
With ENABLE_PERSISTENT_CONFIG=False the admin's model order is reset on every restart because ui.model_order_list falls back to its DEFAULT_CONFIG default, and unlike every other Models setting (DEFAULT_MODELS, DEFAULT_PINNED_MODELS, DEFAULT_MODEL_METADATA and DEFAULT_MODEL_PARAMS) that default was hardcoded to an empty list with no environment variable to source it from. This adds a MODEL_ORDER_LIST environment variable parsed as a JSON array using the same guarded pattern as the neighbouring DEFAULT_MODEL_METADATA and DEFAULT_MODEL_PARAMS defaults, falling back to an empty list on parse errors. Behaviour when the variable is unset is unchanged.

Fixes #27206
2026-07-26 18:34:18 -04:00
Classic298 0116c6e1b9 perf: stop running chardet over entire uploaded files (#27445)
`_detect_text_encoding()` hands the complete file to `chardet.detect()`. chardet is pure Python and costs roughly 1.3 seconds per megabyte, so uploading a large non-UTF-8 text file stalls for seconds inside encoding detection alone. A 4 MiB Shift-JIS file spends 6.4 seconds there. The UTF-8 fast path above it means only non-UTF-8 files reach this, which in practice are exactly the CJK documents the surrounding code was written to handle, so the slow case and the case that matters are the same case.

Detection does not need the whole file. It needs the bytes that are actually not UTF-8, and `UnicodeDecodeError.start` from the fast-path decode already says where those begin, so this samples a 256 KiB window around that offset.

Two things make that safe rather than merely fast.

Centring the window on the first non-UTF-8 byte instead of the file head is what keeps the common case correct. A plain head sample makes chardet report ascii for a file that is ASCII for its first few hundred KiB and only turns CJK later, and the method then falls through to latin-1 instead of the right codec.

The window still cannot help when a stray byte, a pasted Windows-1252 artifact for example, sits hundreds of KiB ahead of the real payload: the sample is then almost pure ASCII and carries no signal. So when the sample holds almost no non-ASCII bytes and is a strict subset of the file, detection falls back to the whole buffer. That case pays the old cost, which is the right trade, because it is precisely the case where sampling would otherwise be wrong. Without this guard a Cyrillic document with a stray leading byte was detected as ISO-8859-1 rather than windows-1251, which is silent mojibake.

Measured, with the encoding returned identical in every case:

| file | before | after |
|---|---|---|
| shift_jis 4 MiB | 6402ms | 755ms |
| gb18030 4 MiB | 3199ms | 449ms |
| big5 4 MiB | 2926ms | 413ms |
| euc-jp 4 MiB | 2456ms | 413ms |
| euc-kr 4 MiB | 2382ms | 468ms |
| latin-1 4 MiB | 1902ms | 394ms |
| gb18030 1 MiB | 807ms | 376ms |
| ascii head then gb18030 tail | 533ms | 294ms |
| stray byte then cp1251 payload | 496ms | 1051ms |
| any UTF-8 file | 8ms | 0ms |

29 cases, all returning an identical encoding before and after: six encodings at 100 KiB, 1 MiB and 4 MiB, three layouts where the non-UTF-8 bytes only begin beyond the window, four where a stray byte is separated from the payload, plus plain UTF-8, UTF-8 CJK and an empty file. The stray-byte rows are slower than before because they scan twice, once over the window and once over the whole buffer. They are the pathological shape, and correctness wins there.

The residual time is now the decode-and-validate loop below, which walks the file once per candidate codec, and `_has_cjk_characters`, which is a per-character Python loop over the decoded text. Both are the same "full scan for a detection decision" pattern and could take a bounded prefix too. That is left alone here.
2026-07-26 18:34:02 -04:00
Classic298 bc948f8f22 perf: parse scraped web pages off the event loop (#27446)
`alazy_load()` builds every BeautifulSoup tree inline in an async function, so a web search that pulls in ten pages stops the entire worker for the whole time it spends parsing. Nothing else on that worker runs during it: not other users' token streams, not health checks, not socket.io traffic. Parsing is CPU work and it belongs in a thread.

Measured over 37 real pages, 13.5 MiB total, with a 5ms ticker sampling event-loop lag:

| | wall | worst loop stall | ticker fired |
|---|---|---|---|
| inline, html.parser (today) | 1793.8ms | 1788.8ms | 1 time |
| offloaded, html.parser | 1872.9ms | 82.9ms | 88 times |
| inline, lxml | 1346.7ms | 1341.8ms | 1 time |
| offloaded, lxml | 1445.4ms | 37.0ms | 118 times |

Today the loop is not merely slow during a batch, it is gone: a 5ms timer fired exactly once across 1.8 seconds. After the change it fires normally and the worst single stall drops by a factor of 20 to 36. The cost is 4 to 7 percent more wall time for the batch itself, from the thread handoffs, which is the right trade for a server handling more than one user.

Three details behind the shape of the change:

`get_text()` is only 2 percent of the cost (34ms against 1706ms of parsing over the corpus), so the whole per-page unit moves into the thread rather than the parse alone. Splitting them measured worse on both axes.

The offload is per page, not per batch. Handing the whole batch to one thread measured worse than either (2081ms wall, 235ms worst stall), so the loop is yielded to between pages.

The metadata block in `alazy_load()` was a duplicate of the module-level `extract_metadata()`, field for field, and `lazy_load()` was already using the shared helper. The new helper calls it too, which is why the diff removes more lines than it adds. The `ascrape_all()` override goes with it: it was a verbatim copy of the inherited implementation and `alazy_load()` was its only caller, so anything still calling it now gets the identical parent method, which resolves `self._unpack_fetch_results` to the override this class keeps.

Verified by feeding the real loader a 37 page corpus and comparing every resulting Document against the implementation this replaces:

```
PASS  one Document per url (37)
PASS  every Document identical to the pre-change implementation (0 differ)
PASS  parsing ran off the main thread
PASS  event loop kept running during parsing (90 ticks)
```

Both `page_content` and `metadata` are byte-identical on all 37 pages. This is independent of the parser in use and composes with switching the default parser to lxml: that change makes the stalls shorter, this one takes them off the loop.
2026-07-26 18:33:27 -04:00
Classic298 b9cfba62d7 chore: regenerate uv.lock (#27557)
Regenerated with the current uv so `uv lock --check` passes again. The committed lockfile was written by an older uv and every run since has reported it as needing an update, which makes it impossible to tell real drift apart from format drift.

No resolved dependency changes: all 354 packages keep their versions, and no existing artifact URL or hash changes. The diff is almost entirely `upload-time` annotations added per artifact. The remaining changes are the `revision = 3` format marker, a `provides-extras` entry on the project stanza, additional GraalPy wheel URLs for already-locked versions of jiter, pybase64, pydantic-core and ujson, and the removal of the hardcoded `version = "0.10.2"` from the project's own stanza, which was stale metadata since pyproject.toml declares `dynamic = ["version"]`.

Nothing that gets installed changes: the Dockerfile installs from backend/requirements.txt and no workflow runs `uv lock` or `uv sync`.
2026-07-26 18:31:12 -04:00
Timothy Jaeryang Baek 4856afcef8 refac 2026-07-26 18:30:58 -04:00