feat: Jan Integrates Cortex.cpp as Provider

[louis-jan]

Implementation Specs

Migration Path:

1. App 0.5.8 opens
2. Return model list from cache (given users are on 0.5.7) -> function normally.
3. Scan JSON models (legacy logics - fresh install or older versions) -> function normally.
4. In background, app attempts to import models and merge with legacy downloaded models (failed to import models)
5. The app combines models returned by cortex.cpp and legacy JSON models. Cortex.cpp models are prioritized in case of the same ID (Models are imported successfully.)

Changes

Naming convention

• inference-nitro-extension is renamed into inference-cortex-extension.
• cortex.cpp binaries have the same name as engine releases.
• Pre-package everything, include cuda dependencies (dll, so) so users don't have to install separately.
• Support noavx-cuda binaries as a fallback

Simplifed

• Deprecated ModelFile. It's no longer relevant. Now, providers define models, so it should manage how to run itself.
• Remove install cuda toolkit UX, should be ready after installed.

Downloader

App proxies to cortex.cpp or app's downloader, depending on the cortex.cpp model support capability.

Model Hub

• App allows extensions to register models available for download in RAM. After downloading them, the models will have their yaml or json persisted along with the model files.
• App priorities model hub decoration (previous json metadata) over cortex.cpp metadata (such as name, size, tags)

Observability

• cortex-extension should watch cortex.cpp server upon launch. It ensures that the cortex process runs with the application.
• All requests will be queued and run when the server to come online, ensuring the UX remains the same. So there would be no asynchronous requests and server run introduced. E.g. Model import or start should not fail due to server not being online in time.
• So there would be no attempt to kill the cortex process on model start every time. It is just a stop and start model, so it will not block other API requests.

Goals

• Updated from the older version to this version, models will be imported and run normally. Models are not imported will still able to run since we will attempt to do preflight before running.
• Users can download models or app proxies to cortex.cpp, or use the app downloader, depending on the cortex.cpp model support capability.

Subtasks

• Pull latest cortex.cpp and engines to package
• Keep binaries with the same name as their release name
• Support noavx-cuda binaries as a fallback
• Rename nitro-extension to cortex-extension
• Deprecate ModelFile
• Pre-package Cuda dependencies
• Registered models should be persisted in memory.
• App manages to download models using cortex.cpp downloader and it's legacy downloader (tensorrt-llm and clip models)
• App gets persisted models from cortex.cpp and scans JSON models itself (legacy)
• App prioritize model decoration metadata from Hub over cortex.cpp
• cortex-extension watches cortex.cpp server on launch
• Model-extension queues cortex.cpp API requests with health checks so requests do not fail due to server uptime
• cortex.cpp supports legacy model load parameters (NGL, context-length, cache enable, clip mmproj)
• App shares CUDA dependencies with extensions (asar.unpackaged) to improve load time and reduce app size when multiple extensions share the same artifacts (tensorrt-llm and llama-cpp)

@janhq/jan @janhq/cortex

#3825

[github-actions]

Preview URL: https://5239fa1d.docs-9ba.pages.dev

[louis-jan]

Update scenario

1. I have an old Jan version with downloaded models.
   

2. Updated to the newer Jan version, should maintain the models without delay or concern about cortex server corruption.
   

3. I can load models from older versions.
   

4. I can download new models using the Jan version.
   

5. Switch back to old Jan versions. Still see old downloaded models (not models downloaded on a newer version).
   

6. Switch back to the new Jan version, all downloaded models are visible
   

[louis-jan]

Rebased dev

[github-actions]

Barecheck - Code coverage report

Total: 69.62%

Your code coverage diff: 0.18% ▴

Uncovered files and lines

File	Lines
core/src/browser/extension.ts	105, 113-114, 116, 126-127, 145-146, 148-150, 156, 200, 202, 209-213, 215-216
core/src/browser/extensions/model.ts	12
core/src/browser/extensions/engines/EngineManager.ts	38
core/src/browser/extensions/engines/LocalOAIEngine.ts	70-71
core/src/browser/extensions/engines/OAIEngine.ts	59, 71, 108, 139-141, 169
core/src/browser/extensions/engines/helpers/sse.ts	37-38, 72-73, 80, 86, 93
core/src/browser/models/manager.ts	8, 11-12, 21-22, 27, 29, 38, 45
core/src/browser/models/utils.ts	90, 111, 130, 132-135, 137-138, 144, 152
core/src/node/api/processors/download.ts	18-20, 68, 73-74, 80-82, 97-98, 104-105, 110-111, 153-154
core/src/node/api/restful/helper/builder.ts	45, 52-53, 61, 100, 116-117, 131-132, 175, 205, 250, 271, 284, 287, 290, 326-327, 335-336, 341, 346
core/src/node/api/restful/helper/startStopModel.ts	21

[louis-jan]

Jan's API Server works with Cortex.cpp. Later, proxy everything.

[dan-menlo]

Nice 👀

Barecheck - Code coverage report

Total: 69.88%
Your code coverage diff: 0.37% ▴

Uncovered files and lines

[louis-jan]

Jan can handle multimodal download, even if cortex.cpp is not supported.

Jan can handle its downloaded models aside from cortex.cpp /models

LlaVa 7B is downloaded by Jan, works with legacy model.json, others are handled by cortex.cpp.

[louis-jan]

Feature test build
https://github.com/janhq/jan/actions/runs/11675880590

[dan-menlo]

prays lgtm