Skip to content

Tags: danbev/llama.cpp

Tags

b7938

Toggle b7938's commit message
vendor : try to suppress boring ssl warnings

b7731

Toggle b7731's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
ggml-metal: do not copy headers for embedded, use current binary dir …

…for embedded (ggml-org#18705)

b6946

Toggle b6946's commit message

Verified

This commit was signed with the committer’s verified signature.
danbev Daniel Bevenius
sampling : add support for GPU sampling (wip)

This is a work in progress to add support for GPU sampling.

The motivation for this feature is to enable sampling to be performed
directly on the GPU as part of the computation graph being executed,
allowing for some or all of the sampling to be done on the GPU.

For example, the GPU sampler chain might select/sample a token directly
in which case only the sampled token needs to be transferred from
device memory to host memory.

It is also possible for the GPU samplers to perform filtering of the
logits, or compute and filter the probability distribution, in which
case only the filtered logits or probabilites need to be transferred
back to system memory for further processing by CPU samplers.

Currently the GPU sampling works in a similar manner to how pooling
works, it is a function that is called by build_graph:
```c++
    // add GPU sampling layers (if any)
    llm->build_sampling(*this, params);
```

GPU samplers can be configured by creating sampler chains, where each
sampler chain is associated with a specific sequence id:
```c++
    struct llama_sampler_chain_params params = llama_sampler_chain_default_params();
    struct llama_sampler * chain = llama_sampler_chain_init(params);
    llama_sampler_chain_add(chain, llama_sampler_gpu_init_greedy());
    std::vector<llama_sampler_seq_config> sampler_configs = {
        { 0, gpu_sampler_chain }
    };
```
The struct is defined as:
```c++
    struct llama_sampler_seq_config {
        llama_seq_id           seq_id;
        struct llama_sampler * sampler;
    };
```

These sampler configs are then passed as context params:
```c++
        llama_context_params cparams = llama_context_default_params();
        cparams.samplers = sampler_configs.data();
        cparams.n_samplers = sampler_configs.size();
```

When the graph is built, the configured sampler's _apply function is
called which allows them to add operations/nodes to the computation
graph.

This enables the sampling to happen fully, or partially on the GPU. The
samplers could sample a single token in which case that is what will be
transferred from the device memory to host memory after llama_decode has
been called. The sampled token can then be retrieved using:
```c++
    llama_token id = llama_get_sampled_token_ith(test_ctx.ctx, index);
```

Is it also possible to run a GPU sampler that only filters the logits
and then only the filtered logits are transferred back to the host and
the sampling can proceed on the CPU with the normal (CPU) sampler chain.
In this case the CPU samplers are configured as usual but they will now
operate on already filtered logits.

Similar to the above handling of logits, it is possible for a GPU
samplers to compute the full probability distribution and transfer that
to the host. And the CPU samplers can then operate on the those
probabilities.

Building and running the tests:

Download a model for testing:
```console
$ cd models && wget https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories15M-q4_0.gguf
```
Building the test:
```console
$ cmake --build build --target test-gpu-sampling -j8
```
Runing all tests:
```console
$ env LLAMACPP_TEST_MODELFILE=../models/stories15M-q4_0.gguf \
    ctest --test-dir build -R '^test-gpu-sampling$' -V
```

The following individual tests are available:
```console
$ ctest --test-dir build -N -R test-gpu-sampling-
  Test 35: test-gpu-sampling-greedy
  Test 36: test-gpu-sampling-temp
  Test 37: test-gpu-sampling-softmax
  Test 38: test-gpu-sampling-top_k
  Test 39: test-gpu-sampling-top_p
  Test 40: test-gpu-sampling-mul_seq

Total Tests: 6
```
These can be run individually, for example:
```console
$ env LLAMACPP_TEST_MODELFILE=../models/stories15M-q4_0.gguf \
    ctest --test-dir build -R 'test-gpu-sampling-temp' -V
```

TODO:

- [ ] Allow GPU samplers to pre-allocate state tensors
- [ ] Integrate GPU samplers with llama-server
- [ ] Implement true top-p sampler on GPU
- [ ] Add missing GPU samplers (e.g. typical, mirostat, etc)

b6883

Toggle b6883's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
llama : use std::abs instead of abs (ggml-org#16853)

b6708

Toggle b6708's commit message
ci : add option to disable CPU repack in test workflow

b6688

Toggle b6688's commit message

Verified

This commit was signed with the committer’s verified signature.
danbev Daniel Bevenius
ci : add job for testing AMX

b6686

Toggle b6686's commit message

Verified

This commit was signed with the committer’s verified signature.
danbev Daniel Bevenius
ggml : check src[1] does not have more than 2 dimensions

b6674

Toggle b6674's commit message

Verified

This commit was signed with the committer’s verified signature.
danbev Daniel Bevenius
ci : change macos-13 to macos-15-intel

This commit updates the macos-13 runners to macos-15-intel.

The motivation for this changes is the macos-13 runners are scheduled
to be retired on 2025-12-04.

Refs: https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

b6672

Toggle b6672's commit message
switch to use ubuntu-22.04-arm [no ci]

b6440

Toggle b6440's commit message

Verified

This commit was signed with the committer’s verified signature.
danbev Daniel Bevenius
ci : add caching for ROCm installation in release workflow

This commit applies the same caching to the release workflow which
currently exists for the main CI workflow that was introduced in Commit
ff02caf ("ci : cache ROCm installation
in windows-latest-cmake-hip (ggml-org#15887)").