Examples
Matmul and reduction
Section titled “Matmul and reduction”Save this as operations.py in the installed source checkout, then run
uv run --no-sync python operations.py:
import tensorcx as cx
a = cx.tensor([[1.0, 2.0], [3.0, 4.0]], device="cpu")b = cx.ones((2, 2), dtype=cx.float32, device="cpu")
product = cx.matmul(a, b)print(product.numpy())# [[3. 3.]# [7. 7.]]
print(cx.sum(a, axis=1).numpy())# [3. 7.]
print(cx.softmax(a, axis=-1).shape)# (2, 2)
print(((a - 1) / 2).reshape((4,)).numpy())# [0. 0.5 1. 1.5]
print(a.sum(axis=1, keepdims=True).shape)# (2, 1)
values = cx.tensor([[1, 2, 3], [4, 5, 6]])bias = cx.tensor([0.25, 0.5, 0.75])result = values.astype(cx.float32) + biasprint((result - result.mean(axis=-1, keepdims=True)).numpy())# [[-1.25 0. 1.25]# [-1.25 0. 1.25]]
print(values.T.numpy())# [[1 4]# [2 5]# [3 6]]
expanded = values.expand_dims((0, -1))print(expanded.shape) # (1, 2, 3, 1)print(expanded.squeeze().shape) # (2, 3)
print(a.sum().numpy()) # 10.0print(a.mean(axis=(0, 1), keepdims=True).numpy()) # [[2.5]]
print(values[:, ::-1].numpy())# [[3 2 1]# [6 5 4]]print(cx.concat([values, values], axis=0).shape) # (4, 3)print(cx.stack([values, values], axis=1).shape) # (2, 2, 3)parts = values.split([1], axis=1)print([part.shape for part in parts]) # [(2, 1), (2, 2)]
mask = values > 3print(mask.dtype) # boolprint(values[mask].numpy()) # [4 5 6]print(cx.where(mask, values, 0).numpy())# [[0 0 0]# [4 5 6]]print(mask.any(axis=1).numpy()) # [False True]
# Leading batch dimensions broadcast; the last two axes form each matrix.batch = cx.ones((2, 1, 3, 4))weights = cx.ones((1, 5, 4, 6))print((batch @ weights).shape) # (2, 5, 3, 6)print((cx.tensor([1., 2., 3.]) @ cx.tensor([4., 5., 6.])).numpy()) # 32.0
scores = cx.tensor([[1., 4., 9.], [16., 25., 36.]])print(scores.sqrt().numpy()) # [[1. 2. 3.], [4. 5. 6.]]print(scores.log().shape) # (2, 3)print(cx.abs(-scores).min().numpy()) # 1.0print(scores.argmax(axis=1).numpy()) # [2 2]print(scores.clip(4, 25).numpy()) # [[4. 4. 9.], [16. 25. 25.]]top_values, top_indices = scores.topk(2)print(top_values.numpy()) # [[9. 4.], [36. 25.]]print(top_indices.numpy()) # [[2 1], [2 1]]To move the example to an available accelerator, give both constructors the
same device="metal" or device="cuda". Check
backend support first.
A small inference pipeline
Section titled “A small inference pipeline”The repository includes a deterministic float32 example in
examples/cuda_mlp.py.
It runs matmul → LayerNorm → GELU → matmul → softmax with generated inputs and
weights. No model download is required.
Start on CPU from the repository root:
uv run --no-sync python examples/cuda_mlp.py --device cpu --repeats 3After installing a CUDA-enabled build on a supported host:
uv run --no-sync python examples/cuda_mlp.py --device cuda --repeats 3The script checks CPU parity and probability row sums, then prints a JSON
summary including shape: [32, 10], cpu_parity: true, and a local latency
measurement. A missing CUDA device is an error, not a silent CPU fallback.
This is an inference mechanics example with untrained weights. It does not train a model or demonstrate prediction quality. Its short timing is a smoke check, not a general performance claim; see measurement guidance.
Embedding and causal attention
Section titled “Embedding and causal attention”This example combines the inference primitives without downloading weights.
Save it as inference.py, then run uv run --no-sync python inference.py.
Use --device metal or --device cuda with an appropriate native build. It
compares the selected backend with CPU and reports shape [2, 3, 8].
"""Deterministic inference mechanics with CPU parity, without model downloads."""import argparseimport json
import numpy as npimport tensorcx as cx
def run(device): rng = np.random.default_rng(42) indices = cx.tensor([[0, 2, 1], [1, 3, 0]], dtype=cx.int32, device=device) table = cx.tensor(rng.normal(size=(4, 8)).astype(np.float32), device=device) weight = cx.tensor(rng.normal(size=(8, 8)).astype(np.float32), device=device) bias = cx.zeros((8,), device=device) gain = cx.ones((8,), device=device) x = cx.embedding(indices, table) x = cx.linear(x, weight, bias) x = cx.attention(x, x, x, is_causal=True) x = cx.layernorm(x, axis=-1, weight=gain, bias=bias) return cx.linear(x, weight)
if __name__ == '__main__': parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--device', default='cpu', choices=['cpu', 'metal', 'cuda']) args = parser.parse_args() actual, reference = run(args.device), run('cpu') np.testing.assert_allclose(actual.numpy(), reference.numpy(), rtol=2e-4, atol=2e-5) print(json.dumps({'device': args.device, 'shape': actual.shape, 'cpu_parity': True}))Weights are deterministic random values; this demonstrates execution and ownership, not model quality or performance. Attention uses a synchronous score-buffer implementation. See inference semantics.
Inspect and reuse storage
Section titled “Inspect and reuse storage”This example uses native host profiling on CPU or CUDA and enables the bounded
memory pool only on CUDA. Save it as runtime_tools.py and run
uv run --no-sync python runtime_tools.py for CPU, or add --device cuda with
a supported CUDA build. An unavailable selected device is an error.
"""Inspect CPU/CUDA host work and opt into bounded CUDA allocation reuse."""import argparsefrom contextlib import nullcontextimport json
import numpy as npimport tensorcx as cx
def run(device): x = cx.tensor([1.0, 2.0, 3.0, 4.0], device=device) scope = cx.memory_pool(device) if device == "cuda" else nullcontext(None) with scope as pool: with cx.profile() as profile: result = (x + x) * x scratch = cx.empty(x.shape, dtype=cx.float32, device=device) assert scratch.shape == x.shape # Initial contents are unspecified. del scratch # The host copy and CPU comparison are outside the profiling scope. np.testing.assert_array_equal(result.numpy(), [2.0, 8.0, 18.0, 32.0]) del result before_close = pool.stats() if pool is not None else None report = profile.report() assert report["complete"] return { "device": device, "cpu_parity": True, "profile_complete": report["complete"], "native_regions": profile.summary(), "pool_before_close": before_close, "pool_after_close": pool.stats() if pool is not None else None, }
if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--device", choices=["cpu", "cuda"], default="cpu") print(json.dumps(run(parser.parse_args().device), indent=2))The JSON contains cpu_parity: true, native host-region counts/times and, for
CUDA, pool snapshots before and after exit. Its scratch empty tensor is checked
only for metadata and never read. Exit frees idle blocks; this example deletes
its participating results first, so checked-out bytes also reach zero. CPU uses
no pool. Metal host profiling and pools are not implemented.
This is a functional example, not a latency benchmark. Profiled calls include instrumentation overhead. See profiling and memory reuse for timing boundaries, DLPack exclusions and cache limits.