A developer has managed to increase the text generation speed on an Nvidia GeForce RTX 3090 graphics card by 4.6 times, running the Qwen3.8-27B language model.

With this adjustment, performance jumped from a modest 82 tokens per second to a remarkable 382 tok/s. What’s truly remarkable about this achievement is that it was accomplished solely through optimizations to the inference engine, without any other modifications to the GPU.

The main bottleneck in AI-powered text generation isn’t computing power, but memory speed. For each new token, the GPU must retrieve the model weights stored in VRAM, causing the processor to spend much of its time idle, waiting for data. To overcome this limitation, the user applied a set of techniques, most notably speculative decoding.

This strategy employs algorithms to propose or guess multiple consecutive tokens at once. Subsequently, the main model validates the entire block in a single pass. Since the main model retains the final decision, the technical accuracy of the response is not degraded. A lookup drafter was added to this structure, scanning previous text in the conversation to reuse repeated sequences, which proved particularly effective in reading and quoting long documents.

The optimizations allowed the context window to be expanded to nearly 139,000 tokens using a more compact working memory format. However, the method has some limitations, such as the initial document read taking twice as long and extended verification consuming more context memory.

Even so, for everyday chat interactions, the system maintains a speed of 133 tok/s, offering a significant improvement that extends the lifespan of a GPU like the RTX 3090, which wasn’t specifically designed for AI.