News

Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

Amazon SageMaker Inference now offers prefix-aware routing for requests that share the same prompt beginning. It sends those requests to the same instance so they can reuse the KV cache. AWS benchmarks on Llama 3.1 70B showed up to 77% lower median time to the first token, with cache hit rates rising from roughly 25% to over 80%.

Shared caches come with an escape route for busy instances

Testing used Llama 3.1 70B Instruct on 7 ml.p5.48xlarge instances with vLLM prefix caching enabled. Throughput increased by up to 16%. All 16 test configurations completed with a 100% success rate. These covered single-model and inference-component endpoints, using native Invoke and OpenAI-compatible APIs. Overload protection redirects requests to a less busy instance when the configured concurrency limit is reached. When instances are added or removed, most requests keep their previous instance assignment. (AWS Machine Learning)

thinkidiot take: Up to 77% lower median time to the first token is the result I would put first. I would test this routing strategy on requests with shared prompt beginnings, where cache reuse is the stated benefit. The overload redirect also gives busy instances a release valve. For that workload, I would prioritize testing request placement before adding instances.

3 ways to prep for your next big race with Search

Google's September 10, 2026 article describes three ways to prepare for races using Search. Canvas in Search's AI Mode can build personalized training schedules. Users who connect a YouTube Music account can also ask AI Mode to create custom running playlists.

Gear searches can start with a firm spending limit

Training schedules can include local routes. They can also incorporate cross-training and strength-building strategies. Google also points to registration alerts for race preparation. Gear requests can specify constraints, such as lightweight hydration vests under $80. Google's Shopping Graph contains over 60 billion product listings. It supports tailored recommendations, side-by-side comparisons of in-stock options, and checks on local availability. (Google)

thinkidiot take: Search supports a concrete request for lightweight hydration vests under $80. I would start there and compare in-stock options with local availability. That gives me a defined budget and specific products to assess. For my race preparation, this is a more useful starting point than asking for a whole training schedule.

Jensen Huang explains why Nvidia will grow an astounding 70% next year

On September 10, 2026, Jensen Huang reiterated a forecast of potential 70% year-over-year revenue growth for Nvidia next year. Analysts expect about $400 billion in revenue for its current fiscal year. TechCrunch calculates that growth at that rate would bring next year's revenue to around $680 billion.

Huang points to sales of complete systems

Huang said a system combining 36 Grace CPUs and 72 Blackwell GPUs was seeing 27% month-to-month sales growth. He also described an NVLink-connected system costing $8.5 million. That system contains 2 million parts. Huang said Nvidia supports models from Anthropic, OpenAI, and Google. Its support also extends to open-weight models. He insisted that Nvidia's deals are not circular. (TechCrunch)

thinkidiot take: Huang's reported 27% month-to-month sales growth for a Grace and Blackwell system is the figure I would examine first. I would give that system-level sales claim more weight than the annual revenue forecast when assessing his case. His separate description of an $8.5 million system puts a concrete price on the hardware being discussed. The sales figure is his strongest evidence here, but I would not treat it as proof of the full-year forecast.

OpenAI puts Pro subscriptions on hold due to Astra demand

OpenAI temporarily paused new subscriptions to its $200-per-month Pro plan because Astra demand was straining its infrastructure. Product leader Thibault Sottiaux said Pro puts the most strain on the company's systems. OpenAI is adding capacity while sign-ups are paused. The API and Go and Plus plans remain available.

No reopening date as Astra's rollout continues

Astra launched on September 3. It has been rolling out across Pro, Plus, Enterprise, and Business accounts. OpenAI described it as promising advances in reasoning, coding, and computer use. The article says the company raised usage limits for Codex users the previous month. OpenAI did not disclose how long Pro sign-ups would stay disabled. It also did not disclose how many people were signing up daily. (TechCrunch)

thinkidiot take: New users cannot currently subscribe to OpenAI's $200-per-month Pro plan. For work I wanted to start now, I would assess the available API and Plus options before planning around Pro access. With no disclosed reopening date, a new Pro subscription has no usable start date for that plan. I would make availability the first selection criterion here, ahead of Astra's promised advances.

Trending AI Papers

Ranking source: Hugging Face Papers for 2026-09-11.

NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

NCP-ArchPreview explores whether a language model can learn more efficiently by looking beyond the next small piece of text. It learns to anticipate larger chunks as well as individual tokens, the units used to build its output. Those larger predictions then help it choose what to write next. The aim is to improve learning while keeping the familiar process of generating text one token at a time.

• Problem: Standard next-token training asks a model to predict the next unit of text. That task does not explicitly require it to predict concepts covering several tokens. This work adds that broader target to test whether it improves learning and later performance.
• New idea: The model builds a vocabulary of concepts, discrete internal representations that cover multiple tokens. It constructs that vocabulary from hidden states, the internal values it computes while processing text. A dedicated component learns to predict upcoming concepts and passes those predictions back to the part that generates tokens. Both prediction tasks are learned together during training.
• Simple example: Think of writing a sentence while also deciding what the next paragraph should cover. The paragraph idea helps guide each word, even though you still write the words one at a time.
• Evidence: The 8.9B-parameter model was trained on 5.73T tokens. It matched OLMo-3-7B's final pretraining loss, a measure of prediction error, after consuming 51.3% of the total training tokens. After full training, it exceeded that model's downstream macro-average by 2.45 points, including a 5.99-point gain on GSM8K. It also approached the training loss of a parameter-matched 8.9B baseline using 85% of the standard computation.
• Limitation: The headline comparison is against OLMo-3-7B, a smaller model. For the parameter-matched baseline, the abstract reports training loss but gives no downstream task comparison.
• Why it matters: Learning to predict larger chunks of meaning could help language models get more out of their training.
• Paper: NCP-ArchPreview Technical Report: Moving towards Latent

SenseNova-U1.5: Towards Native Unified Visual Intelligence

SenseNova-U1.5 brings image understanding and image creation into the same model. It is designed to interpret visual requests, make pictures and edit existing ones. The work focuses on difficult details such as readable text, complex layouts and keeping a subject consistent through edits. It tests whether understanding a request can help the model organize the image it produces.

• Problem: Visual creation requires more than producing an attractive picture: a model must also follow detailed instructions and preserve parts that should stay unchanged. The paper targets weaknesses in text, layout and editing consistency. The abstract does not identify a specific earlier approach as the cause of those weaknesses.
• New idea: The model handles image understanding and creation without a separate image encoder, which converts images into internal representations, or a variational autoencoder, which compresses and reconstructs them. Its training includes rebuilding image patches, small sections of a picture, while keeping their spatial relationships consistent. Specialized experts, versions trained for particular visual tasks, focus on appearance, text in two languages, infographics and editing. Their abilities are combined through distillation, a process that trains a model using behavior produced by those experts.
• Simple example: Think of revising an illustrated poster: change the wording and rearrange the layout while keeping the featured person recognizable and leaving untouched areas intact. That brings together the understanding, planning and editing abilities this work targets.
• Evidence: The abstract reports improvements in image fidelity, text rendering, complex composition and editing guided by multiple reference images. It also reports better instruction following and preservation of subject identity, geometry and unchanged regions. Training supports native resolutions up to 4K. No numerical evaluation scores or improvement margins are provided.
• Limitation: The abstract gives no benchmark scores or named comparison models, so it does not establish the size of the reported gains.
• Why it matters: Combining image understanding with creation could help a model carry detailed requests through to the finished picture.
• Paper: SenseNova-U1.5: Towards Native Unified Visual Intelligence

EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

EvoSafeHarness builds safeguards for AI agents that can take actions. It adapts those safeguards to the particular model and the setting where it works. The goal is to stop harmful behavior while allowing useful tasks to proceed. It changes the surrounding rules and enforcement code without retraining the model itself.

• Problem: Agents face both harmful requests and malicious instructions hidden in material they encounter. Fixed safeguards can block too much for one model while missing risks specific to another application. Protection must account for what actions mean in a setting and how earlier actions affect later ones.
• New idea: EvoSafeHarness searches for a safety harness, an external layer that checks and constrains an agent's behavior. It develops both a policy written in ordinary language and executable code that enforces the rules. The search uses observed model behavior and descriptions of the target setting while leaving the model unchanged. Adversarial review, which tries to find weaknesses, uses fresh contexts to reject rules tailored only to a particular test.
• Simple example: Think of fitting guardrails to a particular workshop. Rules must reflect the equipment and the order of operations, while still letting ordinary work proceed. A rule copied from a different workshop can obstruct useful work or overlook a local hazard.
• Evidence: On DecodingTrust-Agent, average attack success fell from 45.6% to 10.0% at a 3.3-point cost in utility, the measure of useful task performance. It achieved the best score in 14 of 15 cells. On AgentDojo, it reached 82.8% utility with a 0.0% attack success rate, twice CaMeL's utility at the same operating point. The harness also transferred unchanged to unseen AgentDyn suites. Under adaptive PAIR attacks with a refinement budget of 16, mean attack success stayed below 20%.
• Limitation: The reported tests cover four benchmark families, rather than establishing protection across all deployments. Attacks still succeeded in some evaluations, so the results do not show complete protection.
• Why it matters: Safeguards tailored to an agent's model and tasks can reduce successful attacks while preserving useful work.
• Paper: EvoSafeHarness: Evolving Model- and Domain-Specific

Trending AI Repositories

Ranking source: GitHub Trending.

alsk1992/CloddsBot

CloddsBot is a trading terminal for prediction markets, crypto and futures. It brings those trading categories into a shared interface.

• What it is: This TypeScript terminal brings prediction markets, crypto and futures into one interface. The project advertises autonomous trade execution and risk management.
• What it does: Open Source AI trading agent that operates autonomously across 1000+ markets - Polymarket, Kalshi, Binance, Hyperliquid, Solana DEXs, 5 EVM chains. Scans for edge, executes instantly, manages risk while you sleep. Agent commerce protocol for machine-to-machine payments. Self-hosted. Built on Claude.
• Who it helps: It serves traders working across prediction markets, crypto and futures. They can use a terminal that covers all three.
• Limitation: The supplied excerpt does not explain how its risk controls work.
• Repository: alsk1992/CloddsBot

diegosouzapw/OmniRoute

OmniRoute connects AI coding tools to model providers. Its focus is keeping those tools working when a provider's quota runs out.

• What it is: This TypeScript gateway connects AI coding assistants to model providers and falls back to another provider when a quota runs out. Its README shows a dashboard, and the project claims token savings.
• What it does: Never stop coding. Free MIT AI gateway: one endpoint, 352 providers (150+ free), 1200+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 550+ contributors
• Who it helps: It helps developers who use AI coding assistants. They can route requests through a shared gateway with fallback between providers.
• Limitation: The supplied description and README give different provider counts.
• Repository: diegosouzapw/OmniRoute

nashsu/llm_wiki

LLM Wiki is a personal knowledge tool built around your documents. Its focus is keeping that knowledge current as well as giving it structure.

• What it is: This TypeScript desktop application uses an LLM to build and maintain an interlinked wiki from documents.
• What it does: LLM Wiki is a cross-platform desktop application that turns your documents into an organized, interlinked knowledge base , automatically. Instead of traditional RAG (retrieve-and-answer from scratch every time), the LLM incrementally builds and maintains a persistent wiki from your sources。
• Who it helps: It helps people who want a structured reference built from their documents. They can have an LLM read those materials and keep the resulting wiki current.
• Limitation: The supplied excerpt lists PDF, Office documents, and EPUB/MOBI but cuts off before completing the format list.
• Repository: nashsu/llm_wiki