How to Implement an LLM in Python With a Hosted API
For most projects, how to implement LLM in Python comes down to calling a hosted model over HTTP, not training or serving one yourself. With an OpenAI-compatible API it takes about ten lines, and the same code grows into streaming, JSON output and tool calling.
Updated

Key points
- Install the openai package, set a base URL and a key, and use the model ID uncensored.
- Streaming, JSON mode and tool calling are all supported and use the standard SDK arguments.
- Keep chat history in a list and watch the 100,000-token window.
- Handle HTTP 429 with a short retry; failed or refused requests are not charged.
Hosted API or your own model
There are two ways to put a language model into a Python program. You can run the weights yourself, which means a GPU, a serving stack, quantisation choices and a lot of waiting for downloads. Or you can send requests to a model someone else already runs. The first route makes sense for research, strict data isolation or very high steady volume. The second suits almost everything else: a script, a bot, a product feature, a prototype you want working by lunch.
This guide takes the second route. The service here is a hosted, OpenAI-compatible text API with one open-weight model, run by an independent provider and tuned to answer without content refusals for lawful adult use. It is not an official API of any model vendor. Because it follows the OpenAI request format, the standard openai package works, and so does plain requests if you prefer no dependencies.
Step 1: install and make the first call
Install the SDK and store your key in an environment variable, so it never lands in your source code.
pip install openai
export API_KEY="your-key-here"
Then create a client with the service's base URL. The only things you change compared with any other OpenAI-style setup are the URL, the key and the model name.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.fluxapikey.com/v1",
api_key=os.environ["API_KEY"],
)
resp = client.chat.completions.create(
model="uncensored",
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "Explain list comprehensions in two sentences."},
],
)
print(resp.choices[0].message.content)
print(resp.usage)
That is a complete working program. The endpoint behind it is POST /v1/chat/completions, authenticated with Authorization: Bearer <key>, and GET /v1/models lists the model ID. You can get a key and a small trial credit on the key page.
Step 2: stream the answer
Waiting several seconds for a full reply feels slow in a terminal or a web page. Set stream=True and print tokens as they arrive over server-sent events.
# client from step 1: base_url https://api.fluxapikey.com/v1
stream = client.chat.completions.create(
model="uncensored",
messages=[{"role": "user", "content": "Write a short story opening about a lighthouse keeper."}],
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
Usage numbers arrive at the end of the stream, so the last chunk may have no choices. That is why the loop checks chunk.choices first. One thing to remember: a streamed request keeps its concurrency slot until the stream ends, so always read it to the end or close it.
Step 3: keep a conversation
The model has no memory between calls. A chat is just a growing list of messages that you resend each time. A small class is enough:
# client from step 1: base_url https://api.fluxapikey.com/v1
class Chat:
def __init__(self, system):
self.messages = [{"role": "system", "content": system}]
def say(self, text):
self.messages.append({"role": "user", "content": text})
r = client.chat.completions.create(model="uncensored", messages=self.messages)
answer = r.choices[0].message.content
self.messages.append({"role": "assistant", "content": answer})
return answer
bot = Chat("You are a patient Python tutor.")
print(bot.say("What is a generator?"))
print(bot.say("Show me one with yield."))
The context window is 100,000 tokens, shared by the prompt and the completion. That is roomy, but a long chat grows with every turn, and you pay for the whole history on each request. When it gets large, drop the oldest turns or replace them with a short summary you ask the model to write.
Step 4: get structured output
Programs want data, not paragraphs. JSON mode makes the reply valid JSON. Say in the prompt which keys you expect.
# client from step 1: base_url https://api.fluxapikey.com/v1
import json
r = client.chat.completions.create(
model="uncensored",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": 'Return JSON like {"title": str, "tags": [str]}.'},
{"role": "user", "content": "Summarise: a tutorial on async IO in Python."},
],
)
data = json.loads(r.choices[0].message.content)
print(data["title"], data["tags"])
Valid JSON does not guarantee your keys. Check for them in code and retry once if one is missing.
Step 5: let the model call your functions
Tool calling lets the model ask your program to run a function, then continue with the result. You describe the tool, check whether the reply contains a call, run it and send the output back with the role tool.
# client from step 1: base_url https://api.fluxapikey.com/v1
import json
tools = [{
"type": "function",
"function": {
"name": "get_word_count",
"description": "Count the words in a text.",
"parameters": {
"type": "object",
"properties": {"text": {"type": "string"}},
"required": ["text"],
},
},
}]
messages = [{"role": "user", "content": "How many words are in: the quick brown fox?"}]
r = client.chat.completions.create(model="uncensored", messages=messages, tools=tools, tool_choice="auto")
msg = r.choices[0].message
if msg.tool_calls:
messages.append(msg)
for call in msg.tool_calls:
args = json.loads(call.function.arguments)
result = len(args["text"].split())
messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
final = client.chat.completions.create(model="uncensored", messages=messages, tools=tools)
print(final.choices[0].message.content)
Always validate the arguments before acting on them, because the model can produce a call you did not expect. Tool calls also stream, as tool_calls deltas, if you need that. The documentation lists every field.
Step 6: handle limits and errors
A full key allows 8 concurrent requests and 300 per minute; a trial key allows 2 and 60. Go over and the API answers HTTP 429. The error code is rate_limited for too many requests per minute and concurrency for too many at once. On temporary overload you also get a 429, with a Retry-After: 5 header. A small retry wrapper covers all three:
# client from step 1: base_url https://api.fluxapikey.com/v1
import time
from openai import RateLimitError
def ask(messages, tries=4):
for i in range(tries):
try:
return client.chat.completions.create(model="uncensored", messages=messages)
except RateLimitError:
time.sleep(5 * (i + 1))
raise RuntimeError("still limited after retries")
If you run work in threads, cap them at 8 with a ThreadPoolExecutor(max_workers=8) so you rarely hit the concurrency limit. Each response carries X-Request-Id, which is what to quote if you contact support, and X-Balance-USD shows your remaining credit. The maximum body size is 8 MB.
Settings that change the output
Most of the day-to-day tuning happens in a handful of parameters, all passed through to the model.
temperaturecontrols randomness. Use 0.2 or lower for extraction and classification, around 0.7 for chat, and 1.0 or more for brainstorming and fiction.top_ptrims the unlikely tail of word choices. Change either it or the temperature, not both at once, so you can tell which one did the work.seedmakes repeat runs more comparable. It is useful in tests where you want to see the effect of a prompt change and not the noise.stopends generation at a string you choose, which is handy for cutting off a model that keeps adding commentary.max_tokensis optional. Without it the completion may use whatever is left of the 100,000-token window. Set it when you want a hard cap on length and cost.- The frequency and presence penalties discourage repetition, which helps with long creative output that starts to loop.
Write down the settings you used next to any output you care about. Months later, "why did it answer differently" is almost always a changed parameter or a changed system message.
Running many requests at once
A loop that sends one request at a time wastes most of your allowance. With a full key you can have 8 requests in flight, so a thread pool brings a long batch down to roughly an eighth of the wall-clock time.
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI
client = OpenAI(base_url="https://api.fluxapikey.com/v1", api_key="YOUR_KEY")
def summarise(text):
r = client.chat.completions.create(
model="uncensored",
messages=[{"role": "user", "content": "Summarise in one line: " + text}],
max_tokens=60,
)
return r.choices[0].message.content
texts = ["first document", "second document", "third document"]
with ThreadPoolExecutor(max_workers=8) as pool:
for line in pool.map(summarise, texts):
print(line)
Keep the worker count at or below 8, or at 2 on a trial key. If you still see 429 responses with the code concurrency, lower the number by one or two. The 300 per minute limit matters for fast, short requests, so a batch of tiny calls can reach it before the concurrency cap does.
Common problems and fixes
Most first-day errors come from a short list of causes.
- 401 or authentication error. The key is missing, mistyped or sent to the wrong host. Print the base URL your client actually uses and make sure it ends in
/v1. - Model not found. The model ID must be
uncensored. CallGET /v1/modelsif you want to confirm what the server lists. - Empty or cut-off reply. A low
max_tokensstops the answer mid-sentence. Raise it or remove it. Checkfinish_reasonon the choice:lengthmeans the cap was hit. - JSON that will not parse. Either the reply was truncated, or the prompt never described the shape. Fix the prompt first and wrap parsing in a try block.
- Slow first token. A very long prompt takes longer to process. Trim history or send less context.
- Request too large. The body limit is 8 MB, and the token window is 100,000. Split large documents into pieces and summarise each.
When something still looks wrong, log the X-Request-Id header with the failing input. It lets support find the exact request.
What it costs
You pay $0.15 per 1M input tokens and $0.50 per 1M output tokens from prepaid credit. A typical call with 2,000 tokens in and 500 out costs 0.002 x $0.15 = $0.0003 for input plus 0.0005 x $0.50 = $0.00025 for output, so $0.00055 in total. Ten thousand such calls cost $5.50. Failed or refused requests are not charged and credit never expires. Top-ups run from $20 to $500; see pricing for the bonus tiers. Prompts are not used for training.
Output tokens cost more than input tokens, so the cheapest habit is to ask for short answers: say two sentences, or JSON only, and set max_tokens to a sensible ceiling. Read resp.usage during development to see real token counts for your prompts, and multiply by the rates above before you scale a job up. The X-Balance-USD header shows what is left after each call.
More guides
Questions and answers
Do I need a GPU to implement an LLM in Python?
Not with a hosted API. Your Python code only sends HTTP requests, so any laptop works.
Can I use the official openai package?
Yes. Set base_url to the service URL and api_key to your key, and use the model ID uncensored.
Does it support async code?
The SDK's AsyncOpenAI client works with the same base URL and key. Keep concurrency at or under 8 requests.
How do I try it without paying?
Sign up and claim the $0.50 trial credit on the key page. It is valid for 7 days and needs no card.
Your key is one form away
Create an account, copy the key, change the base URL. That is the whole setup.