Credit use of AI models for bioinformatics: a tiny benchmark

Agentic coding models such as codex and claude code are really useful for software engineering tasks. We sadly (and unsurprisingly) seem to be exiting the era of highly subsidised costs to get us hooked. Currently on my work account I have a measly 300 credit per month maximum.

I immediately burned through 50 credits with a complex prompt to GPT-5.6 Sol with the chatbot, whoops. Now that being more careful with credits feels increasingly important, I wanted to try and benchmark credit use for some of the current models for the tasks I’ve been playing around with.

There are good overviews of general coding benchmarks comparing all the models at different reasoning levels¹, but I find it quite hard to know which of my problems are easy / hard, and require more or less reasoning. Most of the things I want to do are reengineer some existing code to try an idea about making it more efficient, add a new feature, or fix a bug (mostly in rust). Which of these are complex, moderate or everyday tasks?

Overall I’m motivated to try and improve on the ’try the cheaper model first, and if it fails use the more expensive one’ approach. This will always cost more, and evaluating failure is hard. If you can make a good guess about where to start with, this would be better. It’s a lot like requesting HPC resources.

Problem summary

I decided to benchmark on a fairly typical task for me in bioinformatics, which was to attempt to improve performance of one of our tools, ska.rust.

With large numbers of samples much of the internal information stored in a dense matrix is redundant, and memory use can grow very high in large and diverse input populations. Using a sparse representation would use less memory. Some more notes here: https://github.com/bacpop/ska.rust/issues/47.

I think it’s a good question for agents as it’s: not worth it for me to spend much time on, fairly tedious to implement, only impacts some users so I don’t want to spend too long on it, we have good tests which should remain unchanged, I have a reasonable idea of what implementations to try.

I’d tried this once before back in 2023 but got caught up in annoying fiddly changes, and gave up as it wasn’t a priority to fix.

Models tested

The current codex /model prompt gives the following descriptions:

  1. gpt-5.6-sol (current)  Latest frontier agentic coding model.
  2. gpt-5.6-terra          Balanced agentic coding model for everyday work.
  3. gpt-5.6-luna           Fast and affordable agentic coding model.
  4. gpt-5.5                Frontier model for complex coding, research, and real-world work.
  5. gpt-5.4                Strong model for everyday coding.
  6. gpt-5.4-mini           Small, fast, and cost-efficient model for simpler coding tasks.

  1. Low               Fast responses with lighter reasoning
  2. Medium (default)  Balances speed and reasoning depth for everyday tasks
  3. High              Greater reasoning depth for complex problems
  4. Extra high        Extra high reasoning depth for complex problems
  5. More reasoning…   Max consumes usage limits faster

My main reason for doing this comparison is that I still have little intuition for how these descriptions map on to the things I actually want to do.

Costs for Sol, Terra, and Luna are $5 / $30, $2.5 / $15, and $1 / $6 respectively per million input/output tokens, so 1x, 2.5x and 5x costs.

I was only able to compare Sol and Luna before running out of credits, and was only able to look at medium reasoning.

I’ve not tried Claude 4.8 opus or 5 fable due to no subscription, and apparently fable doesn’t do biology.

Approach

For all prompts, I asked the chatbot instant model (‘free’) to help me improve the prompt clarity for codex.

My initial prompt is at the end of the page.

Credit costs and narrative

5.6-luna medium

• Plan: 4 credits
• Follow-up: 2 credits
• Implementation: 11 credits

The first plan proposed an inefficient array of structs, which was then corrected after I used a second prompt.

The implementation only made one object sparse, but forgot that the objective was the memory whole program so was a failure. This was, at least, obliquely noted in the output.

The model’s suggestion was to include drift prevention in the plan (acceptable and unacceptable shortcuts):

One suggestion beyond the prompt itself: for architecture-heavy work, it’s often useful to require the model to state a small set of non-negotiable invariants before it starts coding. In your case, one of those would have been:

No dense per-split-k-mer representation may exist after a SkaDict has been incorporated into the merge pipeline.

To correct this with a follow-up plan and implementation:

• Follow-up plan: 2 credits
• Write plans: 5 credits
• Implement plan: 50 credits

(side note: saving plans is not free or automated. Agent can do it, but no automatic write feature as yet. Plan mode cannot write)

This then produced a plausible implementation where the tests passed without changes.

• Planning: 8 credits
• Logging: 5 credits
• Implementation: 61 credits

Total: 74 credits (about 30 minutes of run time)

5.6-sol medium

• Plan: 16 credits
• Plan follow-up: 6 credits
• Implementation: 153 credits

Total: 175 credits (about 20 minutes of run time)

The first plan was obviously better and more detailed. It correctly rejected the per variant RoaringBitmap luna first suggested due to overheads (i.e. rules out array of structs in favour of struct of arrays), but did not consider an overall RoaringBitmap which I suggested in a short follow-up.

Without specifically being in the prompt, the model plan correctly notes that making only the final object sparse after dense construction fails the primary objective.

This then one-shotted the implementation (which was different to Luna’s), but did not run the memory benchmark as it promised. (Although I think I’m still a lot better, or at least cheaper, when running these benchmarks myself anyway).

General lessons

• Luna medium is not good enough for this kind of task, even with guidance.
• Sol medium worked, but used half my credits in one afternoon.
• Refining prompts with the instant model was helpful and saved me time, but I wasn’t able to measure the effect on credit use and outcome.
• Having tests I trusted (and the model was able to pass untouched) made this a lot quicker.
• Getting to the conclusion of “this won’t work”, see below, was still at least 10x faster than my previous manual attempt (and consequently >10x less upsetting).
• This experiment cost about $20 of credits.

I still don’t have a good general idea of which model for which task, but did learn about likely costs, and what kind of effect using a lower model produces. To guess:

• This is probably at the higher end of complexity of what I ask these models to do.
• New implementations are of similar complexity, but run for a lot longer.
• Bug fixes might be a bit easier (when given clear tests)
• Scripting and plotting should be a lot easier

For future (when I have more credits)

I really wanted to compare implementing the Sol plan with Luna, and vice-versa. The Sol plan already avoided the issue of Luna’s first attempt, so if Luna could have implemented it that might have been possible in about 70 credits (and probably still a saving if this required a higher reasoning level).

I should also compare Luna on high/xhigh and Sol on low, as these have more similar costs and performance in the general benchmarks.

Keeping track of credit cost across and model performance other bioinformatics tasks would be useful too.

The proof is not in the pudding

Anyway, none of this actually achieved an improvement!

On the typical use case SKA2 is designed for, a small number of related samples (Listeria monocytogenes strain, 28 samples, 3 264 141 k-mers):

(multithreading has never been efficient for this task, I benchmarked in case any of the changes improved it)

On the face of it the Sol attempt does reduce memory cost at the expense of longer processing time and more code complexity, which is what I asked it to do. Luna’s attempt seems to improves the final representation size but doesn’t help with building it.

However, on a larger benchmark across a whole species (Streptococcus pneumoniae, 616 samples, 11 855 953 k-mers):

Here only the Sol model makes an improvement to the underlying struct, but the implementation still didn’t reduce the max memory use during its construction. I was actually pretty surprised how little difference the changes made, which made me think that the underlying assumption that a sparse representation would work better in such a diverse dataset is wrong.

Oh well, I think this is a dead-end, and the use I’m optimising for is a bit ‘off-label’ anyway! I think if the code had been slower but reduced memory use for the very large collections I probably still wouldn’t have merged it.

Prompt details

Initial prompt:

Even when a split k-mer does have variation at its middle base, most samples have the same base. I would therefore like to use a sparse representation, and introduce a reference middle base (which is the most common middle base) to be stored along with the split k-mer outside of the variant matrix.

I propose using split_kmers: HashMap<IntT, u8>, to stored the split k-mer (IntT) and reference base (u8). We can then store alternate bases in a sparse structure which keeps track of their positions (e.g. expressed column major n-samples * variant index + sample-idx) and the bases. The support functions for MergeSkaArray e.g. filter, distance weed etc will need to be updated to support this new structure.

As MergeSkaArray is built from MergeSkaDict, the underlying changes need to either be made there (to split_kmers: HashMap<IntT, Vec<u8>>,), or we simply remove the intermediate MergeSkaDict and put all the building and merge functions into MergeSkaArray which needs to support append, merge, extend and build from SkaDict (the input for a single sample).

One complication is that the merge/append used to support parallel construction needs to drop the individual SkaDict from memory during construction to have a meaningful drop in memory use, and as such a global variant index won’t be possible to calculate upfront. So these positions may need a constant offset adding on merge/append. Also, different subsets of the data may have chosen a different ref base, which would then need to be resolved if a matching split k-mer is being merged (and positions updated). We could use roaring::RoaringBitmap (automatically memory efficient) to store non-ref positions with a Vec for the middle bases, or we could use a HashMap (fast key access) or BTreeMap (supports range queries, automatically sorted) which store positions as keys and have non-ref bases as the values – please evaluate which is best.

In your plan, consider both complexity of updating MergeSkaArray support functions, and efficiency of building, in making choices. Fast implementation and reduced memory use should be prioritised over careful runtime error checking."