StackOverflow on steroids

Like many others, I started with ChatGPT web usage, somewhere in 2023 with questions like “How do I create 3d Line plot with plotly?”

That helped me a lot - I am not good in interactive demos and web-stuff even though I constantly use them for the work. Yet, I could not call this transformative or improving my productivity — all those things I still could solve on my own and not that much slower. Yes, my demos would be uglier, yes, more time for debugging, but nothing dramatic.

Demos, boilerplate and constant fixing

The first “enablement” feeling soon after - I needed to make iPhone capture app for some demo.

ChatGPT not only created some code for me — that I could do via copy-pasting from tutorials, but found the problem with tutorial I was following: my XCode version was different, and the menu items have changed their location. The process was still painful, but at least I had a good helper.

2024 was kind of the same — occasional questions, debugging, maybe code optimization. All good, but I had to manually copy/paste, fix mistakes, etc. It was still kind of small extension, which still felt good. Nothing to write home about.

Then Cursor came. You don’t need to copy-paste anything anymore, although highlighting context for the question/task was super important. The brightest experience was the work on our ICCV 2025 paper about wireframe metrics.

The paper idea was to ask modelers which roof wireframes look closer to the ground truth and then run bunch of metrics to figure out, which of them are aligned with human judgement the most. This required to implement many metrics from scratch, which was done with help of Claude. This was an enormous work, but unfortunately, there were issues.

First, the Claude-2024 written tests were mostly smoke or useless. Second, there were mistakes in implementation, uncaught by the tests. Finally, the implementation was prohibitively slow for the quick iteration.

We have used AI to solve it - in a very precise direct fashion. First, the numerical, hand-derived unit-tests. Second, fix/optimize every function separately, focused. It faster than doing it by hand, but required a lot of focus. Each attempt to let the AI alone to “fix the problem” was a failure.

Next thing, which delivered a different experience, was Collada DAE parsing optimization for my custom 3D format convertor. Collada DAE is 3D asset format –  a simple XML, easily openable by Blender and Apple Preview (until the last MacOS update, at least). There is also an official pycollada python library to parse it.

But the parsing it is not that fast. The good thing — pycollada is open source library, we can optimize it and make a PR into upstream. That’s the first time agents in autonomous loop were working well. There is also no problem with unit-tests — pycollada already provides them, as well a reference implementation.

The profiling and optimization turned out harder than I expected - the model and agents relied on its prior knowledge of the library and its weak spots, so tried to optimize the same things over and over again. It also over-used profiler and under-used simple tools like timeit. In the end of the day, the following solution worked out well: I did the global profiling, and made decisions about which hot spot to optimize, and the model focused on the narrow optimization sprints. We delivered 20% speed-up for typical 3D model DAE files parsing and ~50% speed-up for low-poly ones. PRs: 1, 2

What I have forgot to mention, that so far all those experience came with the sense of accomplishment and satisfaction. I did high level planning and decision making, low-level coding, and used AI to help with unknown parts for me, like implementing things in new languages, doing parsing work (I HATE this part of working with data), etc.

The handling parsing was probably the most enabling thing so far for me. I wouldn’t dare to write a format convertor on my own - not because I am not capable of - but because I would be making tons of mistakes and most importantly bored to death.

Sinister Superpowers

The next step was Claude Code + superpowers — kudos to Manlio for introducing me to them. With help of them, the development of my supporting libraries and models was faster than I ever could do on my own and more robust — the number of tests were enormous.

Claude Code + superpowers helped me to write some custom software from scratch – think about the vibesac I did for ICCV talk, but 10x more complex. That was enormous project, which also required a lot of trial and error, mostly in prompting and formulating what to explore. While Opus was clever enough to implement cool stuff, speed-up directions it explored often were pretty wrong. What helped, is to ask it to impersonate John Carmack, and that was a breakthrough in speed. I also used ChatGPT to balance out Claude occasional blindness.

I also started to give it images as debugging tool - like “see that image, it should have this thing on the left, not on the right”.

That was next enabling moment - now I could tackle hard technical tasks, not just boring to death tasks. Thanks to the said software from above, right now my flagship model is better, than two specialized sota models. It was also time, I started to pay for Claude for personal usage — for the Claude Code — in additional to ChatGPT. I started to implement one research idea with help of it.

The development went well, so did my productivity. My mental health - not so much.

The idea of “iterate on the spec, then hand-off to the agents” never worked to me — long sessions usually meant wasted time and tokens. I had to baby-sit ideas and experiments, answering new questions - on the 10-20 min basis. That is long enough to be distracted, but not long enough to do something else meaningfully. I tried to work on something in parallel - even more exhausting. I played bullet check in the waiting time - I lost 250 rating points.

While the code and feature grew, I didn’t have any sense of accomplishment — only emptiness. I was not able to keep up with arXiv papers - my brain was completely drained. Morning sessions to read were fine — but only when I started my day with the papers, not code. Doing code by hand was not an option either - I am really not good at low level numba/C code. Distractions like working on CVPR S23DR Challenge were weird. On the one hand, they helped with sense of accomplishment and some work done manually. Being able to deliver something in a short term, while main work things were cooking, also helped. But on the other hand, that meant EVEN MORE context switches, more tasks to keep in mind.

Remember that research idea on the personal side? I dropped it, tired of baby-sitting.

I was counting on vacation to breathe.

Light again?

Vacation helped, but was that for a long? Fortunately, something else happened as well — Fable was un-banned by US government. That is the model I was using almost exclusively last several weeks, and that was a game changer.

First and foremost it was able to work on their own. If previous models were like dogs now, the Fable was like dogs 300 years ago:

To give you few examples:

• while the my model was training using specific branch of one python library, I forgot about that and started to work on another feature in the same dir. Fable caught the problem with training, understood the cause, and MADE ANOTHER LIBRARY CLONE TO SUPPORT THE TRAINING. All those unprompted.

• when I asked it to release the next model version to the internal repo with benchmark and compare to previous best, it figured out the dataset on its own, checked if there is any overlap between test and training set for all models, adapted the evaluation scripts for all models including some recent CVPR stuff (absolutely different from my model in output format), made nice table and uploaded.

• during a brainstorm, it recognized a licensing constraint — we needed to inference on data we couldn’t train on — found 27 open-license alternative sources, ran the filtering, and assembled a training dataset.

Basically, that was the first time, I could genuinely focus on the high-level task, delve into details myself only when it is really needed. I started to read papers again. I am paying for Claude Max for personal use and we are doing kornia audit and overhaul right now.

I started to scope more impactful and ambitious work than any time before, and genuinely excited about it. I still feel mental tiredness after the day of work, but now it is the kind of “I was playing chess whole day”, not “I was watching reels whole day” kind of tiredness.

Finally, I feel that AI is a force multiplier, not “small addition”, or “eating my brain”.

To achieve that, besides the Fable itself required to change how I am thinking about the task, how much context I give, and which context I do NOT give to not bias the model first. For example, previous experiments results – models trust them too much, if the info comes from the user. If it comes from the journal log from itself, model can spot problems with them and propose new, cleaner, hypothesis. On the other hand, having practical computer vision knowledge still helps to judge ideas better. And - more importantly - to quickly understand when something is going wrong.

To trust the model, and to learn where I cannot trust it. The model is great at designing inspection tools and metrics – better, than I am. Images, on the other hand, are still the weak point. Recently Fable told me that new algorithm helps, judging from the images and metric. It was not - it was making everything worse. Nor can you trust it without full review. Kornia audit - a small PR of documenting the conventions (and another one ) - took seven review waves from 5.6 Sol and another Fable 5 instance until the convergence – all of that with green CI throughout.

Also, the new worry came - I am probably over-spending my token limit. Should I try to cut? Mindfully select the model? I tried to work the same high-level way with Opus 5 – it was terrible. Opus 5 twice hit 1M context limit and had to compact the conversation – without finishing the work. Fable did the same task with 500K tokens or so. Going back to the Sonnet-on-Superpowers time? Hell, no. So my hope is to prove that the impact worth the cost. But that is a good problem to have.

The daylight is bright again, and I may not only work, but also plan, think and dream. Will see how it goes.

Thanks to Olya, Anastasiia and Ian for the feedback on first drafts of the post.

The task

On the day of my course I have delivered some (quick) talk on the deep learning, and we proceeded to the computer class. The students got the link to the Google Colab with fastai working example of training image classification model. No coding is required, but they are free to do it if they want, e.g. to explore various CNN/ViT architectures etc. The task is the following:

You have to train a two-way image classification model.

Class 1 is “Beer” – hardly can be more Czech than this.

Class 2 is “Trdelnik” or “Chimney cake” as someone translates this into English – tourist-trap pastry, not a traditional at all, but sold everywhere in Prague. I actually like it - what can be wrong about dough + sugar, right?

Your job is to produce the model, which is as good as possible. You may write a google search scrapper, look for the ready datasets, draw images yourself, generate synthetic data – I do not care. You can use ImageNet-pretrained models, or anything else.

When you think, you are done, you call me. I came to your desk, download the test set from my super secret website, and we benchmark it together (but nobody else can see the images).

The leadearboard with the best score is written on the physical whiteboard in the class. The leaderboard is initalized with two entries: 0% – best result so far.

The person, whos model performs the best, wins.

What is the catch?

The next hour or so, people were constructing their datasets. Most advanced ones even created their validation sets. When the accuracy on the validation set reached 98%, someone got enough confidence and called me.

We did a benchmark and the result was 28.5% accuracy. The class giggled a bit. Next 20 minutes nobody could cross this line. The random chance would do better than any of the models. The finally new breakthrough came - 42%. People started to realise the task was not as easy as they thought.

Next hour people finally got 57%. The class was giggling and laughing with every trial – the number on the leaderboard was unchanging. Finally someone used CLIP model and got astounishing 71%.

The catch is the data

What do you usually imagine, when someone says “beer”? Maybe a nice glass of pilsner or IPA. Or maybe a bottle. Rarely a can. What do you imagine, when you hear trdelnik? Probably nothing, unless you live in Prague or visit the city. Then you see that special fire cage, where the pastry is done.

So, I have selected the can beer, the beer near the fire-pit for barbeque, Duff from The Simpsons, and top-down view of the beer glasses near the shashlyck for the Beer class. For the Trdelnik I selected close-up view with a toppings – not popular at that time yet – similar to the beer foam, and the kiosk selling the trdelnik. Finally, to make people less sad, I have found the image, which contains both, so any answer would be correct. Here is my test set in full:

I have also create another test set with similar ideas of naturally adversarial examples – for those, who wanted to try again. It was a bit simpler, but also challenging.

Everything good comes to an end

This trick worked for couple more years – and then VLMs came. I have tested PaliGemma on my tiny dataset – and it gave me 85% accuracy out of the box. Time to invent something new.

What is (part of) the course about? Image matching

I do the lectures and labs on the image matching part of the computer vision. During the series of 5 assignments, students have to implement:

1. Gaussian blurring and patch extraction
2. (Multiscale) Harris corner detector
3. SIFT descriptor and patch dominant orientation estimation
4. SNN matching and homography RANSAC
5. Assemble altogether into image matching pipeline.

Students have got the lectures with a theory, course wiki with task descriptions and assignment templates with API of the functions they have to implement. Most of the assignments are evaluated automatically by unit-tests and benchmarks on university server.

In the end, result should look like this (the widgets are provided with templates).

How to test LLMs for this?

Obviously, there are many possible ways of doing it. I have selected the following: I copy-paste 3 verbatims in a single message. The structure is :

• Please, implement {ASSIGNMENT-DESCRIPTION}
• {ALL-ASSIGNMENT-RELATED-PAGE}
• Code template: {TEMPLATE.PY}

This would be (almost) fair to an LLM, as it would have the same information, as the student, except the images. It also would imitate a prompt from a reasonable, but lazy student.

An real (shortened) example would be:

Please, implement imagefiltering.py - file with the following methods implemented:
gaussian1d, gaussian_deriv1d - functions for computing Gaussian function and its first derivative.

import numpy as np
import math
import torch
import torch.nn.functional as F
import typing


def get_gausskernel_size(sigma, force_odd = True):
    ksize = 2 * math.ceil(sigma * 3.0) + 1
    if ksize % 2  == 0 and force_odd:
        ksize +=1
    return int(ksize)


def gaussian1d(x: torch.Tensor, sigma: float) -> torch.Tensor: 
    '''Function that computes values of a (1D) Gaussian with zero mean and variance sigma^2'''
    out =  torch.zeros(x.shape)
    return out


def gaussian_deriv1d(x: torch.Tensor, sigma: float) -> torch.Tensor:  
    '''Function that computes values of a (1D) Gaussian derivative'''
    out =  torch.zeros(x.shape)
    return out

Evaluation

During the course of the semester, I have prompted the 4o, o3mini, o3 mini-hight, and o1 OpenAI models the same way, saved the .py file and uploaded it to the automatic evaluation system. When the current assignment required the functions from the previous weeks, I imported the ground truth solution – assuming that the students could perfect them via multiple attempts (we allow up to ~50 attempts to reupload homework).

Here are the results (I added GPT5 later):

Overall, LLMs did better, than an average student on their first attempt. If prompted with scores and errors, they were pretty good at fixing their mistakes. This makes me wonder, why people, who obviously used LLM for their assignment still struggle to get high scores. And those, who get the highest scores are mostly NOT using LLMs, given the style and the iterations required to get to the answer.

Funny enough, the tasks, which students usually struggle with, gave LLMs hard times as well.

For example, almost everyone missed the following sentence about strict maximum in the nms2d function description.

 (i.e. take to consideration all 8 points in the 3×3 neigborhood and output non-zero only if the center is strictly larger than the neighborhood)

Two of the tasks are actually benchmark based – patch orientation datection, and SIFT patch descriptor. Here are the results:

Originally I wanted to evaluate all CV tasks from our course – including retrieval, CNN training, tracking. However, dirung the semester, OpenAI has been updating and changing their models a couple of times, and I lost the motivation to redo the work.

Conclusion?

I believe, modern LLMs – even if in “single attempt” mode could easily ace university coding assignemnt. Even more so in “agentic” mode. Relying on ban for LLM usage is futile. Instead we should ask questions like - how do we teach people to do software engineering, and especially deep understanding of the algorithms in the ChatGPT era. The emphasis here on the understanding and the final result - good engineer, and not on the stupid things like “how can we prevent students from using LLMs”. We cannot.

What is troubling to me, is when student could not get good results even with ChatGPT help.

How and where to look?

1. Coarse-to-fine or Funnel. I check 100 paper titles + abstracts, I skim through maybe 10, I read one or two papers carefully .

2. “Features” I looks for: topic, theory/practice, figures quality, field crowdness, potential impact.

3. arXiv-sanity as a main provider and the other sources.

Questions to ask yourself

I will try to show what I am looking for and also a formulate for myself what is going on inside of my “wet neural network” when staring on arXiv-sanity page. I ask myself the following questions, and if the answer is “yes” on some of them, the paper is going to be downloaded. By “read” in this section I mean “skim through abstract, conclusions, tables and figures to decide”.

• My areas? Yes -> read.

WxBS (image matching), 3d reconstruction, image retrieval, metric learning, RANSAC, CNN initialization.

• Opens a new (sub-)area of research? Yes -> read.

E.g. first papers on GAN, NERF, Transformers, Lottery ticket hypothesis. It does not matter if the area is not relevant for me, e.g. image-to-image translation. New area papers are always worth to read.

• From the crowded area? Yes -> skip.

How many papers on the topic are published every day? E.g., in 2014-2015 I have followed the research on semantic segmentation, object detection and GANs. Now I mostly skip all the papers related to GAN, segmenation and so on, because the improvements become incremental AND that areas are not mine. Ofc, from time to time I read some paper on GAN, but only if it comes by other channel - e.g. recommended to me by a colleague.

Two previous points can be seen as an idf – paper score normalization by average number of the papers on the topic.

• Dataset or large-scale benchmark paper? Yes -> read, regargless of the topic.

Why? It is useful to see how people gather data, clean the data, come up with a metrics and so on.

• Simple baseline? Yes -> read, regadless of the area.

• About understanding some aspect of machine learning? Yes -> read if have time

E.g. padding, double descent, over-parametrization.

• Theory paper? Yes -> skip. Unless it touches very important topic for me.

• Relevant for me as a user? Yes -> read if have time.

E.g. new non-linearity, optimizer, etc.

• Am I a reviewer of that paper? If yes -> bad luck.

I have to really read this paper several times regardless of anything.

In addition, I use some kind of “paper gestalt” – does it looks as high quality work? Isn’t the title over-keyworded and so on – this kind of things are hard to verbalize.

Use arXiv-sanity, not arXiv feed

First, given the over-whelming popularity of this site, sometimes it is down. That is good - if it is down, then I do not check papers today. Nothing bad would happen is I skip paper reading today.

Second, it shows first 8 pages thumbnails and the abstracts. This helps to make a more informed decision on whether to download the paper or not.