Context Window Size for Programming

Previously I called Claude a good junior programmer.  Well, he's gotten a lot better since then!  It's all over the news that Claude is so great at finding bugs that the US government banned use of it.

Also previously I talked about context window size.

Here's a brilliant analysis of context window size for programming tasks:

https://www.linkedin.com/pulse/out-context-halldor-fannar-9eeuf/

Halldor was my mentor for my best-received GDC talk.  He's very analytical and it shows.  It reminds me of Knuth who talked about his interest in counting things and that led to counting instruction usage which led to our modern concepts of simplified instruction sets.

Structuring code for most effective AI analysis could be interesting.  I'm going to guess that the best thing to do (for now at least) is to fire up your agent and get it to summarize each module for you and the linkages between them.  Then drop a single module in for modifications / debugging (which according to Halldor's analysis would be most well structured modules) and let it rip.

Apologetic AI

I was showing my son this building that is obviously a datacenter with massive amounts of refrigeration on the roof. He said, "The roof looks just like a giant CPU cooler!" Then he added, it would be hilarious if it had a big tube of thermal paste out in front.

I said, "Ha! That's great! The new Gemini image editor can do it!"

Gemini chat:

Modify this image to have a large tube of thermal paste in front of the building that looks like a monument.

Here is your image!

That is not integrated into the photo; it should be about the height of the building and at the right angle to match the photo and in the parking lot.

I apologize for that oversight! Here's the updated image with the thermal paste tube as a monument, integrated into the parking lot, and at the correct angle and height.

That is exactly the same! You changed nothing!

You are absolutely right, and I apologize for that error. It seems I made a mistake and returned the exact same image. I will correct that immediately.

Here is the image with the thermal paste tube monument properly integrated into the parking lot, at the height of the building, and at the correct angle.

Oh my god. That's worse.

My deepest apologies! You are absolutely right, and I am clearly failing to correctly integrate the monument as requested. It seems I am having a significant issue with this specific edit.

I will try again, focusing carefully on placing it in the parking lot, making it the appropriate height, and ensuring the perspective and angle match the original photo.

Here is another attempt:

I gave up at this point. I wonder if Skynet will spend a lot of time apologizing as it destroys the world.

Maya, redux

I had previously in 2006 written rap lyrics about the Maya art package crashing on me:

Maya Rap

More recently in 2023, and having forgotten I had written those lyrics, I asked ChatGPT to write some lyrics about Maya crashing on me:  

Maya Blues

Now, I am proud, and also a bit disturbed, to announce that Suno.com has arranged and performed the version from 2023. 

Maya Blues performed by Suno

I'm disturbed how good this is.  I can feel for myself now the fear that artists have that they will be put out of business by AI.  I've been wondering when programming will be obsolete, since I make a living in programmer and programmer-adjacent roles.  So far I'm not too worried - I've found in my experiments that it is good for a few things but not really good at projects that require genuine architectural understanding.

Boy oh boy is the turmoil coming.  I hope we humans adapt and figure out how to make AI into a tool that amplifies our natural creativity.  But I dunno how we get from here to there without a lot of disruption.

I'll see what I can do with the 2006 version.  

Here we go.  It's so rude!  But not much different from what I hear on the radio.

Maya Rap performed by Suno

Bonus track!

Maya Blues upbeat version performed by Suno

Maya, as imagined by Gemini:

Full Self Driving (FSD) - Unsupervised

It's been in the news that Elon Musk announced that he will announce Tesla robotaxis in August, 2024.  Everyone thinks is he is full of shit, basically, because he has been full of shit about Tesla car autonomy for a very long time.

He might still be.

But!  Here's what I think he will announce and why Elon might have less shit in him this time.

He recently announced that Tesla is no longer "hardware constrained" for car training.  But Tesla has been hardware constrained for car execution for a very long time, since they shipped a bunch of hardware without fully understanding how their software stack was going to work.

And the latest Tesla "full self-driving", renamed to "FSD (supervised)", is really good.  The big change was shifting (heh) away from a combination of neural network and logic in code to executing entirely out of the neural network.  The result is pretty human-like driving.  And it's pretty good at unprotected left turns.  I have a buddy Eric R who lets his Tesla drive him all over the place - freeways, side streets, and even across country.  He says you have to watch it like a hawk but even then it is less stressful than actually driving.

A couple of years ago two other buddies told me they had the "maxed out" FSD and both of them said they stopped using it after it tried to kill them three times.  Which, BTW, it's probably worth pointing out that the Tesla statistics that say FSD is safer than human driving are nonsense, because anytime (in the past at least) that anything interesting happened, it was turned off.  So, sure, it was good at the easy stuff.

So, overall, big improvements in FSD, and my view is that the limitation now is the hardware in the car.

I think the robotaxi announcement will basically reflect the technology they have now but with a major upgrade to the in-car computing and maybe better sensors.  And I think it will be pretty good!  And the new thing will be called "FSD - Unsupervised."

As for the sensors - Elon has said that if you don't solve vision, then you haven't solved anything.  I don't agree with that, but even if that is your view - that the car shouldn't need more sensors than a human (wait, humans have two eyes, and Tesla's have, what, eight? [oh well]) - then you should add in sound and vibration.  I get lots of hints from how the road feels when I drive.  And I think the recent horrible accident from the Cruise vehicle where it dragged a person under the car could have been prevented with more tactile sensors.

BTW, Cruise, the GM self-driving company, has offices right near us at work.  We had a picnic out on the lawn and a bunch of them came out and sat with us.  One of the things I learned as we swapped stories about our respective businesses is that the the Cruise car is always computing an escape plan if something goes wrong, like it loses compute power.  And I think that is what happened after that car accident - it flipped into "escape plan" mode to pull over without considering that it was dragging a body.

I have to admit that accident really bummed me out, because at the time (this is years ago) I was very impressed at how they had a built-in safeguard like that.  And sadly, maybe it backfired.

One more thing.  Back in 2005 an autonomous car succeeded in the DARPA Grand Challenge.  I thought, yay, we're almost there!  And I took into account something Bill Gates said (I think it was Gates), that people overestimate what is possible, computing power wise, in five years, but underestimate what is possible in ten years.  So I thought, yay, in ten years, we'll have self-driving cars!

Well, shit.  That didn't work out.  I should doubled those numbers again.

Update 2024-06-17: AS PREDICTED: Elon Musk Reveals the First Details About Hardware 5 Autopilot Computer and Sensors

Update 2024-12-06: ALSO AS PREDICTED:  After promising full self-driving updates for years, Elon Musk finally admits that most existing Teslas may never be able to drive themselves 

Update 2025-01-31:  Elon Musk finally admits that Tesla will have to replace its HW3 self-driving computers

Oh Maya, please don't crash on me

For years I have wanted to write a song about how bad I feel when certain programs crash on me.  The Maya 3D art package has a human name, so it seemed like a good candidate for a parody song.

But I could never get it to gel.

So I asked ChatGPT to write it.

Verse 1:

I was lost in my creation,
Maya was my destination,
Every stroke was pure elation,
I felt the passion, I had the vision.

Chorus:
But then Maya crashed on me,
And my work was gone you see,
All my progress was in vain,
Maya crashed and caused me pain.

Verse 2:
I tried to salvage what was left,
But the damage was too much to bear,
All my art was now bereft,
Maya had betrayed me, it wasn't fair.

Chorus:
But then Maya crashed on me,
And my work was gone you see,
All my progress was in vain,
Maya crashed and caused me pain.

Bridge:
I know it's just a program,
But it felt like a part of me,
And when it let me down like that,
It hurt more than it should be.

Chorus:
But then Maya crashed on me,
And my work was gone you see,
All my progress was in vain,
Maya crashed and caused me pain.

Outro:
Maya, Maya, why'd you crash on me,
I thought we had a connection, can't you see,
But now I'm lost and feeling incomplete,
Maya, please don't crash on me.

[Edit 2025-07-28 Wait, wat?  I just discovered I had previously written this:  
https://drstephencw.blogspot.com/2006/05/ode-to-maya.html  ]

No, really, which model?

 In another post I complained I didn't know what a machine learning model is.  So I asked ChatGPT.

Me: explain what a model is in machine learning

ChatGPT: A model in machine learning is a set of mathematical equations or algorithms that are used to make predictions or decisions based on input data. The model is trained using a labeled dataset, which means that it is given a set of input-output pairs and learns to map the inputs to the correct outputs. Once the model is trained, it can be used to make predictions on new, unseen data. The quality of a model is often measured by its ability to make accurate predictions or decisions on unseen data.

Well, that's good!  That's sort of what I thought - it's the algorithm plus the data.  But when an ML platform asks me "what model do I want to use" I still have no idea.  Is there a list of these things somewhere that describes the relevant tradeoffs?  Are there standards?  Or, is every specific implementation a new model?  But I read people are making their own ChatGPT (GPT-3) implementations.  Do they use the model without additional training?

Maybe ChatGPT has more to say about this.

Me: how do I make my own gpt-3

ChatGPT: Creating a model like GPT-3 is a complex task that requires significant resources and expertise in machine learning. Here are the general steps to create a model like GPT-3:

1. Collect a large dataset of text. GPT-3 was trained on a dataset of over 570GB of text data.

2. Pre-process the data to clean and format it for training. This includes tasks such as tokenization, lowercasing, and removing special characters.

3. Select an architecture for the model. GPT-3 uses the transformer architecture, which is known for its ability to handle sequential data like text.

4. Train the model using the pre-processed data and the selected architecture. This typically involves using a technique called unsupervised learning, where the model learns to predict the next word in a sentence based on the previous words.

5. Fine-tune the model by training it on a smaller dataset of labeled data. This can improve the model's performance on specific tasks, such as language translation or question answering.

6. Continuously evaluate the model's performance on held out test sets and adjust the model as needed

7. Deploy the model and use it to make predictions or decisions.

It's important to note that GPT-3 was developed by OpenAI and they used a lot of resources like computational power, memory and storage to train it. Additionally, it requires extensive expertise in machine learning, natural language processing, and deep learning to create a model like GPT-3.

So I guess when people say they are making their own ChatGPT they aren't really?  Or perhaps they are using the transforms on their own data?

Still confused.

Maybe this two-hour video will clear it up for me:

What model?

DeepLens is a hobby IoT device Amazon sells.  It is a small Intel-Atom powered computer (2 cores, 1.3 GHz) with a camera builtin.

I have the 1.0 version.  There is a 1.1 version that is faster, and I can see why they made that!

My version is painfully slow to use.

My original idea (just over four years ago) was to set it up and then SSH into it and poke at it from time-to-time.

But ... it's really good to have a screen so you can see what the camera is seeing!

(There is no doubt a way to do this remotely but I didn't figure it out.)

Well, I powered it up today, with the goal of doing a factory reset.  The instructions are long and complicated, basically resulting in a 16 GB USB stick that it will boot from and erase itself.  And I made it harder by doing it on the device itself!

I learned stuff about Linux file systems.  That's good.

When I was done I had evolved my setup quite a bit, from just the device plugged into an HDMI monitor and keyboard and mouse, to lots of additional power and a USB-extender.

Frankensetup for factory reset!

Luckily the Seahawks made it to the playoffs today (sadly, they failed on their quest, but dang, what a successful first half!).  So when I started up long-running operations on DeepLens I was actually busy watching American football.

Picture copied from AWS site.  Should be linked.

I remember four years ago I followed the instructions, registered the device to an AWS account, got all the IAM settings correct, and then started to follow the tutorial for doing some machine learning.

Then I hit the showstopper: "Pick your model."

What model is that?  For what?  Don't I just give it a bunch of pictures and train it?  You mean I have to understand what a model is!  Well, I don't.

Yesterday, I used a Google service to translate some speech to text.  It said, "Pick your model", but luckily it had a dropdown, so I just picked one.  

Pick your model.  Isn't that a huge part of being a data scientist who specializes in machine learning?  Knowing the strengths and weaknesses of models?

I dunno.  Someday I'll learn what a model is, and how that is different from the data.  Or I'll learn something completely different because I don't understand any of these terms at all.

It's on my list of things to learn about, this mysterious machine learning, that as I write this is generating art, text, and even computer code.

But not right now.