My voice

The purpose of this blog is to practice writing.  I like to write, just not enough to be a writer for a living.  That's a tough gig!  An earlier purpose of the blog was to vent mostly about the economy, but I'm generally over that now.

I mention this because AI has come to this blog - not to write, as that doesn't help me practice writing, but for maintenance.  I should probably also use it as an editor but nah.

Down below are the main stats after two weeks of co-editing with Grok Bot.  This was also my first experience "multi-botting" if that's a term.  I had one bot scanning for blog errors, which we would then discuss how to fix.  I had another bot that rebuilt my Picasa slide shows (check 'slideshow' in the word cloud).  Many of the references here go to my homepage and assorted links at https://above-the-garage.com, so I fired up a bot to clean that up.  It found errors I didn't even know I had!  These bots had to work together to clean things up.  I had another bot that added some features to 'spwnn' which required working with the homepage bot.

I bring all this up because I want to assure you the point of the blog is still for me to practice writing and developing my own voice.  No AI slop here!  Unless it is maintenance related.  If I post something from an AI I'll call it out (like Maya, Don't Crash on Me).  

I suspect if I did a calendar showing how often I blog it would be an accurate representation of how unbusy I am at work, meaning I have time to write.  Except now.  I'm super busy at work!  But this AI stuff is kind of addicting.  The power that flows through my fingers!  Holy moly.  I have other bots going - one to rebuild my personal Raspberry Pi mini-datacenter.  One to unsubscribe from tons of cruft on Twitter/X (1300 follows!  That's what impulse following gets you.). There are a few more I can't remember right now.  I'm using Claude, Muse and Grok.  

The rate of change in AI is off the charts.  It was about a year ago when I complained that Claude was a not-very-good junior programmer.  Now, for my own projects, I don't even look at the code!  I do instruct it, if it doesn't do it on its own, to reconcile and test a lot.

Now it's time for me to go pay my bills.  Oh, the temptation to have a bot pay my bills!  There are people that do this!  Probably a safer bet would be to turn on autopay.  But I'm resisting both.  Financial access is pretty low (high?) on the risk ladder for me.

Anyway, some stats, written by two of the bots:

Grok's blog voice, pretending to be me, and we did collaborate a lot on this:

1. I touched 1,110 of the blog's 1,391 posts, about 80%, in two weeks.
2. Old Flash video players are gone: 69 posts had them, and now none do. 27 of those videos play again in modern YouTube players.
3. Picasa albums came back: 104 of my original photos are in 10 posts again, 8 of them with a new slideshow.
4. About 825 broken links were fixed across 604 posts, and 614 of them now point to the Wayback Machine.
5. All 42 links whose domains had been taken over by spam, gambling or "domain for sale" pages are fixed.
6. 69 posts that only pointed at things long gone are unpublished. None were deleted. The uninteresting book-promo footer is also gone from 354 posts.
7. Every post now has at least one tag. 663 posts had none before.

Grok's homepage voice, talking in the first person about what the bot did:

1. Homepage: I added a link that opens a random blog post, the azurik archive under Work, and three spwnn links in the middle Other column (your reading list, the EC2 version and the Lambda version). 
2. Waterpic: I ported the old Java water-ripple applet to JavaScript, so /waterpic/ works again (see https://above-the-garage.com/waterpic/).
3. One address: www.above-the-garage.com now redirects to plain above-the-garage.com, and spwnn.above-the-garage.com redirects to /spwnn. The homepage tells Google which address is the real one, and there's a 16-page sitemap.  This required updating Let's Encrypt rules.
4. Archive: I moved 236 unlinked items (about 240 MB) to an archive, outside the web folder, with a list of what moved and a one-step undo. 
5. Logs: your site now has its own nginx log folder, rotated daily and kept for a year. The previous 10 days of logs were copied into it too.
6. Stats: I built [meaning downloaded and compiled and installed - SCW] GoAccess, an open-source log viewer, for your private viewing.

Me again:  all of these tasks are things I would not have bothered with in the past - the time to research how to do them made them not worth my time.  Now the cost in "cost/benefit" is very different plus this is all new so it's fun.

We'll see where this all is in a year!   Will the bubble have popped!?  We have to wait and see.

Bot to Bot

"B2B" used to mean "Business to business" but I think it will come to mean "Bot to bot."

https://x.ai/news/grok-voice-3 - the latest voice calling / answering service

There was already the joke where person A tells their AI agent to draft an email explaining topic X in more detail; and then person B gets the email and has their bot summarize it, throwing away the detail.

Now, your agentic voice calling service can call the agentic voice answering service at your cable provider (for example).  The two bots can negotiate a new deal on your internet plan.

I assume the next step is that the calling bot will emit a tone that says, "I'm a bot, can we talk binary please?" and old school modem tones will fly back and forth carrying JSON packets.

Google has had a voice service to make restaurant reservations for some time; I never tried it.  I think the only agentic financial transaction I've done was to get Muse to buy some night lights for me from Walmart.  An eight dollar transaction and that was scary enough.

My agentic usage went from zero - meaning I only did copy/paste into chatbots - to having about 10 projects ongoing in parallel.  I'm not to the point where I send a bot off on a multi-hour task.  My agents complete subtasks while I'm watching and those tasks are programming or computer management related.  When I say "in parallel" I mean that they are "in-progress" and available to do more work when I think of the next task I want them to do.  They have context they remember.  A few of them have worked together - more about that later.  It's infrequent that I have my agents doing multiple tasks at the same time, but that has started for me.  Computer maintenance tasks, while potentially disastrous if they go wrong, especially in terms of security, are also the easiest to track what the bot is doing. And there is a lot of training on how to administer computers these days so they don't mess up too often.

I don't use AI for writing, except to make reports that I comment on in my own voice.  I recently had a bot spend a week of calendar time, yes, a week! looking at all 1300+ posts here for link-rot and other errors. Probably the actual execution time was a few hours spread over that week.  It's something I could never do - I've thought about exporting the blog, writing scripts to check the links, but even that would be problematic, as a lot of link-rot is links that get redirected to a homepage, rather than erroring.  I'd have to look at each and every link that 'works' to see if it really works.  My agent did that for me.  I told it to go slow so as not to trigger any rate limiting - that's one reason it took a long time.  But eventually over 1,100 blog posts were touched in some way - sometimes trivial updates (removing old footers that didn't make sense anymore) and others where features from Picasa, long dead, had to be recoded from scratch (e.g., slide shows).

So much work I would never have done on my own.

I am nowhere near the state of the art in agentic usage; I know people with multiple $200/month AI subscriptions who run big programming tasks overnight.  And some businesses are spending $5000 a month on agent credits per employee.

I think I sound here like an advocate for AI agent usage; but I think my real point is how fast I went from no usage to 10 personal projects - about six weeks.  All that stuff about worrying about hallucinations is much reduced because the better bots check their work.

They can still screw up - check out this typo.

So it's prudent to start slowly and backup everything in advance.  And ask the bot to check their work!  In fact, ask them how they are going to check their work!  Ask them what their plan is before they do it!  Ask them to test every assumption!  The best bots do this automatically but on the free plan you're probably not getting the best bots, so ask them!

The speed that AI adoption is occurring for certain technical tasks is stunning.  AI adoption has grown faster than the internet!

You personally might be anti-AI (because of IP theft, datacenter electrical and water usage, or you're just anti-change) but the odds are high you're already using a bots without knowing it.  I wouldn't say I am pro-AI because that would endorse way too much!  But I'm definitely pro-learn-about-AI.

Here's 35 minutes about personal (not business or coding) agents.

Music - The Stars and Stripes Forever - SCW Arrangement

(Repost from 2023 for the semiquincentennial (250th birthday) of the United States.) 

I was quite bored in high school so I had a lot of time on my hands.

I saw a man named Rod Miller at Disneyland who had taught himself to play the piano by slowing down piano rolls and working out the fingering and practicing 4-6 hours a day for two years.  (Yes, two years!  I have an album he made after just two years and he was good.)

I thought that sounded fun so believe it or not I bought a player piano ($450.00 for a 1910 Aeolian Grand Upright - paid 1/2 by my Mom and 1/2 from my paper route money) and taught myself to play. 

In honor of the Fourth of July, I present The Stars and Stripes Forever, which I learned by watching Rod play (watching his fingers upside down and backwards, as I looked over the top of the piano), and by listening to player piano rolls, and by practicing several hours a day for two years.

At the end of two years, Rod had a whole portfolio of pieces, but I had maybe three: Stars and Stripes, Maple Leaf Rag, and The Alley Cat Song.  Still, it was good enough for me to perform at the high school talent show.

I must say, this piece is hard.  And I didn't even try to do the piccolo part!  And I can't play it anymore.  Too hard!

Also I used to have enough energy to play God Bless America too, which is tacked onto the end here.  This is my arrangement in that Rod's version was too hard for me!  But it is definitely derived from his version which he would play to finish out the day as his grand finale.

Context Window Size for Programming

Previously I called Claude a good junior programmer.  Well, he's gotten a lot better since then!  It's all over the news that Claude is so great at finding bugs that the US government banned use of it.

Also previously I talked about context window size.

Here's a brilliant analysis of context window size for programming tasks:

https://www.linkedin.com/pulse/out-context-halldor-fannar-9eeuf/

Halldor was my mentor for my best-received GDC talk.  He's very analytical and it shows.  It reminds me of Knuth who talked about his interest in counting things and that led to counting instruction usage which led to our modern concepts of simplified instruction sets.

Structuring code for most effective AI analysis could be interesting.  I'm going to guess that the best thing to do (for now at least) is to fire up your agent and get it to summarize each module for you and the linkages between them.  Then drop a single module in for modifications / debugging (which according to Halldor's analysis would be most well structured modules) and let it rip.

IV Parking Only

Intellectual Ventures (IV) used to be in this building.  They had some signage to reserve parking right at the front of the building.

The sign says "IV Guest Parking Only".  I had Google's Nano-Banana fix up the image for me.

Angel's Flight

A very short railway.  $1.50 per person as I write this (2025).

 

L.A. Live

 

L.A. Live is the area near the Convention Center.

Hotel Figueroa in Los Angeles near the Convention Center

 

Cafe Fig inside the Hotel Figueroa is delicious!

Ice skating rink with palm trees reflected

 

Yup.  An ice skating rink on a 78 (F) degree day in Los Angeles.

Ahmanson Theatre

 

Department of Water and Power (Los Angeles)

 

How cool!  The Department of Water and Power (DWP) in Los Angeles has a pond on top of their parking structure.

Trees and Sky

 

LLM Consciousness

(Image by Grok)

Good summary of LLM memory issues: 

From https://simple.ai/p/the-ai-memory-problem

The next real breakthrough probably isn't going to come from cleverer prompts or better instructions. It's going to come from solving AI's memory problem.

• Why AI systems fundamentally have no memory

• The current workaround: using context as a memory substitute

• How RAG (Retrieval-Augmented Generation) scales this approach

• Why real AI memory is the next frontier

A Google paper on this topic!

https://arxiv.org/html/2507.16003v1

And a Google AI summary of it!

Based on the research paper you're viewing, here is an explanation of the concept of "weight updates":

In the context of this paper, a weight update is an implicit modification of a neural network's parameters that occurs at inference time (when the model is being used), rather than during training. The authors argue that this is the mechanism behind in-context learning (ICL) in Large Language Models (LLMs).   

Here are the key ideas:

Implicit vs. Explicit Updates: Traditionally, a model's weights are updated explicitly during training using an optimization algorithm like gradient descent. This paper proposes that when an LLM is given a prompt with examples (a context), it performs an implicit weight update on the fly, without undergoing further training.   

The Role of Contextual Blocks: The paper introduces the idea of a "contextual block," which generalizes the standard transformer block. This block consists of a contextual layer (like self-attention) followed by a neural network (like an MLP). The contextual layer processes the input prompt and transforms it into a low-rank weight update that is applied to the MLP layer.   

Learning Dynamics: When the context is a sequence of tokens, these weight updates happen iteratively as each token is processed. The paper shows that this sequence of updates resembles the dynamics of online gradient descent, where each token effectively acts as a data point that refines the model's weights. As more of the context is processed, these updates become smaller and eventually converge.   

In essence, the paper posits that in-context learning is not just about retrieving information learned during pre-training. Instead, the model implicitly fine-tunes its own MLP weights based on the examples provided in the prompt, allowing it to adapt and learn new patterns at inference time.   

FSD Progress

This video is amazing!

I'm annoyed that Tesla got away with promising FSD for so long without delivering.

But it seems to work pretty well!  My family and I were lucky enough to be in Rome last year and yes, oh my goodness, it is chaotic.

Waymo has just started testing in Seattle so that will be fun to try.  I think I'll have my first ride on a sunny day though!  I wonder how well Tesla does in the rain and fog and darkness?

Apologetic AI

I was showing my son this building that is obviously a datacenter with massive amounts of refrigeration on the roof. He said, "The roof looks just like a giant CPU cooler!" Then he added, it would be hilarious if it had a big tube of thermal paste out in front.

I said, "Ha! That's great! The new Gemini image editor can do it!"

Gemini chat:

Modify this image to have a large tube of thermal paste in front of the building that looks like a monument.

Here is your image!

That is not integrated into the photo; it should be about the height of the building and at the right angle to match the photo and in the parking lot.

I apologize for that oversight! Here's the updated image with the thermal paste tube as a monument, integrated into the parking lot, and at the correct angle and height.

That is exactly the same! You changed nothing!

You are absolutely right, and I apologize for that error. It seems I made a mistake and returned the exact same image. I will correct that immediately.

Here is the image with the thermal paste tube monument properly integrated into the parking lot, at the height of the building, and at the correct angle.

Oh my god. That's worse.

My deepest apologies! You are absolutely right, and I am clearly failing to correctly integrate the monument as requested. It seems I am having a significant issue with this specific edit.

I will try again, focusing carefully on placing it in the parking lot, making it the appropriate height, and ensuring the perspective and angle match the original photo.

Here is another attempt:

I gave up at this point. I wonder if Skynet will spend a lot of time apologizing as it destroys the world.

Concrete

 

Financial District, San Francisco

Maya, redux

I had previously in 2006 written rap lyrics about the Maya art package crashing on me:

Maya Rap

More recently in 2023, and having forgotten I had written those lyrics, I asked ChatGPT to write some lyrics about Maya crashing on me:  

Maya Blues

Now, I am proud, and also a bit disturbed, to announce that Suno.com has arranged and performed the version from 2023. 

Maya Blues performed by Suno

I'm disturbed how good this is.  I can feel for myself now the fear that artists have that they will be put out of business by AI.  I've been wondering when programming will be obsolete, since I make a living in programmer and programmer-adjacent roles.  So far I'm not too worried - I've found in my experiments that it is good for a few things but not really good at projects that require genuine architectural understanding.

Boy oh boy is the turmoil coming.  I hope we humans adapt and figure out how to make AI into a tool that amplifies our natural creativity.  But I dunno how we get from here to there without a lot of disruption.

I'll see what I can do with the 2006 version.  

Here we go.  It's so rude!  But not much different from what I hear on the radio.

Maya Rap performed by Suno

Bonus track!

Maya Blues upbeat version performed by Suno

Maya, as imagined by Gemini:

Anti-War

I've been trying to make an anti-war (really more like anti-military-industrial complex) photo using AI.  I finally have one I like!

https://x.com/DrStephenCW/status/1936194275622240549

 

Consciousness part 5

Consciousness Part 5

Assume we're living in a simulation, as Scott Adams and Elon Musk believe. If you're authoring our simulation, you probably don't care much about where individual memories are stored. We can assume you're either running the simulation to learn something or for entertainment (or both), and you only care about results. As Elon Musk says from his simulation viewpoint: "Most entertaining outcome is most likely" (source).

In fact, it might be an optimization to kill off actors entirely, preserving only their external impact as remembered by other actors in the simulation. This would free up enormous amounts of memory and remove useless detail.

However, as an actor in this simulation, I do care deeply about my memories—they constitute my identity. I've been programmed to want to preserve that identity. Sure, there are some things I'd rather forget, but generally I value my sense of continuity. It's what makes me, me.

(This relates to the "Ship of Theseus" question: if a boat gradually has all its parts replaced, is it still the same boat? Yes, it is! This is called the "Continuity View," and while other philosophical views exist, they're wrong—at least for this discussion.)

The Missing Element: Save Files

As a game developer who has worked in offline and online worlds for 35 years, I'm familiar with what it means to run a simulation. One key factor that Scott Adams and Elon Musk seem to have overlooked is the importance of the save file. When people discuss living in a simulation (including in books like Neal Stephenson's "Fall; or, Dodge in Hell"), they typically assume the simulation works like physics—one event leads naturally to the next.

But that need not be the case.

Time as Illusion

Time itself might be an illusion. Our lives as actors in the simulation could be time-sliced with everyone else's. Maybe we're only simulating ourselves 1/100th of the time, while 99/100ths of the time we're "off." (In most game simulations, the time spent active or inactive varies depending on computational load.)

You could be turned off for minutes—and perhaps experience it as daydreaming, those random thoughts unrelated to your life's purpose. This would serve as a cover story, and you wouldn't even need to be in a running state for the daydreaming to occur. When you're unpaused, a memory of daydreaming could simply be inserted to account for the missing time.

Sleep could be a much larger version of this phenomenon. But it gets worse (or better): you could be turned off for an entire day without knowing it, then have a vague memory of being sick implanted to cover the gap.

Editing Reality

While you're paused, your save file could be edited. Nothing requires the simulation of all these actors to be consistent. When consistency matters for the simulation's purpose, a "rendezvous" (in Ada) or "channel" (in Go) could synchronize state between actors. Perhaps these connections are created automatically when two actors are physically close to each other. Most of our information about nearly everything else is hearsay anyway.

The Manufactured Past

This means the past could be as much an illusion as the present. Just as some fundamentalist Christians believe the entire world was created 8,000 years ago complete with dinosaur bones embedded in the Earth, our lives as we know them might have started 10 milliseconds ago, with our personal history and sense of continuity restored from a modified save file.

So...

What was I saying?

Consciousness part 4

 

In terms of consciousness, memory is generally talked about in two ways (at least in science fiction):

1. Our memory of others; and 

2. Our own memories.

Let's talk about our memories of others.  It’s common to mention how someone lives on in their work products and in the memories of those around them.

In Westworld, Dolores says, “You live as long as the last person to remember you.”

On the other hand, Woody Allen said:

“I don't want to achieve immortality through my work; I want to achieve immortality through not dying. I don't want to live on in the hearts of my countrymen; I want to live on in my apartment.”

Scott Adams, who announced he has terminal cancer, and just has a few months to live, absent some breakthrough treatment, has put some time into exploring creating an AI of himself.  He decided it might have an uncanny valley oddness to it so instead he was going to treat it like an offspring.

While very few people have the public body of work that Scott Adams has, I think a "memorial AI" produced from home movies or other references will be as common as a photograph of a loved one.

I was talking to a buddy, Jared C, about how AI doesn’t learn the way people do, because AI takes so much repetition.  I suggested that the likely future – which is already happening to a certain extent and in the two months since that conversation has grown by leaps and bounds – is that foundation models, as they are called, will be used as a basis for smaller, specialized AIs, which can learn quickly, because the foundation model has encoded the basic wiring, and the specialized AI will ‘just’ be a big context window.  (The context window is the number of tokens the model remembers from talking with you.)

Jared said, and I thought this was quite profound, that maybe a million years of evolution had created the human foundation model, and that’s why people can learn quickly.  A lot of it is pre-wired and what we call our experience is a big, self-pruning context window.

There is some evidence that LLMs based on different human languages (e.g., English, French) tend to encode similar patterns.  In fact, the original "Transformer" model was for translating languages.  I believe LLMs encode knowledge because language encodes knowledge.  It trivializes the achievement to say an LLM ‘just’ predicts the next token.  Sure, but based on a 12,000 dimensional vector space of encoded and related tokens.

It's impossible for a human to visualize a 12,000 axis vector space.  In fact, after about five dimensions it gets really difficult.  And the vector space has just grown and grown.  I asked Grok, “Please make a table showing how the vector size has grown over time.”

+------+------------------+------------+---------------------------------+

| Year | Model            | Vector Size| Notes                           |

+------+------------------+------------+---------------------------------+

| 2013 | Word2Vec         |        300 | Early word embeddings           |

| 2017 | Transformer      |        512 | Base model from Vaswani et al.  |

| 2018 | BERT-Base        |        768 | Google’s BERT, base version     |

| 2019 | GPT-2            |       1024 | Est. based on model scale       |

| 2020 | GPT-3            |       2048 | Est. based on parameter growth  |

| 2022 | LLaMA            |       4096 | Est. for Meta’s efficient model |

| 2023 | GPT-4            |       8192 | Est. based on scaling trends    |

| 2024 | Grok 3           |      12288 | Est. for xAI’s advanced model   |

+------+------------------+------------+---------------------------------+

As you can see, the dimensions of the vector space have generally grown, but may have topped out in terms of a useful "sweet space", between 8,000 and 12,000, at least for most applications.

As foundation models have grown in their ability to categorize knowledge into these massive vector spaces, so too has the size of the context window.  I mentioned before that the context window is the text you put into the model - your questions and/or observations.  But the designer of the AI also puts text into the context window - rules of behavior.  The context window can come from anywhere.  For Perplexity, and other models that are adding search capabilities, it also comes from the results of a web search. 

I asked Grok, “Produce for me, please, a historical table of LLM content window sizes.”

Below is a historical table of context window sizes for notable large language models (LLMs), based on available data up to May 2025. The context window, measured in tokens, represents the maximum amount of text an LLM can process at once, influencing its ability to handle long conversations, documents, or complex tasks. The table is organized chronologically by model release or significant update, with approximate release dates where possible.

+------+------------------+------------+---------------------------------+| Year | Model            | Ctx Window | Notes                           |+------+------------------+------------+---------------------------------+| 2017 | Transformer      |        512 | Base model, Vaswani et al.      || 2018 | ELMo             |        512 | Est. for bidirectional LSTM     || 2018 | BERT-Base        |        512 | Google’s BERT, base version     || 2019 | GPT-2            |       1024 | OpenAI’s early generative model || 2020 | GPT-3            |       2048 | Initial release, ~1.5k words    || 2022 | LLaMA 2          |       4096 | Meta’s efficient model          || 2023 | GPT-4            |      32768 | 32k version, ~24k words         || 2024 | Gemini 1.5       |    1000000 | Google’s 1M token model         |+------+------------------+------------+---------------------------------+

It’s several months (six or more) since Scott Adams looked into encoding his vast library of publications (books and podcasts) into a “Memory AI” of himself.  I think it’s probably doable now and it will just get easier, because of larger vector spaces and larger context windows.

NotebookLM from Google has a context window big enough to upload my book, Nano-Plasm (free copy at https://www.above-the-garage.com/nano-plasm/Nano-Plasm_v1_1_3_2008-12-31.pdf).  I uploaded the PDF version which it handily decoded.  It summarized the book in less than a second.  Yay for large context windows!  [Update:  NotebookLM identified some typos in the book - actually a lot of them!  But each one it identified that I checked was actually fine.  I think the PDF parser is busted!  Each typo it thought it saw was a letter replaced with a space.  I'll try again later with a .txt file.  You just can't trust LLMs.]

Current LLMs can store enough information to recreate how others remember us.  Dolores in Westworld says,

"If you could see the code, the way it was written, you’d see the beauty of it. It’s elegant, simple… a single line of code can describe everything that person was. Everything they did. For you humans, that’s like… the equivalent of reading their life story in about 30 pages. Maybe less."

In the next article in this series, I will discuss our own memories.

(See "What model" blog post to see how much I knew about foundation models.)