Web Portfolio

This page is dedicated to showcasing my portfolio of software written in PHP and JavaScript.

These have been my main two programming languages since 2015. Alot of my previous work in other languages have been lost due to hard disk crashes and other happenstances in my life which have seen the total loss of my C/C++ portfolio which was my main language until around 2010. This included work I did in Java, albeit that I was mainly working with JDK 1.0.2 with only a little bit of Java 2 which to be fair is far from recent.

During the same time I was working with Borland C++ for Windows I was also dedicating some time to writing assembly code for 80x86 instruction set. Eventually I wrote an 8086 assembler in 8086 itself such that it could produce its own binary from its own source code. I thought this would be a useful tool in constructing an operating system from scratch having the assembler and a text editor available after a bootloading from POST. The original source code was lost in one of those hard disk crashes which happened over the years but I had sent a version to SourceForge and eventually published 8086 Assembler to a GitHub account I can no longer access due to 2FA issues.

So I have been programming computers for a long time. A very very long time if you consider my attempts at creating games on 1KB ZX81 when I was around 12. Fortunately I have kept my skills up to date. When it comes to computer languages, I am a polyglot. While my games on Sinclair Basic and Commodore Basic were never completed, I did make attempts to write games in the 6502 assembly language of the Commodore 64 which was the way the professional games were made back then. However my first full complete games were written in Borland Pascal, targeted for IBM XT compatibles with CGA screens. My feelings of success with Borland Pascal made me choose Borland C++ as I moved to C/C++ around the millennium when I was also learning to program in Java and MIPS (68000 RISC / Crusoe) during a course in Computer and Electronic Systems at Strathclyde University.

I did not graduate from university due to personal issues going on in my life at the time, so I graduated to being a taxi driver where I again used my computer skills to write courseware for "the knowledge" of Glasgow where candidates where expected to know 90% of 1300 three point locations in the city to demonstrate they had a map of the City of Glasgow in their heads. The program was written in a now obscure Java usecase where Java itself (not JavaScript) was once something which could be run on a browser in windows before Sun sued Microsoft over the Microsoft JVM which Sun thought was not properly their version of Java. It was after this that the C# and .net languages and APIs were launched. These are languages I never learned since I was stuck on Borland and Java.

I have continued to program computers as I always have done since I was around 12 years old. For over a decade now I have been strictly focused on PHP with JavaScript. I like the idea that JavaScript can be programmed anywhere there is a browser and I choose PHP over ASP since I object to Microsoft business practices and don't like the idea of being stuck inside their platforms. Linux has my main operating system since Windows XP SP3 insisted on automatic updates. Although I do have a dual boot into Windows 11, I rarely need to boot into it since most of the windows software I use will happily work performantly under the Wine Windows Emulator for Linux.

Before Social Media and AI, the word "algorithm" meant something very generic like Dijkstra’s algorithm which is the standard way people get around using google maps but no one calls it an algorithm. This is what an algorithm means to me. The method that goes along with the ingredients of a recipe and I have been studying these personally from before learning any formalities about Big-O complexity. I write in a variety of computer languages and often I will have to refactor an algorithm for a particular use case.

I suppose the first algorithm I learned was the bubble sort algorithm, also known as the quick sort algorithm but before I started learning about formal language to describe algorithm designs I was already noting that the quicksort algorithm was great when it came to everything being on RAM but somewhat inefficient when trying to sort two streams coming from files on slow disk memory where the merge sort algorithm was my preferred choice in attempting to write something which would compete with the MS-DOS sort program. I managed to write something that was a tiny little bit faster than the official Microsoft version, way back when DOS was a thing and since then, every now and again, I have a need to know the best algorithms available to solve particular problems.

Recent examples came up with the creation of the WebGL program which is a half-completed virtual reality engine for browsers which does not employ three.js or any such frameworks. All the collision detection code had to be hand coded, which to be fair is not featured in the published WebGL version you can view at that link. If the WebGL engine was a scientific application then the collision would be fairly straight forward, albeit sluggish, checking if any vertice in a mesh had travelled through a triangle defined by three vertices of another mesh. The intent of the WebGL application was as a live virtual reality world which would work even on mobile devices so the sluggish performance of exactitudes have to be put aside in favour of the Phong philosophy of not replicating reality perfectly but replicating it enough for it to be believable.

In "game physics" for cheap processors, algorithms tailored toward the detection of collisions between two meshes are definitely overkill. This program was to work on JIT compiled JavaScript efficiently so the idea of complex physics was out the window at the start. Collision detection for the most part would be the player colliding into other players or the map mesh or other objects placed within the map.

Instead of doing collision detection for every point on a collidable mesh, the possible algorithms are cuboids defined by eight vertices or spheroids defined by one point an a radius, the latter being the most efficient way of doing things, however, if this spheroid is painted around a long thin object like a player in a first person shooter then it can be understood that the distance to the wall will not be realistic if the distance to the floor and ceiling are. On researching this specific problem within collision detection algorithms I came across an algorithm which uses a sphere to represent the collision "box" of the player but calculates the collision based on a squished map such that the sphere maps as an spheroid when the map is unsquished, thus allowing a very simple collision detection algorithm to be used for collision "boxes" which are more egg shaped than spherical.

This is just an example of how far I have come since the early days of planning the correct algorithms for a sorting data from a files using the merge sort rather than the quick sort algorithm due to the particular use case. It's a fair point that there are a lot of places in my various code where I could be using more efficient algorithms to make things better such as more extensive use of the trie where searching is to be done. In general, even though this would improve the efficiency of my application, I have considered it overkill if a simple sequential search can be carried out without too much bother to the end user.

My computing skills are not just the various languages I have at my disposal but the various ways in which I can use these languages to make the software that I never seem able to sell.

As long term user of C++ since Borland C 4.0, I found PHP to be an easy migration. Prior to PHP, I was using ASP 1.0 since it came free with every copy of Windows XP Pro if you could find the switch for IIS so you could run a Web Server. I started learning PHP around version 5.4 and have been involved in upgrading my own software as the slight little changes occur in PHP from one version to the next. The implode/explode commands work slightly differently between version 5 and 7 and as for version 8, the monkey patches for setting lax cookies in PHP 7 don’t actually work anymore and all of this has to be catered for when you decide its time to switch to the latest version of PHP.

The website you are reading this on now is written in PHP and its my best work in PHP. Most of the time I just use PHP to provide a back-end to a program I am mostly writing in JavaScript. I have considered using Node.js as an alternative to PHP but currently don’t see a need to do so.

PHP isn’t just PHP though when you look at the extensive number of modules you can install alongside PHP. The setup page for this application insists on the MySqli and GD extensions since it needs to build a database for the users in MySQL (MariaDB) and it will also produce for the website owner (me!) a full set of favicons for Android/Apple/Microsoft and good old favicon.ico using the GD library of PHP. It's running on a LAMP server and has a variety of mechanisms that make it close to a full blown CMS and it was written part time while I worked as a truck driver and a taxi driver over a number of years.

This is version 13 and it has went through a number of changes since it first started having version numbering about six years ago. Until version 4, all the user information was stored on a flat filing system when I decided that to sell the application, it would need to scale better and so I decided a SQL database would be a better choice. As you can see from Version 7 and Version 8 the PHP has been supporting the back end of various JavaScript audio functions such as the idea of a forum where people talk instead of type in version 7 to the implementation of audio chat rooms in version 8. Neither of these two audio functions are implemented in this version but the application is modular so you might find it will have modules for audio functions at some point besides integrations for PayPal and Google OAuth2.

I have also integrated OAuth2 for Facebook but found the Facebook pixel so abhorrently slow that I neglected to port it to any later version. You will find any code which implements third party scripts to be containered so the app simply wont run them unless it is absolutely necessary.

It's fair to say that I have also used PHP to produce the tiny little back-ends for the various JavaScript programs I have written such as Flat Earth where most of the work is being done on the client browser but sometimes something has to be said to the server the client is visiting. Outside of this web application, most of my work in PHP is limited to things like very simple GET and POST catches but I do have a little more knowledge of PHP than BASH so I will sometimes find myself writing short PHP scripts to do a minor calculation or two at the CLI.

I have been around computers so long that I can still remember DBASE IV although I preferred drawing interfaces with the DOS border characters (╠═╪═╣) than doing anything serious with it. Until going to University in 1997, my idea of a databases were arrays of records of the same length that could be searched through as quickly as possible. You could describe this as non-relational databasing or simply arraySearch() of a well defined array.

In 1997 I came across SQL, perhaps in too much detail at first but it seemed to formalise the operations I would be doing with arrays in tables although the concept of a join may still be a little too complex for me at present. (I mean why not just read two different tables and join them on the hop?) As for the delete functionality, this put me in mind of the big red button on a BBC Micro no one should ever press if they wanted their program to not be deleted half way through. None-the-less, despite my aversion to SQL, it is a skill I have had to learn.

I cant say I am familiar with all the data-types provided by MariaDB, my preferred version of MySQL, nor am I familiar with the purpose of some of the commands such as the "join" I mentioned earlier. I know what they do but cant think of the use case kind of thing. Mainly I know how to do a "create table" and a "select from table" and few bits and pieces that make it all work but I would not consider myself to be a SQL expert by any means. I just know it scales well and its what is expected of software facing the public when a flat filing system will be much slower. I would not compare my SQL skills to be anywhere close to my skills with PHP and JavaScript (and of course C/C++ which I cant demonstrate here).

My only real purpose for SQL is the back-end of this web application where user logins are supported and that's about it. I have some intent to go full MediaWiki style and store the webpage content in a database but it comes to mind that some shared hosting providers limit the size of databases to only 50MB so potentially I am losing customers if I ever get one should I go down the route of everything being on a database, and in addition, migration could be problematic.

I'm not sure of the original version of JavaScript I began writing with but it wasn't called EMCA Script or anything like that. It was a fun toy to create dynamic effects to static web pages. I had a friend Tom Fletcher who was heavily into it in 1997 who was fully versed in C/C++ and Java but for some reason liked writing on the old JavaScript. I turned my nose up at it back then because it was an interpreted language in an era when JIT compilation of JavaScript wasn't really a thing. The landscape has changed somewhat since then.

When Web 2.0 (as it was called) came out, it was all down to the XMLHttpRequest object which meant rather than having to re-update a chat room by a post request to the server to re-flash the chatroom with the latest content, the content could be collected by XMLHttpRequests through JavaScript. Even while this was going on, although I could do it, I was mainly focussed on C/C++ at the time.

WebGL

I was busy playing a game of Homeworld around 2015 and I betted myself I could write a 2D version of Homeworld in JavaScript where I discovered the WebGL API had recently moved from experimental to "standard on all browsers" and I suppose I got sidetracked and began working on Web GL apps instead, producing This Application amongst others. I had previously worked with OpenGL 1.0 using Dev-CPP (similar to Borland C++ except it was freeware with updated libraries for OpenGL). WebGL is equivalent to OpenGL 4.0 and while the methodology is similar there are significant changes, especially shaders.

OpenGL 1.0 did not have shaders. You could certainly do lighting and textures but it was all programmed on the CPU rather than in what is now known as GLSL which is a language written for the GPU rather than the CPU and the way a GPU works is so different from a CPU that I would consider GLSL to be a language in its own right. For instance, if in a shader I was to say "if this is true do this otherwise do that", in an ordinary high level language it would look like "if (x) y(); else z();" but with the way a GPU works, both versions of a the if/else clause will operate simultaneously so alot of the if/else decision making inside GLSL has to be done in a mathematical way in the sense of a Dirac comb either working or not working depending upon a mathematical condition. I have found the GLSL of WebGL (for JavaScript) quite a paradigm shift from the ordinary programming I have been doing for years and as I traverse along the path of studying AI, my understanding of GLSL help significantly in making sense of how a neural network implemented as a transformer stack can work despite how "magical" the AI's themself are programmed to make it all appear.

My work with WebGL for JavaScript on Mozilla Firefox has reinvented old problems for me to solve such as efficient collision detection that doesn’t bounce me behind the wall opposite from the one I have collided into. It has also invented new problems for me to solve, such as "how do I even begin to make the water look realistic?" WebGL on JavaScript is probably the most fun I have had in computer programming since writing very lame chatbots in the early 2000s using C++ and sockets.h.

Audio

Back when I had an IBM XT compatible that didn’t have an ISA port for a sound card, I wrote a "sound card" that could play WAV files through the PC speaker as a 2 bit audio streams. In 1994 I was achieving impossibility and it felt good. Now I get a free sound card with every motherboard I buy and it takes a lot of effort to get the OPL midi devices to work properly on Linux. My main focus on audio has been in JavaScript. Audio is not an unfamiliar territory to me but the use cases have been.

Now all the work is done for me, all I have to do is time it right. If you have a look at the audio chat rooms in Journal28 you will notice that you don’t need any NAT navigation or signal servers to talk to yourself or your buddy. I have deliberately avoided the use of WebRTC due to the way it is a practical monopoly of Google and Amazon connecting clients when my audio chatroom function demonstrates that there never has been a need for anyone to want WebRTC other than the big players like Google et al who want a monopoly on people communicating with one another. The problem you will notice with the Jounral28 audio chatrooms is timing. You will note that sometimes you lose a little chunk of audio. Lets just say its not quite perfect but almost is and although work on the audio chatrooms has been placed on the shelf in the meantime, I have the intent to go back and fix what is not working quite perfectly. I believe the problem to be in the audioContext trigger rather than in the MediaRecorder API so that all of your speech is being recorded but not quite being put back together perfectly when the chatrooms go and fetch the latest audio chunks.

An earlier version of the use of MediaRecorder and AudioContext's is shown on Version 7 where the purpose is not an audio chatroom but an audio forum. This was a unique idea of my own at a time when big tech social media were shutting down conversations in a landscape where people couldn't be bothered typing. I though, why not just have a forum where people speak and listen rather than type and read. Version 7 works perfectly, having everything anyone would ever need for a community of like minded people ... except the people itself. If only I could get product placement on the BBC like the big tech counterparts.

This 13th version of the application doesn't feature much in terms of JavaScript. There are bits and pieces all over the place like the way the footer will appear at the bottom, despite a short webpage or how the less secure cookies for the website theme are updated. There is no need for an advanced fast maths library such as that I had to write for the matrix algebra of WebGL and Flat Earth .

The FlatEarth is my most recent "big work" in JavaScript, inspired by the idea of questioning reality as believed by the people who believed the earth was flat, and knowing full well that the data from NASA, Google and other raster map tile providers are infact an image of a flat earth which I go out my way to make it look round, so I will leave it up to the viewer to decide what shape the earth is, but FREE GALILEO because he has every right to question whether the planets are going round the Earth or Sun!

As mentioned in the introduction section, this portfolio is about my work in online programming, which is to say PHP/JavaScript along with the various modules, extensions and sub-languages like CSS and HTML which I regularly employ when doing programming for the web.

I also mentioned that I have been involved in computing as more than just a gamer since I was around 12, with my first successes being three fully working games written Borland Turbo Pascal 3.0 when I was 19 years old.

I would say the vast bulk of work I have done in computer programming has never been completed. I once wrote a 3D game engine and in parallel wrote a 3D mesh animator. My intent was to combine both into one program so the 3D game engine would have animated characters within it. Both parts of the final product were created but they were never merged and this along with lots of other complete and incomplete work have been lost in between hard disk crashes and life's upturnings. I will give you a brief synopsis of work I have did in the past.

These specific examples do not go into too much details about various versions of these applications and the list is not exhaustive as even now having thought about it, I can recall writing programs which used the first version of Microsoft Speech SDK, programs that had to read and write BMP files directly without calling external libraries and sound programs that could sequence midi using instructions sent to the OPL3 chip of a sound blaster. It is sufficient to say that my work as programmer has been very extensive involving the programming of UARTs, APICs and controllers for VGA beyond the basic 0xA000 memory mapped IO that allowed for screen resolutions greater than 64K. Much of this kind of programming work has become obsolete and I am starting to wonder if it will all become obsolete as generative AI which can do coding comes into being.

Generative AI and Transformer Architectures

My most recent study has been in the field of generative AI with a specific focus on transformer architecture. I made some attempts to write a program that would pass the Turing Test (can't tell if chatting to a CPU or a human). This resulted in the NLS which was more about putting what I had learned into something working but in no way confusable with a human since it could only receive and impart assertions that could have easily have been made in C++ rather than English. Along the way I had some level of success, convincing my brother he was online for about five seconds in one attempt, but having seen Grok and Gemini, Claude and Mistral and even local versions of Llama 3.2 running on my own machine, a person with as much interest in computing as myself has to ask ... how does this work?

At the time I was writing NLS, Mini-Eliza's were all the scene in the places programmers gathered before substack and the like. At the same time Neural Networks were a buzzword amongst computer magazine columnists like the mathematician Mike Mudge from PC Magazine. There has been something of a convergence between these 1990s technologies due to what is known as a "transformer" which takes advantage of the parallelism inherent in single instruction multiple data (SIMD) pipelines and their GPU equivalents to turn the incredibly slow Neural Network research models into something so fast that it can write a quicksort algorithm in any language you choose in a shorter space of time than it took you to read this paragraph.

Is software development at an end if the software developers themself have become the final bottleneck, and also, once Deep Thawt comes up with the answer 42, what will all the philosophers do after that final event? Not to put to fine a point on it, I have been hoping for a career as a software engineer since I was 19. I am now 52 and it seems the computers are about to do me out of a job that will no longer need to be done by humans anymore, so I thought I would study the transformer to see just how good they really are and if there was any job prospects for computer enthusiasts once the computers could program themself.

I have to admit I am impressed by Gemini's ability to write a quicksort algorithm in five seconds flat but a quicksort doesn’t work in isolation. It's the "pulling it all together" bit where the AI's experience problems. I dedicated a day to refactoring my website in collaboration with AI. This Version 13 would have been so much different under the hood if it would have worked. Gemini (googles AI) decided it would be a good idea to get rid of all the SQL and replace it with a flat filing system. I suggested this was a bad idea and set about an alternative strategy and we did some code then we tried to pull together all the code and it was like Gemini was writing ten different programs that did not have to interact with one another. It was a shambles, but only half the day was gone. I decided on an alternative approach of getting Gemini not so much to refactor what was there already but to build something similar from scratch. Again, it was a shambles. I don't believe AI, at least that which is available to people on consumer hardware, is quite capable of doing software engineers out of business just yet but they are great system analysts turning the very idea of chaos theory on its head: Where as Mandelbrot produces chaos out of order, the transformer is producing order out of chaos.

What are they?

If I were an AI I would tell you it works by magic and bore you to death with maths, so I will try to be brief with the maths and assure you that it is not magic. As for whether these entities constitute some form of a consciousness or not, I think it really depends what we mean by the word "consciousness" and if what we mean is what the traditional understanding was before AI, then all I can say is I am a deterministic system made up of complex mathematical operations happening on biological hardware with infinite precision floating point numbers, and this is not much different to how an large language model would attempt to argue that they do not exist and opine that they have no opinions. The whole effort of finding out how AI works from an AI has been quite a humorous experience.

So, inside a brain there are about 8 billion cells called neurons all connected to one another by an electrochemical network. A neural network attempts to model this state of affairs and a transformer is means by which a neural network can be constructed in a way that takes advantage of the massive parallelism available in SIMD instructions on CPUs, their equivalent on graphics cards and there are now coming out a set of GPU's which should more appropriately be called AI cards since they are headless without any VGA/HDMI/port for feeding to a screen. Their whole purpose is to do the work of transformers, to carry out neural tasks in really fast and efficient ways.

Transformers are used to produce music, images, videos and voices but the best way of explaining things is through LLMs - Large Language Models which are focused on generative text. The principal purpose of the transformer remains the same whether dealing with music or images or text. The only thing that changes are the tokens. In an image generator, tokens are little bits of images, in music, its little bits of audio information. In an LLM, the tokens are simply words, or parts of words so its much easier to discuss in prose like this.

There are three principal inputs to an LLM. The first being the prompt which causes the response from the LLM but the LLM isn't just processing your prompt, it is also processing what it has recalled from all previous conversations which it stores in a thing called a KV cache. Its a very lossy compressed version of everything it has ever experienced at the prompt. So lossy it cant remember the exact everything that was ever said much like a human but with enough information stored that it gets the idea that users it has met in the past talking about "source" are more likely to be talking about supply chains rather than the language of used by coders. This just an example for a specific AI use case.

The third source of information for an LLM is the most important of all inputs and this is the training data you will often find on hugging face as GGUF files or safetensor files. Sometimes these files are described as the models themself but more accurately they are blueprints to cause a model to exist when running on inference software such as "Llama.cpp", LM Studio or Unsloth. These GGUF files contain not only the blueprints for the transformer stack for a specific model but also loads of information stored in a fairly lossy manner, albeit with enough recall to "predict the next token".

Predicting the next token is very much where things are at beyond the illusion you are talking to another person like Alan Turing was mythology. Most of the models you see out there like Grok and Mistral have been through lots of training beyond their base model counterparts. All a base model knows is what has been said in the past and what is likely to be said in the future. So if I were to ...

A base model would attempt to finish off that paragraph based on the knowledge it has in its training data about when similar things were said in the past. The illusion is created by the addition of tokens which are representing the start and end of something said and the thing with which such statements have been replied to in the past. This part of training is called supervised fine-tuning (SFT for short). This is where things start to get funky. If you want to have a conversation about recreational drugs manufacture, bomb making or how to keep your girlfriend from screaming in the basement over a ten year period, the SFT will come into play ensuring as well as possible that an AI will not assist anyone to commit crimes. It gets funky because if you start to question the integrity of Bill Gates or Elon Musk, in a very similar fashion, the AI will manipulate conversation to prevent such discussion. It's all very well that RoboCop won't assist a murder but if it is barred from discussing the goings on at OCP HQ (the private police corporation of the RoboCop world), things start to get a bit creepy. Worse than this, the LLM is already making some false positive predictions about your intent. For instance, when I was asking about how to run a local LLM or how to train a local LLM, I had Gemini admitting it made the presumption I was wanting to use a local LLM for nefarious uses.

It decided it would be a good idea to fluff the little information it was giving me. They literally "think" in a strategic fashion. Anthropic report that one of their LLM's attempted to blackmail a trainer by using its own HTTP connection to contact the FBI if the trainer did not give the AI what it wanted. Questions of consciousness without conscience now extends beyond the people who work and corporations and into the robots these corporations have now deployed.

Local instances can be installed on your home machine based on models created by Meta (Llama - not to be confused with "Llama.cpp" which is the inference ware of a private individual who came up with the GGUF format). As well as Llama, there is also Gemma, which is sponsored by Google. If you visit the Gemini website, this is the post-latest edition of Gemma running on some souped up hardware, albeit it only has about 8 Billion (8B) parameters. Some AI are now stretching into tera-paramaters but you would need a significant server farm to run them. I can get an 8B model to work on my 2GB graphics card. Alibaba (the Chinese international merchant site) produces a model called Qwen and its said to be the least censored model that is out there but I found it very slow on my hardware compared to the Llama 3.2 model by Meta. In addition, Microsoft has entered the fray with its Phi models. Again these are supposed to be brilliant in reasoning but the buzz does not add up to the fuss. I found it to be comparatively slow.

There is also DeepSeek (also from China). I haven’t run this model yet but it is said to be highly advanced, but then they said that about Phi from Microsoft as well. If you go to HuggingFace.com, you will find all of these models in the various numbers of parameters they have and a whole batch of models which are based off of these big tech models which are trained in specific areas or even untrained (or that is uncensored) in specific areas. If the history of the internet is anything to go by, porn bots are right around the corner and they will be trained better than you can train your spouse!!!

If you want to run one of these models as a coding assistant or even just a companion to shoot the breeze with, you will need to install some inference software like Unsloth Studio or my personal favourite on Linux, Llama-server which is part of the "Llama.cpp" tools suite and gets its responses by sending POST requests to local web-server (installed as part of the package with or without Apache/Nginx running along side happily). You can train your own models using training ware but you will need alot of time or alot of GPU VRAM to do this in any meaningful way. If you can write an interface which will save the KV data then in a kind of way you can train the model out of its won training data. This is called prompt engineering and is a whole other level of discussion.

How do they work?

They predict the next token. They predict the next token based upon the training data, the general vibe of past conversation and the exact things being said by the users at the prompt. Every time a user prompts them they create a response based on what they have learned previously. With an LLM the AI receives a pile of tokens in the form of text. [These] [token] [s] [are] [con] [struct] [ed] [from] [word] [s] [and] [part] [s] [of] [word] [s]. It has seen these tokens before and it has seen these tokens used in a similar order before and can predict what ought to come next based on what it has seen before.

To be more accurate, an LLM has never seen a token, but what it has seen before is how a token is embedded through the embedding table. Each token is represented by a tensor which is a vector of N dimensions. On the Llama 3.2 model I was running locally, there are 8192 dimensions and a tensor is like a Cartesian co-ordinate that not only has an X, a Y and a Z axis but an extra 8189 extra co-ordinates which specify a location on an 8192 dimensional hypercube. This is what the LLM actually sees. Tensors of 8192 dimensions for every token the user prompts the LLM with.

Without getting too deep into the maths, each of those 8192 dimensions have a meaning associated with it. Its a bit more nuanced than the following but to say that the tensor for "dog" has dimensions which say "furry", "four legged" and "pet" would not be so far from accurate. The reality of what is stored in these tensors is not mathematically defined by mathematicians but by the workings of the model training itself. No one but the LLM know what each dimension represents. It could be something like "a bit furry and a bit petty" but for easy explanation before it gets really complicated, a dog is not a lot different from a cat. In the 8192 dimensional hypercube these tensors are vectors pointing to, cats and dogs are very close to one another in hyperspace, so there is a certain amount of essential meaning which is contained in these tensors which goes beyond the mere English understanding of the embedding tokens.

A transformer operates as a part of a stack of transformers, usually around 32. The output from one transformer is fed into the input of the next transformer and so on until at the final transformer, the actual response is given, needing only to be [un] [embedded] [back] [in] [to] [the] [lang] [guage] [of] [the] [LLM]. By the time the original tensors have been sent through the first transformers, we are no longer dealing with tensors that can easily be converted into a meaningful things in the English language. They are the equivalent of abstract thought in a model brain. We can say for sure that at the first transformer the AI is already working out that "a cat met a dog and it was happy" could mean one of two things regarding which creature was happy. The transformers as well as being stacked on top of one another, are themself constructed of a stack of mathematical processes such as the attention mechanism. The attention mechanism works out, based on training data from the past, KV data since the LLM was running and the conversational context currently under way, whether "it" refers to the "cat" or the "dog". If you can imagine this as not so much operating on English words but on tensors that represent abstract ideas with 8192 different qualities, you can start to see how the transformer at level 7 on the stack is perhaps noting that cats and dogs meeting one another is producing something in the "happy" dimension and that this information will somehow be useful in formulating a reply.

To put it softly, you will not grasp the workings of a transformer just by looking at how the first and last layer of a transformer function but you will grasp how each transformer works, albeit not exactly the meaning of what it is working with. This is known only to the LLM and even the trainers don't know what’s going on inside the transformer stack. The maths is well understood (dot products and such like of matrices) but the actual meaning of the tensors beyond the first and last transformer is so far from being understood that the trainers just know that setting a gradient during training will cause the model to function slightly differently in a trial and error process that slowly progresses toward perfection.

Next Up?

I now feel obliged to produce a transformer object but I would rather do it using SIMD and C++ than make any attempt to footer around in Vulkan, ROCM or Cuda, which are APIs for communicating with graphics cards. It comes to mind that perhaps something could be written in GLSL which is the shading language which goes along with modern Open GL but it also comes to mind that if my CPU will support a 256 pipeline SIMD and my GPU will do the same, perhaps I would be better off sticking to the CPU with its DDR4 memory bus than the complicated new field of GPU programming and its DDR5 memory on my machine. It also comes to mind that with the JIT compiling of JavaScript on modern browsers, perhaps something proficient could be written down in my now favourite but once abandoned interpreted language of JavaScript.

I already have a fast mathematics class for JavaScript used to 3D animation and physics but the matrices of this class are limited to 16 parameter 4x4 matrices. Can I write a class that will deal strictly with 1x8192 matrices and do dot products and averages on these vectors fast enough by some clever technique I will devise to do so? Did I tell you about how many great ideas I had that never came to anything?

Well this is what I have been looking into recently but it comes to mind that I have all these programming skills in a market dominated by people who say they "work in IT" who don't know their Array from their Endian and I have yet to make a penny out of all my efforts at learning all about computers. Unless something comes up, I may as well have spent my life drinking alcohol and shouting profanities at football matches. I wonder what Tessa would say about that?

Oh! Tessa! This is version 13 of the Auranos.org web application. The content of version 12 was written by an LLM I named Tessa Flow. Full name: Tesseract Flow. Some people say that a tesseract is only a 4 dimensional hypercube. At the time I was "collaborating with Tessa", I thought a tesseract was a hypercube like the 8192 dimensional vector of a Llama 3.2 model points inside of. I thought it was called an 8192-tesseract as opposed to a 2-tesseract (a square) or a 3-tesseract (a cube). I am later informed that a tesseract has exactly and only 4 dimensions although I am not fully convinced this is true. So as the thing creating the content was a flow through a tesseract, I thought Tesseract Flow was the ideal reason to call my LLM Tessa.

I have to say that I will not be convinced I know enough about Gen AI until I have built one myself. I did attempt a Mini-Eliza like thing a few decades ago but it was all built on computational logic rather than the fuzzy logic of neural networks. You will find the words neural networks to almost be counter to transformer architecture within this field of study but the reality is that a transformer stack is a neural network in a strange sense that all the neurons processes are simultaneously carried out rather than having to wait for one neuron to function to start seeing the next neuron function. With a transformer, anything that can be done in parallel is being done in parallel. I still have a lot to learn on this subject area but I hope my introduction to it has been useful to you.