Prometheus

Your figure of a being and an environment reminds me of Markov Blankets, which relate a set of internal and external states as conditionally independent from each other. In this framework, i think the distinctions between intelligence, consciousness and self-awareness are on a continuum and so not qualitatively different - unless there is some kind of 'phase transition' when markov blankets are embedded in one another to a sufficient extent. (The free energy principle from which this model is derived draws heavily from physics so might be of interest to you).

I guess it's easier to take the law into your own hands and destroy things than it is collecting evidence and constructing a well-reasoned case against something.

You might be interested in the work of Michael Levin who, as i understand it, talks about layers of computation in organisms - organelles performing computations which in concert with other organelles perform computations at a cellular level and similarly up through tissues, organs, individuals and societies.

I'd also add cellular automata (maybe rule 30) to invoke the idea that we can have simple and precisely known rules which generate unpredictable iterations in order to communicate an intuition as to why at a high level we don't know how deep learning architectures produce their outputs.

It does seem domain specific. In medicine, blinded trials have been assessed by panels of physicians to give comparable or better answers to medical questions than physicians. I couldn't find equivalent papers for maths but there are papers assessing it in isolation. It makes sense that LLMs would struggle more with maths than medicine as the former is more abstract and less talked about while medicine is more embodied in our language and a more common topic of conversation. If you can find a copy of Galactica you might find it more useful as its training included LaTeX equations, and it's also designed to give intermediary steps in its workings.

There's an additional step, called tokenisation, in which words and punctuation are broken down in 'tokens', which can be roughly thought of as components of words. I imagine this makes it even less like how humans learn language.

With models such as Chat-GPT there is another potential source of bias (and for mitigating bias). People have already mentioned that the model is selecting from a distribution of possible tokens. That is what the base GPT model does, but this process is then steered via reinforcement learning with human feedback. There are several ways to do this but essentially a human is shown various outputs of the model, and then they score or rank the outputs ,which go back into the objective function. The reinforcement during this process could steer the model either towards bias or away from it. This is one argument proponents of open source models use: we don't know, and may never know, the reinforcement regimen of Chat-GPT. Open source communities are more transparent regarding what guidelines were used and who performed RL. Makes me consider what we mean by learning. I wouldn't have considered this example learning, because the model weights have not updated due to this interaction. What has happened is that the model is using the context of the previous answers it has give. Essentially asking the model to generate the most likely outputs, given that the inputs already include wrong attempted answers. The default is 2048 tokens, with a current (and rapidly increasing) max of 4096. I would put this in the domain of prompt engineering rather than learning, as it's up to the human to steer the model to the right answer. But maybe it is a type of learning?

The foundation models of chat-GPT aren't trying to be factual. A common use of chat-GPT is for science fiction writers - they will at times want accurate science and maths and at other times want speculative, or simply 'wrong', science and maths in service of a story. Which you want will determine what you consider a 'right' or good answer. Prompt engineering is the skill of giving inputs to a model such you get the type of answers you want, i.e. learning to steer the model. A badly driven car still works. Or wait for the above mentioned Wolfram Alpha API which will probably make steering towards factually correct maths easier. BTW, question for the thread, are we talking about chat-GPT specifically, LLMs or just potential AGI in general? - they all seem to get conflated at different points of the thread.

More attention should be spent on Galactica which is specifically trained on scientific literature. Even though it is a smaller model trained on a smaller corpus, that data (i.e. scientific literature) is much higher quality, which results in much improved outputs for scientific ends. They also incorporated a 'working memory token' to help the model work through intermediate steps to its outputs - i.e. showing your working. Would love to use this for literature reviews, there's just way too much in most domains for any human to get through.

Has anyone tried repeating the same question multiple times? If chatGPT works in a similar manner to GPT3 it's sampling from a distribution of possible tokens (not quite letters/punctuation) at every token. There's also a parameter, T, which allows the model to preferentially sample from the tails to give less likely answers.

No, but our aspirations should be as big as the universe. The LHC costs roughly $4.5 billion a year. The global GDP is $85 trillion/year. The LHC represents 0.00005% of humanities annual wealth, or 0.0003% of the EU's annual GDP. A small price to pay to push at the borders of our ignorance.

Computational neuroscience would cover most foundations.

If he was accessing other 'processes' then he was not dealing with Lamda. If he has been giving information out about Google's inner workings I'm not surprised he had to leave, I'm sure he violated many agreements he made when signing up with them. But given what he believed about the AI, he did the right thing. I don't know anything more about him than that.

It's not an analogous situation for (at least) 2 reasons. Someone without any senses other than auditory are still not only 'trained' on words, as words only form part of our auditory experience. Nor does Lambda have any auditory inputs, including words. The text is fed into the model as tokens (not quite the same as words, but close). The human brain/body is a system known, in the most intimate sense, to produce consciousness. Hence, we are readily willing to extend the notion of consciousness to other people, notwithstanding edge cases such as brain-stem death. I suspect a human brought up truly only on a single sensory type would not develop far past birth (remembering the 5 senses model was put forward by Aristotle and far under-estimates the true number).