Cognitive bootstrapping and a priori knowledge

Cognitive bootstrapping and
a priori knowledge
Andrea Kulakov,
Sts Cyril and Methodius University
Skopje, Macedonia
Georgi Stojanov,
American University of Paris
Paris, France

Objectives of the work-package WP5
• To explore the notion of innate knowledge and cognitive
bootstrap within the main goal of XPERO: learning by
experimentation.
• To investigate the state of the art in computational models of
inborn knowledge and internal value systems
• overview of existing models of curiosity
• innate knowledge
• (Guidelines for the XPERO architecture)

WP5 relation to the other WPs
WP 5
Innate Knowledge and
Cognitive Bootstrap
WP 1
Stimulation of
Experiments
WP 4
Gaining Insights
and Representing
Knowledge
WP 6
The Experimental
Loop
WP 3
Observation and
Evaluation of
Experiments
WP 2
Design and Execution
of Experiments

Introduction to Curiosity
• With the advent of developmental (epigenetic) robotics we are
witnessing an increased interest in
for autonomous agents and especially in the notion of curiosity
witnessing an increased interest in motivational subsystems
• This community is particularly interested in agents that, during their
development, exhibit ever more complex behavior via the much sought
after open-ended ended learning
• Sometimes this type of learning is also referred to as task-
independent or
• Curiosity
or task non-specific
specific.
Curiosity then, in this context, is understood to be the
mechanism that would drive these systems to do
something rather than nothing
• Researchers adopting Piagetian schema constructs point out that
schemas are self-motivated
to be executed

Some implementations of curiosity in
artificial agents

Schmidhuber (1991)
• Schmidhuber (1991) introduces the notion of curiosity
in an otherwise Reinforcement Learning (RL) setup.
S
CUR
model ctrlr
M
• In his agent there are 2 recurrent neural networks
(RNN). The first one models the environment, by
implementing a predictor mechanism:
Situation1-Action
Action-Situation2
and the second one (the controller) actually controls
agent’s s behavior (i.e. chooses the next action to be
executed).

Kaplan and Oudeyer (2002)
• There are three essential “processes” that interact with each other:
motivation, prediction, , and actuation.
• Motivation process is based on three motivational variables:
• predictability (how good is the prediction process in guessing the next
S(t) given the previous (SM(t-1)),
• familiarity (how many times the robot has actually experienced that
particular transition SM(t-1) to S(t)), and
• stability (how close remains S(t) to its average value).
The reward function is such that the robot gets positive
• The reward function is such that the robot
reinforcement if it maximizes stability motivational variable,
and when it maximizes the first derivative of the predictability and
familiarity motivational variables.
• Apparently this reward policy is a variation of Schmidshuber’s principle.
Kaplan and Oudeyer also relate their motivational variables to the t
notions of
novelty and curiosity as used by (Huang and Weng, , 2002) and (Kulakov and
Stojanov, 2002).

Weng et al 2001; Huang and Weng 2002
• Working within the context of a research program called autonomous
mental development (Weng
et al., 2001; Weng, , 2002) Huang and
Weng (e.g. Huang and Weng, , 2002) have implemented a motivational
system in a physical robot called SAIL, which
in a physical robot called SAIL, which rewards the robot for
going into novel situations.
• A novel situation is defined in terms of how different are the current c
sensory inputs from the ones encountered in the past. This pushes s the
robot towards regions where its predictor
errors in guessing the next sensory input, and, as expected the robot
indeed improves its performance in environments that are
deterministic and learnable.
predictor makes biggest
• The problem arises in probabilistic and/or noisy environments
where the robot apparently behaves randomly in order to
maximize the prediction error (and the reinforcement with that).

Barto at al, , 2004; Stout et al, , 2005
• Generalize their traditional reinforcement learning approach
(e.g. Sutton and Barto, , 1998) by distinguishing between
external reinforcement (usually given by a teacher
or critic) and internal reinforcement. . The internal
reinforcement allows for intrinsically motivated
learning which would enable the agent to learn
“[The intrinsic reward system] favors the development
of broad competence rather than being directed to more
specific externally-directed goals. But these skills act as
the “building blocks” out of which an agent can form
solutions to specific problems that arise over its
lifetime.” (Barto et al. 2004)

Blank et al (2005)
• Blank et al. . in (2005) identify three essential notions for
autonomous development in robots:
• abstraction,
• anticipation, and
• self-motivation.
• The self-motivation subsystem:
“[…] indicates to the system how “comfortable” it is in the given
environment. If it is too comfortable, it becomes bored, and takes
measures to move the robot into more interesting areas.
Conversely, if the environment is chaotic, it becomes over-excited
and attempts to return to more stable and well known areas.”
• They present the initial results on a simulated agent that solves s the
navigational problem.

A spectrum of competences
• • Every organism is a mixture of both kinds of capabilities:
• pre-configured — constructed (meta-configured)
tribute to A. Sloman
• • Not all of the pre-configured capabilities are manifested at birth –
many are ‘time-bombs’ (e.g. waiting for the season to hibernate, or migrate).
• • Architectures for the more advanced species can do many things
that are not directly biologically useful, , but may provide reusable
information: including (possibly dangerous) exploration of a space of possibilities
lities.
• • Architectures can change over time.
• • Ontologies used can change over time.
• • Forms of representation used can change over time.

On Innate KnowledgeK
• Where does knowledge come form
• Nativists
• Plato
• Chomsky
• Nurturists
• Behaviorists (Brooks, …)
• Cognitive Developmentalist
• Related projects
Related projects
• The phylogenetic abilities (the set of sensori-motor circuits) are
predetermined by the chosen ontology of innate concepts.
RobotCub people decided on the following list of innate concepts:
objects, numbers, space and people.
• innate skills of the COGNIRON robots

Innate skills of the COGNIRON robots

Example: Perceiving causation
tribute to A. Sloman
• Our ability to perceive moving structures, and our
meta-level ability to think about what we perceive,
is intimately bound up with perception of causation
and affordances.
• Sometimes the causal relations are inherent in what
is seen
• Sometimes they involve invisible (hypothesised)
structures and processes

What is the right gear going to do if the left
one is turned clockwise
We do not know the functionality (the mechanism) inside the box!

How about now
This was easier to guess, but …

Why not something like this
… we assumed the rigidity of the materials from which the gears are made!

Precocial/Altricial
• • Precocial
• Some animals are born highly competent: deer, chickens, etc.
tribute to A. Sloman
• • Altricial
• Some animals are born underdeveloped and highly incompetent, but adult
forms can do things precocial species cannot, e.g. hunting mammals, nestbuilding
birds, primates, humans.
• • Even altricial species have some precocial skills or tendencies
• e.g. sucking, stimulating parents to feed, and some ‘delayed’ precocial skills,
like sexual maturation.
• • Architectures and competences may be pre-formed in precocial
species, but slightly adaptable, e.g. by reinforcement learning.
• to contrast learning a language, or learning to program computers
• • Altricial species may be using sophisticated architecture-growing
mechanisms doing far more than varying weights (etc.), when they look
incompetent.
• By collecting chunks of information about affordances provided by the
environment and by their bodies — initially these affordances are stored,
then later are recombined and used.

Biological bootstrapping mechanisms
tribute to A. Sloman
• • There are some species whose needs cannot be served by genetically
ly
determined (preconfigured) competences based on pre-designed
architectures, forms of representation, ontologies, mechanisms, and stores
of information about how to act so as to meet biological needs.
• • Evolution have ‘discovered’ that it is possible instead to provide a
powerful meta-level bootstrapping mechanism for ‘meta-configured’
species:
• a mechanism without specific information about things that exist in the
environment (apart from very general features such as that it includes spatio-temporal structures and
processes, causal connections, and opportunities to act and learn, and that the neonate has a body that is
immersed in that environment)
• with specific information about types of: things to try doing, things to observe,
things to store
• with specific information about how to combine the things done and keep
records of things perceived into ever larger and more complex reusable
structures,
• including a continually extendable ability to run simulations that can be used
for planning, predicting and reasoning.

Biological Nativism: Altricial/Precocial
tradeoffs
tribute to A. Sloman
• • Evolution ‘discovered’ that for certain species which need to adapt relatively
quickly to changing environmental pressures, a kind of learning mechanism is
possible which combines previous knowledge and allows much faster and richer
learning than is possible in systems that merely adjust probabilities on the basis of
observed evidence (statistical data).
• • The altricial species learn a great deal about the environment after birth and in
some cases are able rapidly to develop capabilities none of their r ancestors had
• like young children playing with computer games.
• • This uses an information-processing architecture which starts off with a collection
of primitive perceptual and action competences, but also with a mechanism for
extending those competences by ‘syntactic’ composition
• as a result of play and exploration, which is done for its own sake, not to meet
other biological needs (food, protection from hurt, warmth, etc.)
• • The meta-level features of the mechanism and the initial competences are
genetically determined, but the kinds of composite competences that t
are built are
largely a function of the environment.
• • This requires forms of learning that are not simply adjustments of probabilities,
but involve continual creation of new useful structures, expanding ng the ontology
used.

The developmental architecture
tribute to A. Sloman
• • There is an important sub-class of animals in which competences are not
all pre-configured, whose development makes use of:
• Genetically determined primitive actions, perceptual capabilities and
representations,
• Genetically determined play/exploration mechanisms which drive
learning that extends those actions, etc., using abilities to chunk,
recombine and store
• new more complex action fragments
• new more complex perceptual structures
• new more complex goals
• Creating new ontologies, theories, competences (cognitive and behavioural)
• i.e. new more complex thinking resources,
• • Not restricted to somatic sensorimotor ontologies.
• • Thereby extending abilities to search in a space built on larger chunks:
solving ever more complex problems quickly.
• unlike most statistical forms of learning
• • For AI systems this will require us to discover new architectures s and
learning mechanisms.

XPERO’s s first experiments
• One robot and one object
• Motivation, internal value system, experimental
stimuli: : implicit/nonexistant
• The notion of object is implicit: : the designer
chooses the set of learning attributes/features
(distance, angle…)
• Motor commands generated by the designer: tele-
operated robot

Eventually the embodied robot
• Will need motivation and internal value system:
•Why would the robot do anything
• Self-preservation
• Curiosity
• Explaining unexpected phenomenon
•What would the robot do
• Stochastic experimentation
• “Planned” experimentation

Types of stimuli
• Drives/needs, , a stimulus which arises either extrinsically
or intrinsically to secure the survival and integrity of the
embodied agent, , e.g. obtain food and energy, procure
shelter, liberate from captivity or from an emergency
situation.
• Curiosity, , the innate interest to find and explore the
unknown, may it be unknown physical space, unknown
objects, unknown functions and properties of objects or
unknown own capabilities
• Novelty triggering hypotheses formation about a
physical phenomenon which has emerged from a current
activity of the embodied agent, e.g. during the execution of
a task;

Stochastic vs. planned experiments
• Stochastic experimentation occurs during the
cognitive bootstrap; the chosen elementary actions
(contingent on the embodiment) and the inborn
proto-objects objects provide the robot with the initial
ontology (what type of objects there are)
• Planned experiments to gain insights about
properties and relations among objects

Why innate knowledge
• The robot needs a mechanism for creating the basic
ontology
• The notion of proto-object
object will speed up the
development
• As we are not necessarily doing modeling of
cognitive development we can introduce what we
deem fit (logic, self-preserving behaviors…)

Innate knowledge
• Self-preserving reflex behaviors
• Proto-objects
objects
• Innate gestalt principles (possible point of
interest in the sensory input)
• Internal value system
• Logical inference rules

Representation of Proto-Objects
Feature1:
Symbolic
name
Feature2:
FeatureN:
affordancy1
affordancN
activation

Relations between objects
Feature1:
Feature2:
FeatureN:
Sym
Name
affordancy1 affordancyN
activation
Instance of
A
Feature1:
Feature2:
FeatureN:
Instance of
B
Feature1:
Feature2:
FeatureN:
affordancy1affordancyN
activation
affordancy1 affordancN
activation
C
Feature1:
Feature2:
FeatureN:
D
Feature1:
Feature2:
FeatureN:
affordancy1
affordancyN
affordancy1 affordancyN
activation
activation
Later we will summarize the 4 crucial mechanisms that guide agent’s behavior

We need to think about architectures
tribute to A. Sloman
• • The sort of system we are discussing has many components doing many m
different things in parallel.
• • Putting the pieces together in a working architecture is a non-
trivial task for engineers and for scientists attempting to produce explanatory models
scientists attempting to produce explanatory models.
• • We need good theories about the space of possible architectures,
and good theories about particular architectures in that space in order to explain the wide w
variety of biological phenomena and in order to understand the development of humans, since
we are not born with a fully fledged architecture: their architecture grows in ways that may
partly replicate some of our evolutionary history but will be much influenced by our culture and
physical environment.
• • Different aspects of motivation and emotion relate to different
architectural layers with different competences.

Functionalism Why architectures,
when we can use FSM
tribute to A. Sloman
• • Functionalism is one kind of attempt
to understand the notion of virtual
machine, in terms of states defined by
a state-transition table (Finite-State-
Machines - FSM).
• • This is how many people think of
functionalism: there’s a total state
which affects input/output
contingencies, and each possible state
can be defined by how inputs
determine next state and outputs.
State(t+1) = f(State(t), Act(t), Observ(t))

Deeper notion
of functionalism
• Instead of a single (atomic) state which
switches when some input is received, a
virtual machine can include many subsystems
with their own states and state
transitions going on concurrently, some
of them providing inputs to others.
• The different states may change on
different time scales: some change very
rapidly others very slowly, if at all.
• They can vary in their granularity: some
sub-systems may be able to be only in
one of a few states, whereas others can
switch between vast numbers of
possible states (like a computer’s
virtual memory).
• Some may change continuously, others
only in discrete steps. Some subprocesses
may be directly connected to
sensors and effectors, whereas others
have no direct connections to inputs
and outputs and may only be affected
very indirectly by sensors or affect
motors only very indirectly (if at all!).
tribute to A. Sloman

Under development
tribute to A. Sloman
• A TAXONOMY OF TYPES OF ARCHITECTURE, based on
the analysis of:
• Requirements for architectures,
• Designs for architectures,
• Components of architectures
• Varieties of information structures
• Varieties of mechanisms
• Kinds of control systems
• Ways of assembling components
• How architectures can develop,
• Tools for exploring and experimenting with
architectures

Nervous system of the
“cybernetic animal”
Nerve net
Receptors
Effectors
Environment
adapted from V. Turchin, “The Phenomenon of Science”, Columbia University Press, 1977

The Reactive Architecture

Including alarms (priorities of execution)

More developed architecture

H-CogAff
cognitive architecture

Interactivist cognitive architecture

CoSy Architecture

Proposed XPERO general architecture

The 4 crucial mechanisms that guide agent’s
behavior / development (Kulakov, Stojanov 2002)
• Abstraction mechanism that provides chunking of the sensory
that provides chunking of the sensory-
motor flux from highly dimensional input via proto-objects objects to symbolic compact
descriptions of the objects in the environments. This will be happening during
the cognitive bootstrap phase using the stochastic experimentation. Abstraction
mechanism will enable the agent to deal with more and more complex situations
with the same or less cognitive effort;
• Planning mechanism used during deliberate experimentation. It
combines various previous experiences into new knowledge, for example by
analogy-making;
Mechanism that provides emergence of
• Mechanism that provides emergence of more complex inner
value and motivational systems according to which new
experiences are judged, foreseen and executed;
• Socialization/communication mechanism that enables the
agent to interpret in a special way inputs coming from other agents (possibly
humans) and to provide translation of the newly acquired knowledge into human
understandable form;

A schema
schema links with weights denoting
the reliability of the schema
p1
W1
W2
p2
Condition Percept Schema Node Expectation Percept

simil link
Proto-conceptual network
abstract schema
percept node
schema node
schema link
type/token link

Building an abstract schema
Condition
Percept
Expectation
Percept
Action Sequence
or
Abstract Schema
If long enough sequence of reliable schemas
is detected (3+ actions), then a new Abstract
schema is created with the same Condition
Percept as the first schema and the same
Expectation Percept as the last schema
in the sequence.
The Action Sequence of the Abstract
Schema is left undefined, but
type/token links are created between
the Abstract Schema and the
underlying more
concrete schemas.

Building an abstract schema
The reliability of the abstract schema
takes the value of the lowest reliable
schema at the level below (the weakest
link), while its level of abstraction equals
the level of the most abstract schema
between its constituents, plus one
If the Expectation Percept
matches the Current Percept
derived from the Current Sensory
Input, then the Reliability of the
Schema is increased; otherwise it
is lowered

Part of the internal representation ( (proto-
conceptual network)

Levels of abstraction

Important
• Internal value system and the previous knowledge
influence the perception, the observation and the
evaluation of an ongoing experiment (WP1 and
WP3), as well as the planning and decision making
processes (WP2)
• It is very important to know the architecture of the
whole system (what is learnt, how it is learnt, how
it is represented, which part makes the decision
what to do next and how, etc.)